crawler.store_all_types

If true, override accept/reject rules and crawl and store all file types encountered

Key: crawler.store_all_types
Type: Boolean
Can be set in: collection.cfg

Table of Contents

Description

This parameter can be used to tell the web crawler to store all file types it encounters, regardless of the settings in the crawler.accept_files and crawler.reject_files parameters. You might set this to true if you wanted to make a complete mirror of a site i.e. download all images, media types etc. as well as the standard web pages.

Default Value

crawler.store_all_types=false