crawler.store_all_types

Background

This parameter can be used to tell the web crawler to store all file types it encounters, regardless of the settings in the crawler.accept_files and crawler.reject_files parameters. You might set this to true if you wanted to make a complete mirror of a site i.e. download all images, media types etc. as well as the standard web pages.

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.store_all_types key, and set the value. This can be set to any valid Boolean value.

Default value

crawler.store_all_types=false