crawler.store_headers

Background

This parameter controls whether HTTP headers are stored when downloading content. This should normally be set to true, as these headers are used by the indexer and web crawler when doing incremental crawling.

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.store_headers key, and set the value. This can be set to any valid Boolean value.

Default value

crawler.store_headers=true

See also