Whether HTTP header information should be written at the top of HTML files.

Key: crawler.store_headers
Type: Boolean
Can be set in: collection.cfg


This parameter controls whether HTTP headers are stored when downloading content. This should normally be set to true, as these headers are used by the indexer and web crawler when doing incremental crawling.

Default Value


See Also

© 2015- Squiz Pty Ltd