crawler.header_logging

This parameter controls whether HTTP headers should be written out to a separate log file during a web crawl. If this option is enabled then the headers.log file will be compressed at the end of the crawl.

The default behaviour is false i.e. do not log headers to a separate log file.

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.header_logging key, and set the value. This can be set to any valid Boolean value.