Optional URL prefix to be applied when processing the crawler.request_header parameter

Key: crawler.request_header_url_prefix
Type: String
Can be set in: collection.cfg


This parameter can be used to specify an optional URL prefix to be applied to the crawler.request_header parameter.

If defined and non-empty then only URLs which start with this prefix will be requested with the additional request header. Otherwise the request header will be sent with all outgoing requests if defined.

Default Value



Specify that the defined crawler.request_header should only be sent in requests to URLs which start with