crawler.request_header_url_prefix
Optional URL prefix to be applied when processing the crawler.request_header parameter
Key: crawler.request_header_url_prefix
Type: String
Can be set in: collection.cfg
Table of Contents
Description
This parameter can be used to specify an optional URL prefix to be applied to the crawler.request_header
parameter.
If defined and non-empty then only URLs which start with this prefix will be requested with the additional request header. Otherwise the request header will be sent with all outgoing requests if defined.
Examples
Specify that the defined crawler.request_header should only be sent in requests to URLs which start with http://example.com/forums/
crawler.request_header_url_prefix=http://example.com/forums/