crawler.request_delay

Background

This setting specifies the number of milliseconds to wait before making another request to a single host.

The default behaviour of the web crawler is to dynamically calculate how long to delay based on the time taken for the previous request to a given web server. This is done so that fast web servers can be crawled rapidly while slow servers are given more time to respond (e.g. 10 x previous_request_time).

This means that the actual request delay may sometimes be less than this fixed setting. If you wish to ensure that this fixed setting is always used you will need to set crawler.monitor_delay_type to "fixed".

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.request_delay key, and set the value. This can be set to any valid Integer value.

Default value

crawler.request_delay=250

Examples

If you have an internal server and are crawling it out of hours, then you can reduce the request delay to speed up the crawl:

crawler.request_delay=20
This parameter is similar to the crawler.monitor.* parameters in that it will be checked by the crawler monitor during the crawl. This means that if you modify this value during a running crawl the new value will be used for subsequent requests.