crawler.request_delay
Background
This setting specifies the number of milliseconds to wait before making another request to a single host.
The default behaviour of the web crawler is to dynamically calculate how long to delay based
on the time taken for the previous request to a given web server. This is done so that fast web servers
can be crawled rapidly while slow servers are given more time to respond (e.g. 10 x previous_request_time).
|
This means that the actual request delay may sometimes be less than this fixed setting. If you wish
to ensure that this fixed setting is always used you will need to set crawler.monitor_delay_type
to "fixed".
Examples
If you have an internal server and are crawling it out of hours, then you can reduce the request delay to speed up the crawl:
crawler.request_delay=20
This parameter is similar to the crawler.monitor.* parameters in that it will be checked by the crawler
monitor during the crawl. This means that if you modify this value during a running crawl the new value
will be used for subsequent requests.
|