Milliseconds between HTTP requests per crawler thread.
Can be set in: collection.cfg
Table of Contents
This setting specifies the number of milliseconds to wait before making another request to a single host.
The default behaviour of the web crawler is to dynamically calculate how long to delay based
on the time taken for the previous request to a given web server. This is done so that fast web servers
can be crawled rapidly while slow servers are given more time to respond (e.g. 10 x
This means that the actual request delay may sometimes be less than this fixed setting. If you wish
to ensure that this fixed setting is always used you will need to set
If you have an internal server and are crawling it out of hours, then you can reduce the request delay to speed up the crawl:
This parameter is similar to the