Timeout for HTTP page GETs (milliseconds)

Key: crawler.request_timeout
Type: Integer
Can be set in: collection.cfg


This is the maximum number of milliseconds that the crawler will wait for a HTTP GET request to return. If the web server does not start to return the page in question before this timeout, the crawler assumes the page was unavailable.

This value is multiplied by 4 to give the parser (link extraction) timeout.

Default Value