crawler.request_timeout
Background
This is the maximum number of milliseconds that the crawler will wait for an HTTP GET request to return. If the web server does not start to return the page in question before this timeout, the crawler assumes the page was unavailable.
This value is multiplied by 4 to give the parser (link extraction) timeout. |