crawler.request_timeout

Timeout for HTTP page GETs (milliseconds)

Key: crawler.request_timeout
Type: Integer
Can be set in: collection.cfg

Description

This is the maximum number of milliseconds that the crawler will wait for a HTTP GET request to return. If the web server does not start to return the page in question before this timeout, the crawler assumes the page was unavailable.

This value is multiplied by 4 to give the parser (link extraction) timeout.

Default Value

crawler.request_timeout=15000

© 2015- Squiz Pty Ltd