Specifies the maximum time the crawler is allowed to run. When exceeded, the crawl will stop and the update will continue.
Can be set in: collection.cfg
This option specifies how many minutes, or hours, the crawler is allowed to run. The option
specifies the unit of measure.
This parameter can be left empty, in which case the crawler will keep going until there are no
URLs left in its frontier, or it reaches another limit e.g.
If you are testing a new web data source, then it can be useful to run a short crawl, say 10 minutes: