crawler.num_crawlers
Number of crawler threads which simultaneously crawl different hosts.
Key: crawler.num_crawlers
Type: Integer
Can be set in: collection.cfg
Table of Contents
Description
This option specifies how many crawler threads will be created to download pages from a set of web sites. Note that with the default frontier, only one thread will access a given host at a time (i.e. if you have 10 threads but are only crawling one site then 9 threads will be idle).