crawler.frontier_hosts
Lists of hosts running crawlers if performing a distributed web crawl.
Key: crawler.frontier_hosts
Type: List<String>
Can be set in: collection.cfg
Table of Contents
Description
This option allows you to specify which hosts crawlers are running on when using a DistributedFrontier
(distributed crawling). Each host will be responsible for its own subset of the web servers being crawled.
Examples
Specify the IP addresses of the hosts that the distributed crawl is running on.
crawler.frontier_hosts=12.67.89.8,12,67.89.9