crawler.max_individual_frontier_size
Specifies the maximum size of an individual frontier.
Key: crawler.max_individual_frontier_size
Type: Integer
Can be set in: collection.cfg
Table of Contents
Description
This option limits the number of URLs stored in a frontier. If new URLs are discovered during a crawl but can not be queued because the frontier is full, they will be dropped (though they may be found again later in the crawl).