crawler.max_individual_frontier_size

Specifies the maximum size of an individual frontier.

Key: crawler.max_individual_frontier_size
Type: Integer
Can be set in: collection.cfg

Description

This option limits the number of URLs stored in a frontier. If new URLs are discovered during a crawl but can not be queued because the frontier is full, they will be dropped (though they may be found again later in the crawl).

Default Value

If not defined, it is unlimited.

Examples

crawler.max_individual_frontier_size=1000

© 2015- Squiz Pty Ltd