crawler.max_individual_frontier_size

This option limits the number of URLs stored in a frontier. If new URLs are discovered during a crawl but can not be queued because the frontier is full, they will be dropped (though they may be found again later in the crawl).

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.max_individual_frontier_size key, and set the value. This can be set to any valid Integer value.