crawler.frontier_use_ip_mapping
Whether to map hosts to frontiers based on IP address.
Key: crawler.frontier_use_ip_mapping
Type: Boolean
Can be set in: collection.cfg
Table of Contents
Description
This flag controls whether hosts are mapped to frontiers using IP addresses during a web crawl.
You would set this to true if you know you might be crawling a server which uses a single IP address to serve a large number of web sites. When set to true this will trigger the use of an internal DNS cache to cache the IP address information.