crawler.frontier_use_ip_mapping

Background

This flag controls whether hosts are mapped to frontiers using IP addresses during a web crawl.

You would set this to true if you know you might be crawling a server which uses a single IP address to serve a large number of websites. When set to true this will trigger the use of an internal DNS cache to cache the IP address information.

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.frontier_use_ip_mapping key, and set the value. This can be set to any valid Boolean value.

Default value

crawler.frontier_use_ip_mapping=false