Option to control whether a list of new and changed URLs should be written to a log file during incremental crawling.

Key: crawler.incremental_logging
Type: Boolean
Can be set in: collection.cfg


This parameter controls whether information on different URL types seen during an incremental crawl is logged out. If this setting is enabled then the following log files will be created:

  • new_urls.log: A new URL is defined as one which was not stored in the previous crawl.

  • copied_urls.logs: All URLs whose content was copied from the previous crawl, as they had not changed and so were not downloaded again.

The logs will be located in the log directory in the relevant "view" for the web data source and can be viewed using the log viewer.

Default Value


The default behaviour is false i.e. do not perform this logging.


Turn on incremental logging: