crawler.revisit.num_times_unchanged_threshold
Threshold for the number of times a page has been unchanged when deciding whether to revisit it.
Key: crawler.revisit.num_times_unchanged_threshold
Type: Integer
Can be set in: collection.cfg
Table of Contents
Description
This parameter specifies a threshold to use when looking at how many times the content of a given URL has been unchanged over the course of multiple crawls.
A particular revisit policy might decide that if the number of times a URL has been unchanged is higher than the threshold then a revisit would not be required.
Increasing this threshold would result in more revisits (HTTP HEAD and/or GET requests) being performed.