Help Center

Menu

crawler.max_url_length

Specifies the maximum length a URL can be in order for it to be crawled.

Key: crawler.max_url_length
Type: Integer
Can be set in: collection.cfg

Table of Contents

Description

The crawler will ignore all URLs that are larger than this option’s value. The check is based on the number of characters in the URL e.g. if the limit is 256 then any URL with more than 256 characters will be ignored. Typically if the URL is too long, it is likely to be a crawler trap.

Default Value

crawler.max_url_length=256

See Also