Check if aliased URLs exists - if not, revert to original URL.

Key: crawler.check_alias_exists
Type: Boolean
Can be set in: collection.cfg


This parameter controls whether to check if an aliased URL exists. During the course of a crawl we may detect a server as an alias of another e.g. is an alias of

If this parameter is set to true then we will use a HTTP HEAD request to check if the aliased URL is there before making a full request. If the aliased URL does not exist we will then revert to the original URL.

This results in extra HTTP requests so the default setting is false i.e. we do not normally run this check as it is usually not required.

Default Value


See Also