Help Center

Menu

crawler.follow_links_in_comments

Whether to follow links in HTML comments while crawling.

Key: crawler.follow_links_in_comments
Type: Boolean
Can be set in: collection.cfg

Description

This parameter controls whether the webcrawler should follow links found inside HTML comments.

Default Value

crawler.follow_links_in_comments=false

The default behaviour is false i.e. do not follow commented-out links.

⚠ Caveats

This parameter does not apply to links extracted from Javascript code - please see the separate crawler.extract_links_from_javascript parameter to control the behaviour of following links in Javascript (which sometimes appears in HTML comments).

See Also