crawler.follow_links_in_comments
Whether to follow links in HTML comments while crawling.
Key: crawler.follow_links_in_comments
Type: Boolean
Can be set in: collection.cfg
Table of Contents
Description
This parameter controls whether the webcrawler should follow links found inside HTML comments.
Default Value
crawler.follow_links_in_comments=false
The default behaviour is false i.e. do not follow commented-out links.
⚠ Caveats
This parameter does not apply to links extracted from Javascript code - please see the separate crawler.extract_links_from_javascript
parameter to control the behaviour of following links in Javascript (which sometimes appears in HTML
comments).