Matching is case-insensitive over the length of the name in a robots.txt file

Key: crawler.robotAgent
Type: String
Can be set in: collection.cfg


This is a string that is used to examine any robots.txt files that are found to see if there are Funnelback specific restrictions.

The string is case-insensitive, so it will match

User-Agent: funnelback
User-Agent: FunnelBACK

The user-agent check is equals, not a substring so it will not match

User-Agent: funnelback crawler
The webcrawler will still obey all rules in the robots.txt file which are meant to apply to all webcrawlers regardless of name.

Default Value