crawler.robotAgent
Matching is case-insensitive over the length of the name in a robots.txt file
Key: crawler.robotAgent
Type: String
Can be set in: collection.cfg
Description
This is a string that is used to examine any robots.txt files that are found to see if there are Funnelback
specific restrictions.
The string is case-insensitive, so it will match
User-Agent: funnelback User-Agent: FunnelBACK
The user-agent check is equals, not a substring so it will not match
User-Agent: funnelback crawler
The webcrawler will still obey all rules in the robots.txt file which are meant to apply to
all webcrawlers regardless of name.
|