crawler.verbosity
Background
This parameter specifies the verbosity setting for the webcrawler when it writes to its log files. The higher the number the higher the verbosity and the more detailed messages it will write out.
Verbosity levels
The verbosity is set to a value between 0 and 7. Each level includes messages from the lower numbered levels.
You may set this value higher if you are trying to debug a crawling issue, but normally it should be left at the default as otherwise it may generate very large log files.
-
Level 0: no output, apart from error messages.
-
Level 1: addresses of pages downloaded, redirects, aliases.
-
Level 2: crawler status and memory messages, pages added to frontier, sleep messages, file types, storage, cache, frontier, level messages.
-
Level 3: loading policy messages.
-
Level 4: HTTP header, network byte counts, unacceptable (rejected) files, robots.txt, parameter removal.
-
Level 5: absolute URLs, rule matches.
-
Level 6: extracted text from each HTML document parsed.
-
Level 7: regular expression patterns.