crawler.verbosity

Background

This parameter specifies the verbosity setting for the webcrawler when it writes to its log files. The higher the number the higher the verbosity and the more detailed messages it will write out.

Verbosity levels

The verbosity is set to a value between 0 and 7. Each level includes messages from the lower numbered levels.

You may set this value higher if you are trying to debug a crawling issue, but normally it should be left at the default as otherwise it may generate very large log files.

  • Level 0: no output, apart from error messages.

  • Level 1: addresses of pages downloaded, redirects, aliases.

  • Level 2: crawler status and memory messages, pages added to frontier, sleep messages, file types, storage, cache, frontier, level messages.

  • Level 3: loading policy messages.

  • Level 4: HTTP header, network byte counts, unacceptable (rejected) files, robots.txt, parameter removal.

  • Level 5: absolute URLs, rule matches.

  • Level 6: extracted text from each HTML document parsed.

  • Level 7: regular expression patterns.

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.verbosity key, and set the value. This can be set to any valid Integer value.

Default value

crawler.verbosity=4

See also