crawler.protocols

Crawl URLs via these protocols.

Key: crawler.protocols
Type: List<String>
Can be set in: collection.cfg

Description

This option is a comma-separated list of protocols to process. Note that the protocol must be supported by the crawler.

Default Value

crawler.protocols=http,https

Examples

Crawl HTTP only:

crawler.protocols=http

© 2015- Squiz Pty Ltd