crawler.max_download_size
Specifies the maximum size of files the crawler will download (in MB).
Key: crawler.max_download_size
Type: Integer
Can be set in: collection.cfg
Table of Contents
Description
This parameter specifies the maximum download size for any single file during the crawl. The value is specified in MB. The default value is 10 MB.
This download limit applies to both HTML files and binary files (e.g. PDF, Word etc.).