The web crawler has a comprehensive set of configuration options that can be used to adjust how the web crawler operates.
Options controlling what gets included
Options controlling size limits and timeouts
Web crawler monitor options
Web crawler advanced options