crawler.store_all_types
Background
This parameter can be used to tell the web crawler to store all file types it encounters, regardless of
the settings in the crawler.accept_files
and crawler.reject_files
parameters. You might set this to true if you wanted to make a complete mirror of a site i.e. download
all images, media types etc. as well as the standard web pages.