crawler.accept_cookies

Cookies are small text files which are normally stored by a user’s web browser to save their preferences and other information when browsing a web site.

In some cases e.g. crawling dynamic websites having cookie support turned on may be useful.

Cookie policy. Default is false i.e. do not accept cookies. Requires HTTPClient if true.

To disable this feature set it to "false" e.g.