Help Center

Menu

crawler.classes.URLStore

Background

The crawler stores downloaded documents on a local file system for them to be indexed. This option identifies the Java class to be used to store the documents.

The main store classes are:

  • com.funnelback.common.store.WarcStore: (default) Store cached documents in a single compressed WARC file

  • com.funnelback.common.io.MirrorStore: Store cached documents using a mirror of their URL directory structure.

Setting the key

Set this configuration key in the search package or data source configuration.

Use the configuration key editor to add or edit the crawler.classes.URLStore key, and set the value. This can be set to any valid String value.

Default value

crawler.classes.URLStore=com.funnelback.common.store.WarcStore

See also