Warc storage
| This feature is not available in the Squiz DXP. |
Funnelback can store gathered content in a single WARC file for indexing, and this is the default storage option for most collection types. By compressing the content and avoiding the creation of a large number of files and directories for storage this approach saves on storage space. It also simplifies the transfer of gathered data to another machine in a multi-server setup.
Table of Contents
Configuring WARC storage
You can configure the use of a WARC store by specifying the appropriate Java class in your collection.cfg file:
-
crawler.classes.URLStore for web collections
-
store.raw-bytes.class for filecopy collections
-
store.xml.class for database, connector and directory collections