Specifies the number of top level directories to store disk based frontier files in.

Key: crawler.frontier_num_top_level_dirs
Type: Integer
Can be set in: collection.cfg


This optional parameter can be used to specify the number of top level directories to use when storing files that are required for a disk based frontier.

Default Value



If you are performing a very large crawl you may run into filesystem limits on the number of files allowed in a single directory. To get around this you might split the frontier files among 100 top level directories by setting: