Help Center

Menu

Logs reference

Background

This article describes most of the log files produced by Funnelback.

The following abbreviations are used in this document:

  • SEARCH-PACKAGE-ID - Search package ID, as displayed on the search package manage screen.

  • DATA-SOURCE-ID - Data source ID, as displayed on the data source manage screen.

  • RESULTS-PAGE-ID - Results page ID, as displayed on the results page manage screen.

Search package logs

User interface logs

modernui.Admin.log

  • Location: Search package - collection logs

  • Purpose: Linux only. Contains messages logged by the admin search endpoints. e.g. template error logging.

modernui.Public.log

  • Location: Search package - collection logs

  • Purpose: Linux only. Contains messages logged by the public search endpoints e.g. template error logging.

modernui.SEARCH-PACKAGE-ID.Admin.log

  • Location: System web logs

  • Purpose: Windows only. Contains messages logged by the admin search endpoints. e.g. template error logging.

modernui.SEARCH-PACKAGE-ID.Public.log

  • Location: System web logs

  • Purpose: Windows only. Contains messages logged by the public search endpoints e.g. template error logging.

Analytics update logs

pattern_analyser.log

  • Location: Search package - collection logs

  • Purpose: Logs the output from outliers-log-processing.pl. Logs messages for the pattern analyser reports build.

update_reports_launch.log

  • Location: Search package - collection logs

  • Purpose: Logs messages for the analytics report build.

update_reports.log

  • Location: Search package - collection logs

  • Purpose: Logs the output from reports-load-queries.pl. Logs messages for the analytics reports build.

update_reports_previous.log

  • Location: Search package - collection logs

  • Purpose: Messages from the previous reports build.

Results page logs

Step-BuildAutoCompletion.RESULTS-PAGE-ID.log

  • Location: Search package - live/offline logs

  • Purpose: Logs the output from build_autoc. Logging for each auto-completion index. An auto-completion index is built for each profile.

Data source logs (non-push data source)

Update logs - all data source types

DATA-SOURCE-ID.lock

  • Location: Data source - collection logs

  • Purpose: These files are used to prevent multiple updates running simulatiously (by taking an OS lock on the file). They’re generally empty and hence contain no useful info.

DATA-SOURCE-ID.pre-update.log

  • Location: Data source - collection logs

  • Purpose: Logs the java command that was run for the update.

update-DATA-SOURCE-ID.log

  • Location: Data source - collection logs

  • Purpose: Logs the output from update.pl. Logs messages for the data source update process.

update-DATA-SOURCE-ID.previous.log

  • Location: Data source - collection logs

  • Purpose: Messages from the previous data source update.

update.log

  • Location: Data source - live/offline logs

  • Purpose: Top level log for the update pipeline.

Gather logs - web and matrix data sources

binaries.log

  • Location: Data source - live/offline logs

  • Purpose: List of binary files stored e.g. (PDF, DOC etc.) Appended to on restart from checkpoint.

copied_urls.log

  • Location: Data source - live/offline logs

  • Purpose: Log output if crawler.incremental_logging=true. All URLs whose content was copied from the previous crawl, as they had not changed and so were not downloaded again.

crawler.log

  • Location: Data source - live/offline logs

  • Purpose: Contains filter messages.

crawler_logs_checkpoint_sizes.dat

  • Location: Data source - live/offline logs

  • Purpose: Records the sizes of crawler logs at the time a checkpoint occurred. This allows truncating them back to that size if the crawler is restarted from a checkpoint to avoid inconsistency.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Top level log for the web crawler.

crawl.<N>.log

  • Location: Data source - live/offline logs

  • Purpose: Logs individual messages for each running crawler thread. One log per thread.

domains.log

  • Location: Data source - live/offline logs

  • Purpose: Frontier and stored document counts for each domain encountered during the crawl.

frontier_dump.log

  • Location: Data source - live/offline logs

  • Purpose: Contains a dump of the crawl frontier (set of known, uncrawled URLs).

headers.log

  • Location: Data source - live/offline logs

  • Purpose: Captures HTTPS headers recorded during the crawl. Output if crawler.header_logging=true

manifest.txt

  • Location: Data source - live/offline logs

  • Purpose: Records the order in which bundles were created.

monitor.log

  • Location: Data source - live/offline logs

  • Purpose: Records various crawl statistics.

new_urls.log

  • Location: Data source - live/offline logs

  • Purpose: A new URL is defined as one which was not stored in the previous crawl. Log output if crawler.incremental_logging=true.

redirects.txt

  • Location: Data source - live/offline logs

  • Purpose: Appended to on restart from checkpoint. Captures redirects.

servers.log

  • Location: Data source - live/offline logs

  • Purpose: Frontier and stored document counts for each server encountered during the crawl.

stored.log

  • Location: Data source - live/offline logs

  • Purpose: All URLs stored during a crawl (also used for refresh updates), in chronological order. Appended to on restart from checkpoint.

store-messages.log

  • Location: Data source - live/offline logs

  • Purpose: Records URLs stored into a WARC/Mirror store as well as edit distance calculation logs and certain error/warning states of MirrorStore.

url_errors.log

  • Location: Data source - live/offline logs

  • Purpose: Records errors while processing URLs.

url_no_content.log

  • Location: Data source - live/offline logs

  • Purpose: Contains URLs which are stored despite having no content. Documents with no content are usually the result of a filter returning an empty document. This occurs when crawler.store_empty_content_urls=true.

url_titles.log

  • Location: Data source - live/offline logs

  • Purpose: Lists URLs and their titles.

BroadMIMETypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: The file types by MIME report displays statistics on document types as reported by the web server. A significant difference between the document types reported here and the document types reported by the types by suffix report may indicate a web server serving documents with an incorrect content type.

BroadWebServerTypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: Records web-server types in general categories (anything after a / or ( is truncated) - e.g. Will capture Apache without the version.

CrawlSizeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: Records the total URLs stored by a web crawl.

FileSizeByDocumentTypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: The file sizes by document type report displays statistics on file sizes found, divided by content type.

FileSizeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: The file sizes report displays statistics on content sizes found.

MIMETypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: The file types by MIME report displays statistics on document types as reported by the web server. A significant difference between the document types reported here and the document types reported by the types by suffix report may indicate a web server serving documents with an incorrect content type.

ReferencedFileTypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: Records the file extensions of URLs seen in href/src HTML attributes even if those URLs would not be crawled.

SuffixTypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: The file types by suffix report displays statistics on document types as identified by checking of the suffix.

URLlengthStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: Records the number of URLs of different lengths seen during a crawl.

WebServerTypeStatistic.stat

  • Location: Data source - live/offline logs

  • Purpose: Records statistics the web server types (including versions etc.) seen during the crawl.

Gather logs - database data source

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages from the database gatherer.

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

Gather logs - directory data source

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages from the directory gatherer.

Gather logs - filecopy data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages from file copier data source updates.

stored.log

  • Location: Data source - live/offline logs

  • Purpose: Lists the documents stored by the file copier.

monitor.log

  • Location: Data source - live/offline logs

  • Purpose: Records various statistics about the filecopier update.

url_errors.log

  • Location: Data source - live/offline logs

  • Purpose: Records errors while processing URLs.

Gather logs - custom data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from the custom gatherer.

Gather logs - TRIM (trimpush) data source

trim.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from the TRIM gatherer.

trim-details.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from the TRIM gatherer.

trim-combine-attachments.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages from the combine attachments process.

filter.log

  • Location: Data source - live/offline logs

  • Purpose: Logs filter messages from the TRIM update (filtering via the Funnelback daemon filter service).

monitor.log

  • Location: Data source - live/offline logs

  • Purpose: Provides various statistics about the TRIM gather process.

error.log

  • Location: Data source - live/offline logs

  • Purpose: Logs TRIM IDs of documents that recorded a failure.

Gather logs - Slack (slackpush) data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from the Slack gatherer.

Gather logs - Facebook data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Contains detail for the Facebook gather process.

Gather logs - Flickr data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Contains detail for the Flickr gather process.

Gather logs - Twitter data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Contains detail for the Twitter gather process.

Gather logs - YouTube data source

gather_executable.log

  • Location: Data source - live/offline logs

  • Purpose: Wrapper log for the gather process.

gather.log

  • Location: Data source - live/offline logs

  • Purpose: Contains detail for the YouTube gather process.

Indexer logs - all data source types

Step-AnnieAPrimaryCollection.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from annie-a. Logging for the build of the annotation index.

Step-BuildAutoCompletion.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from build_autoc. Logging for the auto-completion index build.

Step-BuildCollapsingSignatures.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from padre-cc. Logging for the collapsing index build process.

Step-BuildSpelling.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from build_spelling_index. Logging for the spelling index build process.

Step-Index.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from padre-iw. Logging for the data source index build process.

Step-QueryIndependentEvidenceCollectionLevel.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from padre-qi. Logging for the query independent evidence index build process.

Step-SetGscopes.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from padre-gs. Logging for application of data source level gscopes.

Step-FacetBasedGscopes.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output for setting up query facets.

Step-MoveTmpIndexIntoPlace.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output for index move step.

Step-ClearGscopes.log

  • Location: Data source - live/offline logs

  • Purpose: Logs the output from padre-gs when clearing any previously set gscopes.

Step-ExactMatchKill.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from padre-fl for killing documents specified in kill_exact.cfg.

Step-FacetedNavigation.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages relating to the building of query-based faceted navigation.

Step-PartialMatchKill.log

  • Location: Data source - live/offline logs

  • Purpose: Logs output from padre-fl for killing documents specified in kill_partial.cfg.

Step-SecondaryIndex.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages relating to the creation of secondary indexes for an instant update.

Step-SetExtraRegexGscopes.log

  • Location: Data source - live/offline logs

  • Purpose: Logs messages relating to the setting of gscopes from other sources.

Push data source logs

Step-BuildCollapsingSignatures.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from padre-cc. Logging for the collapsing index build process.

Step-BuildMatchOnlyIndexForAutoc.log

  • Location: Data source - push live logs

  • Purpose: Logs message relating to the creation of a match only index which is used to build profile-scoped auto-completion indexes.

Step-ClearGscopes.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from padre-gs when clearing any previously set gscopes.

Step-Index.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from padre-iw. Logging for the index build process.

Step-KillDocuments.log

  • Location: Data source - push live logs

  • Purpose: Logs messages relating to the killing of documents as part of maintaining the integrity of the search index. Note: push data sources do not support the use of kill configuration files - the API should be used for this.

Step-QueryIndependentEvidenceCollectionLevel.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from padre-qi. Logging for the query independent evidence index build process.

Step-SetExtraRegexGscopes.log

  • Location: Data source - push live logs

  • Purpose: Logs messages relating to the setting of gscopes from other sources.

Step-SortDocumentUrls.log

  • Location: Data source - push live logs

  • Purpose: Logs messages relating to the sorting of index.urls in order to speed up document kill operations.

Step-annie-a.log

  • Location: Data source - push live logs

  • Purpose: Logs messages relating to the building of annotation indexes.

Step-BuildAutoCompletion.RESULTS-PAGE-ID.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from build_autoc. Logging for each auto-completion index. An auto-completion index is built for each profile.

Step-BuildAutoCompletion.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from build_autoc. Logging for the auto-completion index build.

Step-BuildSpelling.log

  • Location: Data source - push live logs

  • Purpose: Logs the output from build_spelling_index. Logging for the spelling index build process.

Application logs

Installation logs

post-install-<VERSION>.log

  • Location: System logs

  • Purpose: Output from the post installer script.

post_install.sh-<VERSION>.log

  • Location: System logs

  • Purpose: Output from the post_install.sh script.

upgrade-<VERSION>.log

  • Location: System logs

  • Purpose: Output from the post_install.pl script.

upgrade-utility-v15-to-v16.build.<CLIENT-ID>.<DATE>.log

  • Location: System logs

  • Purpose: Output from running the upgrade-utility-v15-to-v16 script, the <CLIENT-ID> is the client ID provided when running the script, and <DATE> is the time the upgrade script was executed.

upgrade_collection_indexes.log

  • Location: System logs

  • Purpose: Logs messages relating to the upgrading of indexes, as part of a Funnelback upgrade.

Search package and data source management logs

create.log

  • Location: System logs

  • Purpose: Logs the output of creating a search package or data source from the search dashboard.

delete.log

  • Location: System logs

  • Purpose: Logs the output of deleting a search package or data source from the search dashboard.

public-ui.warnings

  • Location: System logs

  • Purpose: Contains administration user interface warning messages. When this log file exists a banner is presented in the search dashboard.

scheduled_tasks_backup.txt

  • Location: System logs

  • Purpose: Records the last change made to scheduled tasks on Unix based OSes (basically a copy of the cron job’s before-and-after states).

update-launch.log

  • Location: System logs

  • Purpose: Logs any errors that may occur when starting an update.

update_reports_errors.log

  • Location: System logs

  • Purpose: Contains errors from the main report update process.

publish.log

  • Location: System logs

  • Purpose: Logs information about files that are published via the search dashboard.

realm_properties_update

  • Location: System logs

  • Location: System logs

  • Purpose: Logs errors that occurred when updating the recommendations database.

Service logs

daemon.log

  • Location: System logs

  • Purpose: Contains messages recorded by the Funnelback daemon service.

daemon-wrapper.log

  • Location: System logs

  • Purpose: Contains messages from the YAJSW wrapper for the Funnelback daemon service.

jetty.log

  • Location: System logs

  • Purpose: Contains messages recorded by the Jetty web server.

jetty-wrapper.log

  • Location: System logs

  • Purpose: Contains messages from the YAJSW wrapper for the Jetty service.

graph-wrapper.log

  • Location: System logs

  • Purpose: Contains messages from the YAJSW wrapper for the Funnelback graph service.

neo4j/debug.log

  • Location: System logs

  • Purpose: Contains log messages from Neo4j.

redis.log

  • Location: System logs

  • Purpose: Contains messages recorded by the Redis service.

redis-wrapper.log

  • Location: System logs

  • Purpose: Contains messages from the YAJSW wrapper for the Redis service.

mail.log

  • Location: System logs

  • Purpose: Logs emails sent out by Funnelback.

Web server logs

Request logs

<YYYY_MM_DD>.error.log

  • Location: System web logs

  • Purpose: Contains errors from the Jetty web server.

access.admin.log

  • Location: System web logs

  • Purpose: Jetty access logs for the administration context. Logs are rotated and compressed.

access.public.log

  • Location: System web logs

  • Purpose: Jetty access logs for the public context. Logs are rotated and compressed.

Audit logs

audit-admin-api.log

  • Location: System web logs

  • Purpose: Log of users who access the admin API (/admin-api) along with the response code the receive.

audit.classic-admin.log

  • Location: System web logs

  • Purpose: Log of users who access the classic-admin (/search/admin) along with the response code the receive.

audit.classic-admin.log.<N>.gz

  • Location: System web logs

  • Purpose: These are older logs of the above rotated away and compressed.

audit.push.log

  • Location: System web logs

  • Purpose: Log of users who access the Push API (/push-api) along with the response code received.

admin-api.log

  • Location: System web logs

  • Purpose: Log from the admin-api web application (/admin-api)

classic-admin.log

  • Location: System web logs

  • Purpose: Log from the classic-admin web application (which takes requests to jetty and passes them to the perl CGIs in classic-admin).

authentication.classic-admin.log

  • Location: System web logs

  • Purpose: Record of successful and failed authentication attempts against the classic-admin (/search/admin) (includes remote IP address and X-ForwardedFor headers).

authentication.modernui.log

  • Location: System web logs

  • Purpose: Record of successful and failed authentication attempts against the modernui (/s) (includes remote IP address and X-ForwardedFor headers).

authentication.mediator-endpoint-http.log

  • Location: System web logs

  • Purpose: Record of successful and failed authentication attempts against the mediator (/search/admin/mediator) (includes remote IP address and X-ForwardedFor headers).

authentication.push.log

  • Location: System web logs

  • Purpose: Record of successful and failed authentication attempts against the push (/push-api) (includes remote IP address and X-ForwardedFor headers).

mediator-endpoint-http.log

  • Location: System web logs

  • Purpose: Log from the mediator web application (this pre-dates the admin API).

push.log

  • Location: System web logs

  • Purpose: Log for the push API.

data_api.log

  • Location: System web logs

  • Purpose: Log for the (deprecated) data API.

Global user interface logs

modernui.Admin.log

  • Location: System web logs

  • Purpose: Log from the modernui web application (/s/) deployed on the admin jetty server (port 8443 by default).

modernui.Public.log

  • Location: System web logs

  • Purpose: Log from the modernui web application (/s/) deployed on the public/search jetty server (ports 80 and 443 by default).

Knowledge graph logs

kg/spring-boot.log

  • Location: System web logs

  • Purpose: Logs messages from the startup of knowledge graph.

kg/cortex-api.log

  • Location: System web logs

  • Purpose: Logs messages from the knowledge graph API.

kg/cortex-neo4j.log

  • Location: System web logs

  • Purpose: Contains the Neo4j queries executed by Funnelback.