Funnelback 16.1.2032 release notes

New features

New administration dashboard, terminology and client separation

Funnelback’s administration dashboard has been overhauled to make search implementation easier.

As part of this improvement, new terminology of search packages, data sources and results pages has been introduced. Broadly, search packages are equivalent to meta collections in earlier versions of Funnelback, and a search package is now required to bundle up the data sources and results pages used to deliver a search. Data sources are equivalent to non-meta collections in earlier versions (but without the ability to serve search results), while result pages are the new equivalent of profiles and are used to define the search results page functionality and formatting.

The new administration dashboard also formalises the separation of implementations in a multi-tenant Funnelback environment with the concept of a client, which groups together all components of an implementation.

Plugins for reusable custom code

A new bundled format for custom code to be run on the Funnelback server, known as a plugin, has been introduced to the product.

Plugins are intended to separate customisations from individual data sources and results pages so that they can be more easily reused and can go through a separate approval process to ensure quality in multi-tenant environments.

Plugins provide similar capabilities to custom Groovy scripts in earlier versions and add the ability to supply some types of data, such as external metadata, dynamically during a data sources update to avoid the need to assume the internal file system layout.

The features superceded by plugins - Groovy document and Jsoup filters, hook scripts, custom workflow and custom gatherers are no available and any existing implementations that are updated to Funnelback 16 must rewrite this custom code using the plugin framework.

Improvements

In general

  • The query_processor_options configuration setting is now supported on results pages, overriding whatever is set on the search package configuration.

  • The individual data source components and associated relative weightings of data sources in search packages can now be set via the configuration APIs with the meta.components and meta.components.[component].weight settings. When modified, these settings will be automatically applied to the search package.

  • Streamlined knowledge graph administration experience.

  • Tuning runs are now subject to the same task queueing system as data source and analytics updates.

  • Updates to the default template to support results page level configuration settings where possible.

  • Added support for knowledge graph scripts at the results page level.

  • The data source components of a search package can now be set within the configuration editing screens.

  • A number of improvements to SAML authentication support, in particular reduction in the number of SAML service providers required for administration setups and to support integration with Auth0.

  • Added a Content-Type response header to the push API endpoint

  • The 'Intercom' support tool has been integrated into the Funnelback administration dashboard.

New APIs

The support for performing the following tasks via REST APIs has been introduced:

  • Creating and deleting results pages.

  • Reading log files (via WebDAV).

  • Reading/editing of gscopes.cfg and external_metadata.cfg (via WebDAV).

  • Determining the update progress of a data source

  • Determining when the last successful update of a data source occurred.

  • Determining whether a running task has been asked to stop.

  • Deleting data sources and search packages.

Bug fixes

  • Prevented WebDAV clients which take long-timeout locks and do not reliably release them from locking out other clients.

  • Fixed recommender operation on filecopy data sources.

  • Fixed the all-results.json endpoint to handle when search sessions are enabled.

  • Fixed presentation of sparklines within trend alerts reports.

  • Fixed possible configuration setting loss when encrypting configuration values for the first time after installation.

  • Prevented creation of users with service user prefixes.

  • Fixed the web crawler to handle responses without a Content-Type header.

  • Improved Padre handling of invalid XML characters.

  • Fixed handling of ui.modern.pseudonymise_client_ips when Funnelback is used behind a proxy or load balancer.

  • Fixed publication of web resources files containing spaces in their filenames.

  • Fixed isAdminUI Freemarker macro when search and administration ports are the same.

  • Fixed consistency of status codes resulted by the update history API.

Important changes

  • Tuning is now a task under the task queue, this means when it runs can be controlled by the task picker. Tuning can no longer be started by post /search-quality/v1/tuning/collections/{collection}/profiles/{profile}/runs?action=START_TUNING, instead it can only be started by using the task queue API: post /task/v1/queue/TUNING.

  • The task queue now allows running tasks to be added to the queue and allows multiple tasks which use the same resources (for example, the same data source), to be in the queue at the same time.

  • query_processor_options now supports configuration environments.

  • The meta.cfg config file no longer exists, being replaced by the meta.components search package configuration setting.

  • The groovy script specified by the auth.admin.saml.groovy-permission-mapper setting now supports defining roles the user is always permitted to edit.

  • The administration dashboard’s edit file-manager rules pages are no longer available. Any remaining cases where custom file manager rules are required must be set directly in the relevant .ini files.

  • Naming of log files has been made more consistent between data source types. For example, crawl.log is now named gather.log to be consistent across the product.

  • Local data sources are not supported in this version, reflecting the restriction on direct filesystem access in the AWS SaaS environment. Existing local collections should be converted to either a web or custom data sources depending on the logic implemented in the local collection.

  • Search package and data source IDs are now limited to 192 characters

  • Freemarker templates are now only able to import/include files which end in .ftl

  • The question.collection.configuration data model exposed to Freemarker no longer contains server (global) configuration settings.

  • SAML authentication now creates user .ini files representing logged in users to support cross-component authentication.

Upgrade notes

Knowledge graph

  • Previous versions of Funnelback used a reserved id metadata class to hold the knowledge graph identifier which caused compatibility issues if id was required for another purpose. This has been replaced with a FUNkgLiveUrl metadata class. Any systems using the knowledge graph nodes endpoint directly will need to account for this change as part of upgrading to this and future versions.

  • All knowledge graphs must be manually updated after upgrading to support the metadata class change noted above.

Document filters

The following methods are no longer available within groovy document filters:

  • com.funnelback.filter.api.documents.BytesDocument.from(FilterableDocument)

  • com.funnelback.filter.api.documents.StringDocument.from(FilterableDocument)

  • com.funnelback.filter.api.documents.StringDocument.from(FilterableDocument, DocumentType, String)

  • com.funnelback.filter.api.documents.StringDocument.getOrGuessCharset(FilterableDocument)

  • com.funnelback.filter.api.DocumentType.fromContentType(String)

Equivalent methods are now available within the FilterContext as detailed in the table below.

Note that the examples assume the FilterContext passed into the filter is the variable context as in FilterExamples

Previously Now

BytesDocument.from(document)

context.getFilterDocumentFactory().toBytesDocument(document)

StringDocument.from(document)

context.getFilterDocumentFactory().toStringDocument(document)

StringDocument.from(document, type, content)

context.getFilterDocumentFactory().toStringDocument(document, type, content)

StringDocument.getOrGuessCharset(document)

No longer supported.

DocumentType.fromContentType(contentType)

context.getDocumentTypeFactory().fromContentTypeHeader(contentType)

No direct replacement for getOrGuessCharset() is provided. Ideally the source should provide the charset, if this is the case use FilterableDocument#getCharset(). It is still possible to use context.getFilterDocumentFactory().toStringDocument(document) to get the document as String where Funnelback will attempt to guess the charset if not set.

The following are examples of the exceptions which may be logged if the removed methods are referenced:

Caused by: groovy.lang.MissingMethodException: No signature of method: static com.funnelback.filter.api.documents.StringDocument.from() is applicable for argument types: (com.funnelback.filter.documents.DefaultBytesDocument, com.funnelback.filter.api.DocumentType$_DocumentType...)
Caused by: groovy.lang.MissingMethodException: No signature of method: static com.funnelback.filter.api.documents.StringDocument.from() is applicable for argument types: (com.funnelback.filter.documents.DefaultBytesDocument)
Caused by: groovy.lang.MissingMethodException: No signature of method: static com.funnelback.filter.api.documents.BytesDocument.from() is applicable for argument types: (com.funnelback.filter.documents.DefaultBytesDocument)
groovy.lang.MissingMethodException: No signature of method: static com.funnelback.filter.api.DocumentType.fromContentType() is applicable for argument types: (String)

© 2015- Squiz Pty Ltd