Help Center

Menu

Funnelback 16.2.0 release notes

Released: 31 Mar 2021

Supported until: 31 Mar 2024

This applies to Funnelback dedicated v16.2. For Squiz DXP please see the Funnelback 16 change log.

The 16.2 Funnelback release includes all changes since the previous 15.24.0 release.

Use the filter to narrow down what’s in the list below, and the sort controls on each column to help locate what you are looking for.

Type Release version Description

1 New and revised features

16.2

Added a new upgrade utility to support the migration and upgrade of collections from Funnelback v15.24 to v16.

Upgrading from Funnelback 15 and earlier into Funnelback 16 is complicated due to a number of architectural changes within Funnelback.

The upgrade utility:

  • automates many of the tasks required to perform the upgrade.

  • can transfer the search from a v15 server. This includes configuration, data and users.

1 New and revised features

16.2

Added a new configuration file, v15collectionremap.cfg, to support the mapping of v15.24 and earlier style collection IDs to v16 search package/data source IDs for public search and push API endpoints.

1 New and revised features

16.2

Added various API calls to support the upgrade tool.

4 Important changes

16.2

Collection scheduled updates times are not preserved when upgrading from 15.24 and earlier.

These must be recreated for all the data sources by applying configuration keys to each data source.

See: scheduling data source updates. In addition, if running and in-place upgrade (running the v16 installer in upgrade mode on an older Funnelback server) any pre-existing cron jobs for collection updates will need to be manually removed by editing the search user’s crontab.

4 Important changes

16.2

Collection ID format change when upgrading from v15.24

There is a breaking change for collection IDs when upgrading from v15.24 and earlier. This will break existing integrations with Funnelback.

System administrators can configure collection remapping using the v15collectionremap.cfg file to partially mitigate these issues.

4 Important changes

16.2

v15 default roles are assigned to a default client on upgrade from Funnelback 15.

Default roles will be prefixed with the default_roles internal client ID when upgrading from v15.

e.g. the v15 role default-administrator is upgraded to default_roles~default-administrator in v16.

4 Important changes

16.2

The Funnelback installer now automatically rebuilds indexes after an upgrade.

In Funnelback 15, this was a step that had to be manually triggered when an administrator logged into Funnelback after an upgrade.

As a result search indexes will be unavailable until the reindexing process is complete.

4 Important changes

16.2

Upgrading to v16 may affect ranking on any collections which were not of the web or meta types due to the automatic wrapping of such collections in a meta collection/search-package. Explicitly setting -SSS=0 as a query processor option on any affected search package will restore the previous ranking behaviour for non-web, non-meta collections.

1 New and revised features

16.1.2108

Added support for Funnelback for higher education in the Squiz DXP.

The v16 release of Funnelback for higher education includes a tool for performing an initial setup of the package.

The setup tool is currently only available for use by Squiz implementation teams.

1 New and revised features

16.1.2108

Added an auto-completion plugin which generates structured auto-completion to be generated from an existing Funnelback index.

The auto-completion plugin enables structured auto-completion to be generated by:

  • Setting a number of configuration options that define the trigger and action

  • Configuring the set of metadata fields to include in the auto-completion JSON response

1 New and revised features

16.1.2108

Added a new enhanced update task scheduler.

The new update task scheduler enables updates to be configured by a search administrator with schedules based on time between updates, or scheduled at a fixed time.

The scheduler replaces the OS scheduler (cron) and existing scheduled updates must be manually set up in the new scheduler.

1 New and revised features

16.1.2108

Added the ability to log in to the administration dashboard using a simple user ID that omits the client ID. This enables login with an email addresses as the ID if it has been used for the username when setting up the account.

1 New and revised features

16.1.2108

Added support for a global plugin to validate all content updates added to the files via the admin-api.

An example use case for this is to provide virus scanning of content that is submitted via the administration interface (or via the API).

1 New and revised features

16.1.2108

Added a new screen to the administration dashboard for the management of client application tokens.

A new user permission, sec.application-token.non-expiring.create, is required for access to this screen and the underlying API.

1 New and revised features

16.1.2108

Added a new API (services/v1/services/{service}/thumbnail) for the management of thumbnails used for results pages.

1 New and revised features

16.1.2108

Added a new API (/plugins/v1/plugins/version/{version}/debug/run-on-index) that enables a user to debug a plugin to see the result of running the plugin on an index.

1 New and revised features

16.1.2108

Added a new API (/config/v2/server/keys/intercom.id) and configuration option (intercom.id) for configuring the Intercom ID. Users require the sec.server.config and edit.key.intercom.id permissions to modify the value.

1 New and revised features

16.1.2108

Added a new API (/clients/v1/clients) for creating clients. A new user permission sec.clients.create has also been added to control access to this API.

1 New and revised features

16.1.2108

Added a new API (/internal/clients/{client-id}/collections/{collection-id}) for adding a collection (search package or data source) to a client.

2 Minor improvements

16.1.2108

Updated the push API (/v1/collections/{collection}/restore/from-remote) to support v15 remote servers.

2 Minor improvements

16.1.2108

Search packages analytics updates are now automatically scheduled when they are created.

2 Minor improvements

16.1.2108

Updated the client~primary administration role so that administration users with this role are able to grant the client~primary role to other administration users or tokens.

2 Minor improvements

16.1.2108

Added support for system administrators to define range restrictions on configuration keys with a duration type value.

3 Bug fixes

16.1.2108

Fixed an issue with accessibility auditor that caused the user interface to break when the user’s web browser language was not set to English.

3 Bug fixes

16.1.2108

Fixed an issue with the web crawler that caused it to ignore the http_source_host configuration option.

3 Bug fixes

16.1.2108

Fixed an issue with best bets where it failed to remove a search result that had the same URL as the best bet URL when the removing matching search result option was selected.

3 Bug fixes

16.1.2108

Fixed an issue with the web crawler form interaction feature that caused it to incorrectly handle form parameters with empty values.

3 Bug fixes

16.1.2108

Fixed an issue that caused invalid XML to be returned when the query contained a vertical tab character.

3 Bug fixes

16.1.2108

Fixed an XXE vulnerability in cached documents that allowed files to be read from disk.

4 Important changes

16.1.2108

Updated the default configuration to enable accessibility auditor by default. Accessibility auditor can be disabled on a data source by setting the accessibility-auditor.check configuration option to false.

4 Important changes

16.1.2108

The outliers (trend alerts) cron job is no longer able to be scheduled via the administration dashboard. A system administrator can update or delete this scheduled update by editing the search user’s crontab.

4 Important changes

16.1.2108

Legacy best bets (from v14.2 and earlier) have been removed.

This should not be confused with, and does not affect, the v15.0 and newer implementation of best bets (which is managed via the marketing dashboard).

The removal of legacy best bets has the following implications:

  • best_bets.cfg is no longer supported. Existing best_bets.cfg files will now be ignored and should be removed.

  • The response.resultPacket.bestBets element has been removed from the search data model

  • The s.BestBets macro has been removed from the funnelback_classic Freemarker library.

  • The best bets auto-completion source supported by build_autoc has been removed.

  • Best bets related options have been removed from padre.

Any legacy best bets should be manually re-entered via the best bets editor located within the marketing dashboard.

4 Important changes

16.1.2108

Funnelback no longer supports additional administration service providers or single logout (SLO) when using SAML authentication.

Existing SAML configurations and the Groovy permission mapper may need to be updated.

1 New and revised features

16.1.2041

Added support for client API access tokens, which enabled applications and scripts to access the admin API using a token that is independent of specific administration users.

2 Minor improvements

16.1.2041

Added warnings to the administration dashboard when license limits are exceeded.

2 Minor improvements

16.1.2041

Improved the admin API’s JSON representation of timezone and other date/time related config settings.

2 Minor improvements

16.1.2041

Introduced support for higher-performance zstd compression in push document storage.

2 Minor improvements

16.1.2041

Improved presentation of errors from SAML user mapper scripts.

2 Minor improvements

16.1.2041

Improved performance of administration dashboard pages showing large numbers of data sources.

2 Minor improvements

16.1.2041

Improved administration dashboard messaging when Funnelback server is in read only mode.

2 Minor improvements

16.1.2041

Eliminated the use of jsessionid cookies when using SAML authentication.

2 Minor improvements

16.1.2041

Product documentation links now refer to the hosted documentation which can be updated when errors are identified.

2 Minor improvements

16.1.2041

Added user interface for enabling plugins.

2 Minor improvements

16.1.2041

Improved administration dashboard breadcrumbs in a number of areas.

2 Minor improvements

16.1.2041

Added client application tokens, an improvement for application tokens, suitable for use on the Squiz DXP

2 Minor improvements

16.1.2041

Added plugin servlet filter hooks, a replacement for custom servlet filter hooks, suitable for use on the Squiz DXP

3 Bug fixes

16.1.2041

Fixed handling of missing originalQuery entries in search query history.

3 Bug fixes

16.1.2041

Fixed preservation of URL parameters through the SAML login flow.

3 Bug fixes

16.1.2041

Fixed broken funnelback_documentation search result template.

3 Bug fixes

16.1.2041

Fixed support for access_restriction settings based on hostnames when Funnelback is deployed behind a load balancer.

3 Bug fixes

16.1.2041

Fixed accessibility auditor to account for the removal of the legacy facet data model.

3 Bug fixes

16.1.2041

Fixed display of default web collection file type restrictions.

3 Bug fixes

16.1.2041

Fixed return key handling when adding web data source URLs.

3 Bug fixes

16.1.2041

Fixed a number of cases of old 'collection' terminology usage within administration dashboard.

4 Important changes

16.1.2041

The legacy metaData section of the result data model has been removed. listMetadata should now be used instead.

1 New and revised features

16.1.2034

Introduced screens for managing plugins and enabling/disabling them on search packages and result pages.

2 Minor improvements

16.1.2034

Added instant-update and push logs to data source log viewing and WebDAV.

2 Minor improvements

16.1.2034

Analytics email settings are now configured in result page config instead of a dedicated file.

2 Minor improvements

16.1.2034

Improved the performance of the tuning dataset administration dashboard when many entries are added.

2 Minor improvements

16.1.2034

Added a message to the administration dashboard when the server is set to read-only mode.

2 Minor improvements

16.1.2034

Simplified the search transaction data model by removing some deprecated sections.

3 Bug fixes

16.1.2034

Fixed handling of profiles containing hyphens in the legacy perl CGI administration pages.

3 Bug fixes

16.1.2034

Fixed handling of permissions within some areas of the administration dashboard to disable inaccessible options.

3 Bug fixes

16.1.2034

Accounted for the new client ID requirement within LDAP authentication.

4 Important changes

16.1.2034

The removal of some data model elements to simplify its usage may cause some freemarker templates, plugins and hook scripts to fail when they reference the removed elements.

These implementation components must be updated to work with this and future versions of Funnelback. Specific data model elements removed include:

  • The legacy facet/categories.

  • The duplicate resultsWithTierBars section.

  • The rawInputParameters and inputParameterMap sections - inputParameters should now be used instead.

See: Data model changes in Funnelback v16

1 New and revised features

16.1.2032

New administration dashboard, terminology and client separation

Funnelback’s administration dashboard has been overhauled to make search implementation easier.

As part of this improvement, new terminology of search packages, data sources and results pages has been introduced. Broadly, search packages are equivalent to meta collections in earlier versions of Funnelback, and a search package is now required to bundle up the data sources and results pages used to deliver a search. Data sources are equivalent to non-meta collections in earlier versions (but without the ability to serve search results), while result pages are the new equivalent of profiles and are used to define the search results page functionality and formatting.

The new administration dashboard also formalises the separation of implementations in a multi-tenant Funnelback environment with the concept of a client, which groups together all components of an implementation.

1 New and revised features

16.1.2032

Plugins for reusable custom code

A new bundled format for custom code to be run on the Funnelback server, known as a plugin, has been introduced to the product.

Plugins are intended to separate customisations from individual data sources and results pages so that they can be more easily reused and can go through a separate approval process to ensure quality in multi-tenant environments.

Plugins provide similar capabilities to custom Groovy scripts in earlier versions and add the ability to supply some types of data, such as external metadata, dynamically during a data sources update to avoid the need to assume the internal file system layout.

The features superceded by plugins - Groovy document and Jsoup filters, hook scripts, custom workflow and custom gatherers are no available and any existing implementations that are updated to Funnelback 16 must rewrite this custom code using the plugin framework.

1 New and revised features

16.1.2032

The support for performing the following tasks via REST APIs has been introduced:

  • Creating and deleting results pages.

  • Reading log files (via WebDAV).

  • Reading/editing of gscopes.cfg and external_metadata.cfg (via WebDAV).

  • Determining the update progress of a data source

  • Determining when the last successful update of a data source occurred.

  • Determining whether a running task has been asked to stop.

  • Deleting data sources and search packages."

2 Minor improvements

16.1.2032

The query_processor_options configuration setting is now supported on results pages, overriding whatever is set on the search package configuration.

2 Minor improvements

16.1.2032

The individual data source components and associated relative weightings of data sources in search packages can now be set via the configuration APIs with the meta.components and meta.components.[component].weight settings. When modified, these settings will be automatically applied to the search package.

2 Minor improvements

16.1.2032

Streamlined knowledge graph administration experience.

2 Minor improvements

16.1.2032

Tuning runs are now subject to the same task queueing system as data source and analytics updates.

2 Minor improvements

16.1.2032

Updates to the default template to support results page level configuration settings where possible.

2 Minor improvements

16.1.2032

Added support for knowledge graph scripts at the results page level.

2 Minor improvements

16.1.2032

The data source components of a search package can now be set within the configuration editing screens.

2 Minor improvements

16.1.2032

A number of improvements to SAML authentication support, in particular reduction in the number of SAML service providers required for administration setups and to support integration with Auth0.

2 Minor improvements

16.1.2032

Added a Content-Type response header to the push API endpoint

2 Minor improvements

16.1.2032

The 'Intercom' support tool has been integrated into the Funnelback administration dashboard.

3 Bug fixes

16.1.2032

Prevented WebDAV clients which take long-timeout locks and do not reliably release them from locking out other clients.

3 Bug fixes

16.1.2032

Fixed recommender operation on filecopy data sources.

3 Bug fixes

16.1.2032

Fixed the all-results.json endpoint to handle when search sessions are enabled.

3 Bug fixes

16.1.2032

Fixed presentation of sparklines within trend alerts reports.

3 Bug fixes

16.1.2032

Fixed possible configuration setting loss when encrypting configuration values for the first time after installation.

3 Bug fixes

16.1.2032

Prevented creation of users with service user prefixes.

3 Bug fixes

16.1.2032

Fixed the web crawler to handle responses without a Content-Type header.

3 Bug fixes

16.1.2032

Improved Padre handling of invalid XML characters.

3 Bug fixes

16.1.2032

Fixed handling of ui.modern.pseudonymise_client_ips when Funnelback is used behind a proxy or load balancer.

3 Bug fixes

16.1.2032

Fixed publication of web resources files containing spaces in their filenames.

3 Bug fixes

16.1.2032

Fixed isAdminUI Freemarker macro when search and administration ports are the same.

3 Bug fixes

16.1.2032

Fixed consistency of status codes resulted by the update history API.

4 Important changes

16.1.2032

Tuning is now a task under the task queue, this means when it runs can be controlled by the task picker. Tuning can no longer be started by post /search-quality/v1/tuning/collections/{collection}/profiles/{profile}/runs?action=START_TUNING, instead it can only be started by using the task queue API: post /task/v1/queue/TUNING.

4 Important changes

16.1.2032

The task queue now allows running tasks to be added to the queue and allows multiple tasks which use the same resources (for example, the same data source), to be in the queue at the same time.

4 Important changes

16.1.2032

query_processor_options now supports configuration environments.

4 Important changes

16.1.2032

The meta.cfg config file no longer exists, being replaced by the meta.components search package configuration setting.

4 Important changes

16.1.2032

The groovy script specified by the auth.admin.saml.groovy-permission-mapper setting now supports defining roles the user is always permitted to edit.

4 Important changes

16.1.2032

The administration dashboard’s edit file-manager rules pages are no longer available. Any remaining cases where custom file manager rules are required must be set directly in the relevant .ini files.

4 Important changes

16.1.2032

Naming of log files has been made more consistent between data source types. For example, crawl.log is now named gather.log to be consistent across the product.

Renamed log files include:

  • crawler.central.log → crawler.log

  • crawl.log → gather.log

  • crawl.log.<N> → crawl.<N>.log

  • dbgather.log → gather.log

  • directory_gather.log → gather.log

  • social_media.log → gather.log

  • filecopier.log → gather.log

4 Important changes

16.1.2032

Local data sources are not supported in this version, reflecting the restriction on direct filesystem access in the AWS SaaS environment. Existing local collections should be converted to either a web or custom data sources depending on the logic implemented in the local collection.

4 Important changes

16.1.2032

Search package and data source IDs are now limited to 192 characters

4 Important changes

16.1.2032

Freemarker templates are now only able to import/include files which end in .ftl

4 Important changes

16.1.2032

Removed the deprecated collection_root and data_root configuration keys.

4 Important changes

16.1.2032a

The question.collection.configuration data model exposed to Freemarker no longer contains server (global) configuration settings.

4 Important changes

16.1.2032

The padre_opts.cfg profile configuration file has been removed. Query processor options specified within this file should now be set in the results page configuration, query_processor_options. The behavior has also changed slightly - the query processor options now override any options specified in the search package or default server configuration. This behavior differs from padre_opts.cfg in Funnelback 15.24 and earlier which merged the options.

4 Important changes

16.1.2032

SAML authentication now creates user .ini files representing logged in users to support cross-component authentication.

4 Important changes

16.1.2032

The _default profile is now hidden in the administration dashboard, but can still be accessed via WebDAV.

4 Important changes

16.1.2032

Added a post_collection_create_script and pre_collection_delete_script server configuration options that run a defined script when a collection is created or deleted. The pre_collection_delete_script supersedes the pre_collection_delete_command from previous versions.

4 Important changes

16.1.2032

Removed several Mediator API calls.

The following Mediator API calls have been removed and can no longer be accessed via the Mediator REST endpoint or via mediator.pl:

  • GetConfiguration

  • GetIndexTime

  • RecordStatistics

  • StartFilecopySlave

  • ValidateExternalMetadata

4 Important changes

16.1.2032

Removed support for the angularJS sessions widget (funnelback-sessions-1.0.0.js or funnelback-sessions.js).

4 Important changes

16.1.2032

Removed support for the search.classic and padre-sw.cgi legacy search endpoints.

4 Important changes

16.1.2032

Removed support for the slack collection type.

4 Important changes

16.1.2032

Removed support for multi-server filecopy gathering.

4 Important changes

16.1.2032

Knowledge graph

  • Previous versions of Funnelback used a reserved id metadata class to hold the knowledge graph identifier which caused compatibility issues if id was required for another purpose. This has been replaced with a FUNkgLiveUrl metadata class. Any systems using the knowledge graph nodes endpoint directly will need to account for this change as part of upgrading to this and future versions.

  • All knowledge graphs must be manually updated after upgrading to support the metadata class change noted above.

3 Bug fixes

16.2.0.25

Prevented the creation of objects within Freemarker template files to ensure that template editors can not cause external code to be executed.

1 New and revised features

16.2.0.25

Added new server configuration keys to configure the Jetty HTTP connection.

3 Bug fixes

16.2.0.24

Fixed an issue where the search results were skewed by ranking option cool.23 'host_incoming_link_score'.

3 Bug fixes

16.2.0.23

Fixed the issue where numbers greater than 1000 were returning NaN while trying to calculate percentages in content auditor.

3 Bug fixes

16.2.0.23

Fixed an issue with the calculation of main host scores that caused inaccurate and inconsistent information to be displayed in SEO auditor.

3 Bug fixes

16.2.0.22

Fixed an issue where the Perl script to trigger analytics update didn’t set the correct update type.

3 Bug fixes

16.2.0.21

Fixed an issue that prevented internal documentation within the administration dashboard from displaying in some circumstances.

3 Bug fixes

16.2.0.20

Fixed an issue that caused administration dashboard labels to display intermittently.

3 Bug fixes

16.2.0.19

Fixed an issue where the Freemarker template upgrader incorrectly upgraded custom variables named metaData.

3 Bug fixes

16.2.0.19

Fixed an issue where trend alerts notifications were not generated.

3 Bug fixes

16.2.0.19

Fixed an issue preventing the correct export of content auditor and accessibility auditor documents in CSV format.

3 Bug fixes

16.2.0.18

Fixed an issue where fetching Facebook comments would cause an infinite loop due to changes within the Facebook endpoints.

3 Bug fixes

16.2.0.17

Fixed a security vulnerability where jackson-databind might allow remote attackers to execute arbitrary code or conduct external XML entity (XXE) attacks.

3 Bug fixes

16.2.0.17

Fixed a security vulnerability where com.google.oauth-client hasn’t implemented PKCE support in accordance with the RFC for OAuth 2.0 for Native Apps.

3 Bug fixes

16.2.0.16

Fixed the security vulnerability where Spring Framework RCE may be vulnerable to remote code execution (RCE) via data binding [CVE-2022-22965]

3 Bug fixes

16.2.0.15

Upgrades log4j2 to version 2.17 to fix the security vulnerability where Apache Log4j2 Thread Context Lookup Pattern is vulnerable to remote code execution in certain non-default configurations.

3 Bug fixes

16.2.0.14

Upgrades log4j2 to version 2.15 to fix the security vulnerability where log4j2 JNDI features do not protect against attacker-controlled LDAP and other JNDI related endpoints.

3 Bug fixes

16.2.0.13

Fixes an issue where the collection tool would return an error for the index presence check.

3 Bug fixes

16.2.0.13

Restored access to data reports from the administration dashboard.

3 Bug fixes

16.2.0.13

Improves access to documentation for individual plugins from extensions administration dashboard.

3 Bug fixes

16.2.0.13

Fixes an issue where perl file manager throws an exception about untainted values when the users try to upload files.

3 Bug fixes

16.2.0.13

Fixes an issue where perl file manager throws an exception about untainted values when the users try to publish or delete files.

3 Bug fixes

16.2.0.13

Fixes an issue where the tuning results administration dashboard couldn’t help apply an outcome of the tuning run.

3 Bug fixes

16.2.0.12

Fixes an issue where the edit metadata mappings administration dashboard wouldn’t display counts of detected sources in searchable documents properly.

3 Bug fixes

16.2.0.11

Fixes an issue where rules defined in redirects.cfg wouldn’t work.

3 Bug fixes

16.2.0.10

Fixes an issue with the administration dashboard which prevents creation of push and custom data sources.

3 Bug fixes

16.2.0.9

Fixes create-collection.pl.

3 Bug fixes

16.2.0.8

Fixes an issue with push replication in SAML mode as push APIs return now 401 rather than 302 for not-authenticated requests.

3 Bug fixes

16.2.0.7

Search session cookies are now explicitly marked with SameSite=None;Secure to fix functionality in partial integrations.

3 Bug fixes

16.2.0.7

Fixes an issue in the DocumentFixerFilterProvider filter in which h1 and h2 elements containing only non-breaking spaces could be used as titles.

3 Bug fixes

16.2.0.6

Fixes an issue where faceted navigation UI would freeze due to numerous API requests done to check templates' backups for the usage of legacy facets.

3 Bug fixes

16.2.0.5

Fixes a cross-site scripting vulnerability in Freemarker templates.

3 Bug fixes

16.2.0.4

Enables users to log into the webdav endpoint without specifying a client id. This is now aligned with other the API endpoints.

3 Bug fixes

16.2.0.3

Fixed a bug with update of YouTube data sources when no channel ID is provided.

3 Bug fixes

16.2.0.2

Fixes a bug in reports-send-email.pl, which causes high load.

3 Bug fixes

16.2.0.1

Fixes a bug in filtering in which outlook files with attachments could not be parsed correctly.