Help Center

Menu

Funnelback 15.20.0 release notes

Released: 15 March 2019

Supported until: 15 March 2022 (Long Term Support Version)

New features

This release of Funnelback introduces:

  • New web-resources and a template editing interface with support for using local text editors via WebDAV.

Improvements

Knowledge graph is easier to implement

  • Documents lacking a node label and/or node name metadata are now ignored, which simplifies the use of single collections for knowledge graph and search.

  • The handling of quotes and spaces within node labels is now better supported.

  • Mentions relationships on any metadata field not yet declared with a relationship type are now created.

Search administration user logins have configurable session options and better auditing

  • Options to control the session lengths of search administration user logins, session refresh periods, and the ability to prevent concurrent logins are now available.

  • A dedicated log of search admin authentication attempts is now available. Repeated failures are tracked and warnings are logged if the failures in a specified time period exceed the threshold, where both the time period and threshold are configurable values.

Incorporating existing branding assets into search results pages is easier

  • To simplify the incorporation of existing branding assets into search result pages, CSS selector support for the IncludeURL template macro is now available.

Miscellaneous improvements

  • The underlying storage for accessibility auditor’s historical data has been redesigned, resulting in greatly improved performance and memory usage.

  • The Twitter collection data model for HTTPS URLs has been improved and now provides more information in retweets.

  • The debounce delay for Concierge auto-completion is now configurable.

  • To minimize the impact of historically clicked search results on search results ranking, -weight_only_fields=[K] was added as the default value to query_processor_options.

  • Added an option to specify the waiting period after accessibility auditor last recorded data, before it records data again. For more information, see accessibility-auditor.min-time-between-recording-history-in-seconds.

  • Auto completion config options (auto-completion.*) are now read from profile.cfg in preference to collection.cfg within FreeMarker templates. Previously, such options set in profile.cfg would have been ignored.

Upgrades to Funnelback components

  • The default version of jQuery used in the default template has been upgraded to 3.3.1.

  • The default version of Font Awesome Free used in the knowledge graph widget has been upgraded to 5.7.2.

  • The Java JDK bundled with Funnelback was updated to OpenJDK11. This should reduce memory usage in many cases.

  • The Groovy language bundled with Funnelback was upgraded to 2.5.6.

Bug fixes

The following bugs and issues were addressed in this release:

  • The case sensitivity in the content auditor’s undesirable text filter was incorrect.

  • The permissions for user management screens were incorrect. The fix results in user management screens being available by default rather than requiring sec.administer-system.

  • The CSVtoXML filter failed on URIs that have an empty host for example file:///path.

Important changes

This section contains important information you should know before upgrading to this release of Funnelback. This information relates to changes that may affect:

  • your existing installation of Funnelback, as well as

  • 3rd-party components Funnelback depends upon.

Changes to configurations

The following changes are automatically performed on all configurations during the upgrade process.

  • New permissions sec.template and sec.web-resources are assigned to users who have file-manager permission to edit either simple.ftl or *.ftl in preview-display-rules.

  • The permission sec.site-profile is assigned to users who have file-manager permission to edit site_profile.cfg.

  • References to ${SearchPrefix}thirdparty in FreeMarker templates are replaced with ${GlobalResourcesPrefix}thirdparty. This was the case prior to Funnelback 15.6 but this functionality was inadvertently removed from version 15.6.

  • All users have their version section updated, setting version=15.20.

Following the upgrade to Funnelback 15.20, configurations migrated from older versions of Funnelback (15.18 and earlier) will need update-configs.pl to be run manually to apply these changes.

Upgrade notes

  • URLs, filenames and metadata classes previously containing the abbreviation 'fkg' for Funnelback knowledge graph are now denoted 'kg'. Therefore, all implementations will need to compensate for the resulting changes to metadata class names, URL paths and file paths.

  • Knowledge graph node labels are now automatically cleaned up to ensure they are valid identifiers. Any implementation relying on labels that do not meet the restrictions will need to be updated to account for this.

  • Directory and DB collections now write URIs with an empty authority e.g. local:///serve-db-document.tcgi instead of local://serve-db-document.tcgi. Padre now allows an authority in all URIs it can canonicalize. Previously, Padre only allowed a host in http, https and ftp URIs. With this change a slash may be added to existing URIs e.g. local://serve-db-document.tcgi > local://serve-db-document.tcgi/ as serve-db-document.tcgi is assumed to be the authority. This can be fixed by a full re-crawl in directory and DB collections. Push will do the same conversion on upgrade, although it may be necessary to empty the push collection and re-add documents in the form scheme:///path?query=part.

  • To turn off default setting of weight_only_field, set it to empty value e.g. query_processor_options=-weight_only_fields=. Note that the -sco=2[K] setting (in contrast to -sco=2[ANYTHING-ELSE]) will be considered as weight-only fields.

  • The option accessibility-auditor.min-time-between-recording-history-in-seconds defaults to 20 minutes for Push collection, previously it was effectively zero.

  • To upgrade jQuery to 3.3.1, update the path to resource in FTL forms from ${GlobalResourcesPrefix}js/jquery/jquery-1.10.2.min.js to ${GlobalResourcesPrefix}thirdparty/jquery-3.3.1/jquery.min.js. Note that previous versions of jQuery will be removed from Funnelback in a future release.

  • To upgrade Font Awesome to v5.7.2, update the path to resource in FTL forms from ${ContextPath}/${GlobalResourcesPrefix}thirdparty/font-awesome-4.7.0/css/font-awesome.min.css to ${ContextPath}/${GlobalResourcesPrefix}thirdparty/font-awesome-5.7.2/css/font-awesome.min.css.

  • Due to the bundled Groovy language upgrade to version 2.5.6, review Groovy’s changelog and in particular the list of breaking changes to assess the impact on any existing Groovy scripts or classes.

  • Due to the bundled Java JRE being to version 11, any custom JVM flags provided via java_options or service files may need to be re-assessed to ensure they are compatible and still beneficial.

  • The $SEARCH_HOME\wbin\java\jre and $SEARCH_HOME/linbin/java/jre files no longer exist in the bundled Java runtime. Any references to these files should be updated to use the java.home system property. For pre and post update commands, $GROOVY_COMMAND can be used instead.

Linux version support

  • Funnelback 15.20 is the last version of Funnelback to support Red Hat Enterprise Linux (RHEL) and CentOS version 6. Future Funnelback versions will require at least RHEL or CentOS version 7.

Known issues

Some components within Funnelback, notably the the Groovy language runtime, produce warnings under JDK11 similar to the following in their current versions.

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by (some class)
WARNING: Please consider reporting this to the maintainers of (some class)
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

These warnings are harmless and will be removed once components such as Groovy are themselves updated.

Patches

Type Release version Description

3 Bug fixes

15.20.0.39

Fixed the security vulnerability where Spring Framework RCE may be vulnerable to remote code execution (RCE) via data binding [CVE-2022-22965]

3 Bug fixes

15.20.0.38

Upgrades log4j2 to version 2.17 to fix the security vulnerability where Apache Log4j2 Thread Context Lookup Pattern is vulnerable to remote code execution in certain non-default configurations.

3 Bug fixes

15.20.0.37

Upgrades log4j2 to version 2.15 to fix the security vulnerability where log4j2 JNDI features do not protect against attacker-controlled LDAP and other JNDI related endpoints.

3 Bug fixes

15.20.0.36

Fixes a cross-site scripting vulnerability in Freemarker templates.

3 Bug fixes

15.20.0.35

Fixes an issue where sessions are not terminated on logout events triggered by perl pages.

3 Bug fixes

15.20.0.35

Fixes an XXE issue where input to the webdav endpoint could be manipulated to trigger http requests.

3 Bug fixes

15.20.0.34

Fixes an issue where Faceted Navigation extra searches may fail because of an index out of bounds error.

3 Bug fixes

15.20.0.34

Improves logging when extra searches take too long.

3 Bug fixes

15.20.0.33

Fixes an issue where deleting the collection property from collection.cfg and then deleting the collection itself would delete other configuration data.

3 Bug fixes

15.20.0.32

Fixes ip pseudonymization when Funnelback is behind a load balancer and client ip details are in the x-forwarded-for header.

3 Bug fixes

15.20.0.31

Reduces memory consumption and improves performance of the purge sessions endpoint

3 Bug fixes

15.20.0.30

Upgrades the version of RestFB library to account for recent breaking changes in the Facebook Graph API.

3 Bug fixes

15.20.0.30

Fixes an issue where Facebook collections gathered less number of documents due to a pagination issue in the Facebook Graph API.

3 Bug fixes

15.20.0.30

Facebook Graph API deprecated fields name, link, app_links and description have been removed from the default values of the facebook.page-fields configuration key.

3 Bug fixes

15.20.0.29

Fixes an XML formatting issue in Faceted Navigation click logs.

3 Bug fixes

15.20.0.28

Fixes a bug in which white space was not preserved in summaries from anchor text when the -map indexer option is enabled.

3 Bug fixes

15.20.0.27

The Push API client used in multi server push now has timeouts enabled allowing it to abandon problematic HTTP requests.

3 Bug fixes

15.20.0.26

Removes the screens for file-manager rule editing which could create security issues

3 Bug fixes

15.20.0.26

Fixes an issue where support packages could contain unintended files

3 Bug fixes

15.20.0.26

Fixes an issue where the running Funnelback jetty web server could retain permissions via supplemental groups after startup

3 Bug fixes

15.20.0.26

Limits an administration CGI script to redirect only within the Funnelback administration interface as intended

3 Bug fixes

15.20.0.26

Removes the unused administration debug.cgi script which reflected input parameters without proper escaping

3 Bug fixes

15.20.0.25

Fixes a bug where a horizontal display of columns in auto-completion dropdown doesn’t work.

3 Bug fixes

15.20.0.25

Fixes a bug where insecure operation on CSS files list was performed when CSS file was exposed via the same domain as auto-completion widget but different port.

3 Bug fixes

15.20.0.24

Fixes a bug introduced in the previous patch in which uploading configuration files in the administration dashboard stopped working.

3 Bug fixes

15.20.0.23

Prevents XSS vulnerabilities found in the classic administration dashboard.

3 Bug fixes

15.20.0.22

Move Funnelback service pid files to /var/run which is required by OS updates to systemd.

3 Bug fixes

15.20.0.22

Include some additional metadata in service template files.

3 Bug fixes

15.20.0.21

Fixed an issue where the crawler would follow <meta http-equiv="refresh"> redirects that appeared within html comments. Redirects inside comments are now ignored.

3 Bug fixes

15.20.0.20

Improves support for running faceted navigation on extra searches.

3 Bug fixes

15.20.0.20

Adds method 'getEffectiveExtraSearchName()' to the search transaction which gets the name of the extra search this search should be considered to be under. The result of this should be used when modifying a particular extra search. As Funnelback may create extra searches under an existing search, for example for faceted navigation, this could be used to work out if the search transaction should be modified.

3 Bug fixes

15.20.0.20

Fixes errors in the sorting of faceted navigation values, which could cause a HTTP 500 error code.

3 Bug fixes

15.20.0.19

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates escaped using output formats by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

15.20.0.18

Fixes a bug introduced in 15.20.0.16, in which all paths in XML would be considered as document titles.

3 Bug fixes

15.20.0.18

Fixes a bug introduced in 15.20.0.16. Empty XML elements mapped as the document URL are now ignored.

3 Bug fixes

15.20.0.18

Fixes a memory leak by disabling the conscrypt SSL provider.

3 Bug fixes

15.20.0.17

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

15.20.0.17

Please note, this patch was retracted due to an incomplete solution causing template errors when used with certain Freemarker escaping modes. The 15.20.0.19 patch, which addresses this issue, should be used instead.

3 Bug fixes

15.20.0.16

Improves the task-picker such that it can load dependencies from custom 'jar' files located '$SEARCH_HOME/lib/java/task-picker/'.

3 Bug fixes

15.20.0.16

Fixes a bug in which spaces would be removed from query biased summaries which came from 'cdata' sections of XML.

3 Bug fixes

15.20.0.16

⚠ Similar to version 15.22.0 of Funnelback, The padre indexer XML parser is now less lenient when indexing numeric, date and geo location metadata. Previously, elements that were not correctly closed, such as <v>123</oops>, could have been mapped to //v. In this version of Funnelback, such mappings will no longer work. This is inline with how other metadata is mapped.</v>

3 Bug fixes

15.20.0.15

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

15.20.0.14

Fixes an issue where relationships labels in the knowledge graph widget have been created incorrectly.

3 Bug fixes

15.20.0.14

Fixes an issue where sorting of related entities has not been applied based on knowledge graph template configuration.

3 Bug fixes

15.20.0.14

Fixed an issue where a double request to search endpoint in the knowledge graph widget has been sent on pressing enter key when search input is focused.

3 Bug fixes

15.20.0.14

Improve the performance of the Accessibility Auditor interface by requesting less data.

3 Bug fixes

15.20.0.14

Fixes an issue where some of the text on the Accessibility Auditor dashboard was showing out of date information.

3 Bug fixes

15.20.0.14

Fixes an issue where the Accessibility Auditor dashboard would not generate the thumbnail screenshots for each domain.

3 Bug fixes

15.20.0.13

Fixes the web crawler so that it uses the 'crawler.user_agent' collection.cfg option.

3 Bug fixes

15.20.0.12

Reduces the memory needed by the Accessibility Auditor history APIs.

3 Bug fixes

15.20.0.11

Fixes an issue where facet navigation URLs in knowledge graph widget search view were double prefixed with a domain.

3 Bug fixes

15.20.0.11

Fixes an issue where knowledge graph public /type and /rels APIs have not been converted to absolute URLs.

3 Bug fixes

15.20.0.11

Fixes an issue where results URLs from filecopy collection have not been resolved correctly due to their relative syntax. New knowledge graph widget parameter urlPrefix has been added to allow converting relative results URLs to absolute ones.

3 Bug fixes

15.20.0.11

Improves the query response time when sorting.

3 Bug fixes

15.20.0.10

Improves the query response time when sorting by title.

3 Bug fixes

15.20.0.10

Fixes an issue where large (>2GB) index.dt files would cause padre-gs to fail when setting gscopes.

3 Bug fixes

15.20.0.9

Fixes an issue where the browser was displaying a mixed-content error when the Knowledge Graph API sometimes returned URLs with mixed protocols.

3 Bug fixes

15.20.0.9

Fixes an issue where the Knowledge Graph widget returned a cross-origin resource sharing policy error. The Knowledge Graph API now supplies Access-Control-Allow-Origin headers.

3 Bug fixes

15.20.0.9

Fixes an issue where upgrading user from an older version of Funnelback would override sec.faceted-navigation from yes to no.

3 Bug fixes

15.20.0.8

Fixes an issue where the WebDAV endpoint would not re-prompt for HTTP Basic authentication when invalid credentials were provided, which could lead to a redirect loop in the Cyberduck WebDAV client.

3 Bug fixes

15.20.0.8

Fixes an issue where the knowledge-graph.max_heap_size was not applied when updating knowledge graph.

3 Bug fixes

15.20.0.7

Fixes an issue where the update could fail during the record Accessibility Auditor history steps after migrating a collection from an older version.

3 Bug fixes

15.20.0.7

Fixes an issue where the deprecated wcag check (replaced by Accessibility Auditor) could cause the update to fail because the 'wcag-journal.log' file could not be found.

3 Bug fixes

15.20.0.6

Fixes an issue where jetty stopped logging after deploying knowledge-graph.

3 Bug fixes

15.20.0.6

Fixes an issue where Facebook collections did not update due to the recent Graph API changes.

3 Bug fixes

15.20.0.6

Fixes an issue in the Knowledge Graph widget where the value of the targetUrl query parameter would be truncated if it contained a '?' character.

3 Bug fixes

15.20.0.6

The API fields that are requested from Facebook can now be specified in collection.cfg using the facebook.page-fields, facebook.post-fields and facebook.event-fields configuration keys. This should enable future changes to Facebook APIs to be handled without requiring further Funnelback patches.

3 Bug fixes

15.20.0.5

Improves how padre query biased summaries are generated such that spaces are added based on the source document. This prevents issues where extra spaces were added or removed.

3 Bug fixes

15.20.0.5

Fixes a bug where Directory collections would strip some unicode characters.

3 Bug fixes

15.20.0.4

Disable HTTP/2 in underlying HTTP library which caused socket timeout errors during crawling

3 Bug fixes

15.20.0.4

Fixes an issue where jetty would terminate on invalid 'index.autoc' (query completion) files.

3 Bug fixes

15.20.0.3

Fixed an issue where some config settings (e.g. ui.modern.search_link) began returning null when empty-string was expected in the template data model.

3 Bug fixes

15.20.0.3

Scoped search triggered in knowledge graph UI to results existed in graph database

3 Bug fixes

15.20.0.3

Ensured initial targetUrl exists in graph database when browsing knowledge graph UI

3 Bug fixes

15.20.0.3

Ensured collection parameter is set correctly in knowledge graph UI when knowledge graph API is returning error

3 Bug fixes

15.20.0.3

Fixed setting of knowledge graph labels in administration dashboard for relationships with undirected direction

3 Bug fixes

15.20.0.3

Fixed validation for detecting duplicated knowledge graph labels in administration dashboard for "metadata" category

3 Bug fixes

15.20.0.3

Added missing translations keys and side help for knowledge graph updates in task queue administration dashboard

3 Bug fixes

15.20.0.3

Fixed copy tool in knowledge graph relationships administration dashboard to create relationship with unique and valid name

3 Bug fixes

15.20.0.3

Prevented from creating self-reference "mention" relationship in knowledge graph

3 Bug fixes

15.20.0.2

Makes the funnelback-graph service run as the Funnelback user on Unix.

3 Bug fixes

15.20.0.2

Fixes an issue that prevents the funnelback-graph service from restarting when requested.

3 Bug fixes

15.20.0.1

Fixes an issue that prevents scheduled tasks from appearing in the Administration interface on Windows Server 2016.