Help Center

Menu

Funnelback 15.24.0 release notes

Released: 23 October 2019

Supported until: 23 October 2024

Improvements

In General

  • Passwords stored in global.cfg and collection.cfg (e.g. passwords to be used when connecting to other systems) are now stored in an encrypted form when they are added or edited.

  • Push collections now provide APIs for restoration to snapshots, including support for fetching the snapshot to be restored automatically from a remote server.

  • Improved include_patterns presentation in configuration interface to better handle large sets of patterns.

  • The default search results template now uses the Freemarker recommended HTML escaping mechanism rather than a legacy one.

  • The implementation of search sessions and cart functionality within the default search template is now simpler and has fewer library dependencies. The upgrading to session history plugin guide can be used for upgrading existing templates to make use of the simpler variants.

  • The configuration settings for contextual navigation, quick links and form interaction are now within the general collection configuration, allowing them to be set via the configuration interface or via the associated REST APIs.

  • Improved the performance of accessibility auditor on collections covering many domains.

  • /s/all-results.csv can now produce a custom csv file name by specifying the 'fileName' url parameter.

  • Improved the interaction with web resources directories containing unpublished changes.

  • Category headings can now be disabled in the list view on a per-entity basis within the knowledge graph widget.

  • Search results data model now includes more useful debugging information related to running internal queries against search indexes.

  • Improved support for faceted navigation with queries that contain bigrams (such as CJKT characters).

  • Improved the task-picker such that it can load dependencies from custom 'jar' files located '$SEARCH_HOME/lib/java/task-picker/'.

  • Improved query performance when many curator rules are defined for any profile under a collection.

  • Improved 'build_autoc' performance for profiles reducing update times.

  • Reduced the time taken by the update step ContentAuditorSummary.

  • Improved support for running faceted navigation on extra searches by adding a getEffectiveExtraSearchName() method to the search transaction object which can be considered when modifying the transaction in a hook script.

  • Web crawling logs now include timestamps and additional frontier debugging information.

Knowledge Graph Specific

  • Knowledge graph now models metadata with multiple values in a list form, allowing for correct presentation of values containing separator characters.

  • Knowledge graph configuration now allows a default template to be configured for entity types which do not have a dedicated template.

  • When sorted by date, knowledge graph entities are now grouped under useful headings showing the relative date.

  • Knowledge graph entities can now define a fallback thumbnail for use if some entities do not define their own thumbnail image.

  • Knowledge graph entities can now be created for items with no logical URL to send a user to, and are presented without a view link in this case.

  • Improved alignment of entity images and data within knowledge graph widget.

Upgrades to Funnelback components

  • The default version of Moment.js used in the default template and knowledge graph widget preview has been upgraded to 2.24.0.

  • Upgraded Freemarker library to 2.3.29 (from 2.3.27) which provides some new template syntax - See https://freemarker.apache.org/docs/versions_2_3_29.html.

  • The Maxmind database used for queries and search analytics has been upgraded from version 1 to version 2.

Bug fixes

  • Fixed an issue where the crawler would follow <meta http-equiv="refresh"> redirects that appeared within html comments. Redirects inside comments are now ignored.

  • The original target page is now loaded if the user is required to log-in before accessing an admin-ui page rather than the home page.

  • Fixed handling of curator rules with invalid null values.

  • Knowledge graph widget now returns to the top when paging through results.

  • Improved handling of knowledge graph entities with no valid node_id values.

  • Restored the remove-headers collection config settings ability to remove security related headers such as X-Frame-Options.

  • Fixed accessibility auditor’s handling of acknowledgement and collection IDs which produced log errors and prevented 'last seen date' data being recorded.

  • The configuration UI now presents very-long profile lists in a scrollable form rather than extending off screen.

  • The default search template now retains the profile URL parameter when a scope removal link is followed.

  • Fixed the lack of redirect information in push collection snapshots.

  • Fixed handling of process-id files for Funnelback services to be compatible with recent changes to systemd.

  • Fixed an issue where relationships labels in the knowledge graph widget were created incorrectly.

  • Fixed an issue where sorting of related entities was not applied based on knowledge graph template configuration.

  • Fixed an issue where a double request to search endpoint in the knowledge graph widget was being sent on pressing enter key when search input is focused.

  • Fixed an issue where the saving of profile and server level configuration parameters were not being executed due to incorrect backup file creation.

  • Tuning now uses the Perl defined in exectuables.cfg rather than any other Perl on the path.

  • Prevented XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets though the default search result template no longer uses AngularJS).

  • Fixed a bug in which spaces would be removed from query biased summaries which came from 'cdata' sections of XML.

  • Empty XML elements mapped as the document URL are now ignored.

  • Fixed a memory leak within the jetty web server by disabling the conscrypt SSL provider by default.

  • Fixed errors in the sorting of faceted navigation values, which could cause an HTTP 500 error code.

  • Fixed an issue where the crawler would follow <meta http-equiv="refresh"> redirects that appeared within html comments.

Important changes

  • Funnelback’s recommended memory requirements have been increased to reflect higher memory usage seen in practice due to new features introduced over the last few releases.

  • Some license keys generated in the past will now trigger a warning saying "This collection is configured with a license in a deprecated form that will not be supported in a future version of Funnelback. Please use the license portal, or contact Funnelback, to request a new license key". To avoid issues in a future upgrade please ensure new licenses are applied to any affected collections.

  • The sec.profile.manage is now used to determine whether a user can create a service rather than the broader sec.administer.system permission.

  • Editing result templates in the administration dashboard now requires only sec.template permission.

  • Editing web resources in the administration dashboard now requires only sec.web-resources permission.

Changes to configurations

  • contextual_navigation.cfg no longer exists. Contextual navigation options are now configured directly from collection.cfg, see here for the new key names. In short, a prefix of contextual-navigation. has been added to contextual navigation keys. Additionally, the following keys have been renamed:

    • contextual_navigation_enabled -> contextual-navigation.enabled

    • type_max_topics -> contextual-navigation.type.max_clusters

    • topic_max_topics -> contextual-navigation.topic.max_clusters

    • site_max_topics -> contextual-navigation.site.max_clusters

  • quicklinks.cfg no longer exists. Quick links options are now configured directly from collection.cfg, the key names and values have been retained.

  • form_interaction.cfg no longer exists. Form interaction options are now configured directly from collection.cfg, see here for the new key names. The following keys have been removed:

    • crawler.form_interaction_file

    • crawler.form_interaction_in_crawl

    The following keys have been added:

    • crawler.form_interaction.pre_crawl.groupId.url

    • crawler.form_interaction.pre_crawl.groupId.form_number

    • crawler.form_interaction.pre_crawl.groupId.encrypted.param

    • crawler.form_interaction.pre_crawl.groupId.cleartext.param

    • crawler.form_interaction.in_crawl.groupId.url_pattern

    • crawler.form_interaction.in_crawl.groupId.encrypted.param

    • crawler.form_interaction.in_crawl.groupId.cleartext.param

Upgrade notes

  • To upgrade Moment.js to 2.24.0, update the path to resource in FTL forms from ${GlobalResourcesPrefix}thirdparty/momentjs-2.22.2/moment.min.js to ${GlobalResourcesPrefix}thirdparty/momentjs-2.24/moment.min.js. Note that previous versions of Moment.js will be removed from Funnelback in a future release.

  • The upgrade process for contextual_navigation.cfg is automatic. The settings configured in contextual_navigation.cfg are moved into collection.cfg when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and contextual_navigation.cfg.

  • The upgrade process for quicklinks.cfg is automatic. Everything from quicklinks.cfg are moved into collection.cfg and preserved when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and quicklinks.cfg.

  • The upgrade process for form_interaction.cfg is automatic. The settings configured in form_interaction.cfg are moved into collection.cfg when an upgrade is run. Additionally, the relevant file permissions will be added to the <user>.ini file - assuming the user had access to collection.cfg keys and form_interaction.cfg. Please note that if there are any custom scripts which generate form_interaction.cfg files automatically, these may need to be updated to use the new implementation.

  • As maxmind has been upgraded, the Location class within the search transaction at transaction.question.location has been modified. The areaCode, dmaCode and region fields have been removed. The longitude and latitude fields are now of type Double. The Location class has had the following fields added: countryGeoNameId and subdivisions. The subdivisions field is a list that generally holds the state or province that the IP address belongs to.

Patches

Type Release version Description

3 Bug fixes

15.24.0.52

Prevented the creation of objects within Freemarker template files to ensure that template editors can not cause external code to be executed.

1 New and revised features

15.24.0.52

Added new server configuration keys to configure the Jetty HTTP connection.

3 Bug fixes

15.24.0.51

Fixed an issue where the post-update hook script was executed even if the knowledge graph import had failed.

1 New and revised features

15.24.0.51

Added a new knowledge graph public endpoint /kg/nodes/version?collection=<collectionID>&profile=<profileID> to access the knowledge graph’s last update version.

3 Bug fixes

15.24.0.50

Fixed an issue where PDF files are not crawled when form interaction is enabled with in-crawl authentication.

3 Bug fixes

15.24.0.49

Fixed an issue where fetching Facebook comments would cause an infinite loop due to changes within the Facebook endpoints.

3 Bug fixes

15.24.0.48

Fixed a security vulnerability where jackson-databind might allow remote attackers to execute arbitrary code or conduct external XML entity (XXE) attacks.

3 Bug fixes

15.24.0.48

Fixed a security vulnerability where com.google.oauth-client hasn’t implemented PKCE support in accordance with the RFC for OAuth 2.0 for Native Apps.

3 Bug fixes

15.24.0.47

Fixed the security vulnerability where Spring Framework RCE may be vulnerable to remote code execution (RCE) via data binding [CVE-2022-22965]

3 Bug fixes

15.24.0.46

Upgrades log4j2 to version 2.17 to fix the security vulnerability where Apache Log4j2 Thread Context Lookup Pattern is vulnerable to remote code execution in certain non-default configurations.

3 Bug fixes

15.24.0.45

Upgrades log4j2 to version 2.15 to fix the security vulnerability where log4j2 JNDI features do not protect against attacker-controlled LDAP and other JNDI related endpoints.

3 Bug fixes

15.24.0.44

Fixes an issue where the edit metadata mappings administration dashboard wouldn’t display counts of detected sources in searchable documents properly.

3 Bug fixes

15.24.0.43

Search session cookies are now explicitly marked with SameSite=None;Secure to fix functionality in partial integrations.

3 Bug fixes

15.24.0.42

Fixes an issue where faceted navigation UI would freeze due to numerous API requests done to check templates' backups for the usage of legacy facets.

3 Bug fixes

15.24.0.41

Fixes a bug with base tags with href not being used correctly within the crawler.

3 Bug fixes

15.24.0.41

Fixes a bug in which instant updates would always include the start URLs.

3 Bug fixes

15.24.0.41

Fixes a cross-site scripting vulnerability in Freemarker templates.

3 Bug fixes

15.24.0.40

Fixes a bug with YouTube collections when no channel id is provided

3 Bug fixes

15.24.0.39

Reduces logging from build_spelling_index

3 Bug fixes

15.24.0.39

Fixes a bug in filtering in which outlook files with attachments could not be parsed correctly.

3 Bug fixes

15.24.0.38

Fixes an issue in which character ’ (RIGHT SINGLE QUOTATION MARK) would be excluded from auto completion.

3 Bug fixes

15.24.0.37

Fixes a bug in which some autocompletion suggestions would be wrongly excluded from the profiles.

1 New and revised features

15.24.0.36

Adds support for parsing MSG (application/vnd.ms-outlook) type files.

3 Bug fixes

15.24.0.35

Improves how meta components are determined, avoiding synchronisation issues in multi-server installations.

3 Bug fixes

15.24.0.35

Fixes an issue where displaying of numerical/date content in administration dashboard was broken when default browser language was not set to English.

3 Bug fixes

15.24.0.34

Fixes a bug in which form interactions may not work with config environments.

3 Bug fixes

15.24.0.34

Fixes a bug in which invalid XML characters in the query could cause queries to fail.

3 Bug fixes

15.24.0.34

Restores support for the web crawler http_source_host parameter.

3 Bug fixes

15.24.0.33

Make it possible to send empty parameters in crawler form interactions.

3 Bug fixes

15.24.0.32

Fixes a bug in which the text "Is it me?" appeared at the end of all query biased summaries.

3 Bug fixes

15.24.0.31

Fixes a bug that prevented access restrictions set by hostname from working correctly when Funnelback was deployed behind a load-balancer.

3 Bug fixes

15.24.0.30

Fixes various XML encoding issues which would cause search not to work.

3 Bug fixes

15.24.0.29

Best Bet option to remove search result if it has the same URL as the best bet is fixed to compare the link URL rather than the URL to display.

3 Bug fixes

15.24.0.28

Fixes the daemon service broken by patch 15.24.0.26.

3 Bug fixes

15.24.0.27

The search interface’s pre_process hook point now has the ability to see all profiles for a collection within the data model, where previously is could see only the user’s initially requested profile.

3 Bug fixes

15.24.0.26

Reduces memory usage when returning search results as XML.

3 Bug fixes

15.24.0.26

Fixes a bug in PDF filtering when the PDF contains invalid XML characters.

3 Bug fixes

15.24.0.25

Eliminate a warning emitted when using the delete-collection.pl command line tool

3 Bug fixes

15.24.0.24

Fixes incremental filecopy gathering to preserve any additional metadata (e.g. metadata added by custom filters)

3 Bug fixes

15.24.0.23

Fixes a cosmetic issue where the Marketing Dashboard tiles were not aligned correctly.

3 Bug fixes

15.24.0.22

Fixes an issue where enabling access restriction was blocking acceess to Content Auditor, Accessibility Auditor and SEO Auditor API endpoints

3 Bug fixes

15.24.0.22

Fixes an issue where searches on collections with sub-searches can fail with a NullPointerException

3 Bug fixes

15.24.0.21

Fixes an issue where sessions are not terminated on logout events triggered by perl pages.

3 Bug fixes

15.24.0.21

Fixes an XXE issue where input to the webdav endpoint could be manipulated to trigger http requests.

3 Bug fixes

15.24.0.21

Fixes an issue with the web-resources interface which could not cope with unusual file names.

3 Bug fixes

15.24.0.20

Fixes an issue in which Push replication would fail because the client would not renew its authentication token.

3 Bug fixes

15.24.0.19

Fixes an issue where the tuning UI may freeze due to the large number of API requests being performed.

3 Bug fixes

15.24.0.19

Fixes an issue in which instant updates would fail due to long log file names.

3 Bug fixes

15.24.0.19

Improves tuning so that it can run when collections have no documents.

3 Bug fixes

15.24.0.18

Fixes admin-ui handling of profiles with hyphens in their IDs.

3 Bug fixes

15.24.0.17

Fixes an issue where Faceted Navigation extra searches may fail because of an index out of bounds error.

3 Bug fixes

15.24.0.17

Improves logging when extra searches take too long.

3 Bug fixes

15.24.0.16

Fixes an issue where marketing dashboard refers a non-existing URL when ui.integration_url is not configurable at the profile level.

3 Bug fixes

15.24.0.15

Fixes ip pseudonymization when Funnelback is behind a load balancer and client ip details are in the x-forwarded-for header.

3 Bug fixes

15.24.0.14

Reduces memory consumption and improves performance of the purge sessions endpoint.

3 Bug fixes

15.24.0.14

Avoids an error in the admin search interface when SAML authentication is used.

3 Bug fixes

15.24.0.14

Adds a tinkey.jar tool for managing password encryption keys.

3 Bug fixes

15.24.0.13

Fixes an issue where knowledge graph update fails when having numbers as metadata class names.

3 Bug fixes

15.24.0.12

Fixes a bug in auto-completion widget where custom URL parameters set in params field haven’t been applied.

3 Bug fixes

15.24.0.12

Fixes a bug where WebDAV client could lock files with long timeouts and not release them.

3 Bug fixes

15.24.0.12

Fixes a bug where a session was not saved if a user is not set.

3 Bug fixes

15.24.0.12

Fixes a bug where trend alerts shapes (graphs) haven’t been displayed in marketing dashboard.

3 Bug fixes

15.24.0.11

Upgrades the version of RestFB library to account for recent breaking changes in the Facebook Graph API.

3 Bug fixes

15.24.0.11

Fixes an issue where Facebook collections gathered less number of documents due to a pagination issue in the Facebook Graph API.

3 Bug fixes

15.24.0.11

Facebook Graph API deprecated fields name, link, app_links and description have been removed from the default values of the facebook.page-fields configuration key.

3 Bug fixes

15.24.0.10

Fixes an XML formatting issue in Faceted Navigation click logs.

3 Bug fixes

15.24.0.9

Fixes a bug with merging under Push.

3 Bug fixes

15.24.0.8

Fixes a bug in which white space was not preserved in summaries from anchor text when the -map indexer option is enabled.

3 Bug fixes

15.24.0.7

The Push API client used in multi server push now has timeouts enabled allowing it to abandon problematic HTTP requests.

3 Bug fixes

15.24.0.6

Removes the screens for file-manager rule editing which could create security issues

3 Bug fixes

15.24.0.6

Fixes an issue where support packages could contain unintended files

3 Bug fixes

15.24.0.6

Fixes an issue where the running Funnelback jetty web server could retain permissions via supplemental groups after startup

3 Bug fixes

15.24.0.6

Limits an administration CGI script to redirect only within the Funnelback administration interface as intended

3 Bug fixes

15.24.0.6

Removes the unused administration debug.cgi script which reflected input parameters without proper escaping

3 Bug fixes

15.24.0.5

Fixes a bug where a horizontal display of columns in auto-completion dropdown doesn’t work.

3 Bug fixes

15.24.0.5

Fixes a bug where insecure operation on CSS files list was performed when CSS file was exposed via the same domain as auto-completion widget but different port.

3 Bug fixes

15.24.0.4

Fixes an issue where the push API failed to start up when using SAML authentication.

3 Bug fixes

15.24.0.4

Fixes an issue where concurrently encrypting passwords for the first time could create multiple master keysets but store only one, remaining encrypted passwords could not subsequently be decrypted.

3 Bug fixes

15.24.0.3

Fixes an issue where a NullPointerException is sometimes thrown when using the country name Curator trigger.

3 Bug fixes

15.24.0.3

Fixes an issue where Filecopier would sometimes log passwords.

3 Bug fixes

15.24.0.2

Fixes an issue where Knowledge Graph groovy scripts are not executed when they are defined at the profile preview level.

3 Bug fixes

15.24.0.2

Fixes an issue where the Knowledge Graph class CsvImporterNeo4j does not create nodes inside Neo4J when executed externally.

3 Bug fixes

15.24.0.2

Fixes an issue where Knowledge Graph API does not work when a JDBC driver is specified for the session database.

3 Bug fixes

15.24.0.1

Fixes a bug in which uploading configuration files in the administration dashboard stopped working.