Funnelback 15.14 patches

Patches

Type Release version Description

3 Bug fixes

Upgrades log4j2 to version 2.16 to fix the security vulnerability where log4j2 JNDI features do not protect against attacker-controlled LDAP and other JNDI related endpoints.

3 Bug fixes

Fixes an issue where sessions are not terminated on logout events triggered by perl pages.

3 Bug fixes

Fixes an issue where deleting the collection property from collection.cfg and then deleting the collection itself would delete other configuration data.

3 Bug fixes

Removes the screens for file-manager rule editing which could create security issues

3 Bug fixes

Fixes an issue where support packages could contain unintended files

3 Bug fixes

Fixes an issue where the running Funnelback jetty web server could retain permissions via supplemental groups after startup

3 Bug fixes

Limits an administration CGI script to redirect only within the Funnelback administration interface as intended

3 Bug fixes

Removes the unused administration debug.cgi script which reflected input parameters without proper escaping

3 Bug fixes

Improves support for running faceted navigation on extra searches.

3 Bug fixes

Adds method 'getEffectiveExtraSearchName()' to the search transaction which gets the name of the extra search this search should be considered to be under. The result of this should be used when modifying a particular extra search. As Funnelback may create extra searches under an existing search, for example for faceted navigation, this could be used to work out if the search transaction should be modified.

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates escaped using output formats by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Improve the performance of the Accessibility Auditor interface by requesting less data.

3 Bug fixes

Fixes an issue where some of the text on the Accessibility Auditor dashboard was showing out of date information.

3 Bug fixes

Fixes an issue where the Accessibility Auditor dashboard would not generate the thumbnail screenshots for each domain.

3 Bug fixes

Improves the query response time when sorting.

3 Bug fixes

Fixes an issue where large (>2GB) index.dt files would cause padre-gs to fail when setting gscopes.

3 Bug fixes

Fixes an issue where jetty would terminate on invalid 'index.autoc' (query completion) files.

3 Bug fixes

Fixes an issue where recording Accessibility Auditor details would fail during the swap views step if the server is in read-only mode.

3 Bug fixes

Fixes an issue where swap-views.pl did not clear the redis state before running the pipeline.

3 Bug fixes

Fixes a bug where query processing may fail when given dates in the query. Failure would always occur when the date was invalid.

3 Bug fixes

Improves the Accessibility Auditor historical data storage. The data is stored in less space while also being significantly faster when storing and retrieving data. The Accessibility Auditor historical data APIs are also improved to reduce the amount of memory needed to help reduce the chance of 'OutOfMemoryError' exceptions from being thrown. The Accessibility Auditor historical data will be automatically moved to the new storage format when Jetty is restarted (one collection at a time) or on the first Accessibility Auditor historical data API request.

3 Bug fixes

The default timeout for 'push.scheduler.delay-between-meta-dependencies-runs' has been increased to '1200' (20 minutes). This has been increased to reduce the frequency at which Accessibility Auditor historical data is recorded. This option will need to be overridden if meta collections containing push collections need a smaller delay in updating the spelling index and auto completion.

3 Bug fixes

Fixes a bug where the query processor may seg fault under some zero result cases, often seen when faceted navigation is used.

3 Bug fixes

Fixes a bug where a 'NullPointerException' may be thrown when a faceted navigation extra search fails or does not run.

3 Bug fixes

Fixes a bug where 'facetScope' URL parameter may not be correctly decoded.

3 Bug fixes

Fixes a bug where queries may not return when instant updates include URLs that contain ampersands.

3 Bug fixes

Prevents creation of objects within Freemarker template files to ensure that template editors can not cause external code to be executed.

3 Bug fixes

Fixes a bug where 'FineTune' may crash when 'query_processor_options' is longer than '1000' bytes.

3 Bug fixes

Push slaves will now actively pull down merge/vacuumed generations, rather than waiting for commits to trigger this. This can help solve problems where the slaves will not reduce the number of generations or re-indexes are not pulled down by the slaves.

3 Bug fixes

Fixes a bug where users that did not have access to all collections would not be given access to the collection they just created.

3 Bug fixes

Ensure the legacy jquery.funnelback-completion.js is included with Funnelback for backwards compatibility. When performing an upgrade, it is still recommended that you upgrade to concierge. See here for more information: https://docs.funnelback.com/15.14/customise/standard-options/auto-completion/upgrading-to-concierge.html

3 Bug fixes

Fixes security issues where:

  • The default form-not-found template reflected the given form id without proper escaping.

  • The default configuration of URL previewing could be used to expose local log file content.

Please ensure any custom form-not-found.ftl templates in collections are updated to perform correct escaping if they were derived from the previously vulnerable form-not-found.default.ftl.

Please ensure that any customised value for the global default_url_renderer.permitted_url_pattern setting in global.cfg prevents access to file:// URLs.

3 Bug fixes

Fixes a bug where the Content Auditor display-metadata config key ui.modern.content-auditor.display-metadata.<metadata class> was not being read correctly.

3 Bug fixes

Fixes a bug where the crawler would not correctly decode links in HTML, XML and plain text documents.

3 Bug fixes

Improves the performance of the directory gatherer.

3 Bug fixes

Fixes support for sort mode '3' in query completion, allowing 'alpha' to be respected.

3 Bug fixes

parent_group Facebook events field has been removed since it requires escalated permissions. On some Facebook collections, this caused crawling of events to fail.

3 Bug fixes

Provides additional metadata for twitter records specifying if a tweet is a reply and to what it is a reply to. This is made available in the XML under 'isReply', 'inReplyToScreenName', 'inReplyToStatusId', 'inReplyToUserId' and 'inReplyToUrl'.

3 Bug fixes

Adds an option to padre-iw to allow control of how the lock string should be modified. The option is "-lock_string_mod_mode=[legacy raw]". By default it is set to "legacy" and that keeps the current behaviour of modifying the lockstring. The "raw" option results in padre-iw not altering the lock string; leaving in white space, new lines and commas.

3 Bug fixes

Adds an experimental DLS plugin called "secBoolExpr" which is able to evaluate lockstrings which are boolean expression. For example, a lockstring where the user must have both B and C or A can be represented as A,(B AND C) or A | (B&C). This lockstring is evaluated as a boolean expression where the user keys are what is true in the expression. For example, if the user had "B" and "C" then they would be able to view that document. To enable the plugin set in collection.cfg "security.earlybinding.locks-keys-matcher.name=secBoolExpr" and set "-lock_string_mod_mode=raw" on the indexer.

3 Bug fixes

Upgrades the version of our internal libraries to account for recent breaking changes in the Facebook Graph API. This will fix issues that caused Facebook collections to fail to update on certain user accounts, when crawling more than 200 posts in an hour, and when crawling events posted by a page. To update existing Facebook collections that may be failing, the changes noted in deployment instructions below will need to be made on each groovy script.

3 Bug fixes

Fixes an issue with SAML authentication where users who did not have access to the funnelback_documentation collection were unable to log out.

3 Bug fixes

Fixes an issue with SAML authentication where some browsers would fail to redirect users completely after login.

3 Bug fixes

Stops the same sitemap.xml file from being processed multiple times on different threads.

3 Bug fixes

Fixes an issue where incremental updates would not copy forward documents.

3 Bug fixes

Fixes an issue where the admin API crawler debug utilities did not respect the 'crawler.form_interaction_file' collection.cfg option. It always assumed a default value of 'form_interaction.cfg'.

3 Bug fixes

Fixes an issue where the crawler did not respect the 'crawler.form_interaction_file' collection.cfg config option. It always assumed a default value of 'form_interaction.cfg'.

3 Bug fixes

Improves the internal re-filter program.

3 Bug fixes

Fixes an issue where configuring a site profile without http basic credentials would cause empty username/password authorization headers to be sent by the web crawler.

3 Bug fixes

Removes a complexity limit in contextual navigation which applied to the entire query rather than only to the terms relevant for contextual navigation. This allows contextual navigation to work on complex queries, such as those generated by facets.

3 Bug fixes

Fixes an issue where the web crawler parser would time out when parsing large (10MB+) HTML pages.

3 Bug fixes

Fixes an issue where two jars provided conflicting classes with the same package name; this results in errors if a filter or groovy script used the 'org.json' package. This patch fixes the issue by forcing the use of 'json-20160810.jar' (org.json:json), which provides everything 'json-1.8.jar' (com.tdunning:json) did and more. See https://issues.apache.org/jira/browse/TIKA-2556 for further details about the change.

3 Bug fixes

Updates the search sessions click history to no longer record all metadata into the DB. Search sessions will only record the metadata classes listed in profile.cfg option 'ui.modern.session.search_history.metadata'. By default this is empty, but can be set with a comma separated list of wanted metadata classes for example:

ui.modern.session.search_history.metadata=a,b,c

3 Bug fixes

Provides a collection.cfg setting (crawler.send-http-basic-credentials-without-challenge) to restore older web-crawler behaviour of sending HTTP Basic authentication details without waiting for a 401 challenge from the web server.

This patch includes a default setting to default to the old behaviour.

3 Bug fixes

Fixes a bug where ratio to run full or incremental updates was not being applied and only a full update was triggered.

3 Bug fixes

Fixes a bug for scheduled updates where the 'schedule.incremental_crawl_ratio' parameter was not being respected.

3 Bug fixes

Fixes a bug in Accessibility Auditor which caused the document audit view to fail when a document contained escaped or unicode characters in their classnames.

3 Bug fixes

Makes the web crawler pre-crawl form interaction accept login forms which are served with an HTTP 401 or 403 code.

3 Bug fixes

Fixes a potential indexer crash introduced in 15.14.0.1, and some additional cases where multiple dots could be shown in summaries.

3 Bug fixes

Fixes query biased summaries so that it doesn’t show multiple dots when the original content contains non breaking spaces as the only value within "p" tags.