Funnelback 15.8.0 release notes

Released: 16 December 2016

15.8.0 - New features

  • Introduced a redesigned facet editing interface.

15.8.0 - Selected improvements and bug fixes

  • Introduced support for HP RM 8.2+

  • Introduced query-gscopes.cfg configuration file allowing gscopes to be set based on a query (without the need for facets).

  • Removed legacy 'htpasswd' files and associated scripts. This fixes an issue where password resets could be overwritten by subsequent user password changes.

  • Improved the compatibility of component collections in meta collections. Padre indexes are no longer made incompatible by varying gscopes, number of metadata classes or the type of metadata classes.

  • Padre recency ranking feature is now based on proximity to the current date, in general the closer the date the higher the score.

  • A new filtering framework has been introduced that is more flexible while simpler to implement.

  • Push has V2 API content calls which allow for multiple values per metadata key.

  • Push PUT document API calls are now able to take even more advantage of multiple CPUs.

  • Implemented a limit on image scaling input to prevent potential OOM situations.

  • Errors from Funnelback’s search interface are now logged per collection (under web/log/ on windows and data/coll-name/log elsewhere) rather than in one shared log file.

  • Addressed a number of cross site scripting security issues.

  • Restructured product documentation, and improved the graphical design.

  • Facet template tags are now able to impose a display order on facets using orderedNames attribute.

  • Upgraded Freemarker library to 2.3.25 which provides some new template syntax - See http://freemarker.org/docs/versions_2_3_25.html.

  • Introduced ability to compare curator configurations between live and preview.

  • Fixes/improvements to the web crawler’s form interaction support.

  • Migrated the functionality of a number of mediator (bin/*.pl) Perl scripts to Java.

  • Fixes a issue where perl publish hook scripts would not be correctly called for curator and synonyms on Windows.

  • Fixed an issue with collections not displaying the correct "last update" date

15.8.0 - Upgrade Issues

  • IsolatedFilterProvider is no longer supported. If this filter is used, filtering will still occur, but will not be isolated in a separate process.

  • FileTypeMappedFilterProvider is no longer supported.

  • JPEGJavaFilter and RTFJavaFilter are no longer provided. TikaFilterProvider is covering JPEG and RTF formats.

  • LotusFilterProvider and TextCleanupFilterProvider are no longer provided.

  • AbstractFilterProvider constructor parameters have changed. Previously (String collectionName, boolean inlineFilterEnabled) is now (File searchHome, CollectionId collectionId). If classes were inheriting from AbstractFilterProvider their constructor will need to be changed accordingly.

  • Post gather filtering (convert.pl) is no longer supported, all filtering must be done inline.

  • Windows now uses symlinks for live/offline views. Existing collections will have symlinks created on upgrade. The Windows user which runs scheduled updates and users which run updates from the command line will need the permission SeCreateSymbolicLinkPrivilege.

  • The JAVA_OPTS environment variable will no longer be respected by groovy commands in pre/post workflow commands.

  • Instant update delete for Filecopy collections no longer require encoding of the system URLs supplied similar to instant update add e.g. \\example\sp%20ce should be supplied as \\example\sp ace.

  • funnelback-reporting.jar has been moved from $SEARCH_HOME/bin/ to $SEARCH_HOME/lib/java/all/.

  • Image scaling is now limited to input images less than 1MB by default. Set default_image_fetcher.max_source_image_bytes to a larger value in global.cfg during upgrade if required.

Patches

Type Release version Description

3 Bug fixes

Upgrades log4j2 to version 2.17 to fix the security vulnerability where Apache Log4j2 Thread Context Lookup Pattern is vulnerable to remote code execution in certain non-default configurations.

3 Bug fixes

Removes the screens for file-manager rule editing which could create security issues

3 Bug fixes

Fixes an issue where support packages could contain unintended files

3 Bug fixes

Fixes an issue where the running Funnelback jetty web server could retain permissions via supplemental groups after startup

3 Bug fixes

Limits an administration CGI script to redirect only within the Funnelback administration interface as intended

3 Bug fixes

Removes the unused administration debug.cgi script which reflected input parameters without proper escaping

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates escaped using output formats by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Prevent XSS AngularJS sandbox bypassing injection in Freemarker templates by inserting zero-width whitespace between consecutive open-curly-brackets.

3 Bug fixes

Prevents creation of objects within Freemarker template files to ensure that template editors can not cause external code to be executed.

3 Bug fixes

Fixes security issues where:

  • The default form-not-found template reflected the given form id without proper escaping.

  • The default configuration of URL previewing could be used to expose local log file content.

Please ensure any custom form-not-found.ftl templates in collections are updated to perform correct escaping if they were derived from the previously vulnerable form-not-found.default.ftl.

Please ensure that any customised value for the global default_url_renderer.permitted_url_pattern setting in global.cfg prevents access to file:// URLs.

3 Bug fixes

Fixes a bug where ratio to run full or incremental updates was not being applied and only a full update was triggered.

3 Bug fixes

Fixes a bug for scheduled updates where the 'schedule.incremental_crawl_ratio' parameter was not being respected.

3 Bug fixes

Fixes a bug where the Admin API was passing the comment to the publish hook as multiple arguments where it should have been passing the comment as a single argument.

3 Bug fixes

Adds time-based reloading of type-caching objects (XStream and Jackson serialisers) to avoid leaking metaspace memory when groovy classes are serialised and reloaded over time.

By default, reloading occurs every 10 minutes, and can be configured in modernui.properties.

3 Bug fixes

Fixed an issue where the user editing interface for a user with no permitted collections would be presented with all collections selected, rather than none.

3 Bug fixes

Allow groovy servlet filters to abort processing in preFilterResponse by returning null.

3 Bug fixes

Changes the click tracking endpoint to no longer depend on the referrer. This does result in the click logs no longer containing the referrer URL.

3 Bug fixes

To minimise the number of false positives reported by XSS testing tools, JSON endpoints have restricted the JSONP callback to only contain A-Za-z0-9 as well as $._-[]".

3 Bug fixes

Restores the behavior of update.pl such that the gatherer (e.g. the web crawler) will use the same collection.cfg file that is passed to update.pl.

3 Bug fixes

Improves the creation of snapshots on empty push collections.

3 Bug fixes

Updates the location of the Push sync restart API call to be consistent with other state changing calls. The existing API call is kept for compatibility.

3 Bug fixes

Adds a new Push sync health API calls that never return null for the value of the boolean in the response. The new calls are under /v2/ of the API.

3 Bug fixes

Adds support for fixing a bug where a data folder for a non-existent collection would be created when a request was made on the search interface to the collection. Affects Linux only.

3 Bug fixes

Avoids the output of the DiskAggregator reports phase being overwritten by the DataMiner phase.

3 Bug fixes

Introduces the ability to customise the jetty access logging configuration with logback.

The default behaviour of logging is unchanged, however with this patch it is possible to configure access log compression, filtering and size-based retention policies if desired.

See Funnelback version 15.12 "Configuring embedded web server" documentation for details and example of how to customise access logging configurations.

3 Bug fixes

Updates the version of restfb so that custom Facebook gatherers may use a later version of the graph API.

3 Bug fixes

Adds a customData map in the SearchQuestion for convenience

3 Bug fixes

Fixes a bug in the query processor when promote URLs was used with URLs that contained double dash.

3 Bug fixes

Fixes a bug in the query processor where sorting on file size did not work.

3 Bug fixes

Fixes a bug where exporting the top queries to csv on the marketing dashboard was not working in Internet Explorer 11.

3 Bug fixes

Fixes a bug where Push Replication would re-attempt a connection to master without sleeping if the response from master was not a 200.

3 Bug fixes

Improves Push collections so that snapshots are marked incomplete during creation to help avoid incomplete snapshots from being used.

3 Bug fixes

Improves Push Replication performance by enabling compression on more files.

3 Bug fixes

Fixes an issue where the Accessibility Auditor would not be able to connecting servers using the SNI extension when checking an individual document. This patch will cause Accessibility Auditor to no longer be able to connect to web servers with untrusted SSL certificates.

3 Bug fixes

Fixes an issue where the mail.on_failure_only collection configuration option was not respected by updated.

3 Bug fixes

Fixes an issue with a spelling mistake in the email subject.

3 Bug fixes

Fixes an issue where instant delete tries to kill documents from an index that doesn’t exist causing the update to fail

3 Bug fixes

Fixes an issue where HSTS was not disabled on all end points.

3 Bug fixes

Fixes an issue where the analytics log was always appended to, resulting in a log file that always grew in size.

3 Bug fixes

Fixes an issue where the URL sent in Trend Alerts emails would not be correctly redirected to the Trend Alerts dashboard.

3 Bug fixes

Updates the version of pdfbox used for filtering so that more PDFs can be correctly filtered.

3 Bug fixes

Fixes two issues with form interaction which could prevent the web crawler logging into authenticated sites:

  1. Updated cookie values would not overwrite the initial values of cookies.

  2. When crawler.accept_cookies=false was set, form interaction cookies were ignored.

3 Bug fixes

Fixes an issue with anchors not being preserved in the displayUrl node of the data model.

3 Bug fixes

Fixes an issue where Analytics would remove the anchor separator from the URL.

3 Bug fixes

Fixes an issue where date sorting in the query processor would not sort future dates correctly.

3 Bug fixes

Fixes an issue where web collections with Accessibility Auditor (WCAG) enabled would not be able to run instant updates.

3 Bug fixes

Fixes an issue where updates could not be started from the Collection Overview section of the admin home page.

3 Bug fixes

Fixes issues with auditing tools when crawling from localhost, non-standard ports or with deep folders documents.

3 Bug fixes

Fixes a problem in the query processor (introduced in 15.8.0.2) which could slow query processing or cause an OutOfMemoryError within the web server.

3 Bug fixes

Fixes an issue where the Recommender database would fail to build on meta collections.

3 Bug fixes

Fixes a bug with promoted URLs where those that were only partial matches would not be promoted to the top position.

3 Bug fixes

Fixes a bug with Trend Alerts links always referring to the ‘Classic UI’ interface. These links will now refer to the collection’s configured search interface.

3 Bug fixes

Fixes a bug with the license usage API which included documents which are not normally searchable e.g duplicate documents and binary documents. This patch also excludes documents in the included funnelback_documentation collection from counting towards the license limit.