Help Center

Menu

Upgrade utility

If you are running a version of Funnelback earlier than 15.24 it is recommended that you first upgrade to 15.24 before attempting to upgrade to Funnelback 16.

The Funnelback upgrade utility automates a number of tasks with the aim of minimizing the work required to import a search from v15. The upgrade tool:

  • Transforms v15 collections and profiles into v16 search packages, data sources and results pages:

    • Converts meta collections to search packages

    • Converts non-meta collections to a meta collection plus results pages (for any pre-existing service-enabled profiles) plus a data source.

    • Updates collection ID references within templates and other configuration files

    • Updates some deprecated data model references in templates.

  • Can pull in v15 collections from a remote Funnelback 15.24 instance (including configuration, data, logs and associated analytics, and related users and roles).

The upgrade utility is not able to upgrade everything, and you will need to make a number of manual changes yourself. This includes:

  • Updating search boxes to reference the new collection ID

  • For any direct integrations with the search JSON (or XML) output, you will need to update your integration to account for data model changes. See: Data model changes in Funnelback v16

  • If you are using legacy facets, sessions or best bets then upgrading these to the current supported versions (see the relevant upgrade guides).

  • Any workflow generation of auto-completion (that calls build-autoc) will need to be replaced with the auto-completion plugin.

  • Although the upgrade utility upgrades your Freemarker templates and Groovy code you should review these to ensure everything is working correctly.

Before you start

Gather information about the collections and customer to be upgraded

  • Put together a list of all collections that should be upgraded as a set.

  • Group the collections into clients. This will normally be a single client that maps back to a customer (or organization). An upgrade should only focus on a single client at a time.

  • Determine if the client exists in the v16 server or if you need to create a new client.

    • For new clients a license is required.

    • For existing clients make a note of the client ID.

  • Put together a list of all client roles (and the role IDs) used by the client in v15 (ignoring default roles).

  • Put together a list of all client users (and the user IDs) in v15.

  • For each collection (non-meta and non-push) note down the current update schedule as these will need to be set up again after the upgrade.

Upgrading all collections, users and roles that belong to a client in a single upgrade operation is recommended.

Review and clean up the collections that have been identified

It is recommended that you perform a review and cleanup of your v15 collections to:

  • Ensure all profiles you want to be converted to results pages are set as services (in v15 select the relevant profile from the profile switcher, then click the enable service button). This includes any profile that is used for system-specific purposes, such as generating auto-completion or for search-powered content.

  • Identify any push collections and put these onto a separate list (push collections should be upgraded separately).

  • Identify any profiles used for generating auto-completion (see the auto-completion upgrade guide for detailed instructions on migrating your auto-completion configuration to use the auto-completion plugin).

Clients

The concept of a client has been added in v16. This extends the v15 concept of collection grouping to collect everything that relates to a particular organisation/client/customer under a single identity.

A client can have the following items associated with it:

  • users and roles

  • API tokens

  • search packages and data sources (and results pages)

  • licences

The client concept introduces a client ID which is prepended to search package and data source IDs and also usernames and roles.

Collections, users and roles that are migrated into v16 must be attached to a v16 client.

When the upgrade tool is run the client must be specified. If the client exists in v16 then the collections/users/roles will be attached to this client. If the client doesn’t exist then it will be created by the upgrade utility.

Collections

v15 collections and profiles are now replaced with v16 search packages, data sources and results pages.

The mapping of these concepts between v16 and v15 is as follows:

v16 v15

Search package

Meta collection

Data source

Non-meta collection

Results page

Service-enabled profile

Key points to note:

  • In v16 results pages are only associated with search packages (data sources cannot have search packages set up on them).

  • v16 search packages and results pages must be associated with a client, and the IDs will change to have the client ID prepended to the pre-existing collection ID.

  • Upgrading non-meta collections into v16 will result in a search package and data source being created in place of the old collection. Any service-enabled profiles associated with the non-meta collection will be attached to the search package.

Collection IDs

The ID of each collection migrated into v16 will change to be prefixed with the v16 client ID.

e.g. Migration two v15 collections, example-collection1 and example-collection2 into v16, attaching to the example-client customer:

v16 v15

example-client~example-collection1

example-collection1

example-client~example-collection1

example-collection2
The v16 search package or data source ID is a concatenation of the v16 client ID, a tilde and the v15 collection ID.

Collection ID references upgraded

v15 collection ID references within the following configuration will be upgraded to v16 IDs:

  • meta.components config key

  • extra_search.*.cfg files. In addition, extra search configuration is moved into collection.cfg as part of the upgrade.

  • faceted navigation configuration

  • license assignment

  • dns_aliases.cfg file

  • redirects.cfg file

  • users and roles

The upgrade of collection IDs contained within users and roles has some limitations and upgraded users and roles should be manually checked after the upgrade. dns_aliases.cfg and redirects.cfg will only be upgraded during an in-place upgraded as these are global configuration files. Any dns aliases and redirects will need to be manually re-configured on the v16 server.

v15 collection ID references in other configuration are not upgraded automatically to v16 IDs and will require manual upgrading. This includes:

  • Freemarker templates

  • Auto-completion widget configuration

  • Filters, workflow scripts, hook scripts

  • linked CSS, Javascript

  • Session and history templates

  • Knowledge graph

Each v16 imported search package and data source ID will be assigned to the client (via the example-client~resources role). Users that are granted this role will have access to the search packages and data sources.

Search package and data source configuration changes

The v15 collection configuration will be upgraded with v16 changes when migrated to v16 search packages and results page. This includes:

  • Update of Freemarker templates and hook scripts to account for changes in the search data model

  • Migration of reporting-email.cfg settings into results page configuration options.

Template and hook script upgrades

In addition to the collection ID changes mentioned above the upgrade tool will attempt to auto-upgrade templates and hook scripts to account for a number of data model changes.

This includes upgrading references of:

  • inputParameterMap with inputParameters

  • rawInputParameters with inputParameters

  • metaData with listMetadata

However, the tool cannot guarantee all syntax variations used in templates and hook scripts will be upgraded correctly and the upgraded templates should also be manually checked for errors. Errors may result in the template breaking and returning a 500 error, or error messages being logged to the modernui logs for the results page.

If the content of any listed above property was assigned to separate variable it may not be upgraded.

e.g. the usage of inputParameterMap won’t be upgraded properly to inputParameters in the groovy code snippet below:

v15

def questionParam = inputQuestion.inputParameterMap
questionParam["f.Residency|3"] = "local"

expected in v16

def questionParam = inputQuestion.inputParameters
questionParam.get("f.Residency|3")[0] = "local"

upgraded with tool

def questionParam = inputQuestion.inputParameters
questionParam["f.Residency|3"] = "local"
If you don’t use Freemarker to display your templates you’ll need to update the data model references yourself. See: Data model changes in Funnelback v16

Administration users and roles

The ID of each administration user and role migrated into v16 will change to be prefixed with the v16 client ID.

e.g. Migration a v15 administration user, example-user and role, example-role into v16, attaching to the example-client customer:

v16 v15

example-client~example-user

example-user

example-client~example-role

example-role
Each user/role ID will be assigned to the client (by inheriting example-client~resources role).

Analytics

Analytics and audit reports will be migrated to the new results pages in v16.

Using the upgrade utility

Basic usage:

$ $SEARCH_HOME/bin/upgrade-v15-to-v16 <parameters> <source> <targetSearchHome> (1)
1 Note: The command can also be called using $SEARCH_HOME/linbin/java/bin/java -cp "$SEARCH_HOME/lib/java/all/*" com.funnelback.upgrade.wrapper.CLI <parameters> <source> <targetSearchHome>

See: Upgrade utility - command line usage for a full list of command line switches.

Defining the source and target locations

This the source (v15) and destination (v16) locations for the upgrade utility

The servers can communicate via ssh or WebDAV.

CLI usage

A URI specifying the source for example:

  • file:///tmp/funnelback will set the source as the search home /tmp/funnelback/ local to the server; For Windows the form is file:/c:/tmp/funnelback/.

  • webdav://fb15-admin.server.com will set the source as a remote WebDAV server assumed to be hosted by Funnelback. By default port 8076 will be used but that can be configured to an alternate port if required. e.g. webdav://fb15-admin.server.com:1234. This will use a system user and expects the server secret of the remote Funnelback to be set in the environment SOURCE_SERVER_SECRET. If the WebDAV connection is slow, set the java option to increase the number of threads:

    -- java -Djava.util.concurrent.ForkJoinPool.common.parallelism=16

  • ssh://search@fb15-admin.server.com/opt/funnelback/ will set the source to be via a ssh server using the user search and expects the remote Funnelback search home to be the path in this case /opt/funnelback/. If the running machine has many cores ( e.g. 32 or more) you may need to limit the number of threads with the java option:

    -- java -Djava.util.concurrent.ForkJoinPool.common.parallelism=1

Interactive usage

There is an interactive mode that can be enabled by providing the --interactive option.

However, due to some limitations it also requires the --client=<clientId> option, even though it asks for the clientId in the interactive mode.

You will also require <source> and <targetSearchHome> to be passed on the command line.

Defining the v16 client

The configuration migrated from v15 must be assigned to a client by providing the --client=<clientId> option.

If the client defined by clientId doesn’t exist, a new client will be created:

  • If --client-human-friendly-name=<clientHumanFriendlyname> is provided, it will be used to set the readable name of the client

  • A license must be specified when creating a new client using --license-id=<licenseId>. The specified license will be assigned to the newly created client

Defining the set of collections to import

The --remote-collection parameter is used to specify the list of collections to import from v15 configuration. The v15 collection ID must be supplied as the parameter value.

e.g. to import collections example-collection-1 and example-collection-2 supply --remote-collection=example-collection-1 --remote-collection=example-collection-2

By default all parts of the collection will be imported. Specific things can be excluded by setting the --exclude to the following values:

  • CONF: The collection configuration files, sourced from v15 $SEARCH_HOME/conf/COLLECTION-ID/.

  • REPORTS: The collection analytics and report databases, sourced from v15 $SEARCH_HOME/admin/reports/COLLECTION-ID/.

  • DATA_REPORTS: The collection data reports, sourced from v15 $SEARCH_HOME/admin/data_report/COLLECTION-ID/.

  • ARCHIVE: The collection archive log files, sourced from v15 $SEARCH_HOME/data/COLLECTION-ID/archive/.

  • DATABASES: The collection database files, sourced from v15 $SEARCH_HOME/databases/.

  • LOG: The collection log files, sourced from v15 $SEARCH_HOME/data/COLLECTION-ID/.

  • VIEW_LIVE: The collection live data files, sourced from v15 $SEARCH_HOME/data/COLLECTION-ID/live/.

  • VIEW_OFFLINE: The collection offline data files, sourced from v15 $SEARCH_HOME/data/COLLECTION-ID/offline/.

e.g. exclude databases and data reports: --exclude=DATABASES --exclude=DATA_REPORTS

Defining the set of users to import

The --remote-user parameter is used to specify the set of users to import from v15 configuration. The v15 user ID must be supplied as the parameter value.

e.g. to import users example-user-1 and example-user-2 supply --remote-user=example-user-1 --remote-user=example-user-2

Defining the roles to import

The --remote-role parameter is used to specify the set of roles to import from v15 configuration. The v15 role ID must be supplied as the parameter value.

e.g. to import users example-role-1 and example-role-2 supply --remote-role=example-role-1 --remote-role=example-role-2

Default Funnelback roles should not be imported as v16 already includes updated default roles.

Defining extra collection, user and role mappings

This option can be used to define additional mappings for collections, users and roles that belong to the client but are not being imported with the defined set of collections, users and roles.

This enables the upgrade tool to correct references within the imported configuration to these additional items.

These options are useful for cases where it is not possible to move all collections/users/roles in one go. e.g. one customer may have few different projects/implementations, or one user/role is managing collections/users/roles from different projects or for a few different customers

These options provide information about already upgraded collections/users/roles and their IDs so that references to old IDs are upgraded properly when the additional collections, users or roles are imported. For example, if collection foo was already upgraded but is referred to in a subsequent run of the upgrade utility, provide the mapping to the new ID using --other-collection-mapping=foo=<clientId>~foo

Command line options

  • --other-collection-mapping=<String=String> defines an additional collection mapping to apply when upgrading collections. This is used to configure the tool for any collections that are part of the set of collections that have already been migrated previously. The CLI tool, com.funnelback.common.upgradeconfig.UpgradeConfUtilityCLI can be used to correct unresolved references (to additional collections) for previously migrated collections.

  • --other-role-mapping=<String=String> defines an additional user mapping to apply when upgrading collections. This is used to configure the tool with mappings for any roles that have already been imported that are referenced from the new set of collections.

  • --other-user-mapping=<String=String> defines an additional user mapping to apply when upgrading collections. This is used to configure the tool with mappings for any users that have already been imported that are referenced from the new set of collections.

  • --new-collection-mapping=<String=String> can be used to define mappings for collections that were previously imported. This is used when importing additional roles and users to ensure that any references to previously imported collections are corrected mapped.

a full set of command line options will be displayed if you run the following command:

$ $SEARCH_HOME/bin/upgrade-v15-to-v16 -h

Detecting usage of deprecated functionality

Detecting usage of scripts

The upgrade utility will list all collections with scripts that need to be updated. In v16 plugins replace custom functionality previously provided by custom gathers, custom filters, groovy hook scripts, workflow scripts etc. Existing custom functionality must be converted to plugins as part of an upgrade to the Squiz DXP. See: released plugins for a list of existing plugins that can be enabled, and developing plugins for information on writing and contributing new plugins.

Detecting usage of deprecated facet navigation

The upgrade utility will list all Freemarker templates that:

  • use legacy facets pre-v15.12 that need to be updated as legacy facets no longer work correctly in v16. See: the following upgrading legacy facets guide for more information.

  • contain removed facets data model features such as queryStringParamName, queryStringParamValue, queryStringParam that need to be updated to work with the new faceted navigation data model.

Detecting v15 collection references in templates and scripts

The upgrade logic provided by the upgrade utility should capture the majority of v15 collection references. However, references will be missed in some instances and manual review of the upgraded functionality will still be required.

The upgrade utility attempts to detect any hardcoded v15 collection IDs used within Freemarker templates and scripts (i.e. workflow, filters). These v15 collection IDs need to be updated to prevent broken functionality.

The upgrade utility will provide options to upgrade all detected collection references in bulk. To re-run detection of collection references, please run the command:

The upgrade utility will store a copy of the output into a log file named upgrade-utility-v15-to-v16.build.<CLIENT-ID>.<DATE>.log, located in the <SEARCH-HOME>/log directory, where:

  • <SEARCH-HOME>: Funnelback installation path, usually /opt/funnelback.

  • <CLIENT-ID>: the provided --client value provided to the upgrade utility script.

  • <DATE>: the date/time that the upgrade utility script was executed.

$ $SEARCH_HOME/bin/upgrade-v15-to-v16 --search-home=<SEARCH-HOME> --collectionId=<TARGET-COLLECTION-ID> --change-collection-ref=<V15-COLLECTION-ID>=<V16-COLLECTION-ID>

Where:

  • <SEARCH-HOME>: Funnelback installation path, usually /opt/funnelback

  • <TARGET-COLLECTION-ID>: is the v16 collection ID the templates will be checked for.

  • <V15-COLLECTION-ID>=<V16-COLLECTION-ID>: collection IDs mapping from v15 to v16, for example foo=client~foo

Copying analytics log files

The upgrade utility will transfer all the analytics log files when the upgrade command is run. However, it is often desirable to re-sync the log files when the upgrade is ready to be released so that all the search logs are transferred to the server.

The logs can be re-synced by running the following command:

$ $SEARCH_HOME/linbin/java/bin/java -classpath "$SEARCH_HOME/lib/java/all/*" com.funnelback.mediator.core.collectionpuller.CLI <SOURCE> <SOURCE-COLLECTION-ID> $SEARCH_HOME --target-collection=<TARGET-COLLECTION-ID>  --sync-archive-logs

Where:

  • <SOURCE>: is a URL specifying the source server and search home (see: Defining the source and target locations above).

  • <SOURCE-COLLECTION-ID>: is the collection ID of the collection containing the search logs (this is the collection ID that is supplied as the collection parameter when you run a search) on the v15 server.

  • <TARGET-COLLECTION-ID>: is the v16 collection ID of the search package that was created from the v15 collection when it was imported.