Plugin: Modify JSON data

Purpose

Use this plugin to modify and transform JSON data that has been downloaded before indexing.

The JSONata Query and Transform language is used to transform the JSON data and query additional metadata from the JSON documents.

When to use this plugin

Use this plugin:

  • To generate additional metadata based on other information contained within the input data. (e.g. create a latlong metadata field from separate latitude and longitude fields).

  • To apply transformations to the input JSON data to data clean, or provide additional compatibility with Funnelback when indexing.

  • To make changes to the JSON structure such as the addition or removal of nodes.

Usage

Enable the plugin

  1. Select Plugins from the side navigation pane and click on the Modify JSON data tile.

  2. From the Location section, select the data source to which you would like to enable this plugin from the Select a data source select list.

The plugin will take effect after setup steps and an advanced > full update of the data source has completed.

Configuration settings

The configuration settings section is where you do most of the configuration for your plugin. The settings enable you to control how the plugin behaves.

The configuration key names below are only used if you are configuring this plugin manually. The configuration keys are set in the data source configuration to configure the plugin. When setting the keys manually you need to type in (or copy and paste) the key name and value.

Create new metadata class

Configuration key

plugin.modify-json-data.config.metadata.*

Data type

string

Required

This setting is optional

Creates a new metadata class with a name defined in 'Parameter 1'. The value of this key should be set to a JSONata expression, and the result of this expression will be the value of your generated metadata class .

Transform JSON input data

Configuration key

plugin.modify-json-data.config.transform

Data type

string

Required

This setting is optional

Specifies a JSONata expression, which will be applied to the input JSON.

Maximum evaluation time

Configuration key

plugin.modify-json-data.config.max_evaluation_time

Data type

integer

Default value

5000

Required

This setting is optional

Specifies the maximum amount of time (ms) that an expression can evaluate for before it terminates.

Maximum evaluation stack depth

Configuration key

plugin.modify-json-data.config.max_evaluation_stack_depth

Data type

integer

Default value

64

Required

This setting is optional

Specifies the maximum number of stack frames allowed during recursive execution.

When configuring the filter chain you must ensure that the JsonataStringFilter is located ahead the JSONToXML filter (but after the ForceJSONMime filter (if you are using that)). This is because this filter operates on the JSON object. Once the JSONToXML filter runs the data is represented as an XML object.

Limitations

This plugin uses the JSONata4Java library which has a few limitations:

There are a few functions that have not been implemented:

  • transformation functions

    ... ~> | ... | ...|
  • $formatInteger()

  • $parseInteger()

Filter chain configuration

This plugin uses filters which are used to apply transformations to the gathered content.

The filters run in sequence and need be set in an order that makes sense. The plugin supplied filter(s) (as indicated in the listing) should be re-ordered to an appropriate point in the sequence.

Changes to the filter order affects the way the data source processes gathered documents. See: document filters documentation.

Filter classes

This plugin supplies a filter that runs in the main document filter chain: com.funnelback.plugin.modifyjsondata.JsonataStringFilter

Drag the com.funnelback.plugin.modifyjsondata.JsonataStringFilter plugin filter to where you wish it to run in the filter chain sequence.

Examples

Example - create a fullname metadata field to use as a record title

A JSON data feed containing directory/people information may contain fields for their first name and last name, but not the full name. The search results output should use the full name as the record title for ranking relevance and ensuring that partial queries for autocompletion produce the expected results.

The following setting would add a new metadata field fullname that is a concatenation of the firstname and lastname fields in the JSON record, given the following input data (JSON object):

{
    "firstname": "Jane",
    "lastname": "Smith",
    "url": "https://www.example.com/1"
}

and the following plugin configuration:

Configuration key name Parameter 1 Value

Create new metadata class

fullname

firstname & ' ' & lastname

The firstname & ' ' & lastname value is a JSONata expression which concatenates the firstname and lastname fields together with a space character in between, thus creating a fullname field without the need for a specialized plugin to be written.

This will create an additional fullname metadata class that can be setup in your metadata mappings.

Remove records based on a criteria

A common need for a JSON data feed is to remove records that fit a given criteria, for example, remove events that have already occurred or remove news articles that have a draft status (that have not been published yet).

Given the following input data (JSON array):

[
  {
    "articleTitle": "...",
    "url": "https://www.example.com/1",
    "status": "published"
  },
  {
    "articleTitle": "...",
    "url": "https://www.example.com/2",
    "status": "draft"
  }
]

and the following plugin configuration:

Configuration key name Value

Transform JSON input data

$filter($, function($article) { $article.status = "published"})

The result of transforming the JSON with this expression is that only records with a status field of published will be added to the search index.

Example - Remove null values from a JSON object

An input data JSON object with null values can be problematic, as the conversion into XML data in Funnelback using the JSONToXML filter results "null" string values which are indexed and will appear in things like faceted navigation..

The following transform expression can be used to remove all null values from a JSON object.

Configuration key name Value

Create new metadata class

$sift($, function($value) { $value != null })

Change log

[1.1.0]

Changed

  • Updated to the latest version plugin framework (Funnelback shared v16.20) to enable integration with the new plugin management dashboard.