Plugin: Modify JSON data
Purpose
Use this plugin to modify and transform JSON data that has been downloaded before indexing.
The JSONata Query and Transform language is used to transform the JSON data and query additional metadata from the JSON documents.
When to use this plugin
Use this plugin:
-
To generate additional metadata based on other information contained within the input data. (e.g. create a latlong metadata field from separate latitude and longitude fields).
-
To apply transformations to the input JSON data to data clean, or provide additional compatibility with Funnelback when indexing.
-
To make changes to the JSON structure such as the addition or removal of nodes.
Usage
Enable the plugin
-
Select Plugins from the side navigation pane and click on the Modify JSON data tile.
-
From the Location section, select the data source to which you would like to enable this plugin from the Select a data source select list.
| The plugin will take effect after setup steps and an advanced > full update of the data source has completed. |
Configuration settings
The configuration settings section is where you do most of the configuration for your plugin. The settings enable you to control how the plugin behaves.
| The configuration key names below are only used if you are configuring this plugin manually. The configuration keys are set in the data source configuration to configure the plugin. When setting the keys manually you need to type in (or copy and paste) the key name and value. |
Create new metadata class
Configuration key |
|
Data type |
string |
Required |
This setting is optional |
Creates a new metadata class with a name defined in 'Parameter 1'. The value of this key should be set to a JSONata expression, and the result of this expression will be the value of your generated metadata class .
Transform JSON input data
Configuration key |
|
Data type |
string |
Required |
This setting is optional |
Specifies a JSONata expression, which will be applied to the input JSON.
Maximum evaluation time
Configuration key |
|
Data type |
integer |
Default value |
|
Required |
This setting is optional |
Specifies the maximum amount of time (ms) that an expression can evaluate for before it terminates.
Maximum evaluation stack depth
Configuration key |
|
Data type |
integer |
Default value |
|
Required |
This setting is optional |
Specifies the maximum number of stack frames allowed during recursive execution.
| When configuring the filter chain you must ensure that the JsonataStringFilter is located ahead the JSONToXML filter (but after the ForceJSONMime filter (if you are using that)). This is because this filter operates on the JSON object. Once the JSONToXML filter runs the data is represented as an XML object. |
Limitations
This plugin uses the JSONata4Java library which has a few limitations:
There are a few functions that have not been implemented:
-
transformation functions
... ~> | ... | ...|
-
$formatInteger() -
$parseInteger()
Filter chain configuration
This plugin uses filters which are used to apply transformations to the gathered content.
The filters run in sequence and need be set in an order that makes sense. The plugin supplied filter(s) (as indicated in the listing) should be re-ordered to an appropriate point in the sequence.
| Changes to the filter order affects the way the data source processes gathered documents. See: document filters documentation. |
Examples
Example - create a fullname metadata field to use as a record title
A JSON data feed containing directory/people information may contain fields for their first name and last name, but not the full name. The search results output should use the full name as the record title for ranking relevance and ensuring that partial queries for autocompletion produce the expected results.
The following setting would add a new metadata field fullname that is a concatenation of the firstname and lastname fields in the JSON record, given the following input data (JSON object):
{
"firstname": "Jane",
"lastname": "Smith",
"url": "https://www.example.com/1"
}and the following plugin configuration:
| Configuration key name | Parameter 1 | Value |
|---|---|---|
Create new metadata class |
|
|
The firstname & ' ' & lastname value is a JSONata expression which concatenates the firstname and lastname fields together with a space character in between, thus creating a fullname field without the need for a specialized plugin to be written.
This will create an additional fullname metadata class that can be setup in your metadata mappings.
Remove records based on a criteria
A common need for a JSON data feed is to remove records that fit a given criteria, for example, remove events that have already occurred or remove news articles that have a draft status (that have not been published yet).
Given the following input data (JSON array):
[
{
"articleTitle": "...",
"url": "https://www.example.com/1",
"status": "published"
},
{
"articleTitle": "...",
"url": "https://www.example.com/2",
"status": "draft"
}
]and the following plugin configuration:
| Configuration key name | Value |
|---|---|
Transform JSON input data |
|
The result of transforming the JSON with this expression is that only records with a status field of published will be added to the search index.
Example - Remove null values from a JSON object
An input data JSON object with null values can be problematic, as the conversion into XML data in Funnelback using the JSONToXML filter results "null" string values which are indexed and will appear in things like faceted navigation..
The following transform expression can be used to remove all null values from a JSON object.
| Configuration key name | Value |
|---|---|
Create new metadata class |
|