Plugin: Modify JSON data
Other versions of this plugin may exist. Please ensure you are viewing the documentation for the version that you are currently using. If you are not running the latest version of your plugin we recommend upgrading. See: list of all available versions of this plugin. |
Purpose
This plugin can be used to transform and modify JSON data before it is indexed.
The JSONata Query and Transform language is used to transform the JSON data and query additional metadata from the JSON documents.
Usage
-
Enable the modify-json-data plugin on your data source from the Extensions screen in the administration dashboard or add the following data source configuration to enable the plugin.
plugin.modify-json-data.enabled=true
plugin.modify-json-data.version=1.0.0
-
Add the modify-json-data filter to the filter chain.
filter.classes=<OTHER-FILTERS>:com.funnelback.plugin.modifyjsondata.JsonataStringFilter:<OTHER-FILTERS>
The modify-json-data filter should be placed at an appropriate position in the filter chain (which applies the filters from left to right). In most circumstances this should be located towards the end of the filter chain and must be placed before the JSONToXML filter).
|
-
Configure the plugin (see the plugin configuration settings section below).
-
Run a full update of the data source. Note: a full update is required as all of your documents must be re-gathered and filtered for any changes to take effect. If you are using this with a push data source then you will need to resubmit anything where you want the new filter to be applied.
Plugin configuration settings
The following plugin configuration settings can be used to configure the plugin:
-
plugin.modify-json-data.config.metadata.[class_name]
: (JSONata expression) Evaluates the JSONata expression and stores the result in theclass_name
metadata class. -
plugin.modify-json-data.config.transform
: (JSONata expression) Evaluates the JSONata expression to transform the JSON input. -
plugin.modify-json-data.config.max_evaluation_time
: (Integer) Maximum amount of time (ms) that an expression can evaluate for before it is cancelled. Default is 5000ms. -
plugin.modify-json-data.config.max_evaluation_stack_depth
: (Integer) Maximum amount of stack frames allowed during execution (only relevant for recursive function). Default is 64.
Any number of plugin.modify-json-data.config.metadata.[class_name]
parameters can be provided, and each is evaluated for that given class_name
metadata class.
Examples
Query - Create a "fullname" field to use as a record title
A JSON data feed containing Directory/People information may contain fields for their first name and last name, but not the full name. The search results output should use the full name as the record title for ranking relevance and ensuring that partial queries for autocompletion produce the expected results.
The following setting would add a new metadata field fullname
that is a concatenation of the firstname
and lastname
fields in the JSON record, given the following input data (JSON object):
{
"firstname": "Jane",
"lastname": "Smith",
"url": "https://www.example.com/1"
}
and the following plugin configuration:
plugin.modify-json-data.config.metadata.fullname=firstname & ' ' & lastname
firstname & ' ' & lastname
is a JSONata expression which concatenates the firstname
and lastname
fields together with a space character in between, thus creating a fullname
field without the need for a specialized plugin to be written.
Transform - Remove Records Based on a Criteria
A common need for a JSON data feed is to remove records that fit a given criteria, for example, remove Events that have already occurred or remove news articles in "draft" status that have not been published yet.
Given the following input data (JSON array):
[
{
"articleTitle": "...",
"url": "https://www.example.com/1",
"status": "published"
},
{
"articleTitle": "...",
"url": "https://www.example.com/2",
"status": "draft"
}
]
and the following plugin configuration:
plugin.modify-json-data.config.transform=$filter($, function($article) { $article.status = "published"})
The result of transforming the JSON with this expression is that only records with a "status" field of "published" will be added to the search index.
Transform - Remove null
values from a JSON object
An input data JSON object with a null
value can be problematic, as when the JSON data is converted into XML data in Funnelback using the JSONToXML
filter, the null
JSON value is converted into a string "null"
value which shows in facets and in the search results output.
The following transform expression can be used to remove all null
values from a JSON object.
plugin.modify-json-data.config.transform=$sift($, function($value) { $value != null })