Custom data sources
The custom data source is a generic data source type that use a plugin to gather and index content from arbitrary sources using methods such as the gathering of content via API, or programmatically via Java libraries or an SDK. The gathering logic for a custom data source uses a plugin or a custom gather script that must be supplied before the data source is usable.
Creating a custom data source
To create a new custom data source follow the data source creation guide selecting custom as the data source type.
Configuring a custom data source
A custom data source uses a plugin gatherer which implements the logic required to connect to and gather content from the custom repository.
Available custom gatherer plugins:
Using a custom gatherer sourced from a plugin
-
Enable the plugin containing the custom gatherer on the custom data source
-
Follow any additional configuration steps as outlined in the plugin’s readme file.
If a custom data source imports multiple plugins that implement custom gathering then you will need to specify which gatherer to use by setting the data source configuration option, plugin.gather-with, to the ID of the plugin.
|
Configuring filters
A custom gatherer is used with a custom data source which has the ability to run filtering (using built-in filters, or filters provided by other plugins)
This enables the content gathered using the custom gatherer to be analyzed or transformed. e.g. If the custom gatherer fetches JSON format data, the custom data source should be configured tp use the JSONToXML filter to transform the JSON into XML that can be indexed.
Filters can be sourced from: