Help Center

Menu

Defining plugin configuration keys

This section describes how to define configuration options for your plugin, allowing it to be configured via the plugin configuration screens of the administration dashboard.

When should I define plugin configuration keys?

Plugin configuration keys are key/value pairs and should be used for plugin configuration where a user needs to provide simple configuration - like a username or password.

If you need to create a set of linked configuration items (say you are writing a plugin where you can define a username and password for specific domains) then wildcard configuration keys (see below) might be appropriate - these allow you to define a set of key/value configuration items that are linked with a common identifier.

e.g. the example below uses 3 configuration keys that would be set up as wildcard keys: plugin.example.config.*.url, `plugin.example.config.*.username` and plugin.example.config.*.password.

  • plugin.example.config.site1.url=https://site1.example.com

  • plugin.example.config.site1.username=crawl

  • plugin.example.config.site1.password=P@ssword

  • plugin.example.config.site2.url=https://site2.example.com

  • plugin.example.config.site2.username=test

  • plugin.example.config.site2.password=letmein

For more complicated configuration, such as defining rules, consider using a plugin configuration file for these definitions.

Your plugin can use both configuration keys and configuration files. For example the three wildcard keys above could be implemented as a rule file that has a rule defined for each of the sets of configuration, and you might also define a default set of configuration keys such as `plugin.example.config.default-username` and plugin.example.config.default-password. This sort of scenario would enable you to write a plugin that could define a default username and password to apply, but allow an override for specific sites that are defined in the rules configuration file.

Configuration key definitions

The configuration definitions will affect how the plugin is configured via the administration dashboard. The dashboard uses these definitions to both present the key editing interface, and also to validate the input. For example, if you define an integer type configuration key, then the dashboard will present an error message if you don’t put a number into the field.

The configuration key definitions are specified in the PluginUtils.java file, by calling a configuration key builder for each key that you wish to define.

Each configuration key requires a block of code similar to the following to build the configuration key object, defining a set of properties for the configuration key.

public final PluginConfigKey<String> KEY_PREFIX = PluginConfigKey.<String>builder() (1)
    .pluginId(getPluginId()) (2)
    .id("regex.*") (3)
    .required(true)  (4)
    .label("Removal pattern (regex)") (5)
    .description("This key defines a Java format regular expression pattern that is compared to the title. Any parts of the title that match this regular expression will be removed.") (6)
    .type(new PluginConfigKeyType(PluginConfigKeyType.Format.STRING)) (7)
    .build(); (8)
1 Defines a String type configuration key which is referenced in your code as KEY_PREFIX.
2 This should be set to getPluginId(), the ID of the plugin you are building.
3 This sets the configuration key’s id. This value is used in the data source or results page configuration key name.
The use of .* in the key name means that you can have multiple keys where the user supplies an identifying parameter. See details below.
4 This indicates if the configuration key is mandatory (required=true) or optional (required=false). If you set this to true the plugin configuration screen in the administration dashboard will not allow you to save the plugin until this is set.
5 This is the name for the configuration key. This value will be shown in the administration dashboard when configuring the key, and also in the documentation.
6 This sets a description that will be shown on the administration dashboard, and also in the documentation. The description should be concise, and must not include any formatting. If you need to add additional information (including asciidoc formatting), also set the longDescription property.
7 This is used by the API to validate the format of the supplied value.
8 This is required and builds the key object.

The order of the properties in the above code is not important, and the set of properties you define will depend on your configuration key.

Encrypted keys

The plugin framework can also define encrypted keys, which should be used when you are storing sensitive information such as a password or API Key.

This follows a very similar builder format to a normal key, except you use the PluginConfigKeyEncrypted.builder() instead of the PluginConfigKey.<TYPE>builder().

    public final PluginConfigKeyEncrypted PASSWORD = PluginConfigKeyEncrypted.builder()
            .pluginId(getPluginId())
            .id("password")
            .required(true)
            .label("Password")
            .description("Password to access the service.")
            .build();

Defining the name used on the configuration screens and the key that is used in data source and results page configuration

When you define a configuration option for a plugin you configure two properties which controls how you identify the key when configuring it.

The label property defines the name that you will see displayed for the option when you use the plugin configuration screens of the administration dashboard.

The id property is used to create a unique configuration key, which is what is saved within the data source or results page configuration. This is the internal identifier used for the option and is determined by the id and pluginId properties that you provide when building the key.

All plugin keys have the following format:

plugin.<pluginId>.config.<id>=<VALUE>

When you define the id property of the plugin you can include wildcards within the value. Keys that contain wildcards can be defined multiple times for the plugin and require the use to input an extra parameter value that replaces the wildcard so that each key that is added is unique for the plugin.

  • Standard keys can be set once in your configuration and have a format similar to:

  • Wildcard keys can be set multiple times in your configuration and are used to group related sets of options together. The wildcards in the ID allow a user to input additional IDs (through parameter 1, parameter 2 fields in the UI) which replace the wildcards in the key names.

Plugin configuration properties

The following properties can be set when calling the configuration key builder. Properties marked as (Mandatory) must be defined for each key.

.id(String)

(Mandatory) The ID is used to define unique configuration keys that identify the configuration key. The ID can include wildcards if you require multiple keys to be defined for this configuration option. The wildcard values are used to group sets of keys together, where you might have several keys that form a piece of linked configuration (like a username key and a password key where you need to link a specific username key to a specific password key).

For example:

  • .id("foo") creates a configuration key: plugin.<pluginID>.config.foo

  • .id("foo.bar") creates a configuration key: plugin.<pluginID>.config.foo.bar

  • .id("foo.*") creates a configuration key: plugin.<pluginID>.config.foo.*. This will result in the configuration UI providing an additional Parameter 1 field that a user can input an identifier. If they input Parameter 1=example then the configuration key written into the configuration will be plugin.<pluginID>.config.foo.example.

  • .id("foo.*.bar") creates a configuration key: plugin.<pluginID>.config.foo.*.bar. This will result in the configuration UI providing an additional Parameter 1 field that a user can input an identifier. If they input Parameter 1=example then the configuration key written into the configuration will be plugin.<pluginID>.config.foo.example.bar.

  • .id("foo.*.bar.*") creates a configuration key: plugin.<pluginID>.config.foo.*.bar.*. This will result in the configuration UI providing an additional Parameter 1 and Parameter 2 field that a user can input two identifiers. If they input Parameter 1=apple and Parameter 2=orange then the configuration key written into the configuration will be plugin.<pluginID>.config.foo.apple.bar.orange.

    Wildcards must be separated by a non-wildcard component in the id. e.g. you can’t have consecutive wildcards such as .id("foo.*.*")
Example: setting the id property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .id("username") (1)
...
    .build();
1 Defines the plugin key ID. This identifies the key and will be result in a plugin.<pluginId>.config.username key being available in your data source or results page configuration.

.pluginId(String)

(Mandatory) This should be set to getPluginId(), which returns the ID of the plugin you are building.

Example: setting the plugin id property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .pluginId(getPluginId()) (1)
...
    .build();
1 This should be set to the ID of the plugin you are developing. The getPluginId() returns this value so you should set the pluginId property to the value returned by this method.

.label(String)

(Mandatory) Label used to for the configuration option. This label is displayed in the plugin configuration screens, and also in the documentation.

Example: setting the label property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .label("Start date") (1)
...
    .build();
1 This sets the label of the plugin to Start date. The label is used in the administration dashboard and the documentation to identify the configuration option.

.description(String)

(Mandatory) Short description describing what this key should be used for. This description is used in the administration dashboard as the plugin description, and also as the first paragraph of the description within the plugin documentation. This field must be plain text only (and not include any Asciidoc formatting).

When defining the plugin description use a short sentence starting with something like: Use this plugin to …​ See the plugins listed in the administration dashboard for examples of what has been defined for other plugins.
Example: setting the short description property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .description("Use this plugin to index content from the LinkedIn social media platform") (1)
...
    .build();
1 This description will appear when you are browsing plugins in the administration dashboard, and is also used as the first paragraph of the description in the documentation.

.longDescription(String)

Additional description text for the plugin configuration key. This text is displayed in following the short description in the plugin documentation. This field can include Asciidoc formatting.

You can include paragraphs of Asciidoc formatted text. To do this you will need to include linebreak characters (\n) in your definition at the end of every line of Asciidoc text (including blank lines).
Example: setting the long description property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .longDescription("The social media presets can be used to quickly apply your date filter to Facebook posts and events, Twitter tweets, YouTube videos and Instagram posts.\n" + (1)
        "\n" + (2)
        "If you select a *CUSTOM* record type, you also need to define the source field and date format using the corresponding configuration settings.")
...
    .build();
1 The \n indicates a line break. The + operators join the strings together in Java.
2 A blank line is required in the Asciidoc to ensure a new paragraph is created.

.defaultValue

This defines a default value for the configuration key.

Example: setting the default value property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .defaultValue("myValue") (1)
...
    .build();
1 The default value argument is the same type as the type of key you are defining. In the example above it is a String key so the default value here is set to "myValue". If the key being defined was an integer you might have a default value like .defaultValue(100).

.required(Boolean)

When set to true, this indicates that the field is mandatory. A user will not be able to save their plugin configuration unless this field is set.

Example: setting the required property
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .required(true) (1)
...
    .build();
1 Indicates that this key must be set when configuring the plugin.

.type(PluginConfigKeyType)

(Mandatory) This defines the data type of the configuration key value.

Example: setting the plugin type
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .type(new PluginConfigKeyType(PluginConfigKeyType.Format.STRING)) (1)
...
    .build();
1 Sets the configuration key type as a String.

The types you can use are:

  • PluginConfigKeyType.Format.ARRAY

  • PluginConfigKeyType.Format.BOOLEAN

  • PluginConfigKeyType.Format.DATE

  • PluginConfigKeyType.Format.INTEGER

  • PluginConfigKeyType.Format.LONG

  • PluginConfigKeyType.Format.METADATA

  • PluginConfigKeyType.Format.PASSWORD

  • PluginConfigKeyType.Format.STRING

If you are defining a configuration key that has an array value, you must also pass a second parameter indicating the subtype:

Example: setting the plugin type to an array
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .type(new PluginConfigKeyType(PluginConfigKeyType.Format.ARRAY,PluginConfigKeyType.Format.STRING)) (1)
...
    .build();
1 Sets the configuration key type as an array/list of strings.

.allowedValue

This can be used to specify a set of allowed values, or validation rule for the key.

Example: Restricting the key values to a list of specified values
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .allowedValue(new PluginConfigKeyAllowedValue<>(List.of("NONE", "FACEBOOK_POST", "FACEBOOK_EVENT", "TWITTER", "YOUTUBE", "INSTAGRAM", "CUSTOM"))) (1)
...
    .build();
1 Adds a restriction to the configuration key so that it only accepts values that you have specified in a set of values.
Regular expression validation pattern
public final PluginConfigKey<String> EXAMPLE = PluginConfigKey.<String>builder()
...
    .allowedValue(new PluginConfigKeyAllowedValue<>(Pattern.compile("ident-\\d+"))) (1)
...
    .build();
1 Adds a restriction to the configuration key so that it only accepts values that match the regular expression ident-\d+. This will match any value of ident- followed by one or more digits. The regular expression must be a valid Java regular expression.

.showIfKeyHasValue

This is used to define a configuration key that is only shown in the administration dashboard, when another is set to a specified value.

Example: setting the show if has key property
public final PluginConfigKey<String> SUB_ELEMENT = PluginConfigKey.<String>builder()
...
    .showIfKeyHasValue(
        PluginConfigKeyConditional.<String>builder()
            .associatedKeyId("plugin.example.config.record_type") (1)
            .associatedKeyValue("CUSTOM") (1)
            .build())
...
    .build();
1 Configures the key so that it is only shown when the plugin.example.config.record_type key (with id=record_type) is set to a value of CUSTOM.

getConfigKeys()

The getConfigKeys() method in PluginUtils.java must return a list of all the configuration keys you have defined. This is used by the administration dashboard when it presents the plugin configuration screen.

Example: Returning the list of configuration keys
    @Override
    public List<PluginConfigKeyDetails> getConfigKeys() {
        return List.of(USERNAME, PASSWORD, REPOSITORY_ID);
    }

Accessing plugin configuration keys from your plugin code

Access to plugin configuration is via the plugin context object that is associated with the interface that you are implementing.

Please see the section on the specific interface you are implementing for the details on how to read configuration keys for that interface.

Configuration keys

Configuration keys defined in the data source or results page configuration can be accessed using a variety of methods that return single or multiple configuration keys.

Please refer to the Plugin shared Javadoc documentation for the interface that you are implementing for the specific methods available. The methods below vary depending on the requirements of the interface.

For example, the gather and index interfaces provide the following methods for accessing configuration keys:

  • Map<String, List<String>> getConfigKeysMatchingPattern(String pattern): Provides config settings which match the given key pattern.

  • Set<String> getConfigKeysWithPrefix(String prefix): Provides a list of configuration setting keys which have a common prefix.

  • String getConfigSetting(String key): Provides access to configurations settings from the current Funnelback installation and data source/results page.

Configuration files

Access to a plugin-specific configuration file is provided using a method in the plugin context object for the interface that you are implementing.

Please refer to the Plugin shared Javadoc documentation for the interface that you are implementing for the specific methods available. The methods below vary depending on the requirements of the interface.

For example, the gather and index interfaces provide the following methods for accessing configuration files:

  • Optional<String> pluginConfigurationFile(String filename): Reads a configuration file for the currently running plugin as a UTF-8 String.

  • Optional<byte[]> pluginConfigurationFileAsBytes(String filename): Reads a configuration file for the currently running plugin as binary data.

Example: Read a configuration key and fall back to a default value

/*
Note: this will require you to add an import to the top of your java file.

import java.util.Optional;
*/


/*
  For boolean values

  Example: Read the plugin.example-gatherer.debug configuration key
   from the data source configuration, and set to false if it's not set.
*/
Boolean debug = Optional.ofNullable(context.getConfigSetting("plugin.example-gatherer.debug")).map(Boolean::parseBoolean).orElse(false);

/*
  For string values

  Example: Read the plugin.example-gatherer.debug configuration key
   from the data source configuration, and set to false if it's not set.
*/

String example = Optional.ofNullable(context.getConfigSetting("plugin.example-gatherer")).orElse("default value");

Dynamically updating the configuration for a plugin

Custom gatherer plugins provide a method that enables the plugin to set or update configuration keys.

Updating of configuration files is not supported.

The gather interface provides the following method:

  • void setConfigSetting(String key, String value): Set the value of configuration key to value in the data source configuration for the running plugin, including the setting of encrypted keys.

The plugin can only set/update configuration keys that directly relate to the plugin (limited to the plugin’s name space). It will throw a run-time exception if it attempts to set a key outside this scope.

Example: Set a configuration key to a specified value

try {
    // Update the last-update-time so that the next plugin execution can perform an incremental update.

    context.setConfigSetting("plugin.example-gatherer.last-update-time", "2022-06-01 13:21:04");

} catch (runTimeException e) {

    // If the the plugin attempts to update a key outside it's namespace then the method will throw a
    // run-time exception containing the message stating that the given key is not allowed for change.

    <Code that implements what to do if the key was unable to be set.>
}

See also