Plugin: Fetch external metadata from URL
Purpose
Use this plugin to download external metadata configuration from one or more publicly accessible URLs during an update.
When to use this plugin
-
To get title, description and other metadata for binary documents that are stored within your DXP Content Management service, or within Squiz Matrix.
-
If you are indexing content from an external system, and you need to download additional metadata to attach to URLs.
The downloaded external metadata must be a valid external-metadata.cfg file.
|
One URL of external metadata vs multiple URLs
Sometimes it is not practical to provide a single URL that contains all the external metadata as the content is too large, or takes too long to generate.
The External metadata file source URL type setting allows you to configure the external metadata fetcher to work with a single URL containing external metadata, or with a URL that contains a list of URLs that contain the external metadata.
If you are downloading a single URL, then the contents returned by the URL must be Funnelback external metadata configuration format.
If you are downloading multiple URLs, then the contents returned by the URL must be a text file containing the list of URLs, one URL per line. Each of the listed URLs is then fetched and must contain Funnelback external metadata configuration.
| Use the multiple URL mode to fetch large amounts of metadata from the Squiz DXP CMS, or Squiz Matrix. |
External metadata feeds requiring authentication
The external metadata fetcher plugin only supports publicly accessible external metadata feeds, and feeds secured using a username and password via HTTP Basic Authentication.
Other forms of authentication are not currently supported, and the plugin authentication must be configured independently of any other authentication configured for the gatherer.
Usage
Enable the plugin
-
Select Plugins from the side navigation pane and click on the Fetch external metadata from URL tile.
-
From the Location section, select the data source to which you would like to enable this plugin from the Select a data source select list.
| The plugin will take effect after setup steps and an advanced > full update of the data source has completed. |
Configuration settings
The configuration settings section is where you do most of the configuration for your plugin. The settings enable you to control how the plugin behaves.
| The configuration key names below are only used if you are configuring this plugin manually. The configuration keys are set in the data source configuration to configure the plugin. When setting the keys manually you need to type in (or copy and paste) the key name and value. |
External metadata file source URL type
Configuration key |
|
Data type |
string |
Default value |
|
Allowed values |
External metadata file,List of external metadata file URLs |
Required |
This setting is required |
Defines the source URL type for the external metadata configuration file.
Possible values:
-
External metadata file: a single external metadata file (default) -
List of external metadata file URLs: a text file containing a list of URLs containing external metadata configuration to download, one URL per line.
External metadata file source URLs
Configuration key |
|
Data type |
array |
Required |
This setting is required |
Defines the source URLs for the external metadata configuration file, or file containing list of external metadata URLs (based on the external metadata file source URL type.
Fail on error
Configuration key |
|
Data type |
boolean |
Default value |
|
Required |
This setting is optional |
Defines if the update should fail with an error or just log a warning if external metadata file is not successfully downloaded.
Possible values:
-
true: The update will fail with an error. (default) -
false: a warning will be logged, but the update will continue.
External metadata file source URL authentication type
Configuration key |
|
Data type |
string |
Default value |
|
Allowed values |
None,HTTP Basic Auth |
Required |
This setting is required |
This option configures the API authentication type.
Only HTTP Basic authentication is supported.
HTTP basic authentication username
Configuration key |
|
Data type |
string |
Required |
This setting is optional |
Defines the HTTP basic authentication username.
This field is required if the External metadata file source URL authentication type is set to HTTP Basic Authentication.
HTTP basic authentication password
Configuration key |
|
Data type |
Encrypted string |
Required |
This setting is optional |
Defines the HTTP basic authentication password.
This field is required if the External metadata file source URL authentication type is set to HTTP Basic Authentication.
Examples
Fetch external metadata from a single URL
In this example https://example.com/extmet.txt contains external metadata definitions that you wish to include in your search.
To fetch the external metadata from https://example.com/extmet.txt, configure the plugin with:
| Configuration key name | Value |
|---|---|
External metadata file source URL type |
|
External metadata file source URL |
|
Fetch external metadata from a feed requiring authentication
This example uses the same feed from the previous example, but is secured using HTTP Basic authentication - with a username of crawluser and password of example123.
To fetch the external metadata from https://example.com/extmet.txt, configure the plugin with the previous details, and also specify the authentication:
| Configuration key name | Value |
|---|---|
External metadata file source URL type |
|
External metadata file source URL |
|
External metadata file source URL authentication type |
|
HTTP basic authentication username |
|
HTTP basic authentication password |
|
Fetch external metadata from multiple URLs
In this example you have external metadata that is contained within multiple URLs.
To fetch this external metadata you need to also have a URL that returns a list of URLs that contain the external metadata.
To fetch a list of external metadata URLs from https://example.com/list.txt, configure the plugin with:
| Configuration key name | Value |
|---|---|
External metadata file source URL type |
|
External metadata file source URL |
|
|
The list of external metadata URLs file For example: |