Help Center

Menu

Plugin: Fetch configuration from URL

Purpose

Use this plugin to download document kill and query independent evidence (QIE) configuration from one or more publicly accessible URLs during an update.

When to use this plugin

This plugin is most commonly required if your content management system has been configured to generate these configuration files, allowing the configuration to be retrieved as part of the update.

Use the Fetch external metadata for URL plugin for external metadata configuration.

One URL of external configuration vs multiple URLs

Sometimes it is not practical to provide a single URL that contains all the external configuration, as the content is too large or takes too long to generate.

The External config file source URL type setting allows you to configure the external config fetcher to work with a single URL containing external config or with a URL that contains a list of URLs containing external config.

If you are downloading a single URL, the contents returned by the URL must be the corresponding Funnelback configuration format.

If you are downloading multiple URLs, then the contents returned by the URL must be a text file containing the list of URLs, one URL per line. Each of the listed URLs is then fetched and must contain the corresponding configuration format.

Use the multiple URL mode to fetch large amounts of configuration from the Squiz DXP CMS or Squiz Matrix.

External configuration feeds requiring authentication

The external config fetcher plugin only supports publicly accessible external config feeds and feeds secured using a username and password via HTTP Basic Authentication.

Other forms of authentication are not currently supported, and the plugin authentication must be configured independently of any other authentication configured for the gatherer.

Push indexes

This plugin can be used with push indexes, however any kill configuration will be ignored (as push indexes do not support kill configuration).

Usage

Enable the plugin

  1. Select Plugins from the side navigation pane and click on the Fetch configuration from URL tile.

  2. From the Location section, select the data source to which you would like to enable this plugin from the Select a data source select list.

The plugin will take effect after setup steps and an advanced > full update of the data source has completed.

Configuration settings

The configuration settings section is where you do most of the configuration for your plugin. The settings enable you to control how the plugin behaves.

The configuration key names below are only used if you are configuring this plugin manually. The configuration keys are set in the data source configuration to configure the plugin. When setting the keys manually you need to type in (or copy and paste) the key name and value.

Kill exact config enabled

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-config-enabled

Data type

boolean

Default value

false

Required

This setting is required

Enable kill exact configuration

Kill exact URL source type

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-url-source-type

Data type

string

Default value

External configuration file

Allowed values

External configuration file,List of external configuration file URLs

Required

This setting is required

Defines the source URL type for the kill exact configuration file.

Possible values:

  • External configuration file: a single external configuration file (default)

  • List of external configuration file URLs: a text file containing a list of URLs containing external configuration to download, one URL per line.

Kill exact URL source

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-url-source

Data type

array

Required

This setting is required

Defines the source URL for the kill exact configuration file, or file containing list of kill exact URLs.

Kill exact fail on error

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-fail-on-error

Data type

boolean

Default value

true

Required

This setting is optional

Defines if the update should fail with an error or just log a warning if kill exact file is not successfully downloaded.

Possible values:

  • true: The update will fail with an error. (default)

  • false: a warning will be logged, but the update will continue.

Kill exact URL source authentication type

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-url-source-auth-type

Data type

string

Default value

None

Allowed values

None,HTTP Basic Auth

Required

This setting is required

This option configures the authentication type.

Only HTTP Basic authentication is supported.

Kill exact URL source HTTP basic authentication username

Configuration key

plugin.fetch-configuration-from-url.config.kill-exact-http-basic-auth-username

Data type

string

Required

This setting is optional

Defines the HTTP basic authentication username.

This field is required if the Kill exact URL source authentication type is set to HTTP Basic Authentication.

Kill exact URL source HTTP basic authentication password

Configuration key

plugin.fetch-configuration-from-url.encrypted.kill-exact-http-basic-auth-password

Data type

Encrypted string

Required

This setting is optional

Defines the HTTP basic authentication password.

This field is required if the Kill exact URL source authentication type` is set to HTTP Basic Authentication.

Kill partial config enabled

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-config-enabled

Data type

boolean

Default value

false

Required

This setting is required

Enable kill partial config

Kill partial URL source type

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-url-source-type

Data type

string

Default value

External configuration file

Allowed values

External configuration file,List of external configuration file URLs

Required

This setting is required

Defines the source URL type for the kill partial configuration file.

Possible values:

  • External configuration file: a single external configuration file (default)

  • List of external configuration file URLs: a text file containing a list of URLs containing external configuration to download, one URL per line.

Kill partial URL source

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-url-source

Data type

array

Required

This setting is required

Defines the source URL for the kill partial configuration file, or file containing list of kill partial URLs.

Kill partial fail on error

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-fail-on-error

Data type

boolean

Default value

true

Required

This setting is optional

Defines if the update should fail with an error or just log a warning if kill partial file is not successfully downloaded.

Possible values:

  • true: The update will fail with an error. (default)

  • false: a warning will be logged, but the update will continue.

Kill partial URL source authentication type

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-url-source-auth-type

Data type

string

Default value

None

Allowed values

None,HTTP Basic Auth

Required

This setting is required

This option configures the authentication type.

Only HTTP Basic authentication is supported.

Kill partial URL source HTTP basic authentication username

Configuration key

plugin.fetch-configuration-from-url.config.kill-partial-http-basic-auth-username

Data type

string

Required

This setting is optional

Defines the HTTP basic authentication username.

This field is required if the Kill partial URL source authentication type is set to HTTP Basic Authentication.

Kill partial URL source HTTP basic authentication password

Configuration key

plugin.fetch-configuration-from-url.encrypted.kill-partial-http-basic-auth-password

Data type

Encrypted string

Required

This setting is optional

Defines the HTTP basic authentication password.

This field is required if the Kill partial URL source authentication type is set to HTTP Basic Authentication.

QIE config enabled

Configuration key

plugin.fetch-configuration-from-url.config.qie-config-enabled

Data type

boolean

Default value

false

Required

This setting is required

Enable QIE config

QIE URL source type

Configuration key

plugin.fetch-configuration-from-url.config.qie-url-source-type

Data type

string

Default value

External configuration file

Allowed values

External configuration file,List of external configuration file URLs

Required

This setting is required

Defines the source URL type for the QIE configuration file.

Possible values:

  • External configuration file: a single external configuration file (default)

  • List of external configuration file URLs: a text file containing a list of URLs containing external configuration to download, one URL per line.

QIE URL source

Configuration key

plugin.fetch-configuration-from-url.config.qie-url-source

Data type

array

Required

This setting is required

Defines the source URL for the QIE configuration file, or file containing list of QIE URLs.

QIE fail on error

Configuration key

plugin.fetch-configuration-from-url.config.qie-fail-on-error

Data type

boolean

Default value

true

Required

This setting is optional

Defines if the update should fail with an error or just log a warning if QIE file is not successfully downloaded.

Possible values:

  • true: The update will fail with an error. (default)

  • false: a warning will be logged, but the update will continue.

QIE URL source authentication type

Configuration key

plugin.fetch-configuration-from-url.config.qie-url-source-auth-type

Data type

string

Default value

None

Allowed values

None,HTTP Basic Auth

Required

This setting is required

This option configures the authentication type.

Only HTTP Basic authentication is supported.

QIE URL source HTTP basic authentication username

Configuration key

plugin.fetch-configuration-from-url.config.qie-http-basic-auth-username

Data type

string

Required

This setting is optional

Defines the HTTP basic authentication username.

This field is required if the QIE URL source authentication type is set to HTTP Basic Authentication.

QIE URL source HTTP basic authentication password

Configuration key

plugin.fetch-configuration-from-url.encrypted.qie-http-basic-auth-password

Data type

Encrypted string

Required

This setting is optional

Defines the HTTP basic authentication password.

This field is required if the QIE URL source authentication type` is set to HTTP Basic Authentication.

QIE config enabled

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-config-enabled

Data type

boolean

Default value

false

Required

This setting is required

Enable query QIE config

QIE URL source type

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-url-source-type

Data type

string

Default value

External configuration file

Allowed values

External configuration file,List of external configuration file URLs

Required

This setting is required

Defines the source URL type for the query QIE configuration file.

Possible values:

  • External configuration file: a single external configuration file (default)

  • List of external configuration file URLs: a text file containing a list of URLs containing external configuration to download, one URL per line.

QIE URL source

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-url-source

Data type

array

Required

This setting is required

Defines the source URL for the query QIE configuration file, or file containing list of query QIE URLs.

QIE fail on error

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-fail-on-error

Data type

boolean

Default value

true

Required

This setting is optional

Defines if the update should fail with an error or just log a warning if query QIE file is not successfully downloaded.

Possible values:

  • true: The update will fail with an error. (default)

  • false: a warning will be logged, but the update will continue.

QIE URL source authentication type

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-url-source-auth-type

Data type

string

Default value

None

Allowed values

None,HTTP Basic Auth

Required

This setting is required

This option configures the authentication type.

Only HTTP Basic authentication is supported.

QIE URL source HTTP basic authentication username

Configuration key

plugin.fetch-configuration-from-url.config.query-qie-http-basic-auth-username

Data type

string

Required

This setting is optional

Defines the HTTP basic authentication username.

This field is required if the QIE URL source authentication type is set to HTTP Basic Authentication.

QIE URL source HTTP basic authentication password

Configuration key

plugin.fetch-configuration-from-url.encrypted.query-qie-http-basic-auth-password

Data type

Encrypted string

Required

This setting is optional

Defines the HTTP basic authentication password.

This field is required if the query QIE URL source authentication type` is set to HTTP Basic Authentication.

Examples

Fetch kill exact (kill_exact.cfg)

Fetch kill exact from a single URL

In this example https://example.com/kill_exact.txt contains kill exact definitions that you wish to kill by URL in your search.

To fetch the kill exact from https://example.com/kill_exact.txt, configure the plugin with:

Configuration key name Value

Kill exact config enabled

true

Kill exact URL source type

External config file

Kill exact URL source

https://example.com/kill_exact.txt

Fetch kill exact from a single URL

This example uses the same feed from the previous example, but you want Funnelback to continue the process even facing the error (e.g. the external configuration URL returns HTTP Status 5xx)

Configuration key name Value

Kill exact config enabled

true

Kill exact URL source type

External config file

Kill exact URL source

https://example.com/kill_exact.txt

Kill exact fail on error

false

Fetch kill exact from a feed requiring authentication

This example uses the same feed from the previous example, but is secured using HTTP Basic authentication - with a username of crawluser and password of example123.

To fetch the kill exact from https://example.com/kill_exact.txt, configure the plugin with the previous details, and also specify the authentication:

Configuration key name Value

Kill exact config enabled

true

Kill exact URL source type

External config file

Kill exact URL source

https://example.com/kill_exact.txt

Kill exact URL source authentication type

HTTP Basic Auth

Kill exact URL source HTTP basic authentication username

crawluser

Kill exact URL source HTTP basic authentication password

example123

Fetch kill exact from multiple URLs

In this example, you have kill exact that is contained within multiple URLs.

To fetch this kill exact you need to also have a URL that returns a list of URLs that contain the kill exact.

To fetch a list of kill exact URLs from https://example.com/list.txt, configure the plugin with:

Configuration key name Value

Kill exact config enabled

true

Kill exact URL source type

List of external config file URLs

Kill exact URL source URL

https://example.com/list.txt

The Kill exact URL source URL, https://example.com/list.txt, is a text file containing 1 URL per line. Each URL in the list must return a valid kill_exact.cfg file.

For example:

https://example.com/kill_exact1.txt
https://example.com/kill_exact2.txt

Fetch kill partial (kill_partial.cfg)

Fetch kill partial from a single URL

In this example, https://example.com/kill_partial.txt contains kill partial definitions that you wish to kill by URL in your search.

To fetch the kill partial from https://example.com/kill_partial.txt, configure the plugin with:

Configuration key name Value

Kill partial config enabled

true

Kill partial URL source type

External config file

Kill partial URL source

https://example.com/kill_partial.txt

Fetch kill partial from a single URL

This example uses the same feed from the previous example, but you want Funnelback to continue the process even facing the error (e.g. the external configuration URL returns HTTP Status 5xx)

Configuration key name Value

Kill partial config enabled

true

Kill partial URL source type

External config file

Kill partial URL source

https://example.com/kill_partial.txt

Kill partial fail on error

false

Fetch kill partial from a feed requiring authentication

This example uses the same feed from the previous example, but is secured using HTTP Basic authentication - with a username of crawluser and password of example123.

To fetch the kill partial from https://example.com/kill_partial.txt, configure the plugin with the previous details, and also specify the authentication:

Configuration key name Value

Kill partial config enabled

true

Kill partial URL source type

External config file

Kill partial URL source

https://example.com/kill_partial.txt

Kill partial URL source authentication type

HTTP Basic Auth

Kill partial URL source HTTP basic authentication username

crawluser

Kill partial URL source HTTP basic authentication password

example123

Fetch kill partial from multiple URLs

In this example, you have kill partial that is contained within multiple URLs.

To fetch this kill partial you need to also have a URL that returns a list of URLs that contain the kill partial.

To fetch a list of kill partial URLs from https://example.com/list.txt, configure the plugin with:

Configuration key name Value

Kill partial config enabled

true

Kill partial URL source type

List of external config file URLs

Kill partial URL source

https://example.com/list.txt

The Kill partial URL source, https://example.com/list.txt is a text file containing 1 URL per line. Each URL in the list must return a valid kill_partial.cfg file.

For example:

https://example.com/kill_partial1.txt
https://example.com/kill_partial2.txt

Fetch QIE by URL (qie.cfg)

Fetch QIE from a single URL

In this example, https://example.com/qie.txt contains QIE definitions that you wish defined in data soruce.

To fetch the QIE from https://example.com/qie.txt, configure the plugin with:

Configuration key name Value

QIE config enabled

true

QIE URL source type

External config file

QIE URL source

https://example.com/qie.txt

Fetch QIE from a single URL

This example uses the same feed from the previous example, but you want Funnelback to continue the process even facing the error (e.g. the external configuration URL returns HTTP Status 5xx)

Configuration key name Value

QIE config enabled

true

QIE URL source type

External config file

QIE URL source

https://example.com/qie.txt

QIE fail on error

false

Fetch QIE from a feed requiring authentication

This example uses the same feed from the previous example, but is secured using HTTP Basic authentication - with a username of crawluser and password of example123.

To fetch the QIE from https://example.com/qie.txt, configure the plugin with the previous details, and also specify the authentication:

Configuration key name Value

QIE config enabled

true

QIE URL source type

External config file

QIE URL source

https://example.com/qie.txt

QIE URL source authentication type

HTTP Basic Auth

QIE URL source HTTP basic authentication username

crawluser

QIE URL source HTTP basic authentication password

example123

Fetch QIE from multiple URLs

In this example, you have QIE that is contained within multiple URLs.

To fetch this QIE you need to also have a URL that returns a list of URLs that contain the QIE.

To fetch a list of QIE URLs from https://example.com/list.txt, configure the plugin with:

Configuration key name Value

QIE config enabled

true

QIE URL source type

List of external config file URLs

QIE URL source

https://example.com/list.txt

The QIE URL source, https://example.com/list.txt is a text file containing 1 URL per line. Each URL in the list must return a valid qie.cfg file.

For example:

https://example.com/qie1.txt
https://example.com/qie2.txt

Fetch QIE by query (query-qie.cfg)

Fetch QIE by query from a single URL

In this example, https://example.com/query_qie.txt contains QIE by query definitions that you wish to apply in your search.

To fetch the QIE by query from https://example.com/query_qie.txt, configure the plugin with:

Configuration key name Value

Query QIE config enabled

true

Query QIE URL source type

External config file

Query QIE URL source

https://example.com/query_qie.txt

Fetch QIE by query from a single URL

This example uses the same feed from the previous example, but you want Funnelback to continue the process even facing the error (e.g. the external configuration URL returns HTTP Status 5xx)

Configuration key name Value

Query QIE config enabled

true

Query QIE URL source type

External config file

Query QIE URL source

https://example.com/query_qie.txt

Query QIE fail on error

false

Fetch QIE by query from a feed requiring authentication

This example uses the same feed from the previous example, but is secured using HTTP Basic authentication - with a username of crawluser and password of example123.

To fetch the QIE by query from https://example.com/query_qie.txt, configure the plugin with the previous details, and also specify the authentication:

Configuration key name Value

Query QIE config enabled

true

Query QIE URL source type

External config file

Query QIE URL source

https://example.com/query_qie.txt

Query QIE URL source authentication type

HTTP Basic Auth

Query QIE URL source HTTP basic authentication username

crawluser

Query QIE URL source HTTP basic authentication password

example123

Fetch QIE by query from multiple URLs

In this example, you have QIE by query that is contained within multiple URLs.

To fetch this QIE by query you need to also have a URL that returns a list of URLs that contain the QIE by query.

To fetch a list of QIE by query URLs from https://example.com/list.txt, configure the plugin with:

Configuration key name Value

Query QIE config enabled

true

Query QIE URL source type

List of external config file URLs

Query QIE URL source

https://example.com/list.txt

The Query QIE URL source, https://example.com/list.txt is a text file containing 1 URL per line. Each URL in the list must return a valid query-qie.cfg file.

For example:

https://example.com/query_qie1.txt
https://example.com/query_qie2.txt

Caveats

Due to bug in plugin configuration UI this plugin v1.0.0 cannot be configured via dedicated UI and as workaround can be set using results page configuration keys UI for results page and data source configuration keys UI for data source.

Change log

See also