Help Center

Menu

Writing search lifecycle code to manipulate the data model

Modifications to the search query and response can be implemented by writing a search lifecycle plugin which contains code that can read and write to the data model.

Creating a search lifecycle plugin

A search lifecycle plugin is created using the Maven archetype template.

The values that need to be set to create a search lifecycle plugin are:

Define value for property Value

groupId

Set this to an appropriate group ID

artifactId

Set this to an appropriate artifact ID

version

Set this to an appropriate version. For a new plugin set this to 1.0.0

package

Set this an appropriate package name

facets

Set this to false

filtering

Set this to false

gatherer

Set this to false

indexing

Set this to false

jsoup-filtering

Set this to false

plugin-description

Set an appropriate description for the plugin

plugin-name

Set an appropriate (human-friendly) name for the plugin

runs-on-datasource

Set this to false

runs-on-result-page

Set this to true

search-servlet-filtering

Set this to false

searchLifeCycle

Set this to true

start-url-provider

Set this to false
A plugin can implement more than one type of interface. The values above are for creating a plugin that only implements a search lifecycle plugin.

General concepts

Search lifecycle plugins provide methods which interact with the data model at different points in the search lifecycle.

Understanding what happens at each stage of the search lifecycle, and when parts of the data model are initialized is critical when writing a search lifecycle plugin.

See:

Search lifecycle plugins allow you to write code that runs in the following search lifecycle phases:

Pre-post phase step Plugin - SearchLifeCyclePlugin interface method

pre-process

void preProcess(SearchTransaction transaction)

pre-datafetch

void preDatafetch(SearchTransaction transaction)

post-datafetch

void postDatafetch(SearchTransaction transaction)

post-process

void postProcess(SearchTransaction transaction)
If you are familiar with writing hook scripts, the contents of these methods match the contents of the equivalent hook script. If you wish to convert an existing hook script into a plugin you can transfer the code from your hook script into your plugin and then convert it to Java. Unlike hook scripts all your code for a single piece of functionality is captured in the four methods above within a single file (instead of code spread across different groovy hook scripts). Note that plugins should be written so that they are reusable and to perform only a single function (or set of related functions) and existing hook script code may need to be converted to multiple plugins.

Manipulating the search query (question)

To modify the search query, implement either the pre-process or pre-datafetch method.

The relevant method to use will depend on what you are modifying. See: Search lifecycle - data model variable initialization for information on when various parts of the data model are initialized.

Key question data model items to modify

These are the more common data model elements that are modified in search lifecycle plugins.

In general, modify in the pre-process method if you want to affect how subsequent items are initialized (e.g. if you wish to change the meta parameters you can modify the meta_X values in the question.inputParameters as a pre-process step and this will mean that the question.metaParameters is generated correctly).

question.inputParameters

Modify this for any parameters that will be used by the modern UI, or for any parameters that are used to setup the elements that are populated between the pre-process and the extra searches/pre-datafetch phases.

question.additionalParameters

Modify this for any parameters that need to be passed directly to padre as query processor options.

This includes the following:

  • origin

  • maxdist

  • sort

  • numeric query parameters (e.g. lt_x)

  • SM

  • SF

  • num_ranks

question.query

Modify this if you need to manipulate the passed in query.

Manipulating the search results (response)

To modify the search response, implement either the post-process or post-datafetch method.

The relevant method to use will depend on what you are modifying. See: Search lifecycle - data model variable initialization for information on when various parts of the data model are initialized.

In general, modify the response in the following phases:

Post-data fetch

use this to modify the raw response values returned by padre. e.g. to update the live URL before the click link is generated, or to make modifications to faceted navigation such as renaming or sorting categories. In particular modify the response in the post-data fetch phase if you wish to affect any of these values (which are populated based on other response items):

  • search result links (click, cache, final live and display links)

  • curator related items

  • HTML encoded summaries

  • extra search responses

  • related document elements

  • faceted navigation elements

  • translations

  • search history

Post-process

Use this to make any other modification to elements that will be displayed.

Search types

By default, search lifecycle code will run on all search requests including those run by content auditor, accessibility auditor and also extra searches.

Each of these searches has a particular search question type that indicates the type of search that is running.

Conditional code based on the search question type can be used to write code that only affects the data model question and response for the specific types of searches.

For most use cases you don’t need to worry about the type of search that is running. However, accessing the search question type allows you to write a plugin that targets only content auditor, accessibility auditor etc.

Restricting search lifecycle code to specific search types

Funnelback search queries are used for a number of different purposes including internal tasks. Every query that runs will apply search lifecycle plugins if they are enabled. Depending on what a plugin does, this can negatively impact these internal tasks.

To mitigate this Funnelback defines a data model variable that tracks the type of search that is running. This can be used to conditionally run plugin code.

In most cases you do not need to worry about the question type unless you have very specific needs, such as modifying the extra search query for when the plugin should run or the code is making modifications that break functionality such as content auditor.

In order to restrict your code to certain question types, you need to import the question types, read the current question type and perform a comparison to decide if you should run your code.

Possible values for the question type are:

Question type Description

SearchQuestionType.SEARCH

A search query submitted to the HTML, XML, JSON endpoints (e.g. search.html, search.json etc)

SearchQuestionType.SEARCH_GET_ALL_RESULTS

A search query submitted to the all-results endpoint all-results.json or all-results.csv.

SearchQuestionType.EXTRA_SEARCH

An extra search configured on a search package.

SearchQuestionType.CONTENT_AUDITOR

A content auditor query.

SearchQuestionType.CONTENT_AUDITOR_DUPLICATES

A content auditor duplicates query.

SearchQuestionType.ACCESSIBILITY_AUDITOR

An accessibility auditor query.

SearchQuestionType.ACCESSIBILITY_AUDITOR_ACKNOWLEDGEMENT_COUNTS

Accessibility auditor query to determine acknowledgement counts.

SearchQuestionType.ACCESSIBILITY_AUDITOR_GET_ALL_RESULTS

Accessibility auditor all-results query.

SearchQuestionType.FACETED_NAVIGATION_EXTRA_SEARCH

Built-in extra search used to obtain faceted navigation information.

Plugin code can target a specific question type using a conditional statement. e.g. only run for the CONTENT_AUDITOR search:

Example: Only run this code for content auditor queries
import com.funnelback.publicui.search.model.transaction.SearchQuestion.SearchQuestionType; (1)

SearchQuestionType questionType = transaction.getQuestion().getQuestionType(); (2)

public void preProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

    if (questionType.equals(SearchQuestionType.CONTENT_AUDITOR)) { (3)
        // Whatever you want to do...
    }
}
1 Import the list of pre-defined question types.
2 Get the question type of the current search.
3 Only run this for content auditor queries.

Testing

When writing unit tests for the SearchTransaction use the following helper classes to generate the mock data sent to your tests:

  • TestableSearchTransaction

  • SearchQuestionTestHelper

Logging

Log messages will appear in the search package’s user interface logs.

When developing a plugin add log messages and use the appropriate log level. The following log levels are defined in log4j:

fatal
error
warn
info
debug
trace
by default you will only see warn or higher level log messages appearing in your log files (or info or higher if your package is in the com.funnelback namespace). When testing use a tool, such as the ModHeader browser extension to set the x-funnelback-request-detailed-logging header to increase the log level for your request. Don’t forget to turn this off when you’ve finished testing.

See: reference/modern-ui/modern-ui-logging.adoc#increase-log-level-for-request

Example: print some log messages
import org.apache.logging.log4j.LogManager; (1)
import org.apache.logging.log4j.Logger; (1)

@Override
public void postProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

    log.info("Running pre process plugin example"); (2)

    if (transaction.getResponse().hasResultPacket()) {
        log.info("Processing search results"); (2)
    }
    else {
        log.error("ERROR: result packet missing - no results to process."); (3)
    }

}
1 Imports required for logging. These should already be imported as they are set in the maven archetype template.
2 Print a log message at the INFO log level. This message will only appear in your logs if your package is within the com.funnelback namespace, or you increase your log level.
3 Print a log message at the ERROR log level.

Examples

Transforming each result

Search result transformations can be implemented in either the postProcess() or postDatafetch() methods.

Example: Convert result titles to uppercase
import com.funnelback.publicui.search.model.padre.Result; (1)

@Override
public void postProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

    if (transaction.getResponse().hasResultPacket()) {
        transaction.getResponse().getResultPacket().getResults().forEach(r -> transformResult(r)); (2)
    }
}

protected Result transformResult(Result result) {
    // Code that processes each search result.

    result.setTitle(result.getTitle().toUpperCase()); (3)
}
1 Imports the Result type used by the transformResult() method.
2 Calls the transformResult() method for each search result in the data model response.
3 Converts the result’s title to upper case.

Working with input parameters

Reading input parameters

Input parameters can be read from the methods corresponding to any of the input or output phases - preProcess(), preDatafetch(), postProcess() or postDatafetch().

Example: read an input parameter value
import static com.funnelback.publicui.search.model.transaction.SearchQuestion.RequestParameters.NUM_RANKS; (1)

@Override
public void preProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

    if (transaction.getQuestion().getInputParameters().containsKey(NUM_RANKS)) { (2)
        var numRanks = transaction.getQuestion().getInputParameters().get(NUM_RANKS); (3)
    }
}
1 Imports the static NUM_RANKS parameter definition. For pre-defined parameters it is best practice to import the definition instead of hardcoding the equivalent value as a string. e.g. use transaction.getQuestion().getInputParameters().get(NUM_RANKS) instead if transaction.getQuestion().getInputParameters().get("num_ranks").
2 Check if an input parameter is defined. Note: it is very important to check for existence before attempting to read or write a parameter to avoid exceptions that will prevent further code for the method from running.
3 Reads the value of a specific input parameter, NUM_RANKS, from the data model. The string NUM_RANKS is defined in the SearchQuestion.RequestParameters class.

Setting the value of existing or new input parameters

Input parameters can be modified from the methods corresponding to input processing phases - preProcess() or preDatafetch().

Example: set or update an input parameter value
import static com.funnelback.publicui.search.model.transaction.SearchQuestion.RequestParameters.NUM_RANKS; (1)

@Override
public void preProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

     if (transaction.getQuestion().getInputParameters().containsKey(NUM_RANKS)) { (2)

        var numRanks = transaction.getQuestion().getInputParameters().get(NUM_RANKS); (3)

        log.info("Updating NUM_RANKS to 25");
        transaction.getQuestion().getInputParameters().get(NUM_RANKS).set(0,"25"); (4)
        transaction.getQuestion().getAdditionalParameters().put(NUM_RANKS, new String[]{"25"});  (5)
    }
    else {
        log.info("Setting NUM_RANKS to 20");
        transaction.getQuestion().getInputParameters().put(NUM_RANKS, "20"); (6)
        transaction.getQuestion().getAdditionalParameters().put(NUM_RANKS, new String[]{"20"}); (7)
    }

    if (!transaction.getQuestion().getInputParameters().containsKey("custom_param")) { (8)
        log.info("Setting custom_parameter to example");
        transaction.getQuestion().getInputParameters().put("custom_param", "example"); (9)
    }

}
1 Imports the static NUM_RANKS parameter definition. For pre-defined parameters it is best practice to import the definition instead of hardcoding the equivalent value as a string. e.g. use transaction.getQuestion().getInputParameters().get(NUM_RANKS) instead if transaction.getQuestion().getInputParameters().get("num_ranks").
2 Check if an input parameter is defined. Note: it is very important to check for existence before attempting to read or write a parameter to avoid exceptions that will prevent further code for the method from running.
3 Reads the value of a specific input parameter, NUM_RANKS, from the data model. The string NUM_RANKS is defined in the SearchQuestion.RequestParameters class.
4 Updates the question.inputParameters["num_ranks"] value of NUM_RANKS to a value of 25.
5 Updates the question.additionalParameters["num_ranks"] value of NUM_RANKS to a value of 25.
6 Sets the question.inputParameters["num_ranks"] value of NUM_RANKS to a value of 20.
7 Sets the question.additionalParameters["num_ranks"] value of NUM_RANKS to a value of 20.
8 Tests for existence of a custom parameter, custom_param.
9 Sets the value of a custom parameter, custom_param, to a value of example.

To modify an extra search query wrap your method code within a conditional statement that only runs for the extra search query.

The conditional statement can appear in any of the search lifecycle methods, depending on what you need your plugin to do.

Extra search configuration allows you to set query processor options that are applied when running an extra search. These can also be set within a plugin (such as applying a gscope or clive, or setting the number of results). Before writing a plugin, make sure that you’re not duplicating something that can be set as a query processor option on the extra search.
Example: Modify an extra search question
import com.funnelback.publicui.search.model.transaction.SearchQuestion.SearchQuestionType;

public void preProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

    SearchQuestionType questionType = transaction.getQuestion().getQuestionType();

    if (questionType.equals(SearchQuestionType.EXTRA_SEARCH)) { (1)

         // Gets the name of the current extra search transaction
            String extraSearchName = transaction.getExtraSearchName().get(); (2)


        // Your plugin code

    }

}
1 Only run this for extra search queries.
2 This will provide the name of the current extra search being processed. Use this in conditional logic to target a specific extra search.

Accessing plugin configuration

You can define configuration keys that can be set in the results page configuration to configure your plugin.

see: Defining plugin configuration keys

Configuration keys are read from the data model using the getCurrentProfileConfig() method:

...
    String maxItems = transaction.getQuestion().getCurrentProfileConfig().get(PluginUtils.MAX_ITEMS);
...

Example: title-prefix-plugin

The developing your first plugin guide runs through the process of creating a search lifecycle plugin. This plugin replaces a pattern at the start of a title in matching search results.

This plugin is explained in detail as part of the developing your first plugin tutorial - please refer to this for detailed notes on various parts of the code.
TitlePrefixPluginSearchLifeCyclePlugin.java
package com.example.plugin.titleprefix;

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

import com.funnelback.plugin.SearchLifeCyclePlugin;
import com.funnelback.publicui.search.model.transaction.SearchTransaction;
import com.funnelback.plugin.search.SearchLifeCycleContext;

public class TitleprefixSearchLifeCyclePlugin implements SearchLifeCyclePlugin {

    final PluginUtils pluginUtils = new PluginUtils();
    private static final Logger log = LogManager.getLogger(TitleprefixSearchLifeCyclePlugin.class);

    @Override
    public void postProcess(SearchLifeCycleContext searchLifeCycleContext, SearchTransaction transaction) {

        String prefixToMatch = transaction.getQuestion().getCurrentProfileConfig().get(pluginUtils.PATTERN.getKey());
        String givenReplaceWith = transaction.getQuestion().getCurrentProfileConfig().get(pluginUtils.REPLACEMENT.getKey());

        if (prefixToMatch == null || prefixToMatch.isBlank()) {
            // There is no prefix configured, we can't do anything. Let's log a warning and abort.
            log.warn("Could not find a pattern specified at " + pluginUtils.PATTERN.getKey() +  ". The TitlePrefixPlugin will do nothing.");
            return;
        }
        // We'll mark replaceWith as an optional key, if someone doesn't supply it, then we'll just remove the pattern configured.
        String replaceWithToUse = givenReplaceWith != null ? givenReplaceWith: "";

        transaction.getResponse().getResultPacket().getResults().stream().filter(result -> {
            return result.getTitle().startsWith(prefixToMatch);
        }).forEach(matchingResult -> {
            String modifiedTitle = matchingResult.getTitle().replace(prefixToMatch, replaceWithToUse);
            matchingResult.setTitle(modifiedTitle);
        });
    }


}
TitlePrefixPluginSearchLifeCyclePluginTest.java
package com.example.plugin.titleprefix;

import org.junit.Assert;
import org.junit.Test;

import com.funnelback.publicui.search.model.padre.Result;
import com.funnelback.publicui.search.model.transaction.testutils.TestableSearchTransaction;
import com.funnelback.plugin.search.mock.MockSearchLifeCycleContext;

public class TitleprefixSearchLifeCyclePluginTest {

    @Test
    public void testSearchLifeCyclePlugin(){
        TestableSearchTransaction searchTransaction = new TestableSearchTransaction()
            .withResult(Result.builder().title("hello").liveUrl("http://example.com/").build());
        MockSearchLifeCycleContext mockSearchLifeCycleContext = new MockSearchLifeCycleContext();
        // Update this to call the method(s) that should be tested.
        new TitleprefixSearchLifeCyclePlugin().postDatafetch(mockSearchLifeCycleContext, searchTransaction);

        Assert.assertEquals("Change this assert statement to check something useful",
            "hello", searchTransaction.getResponse().getResultPacket().getResults().get(0).getTitle());
    }

}

See also