Collection-based auto-completion


Funnelback has a native auto-completion system (suggest.json) that suggests completions as the user types. This auto-completion system uses a JSON REST API to provide suggestions based on the user’s partially entered query using a lightweight search algorithm. This ensures that suggestions are returned quickly and with minimal load on the server.

In some cases however, there may be a need to use the full text search algorithm to provide suggestions to the user. In doing this however, we face a couple of challenges:

  • The user has not entered the full query they are looking for. We may only have queries like "con" or "dav" not giving a lot of context to what the user is searching for yet.

  • The time it takes to run a full query using the standard search algorithm is slower than using the native autocompletion system

To overcome these challenges we can take the following steps:

  • Utilise the top suggestions from the native autocomplete and perform a full text search based on these suggestions

  • Limit the data returned by the result packet

  • Turn off unnecessary search features or settings

Occasionally Funnelback’s auto-completion is too basic to service the needs of a user and it becomes necessary to use a standard Funnelback collection and padre-sw query to supply the auto-completion.

The following process can be used to deploy collection based auto-completion.

The use of collection-based auto-completion is not recommended and should be avoided unless other auto-completion options have been exhausted. Funnelback’s built-in auto-completion system is designed for speed (and minimal system overhead when the request is processed). Using a standard query to produce query completion can result in significant load being placed on the server as a query can be fired for each user keystroke.

Setup process

  1. Create an auto-completion profile on the collection used to deliver auto-completion suggestions

  2. Funnelback 15.24 or newer: Add the following to the query processor options for the profile. Funnelback 15.22 and earlier: Add the following to padre_opts.cfg. This is to optimise the query.

    -stem=0 -SM=off -bb=false -rmcf=[dummyvalue] -spelling=off -contextual_navigation=false -QL=0 -log=false -countgbits=63  -collapsing=false -show_qsyntax_tree=off
  3. Add the following custom Freemarker template to the auto-completion profile:

    <#ftl encoding="utf-8" />
    <#import "/web/templates/modernui/funnelback_classic.ftl" as s/>
    <#import "/web/templates/modernui/funnelback.ftl" as fb/>
    <#escape x as x?jsonstring>
    <#if question.inputParameterMap["callback"]?exists>${question.inputParameterMap["callback"]}(</#if>[
    <#if response.resultPacket.resultsSummary.totalMatching != 0>
    <#if s.result.class.simpleName != "TierBar">
    "key" : "<#if question.inputParameterMap["partial_query"]?exists>${question.inputParameterMap["partial_query"]?json_string}</#if>",
    "disp" : "${s.result.title}",
    "disp_t" : "T",
    "wt" : "${s.result.score}",
    "cat" : "",
    "cat_t" : "1",
    "action" : "${s.result.clickTrackingUrl}",
    "action_t" : "U"
    }<#if (s.result.rank < response.resultPacket.resultsSummary.currEnd)>,</#if>
    <#if question.inputParameterMap["callback"]?exists>)</#if>
  4. Add the following code to the collection’s pre process hook script:

    // Expand a partial query for collection_based auto-completion
    // reads the partial_query CGI parameter and creates a disjunctive query based on the suggestions returned from padre-qs
    // Imports required for access to the padre suggest service
    import java.util.List;
    import com.funnelback.dataapi.connector.padre.PadreConnector;
    import com.funnelback.dataapi.connector.padre.suggest.Suggestion;
    import com.funnelback.dataapi.connector.padre.suggest.Suggestion.ActionType;
    import com.funnelback.dataapi.connector.padre.suggest.Suggestion.DisplayType;
    if(transaction.question.inputParameterMap['profile'].equals("auto-completion") || transaction.question.inputParameterMap['profile'].equals("auto-completion_preview")) {
      def logger = org.apache.log4j.Logger.getLogger("partial query expander")
      def q = transaction.question
      // set query completion to use qc form
      q.form = "qc"
      // Convert a partial query into a set of query terms
      // Maximum number of query terms to expand partial query to - read from collection.cfg partial_query_expansion_index parameter.
      // eg. partial_query=com might expand to query=[commerce commercial common computing]
      def partial_query_expansion_index = 5
      if ((q.collection.configuration.value(["partial_query_expansion_index"]) != null) && (q.collection.configuration.value(["partial_query_expansion_index"]).isInteger())) {
        partial_query_expansion_index = q.collection.configuration.value(["partial_query_expansion_index"])
      if (q.inputParameterMap["partial_query"] != null) {
        //File searchHome = new File("/opt/funnelback") //14.2+
        File indexStem = new File(q.collection.configuration.value(["collection_root"]) + File.separator + "live" + File.separator + "idx","index")
        // NOTE: CONSTRUCTOR depends on the version of Funnelback being run
        List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem, //15.16+
        // List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem) //15.0-15.14
        // List<Suggestion> suggestions = new PadreConnector(indexStem) //14.0 or earlier
        // build the expanded query from the list of suggestions
        def expanded_query = ""
        suggestions.each {
            expanded_query += '"'+it.key+'" '
        // set the number of suggestions to the value of the configured
        q.additionalParameters["num_ranks"] = [q.inputParameterMap["show"]]
        // set the query to the expanded set of query terms ORed together
        if (expanded_query != "") {
            q.query = "["+expanded_query+"]"
  5. Add the following options to the collection configuration:

    #Optional setting to adjust number of suggestions that the partial query is expanded to