Collection-based auto-completion

Background

Funnelback has a native auto-completion system (suggest.json) that suggests completions as the user types. This auto-completion system uses a JSON REST API to provide suggestions based on the user’s partially entered query using a lightweight search algorithm. This ensures that suggestions are returned quickly and with minimal load on the server.

In some cases however, there may be a need to use the full text search algorithm to provide suggestions to the user. In doing this however, we face a couple of challenges:

The user has not entered the full query they are looking for. We may only have queries like "con" or "dav" not giving a lot of context to what the user is searching for yet.
The time it takes to run a full query using the standard search algorithm is slower than using the native autocompletion system

To overcome these challenges we can take the following steps:

Utilise the top suggestions from the native autocomplete and perform a full text search based on these suggestions
Limit the data returned by the result packet
Turn off unnecessary search features or settings

Occasionally Funnelback’s auto-completion is too basic to service the needs of a user and it becomes necessary to use a standard Funnelback collection and padre-sw query to supply the auto-completion.

The following process can be used to deploy collection based auto-completion.

The use of collection-based auto-completion is not recommended and should be avoided unless other auto-completion options have been exhausted. Funnelback’s built-in auto-completion system is designed for speed (and minimal system overhead when the request is processed). Using a standard query to produce query completion can result in significant load being placed on the server as a query can be fired for each user keystroke.

Setup process

Create an auto-completion profile on the collection used to deliver auto-completion suggestions
Funnelback 15.24 or newer: Add the following to the query processor options for the profile. Funnelback 15.22 and earlier: Add the following to padre_opts.cfg. This is to optimise the query.
```
-stem=0 -SM=off -bb=false -rmcf=[dummyvalue] -spelling=off -contextual_navigation=false -QL=0 -log=false -countgbits=63  -collapsing=false -show_qsyntax_tree=off
```

Add the following custom Freemarker template to the auto-completion profile:

qc.ftl

<#ftl encoding="utf-8" />
<#import "/web/templates/modernui/funnelback_classic.ftl" as s/>
<#import "/web/templates/modernui/funnelback.ftl" as fb/>
<#escape x as x?jsonstring>
<#if question.inputParameterMap["callback"]?exists>${question.inputParameterMap["callback"]}(</#if>[
<@s.AfterSearchOnly>
<#if response.resultPacket.resultsSummary.totalMatching != 0>
<@s.Results>
<#if s.result.class.simpleName != "TierBar">
{
"key" : "<#if question.inputParameterMap["partial_query"]?exists>${question.inputParameterMap["partial_query"]?json_string}</#if>",
"disp" : "${s.result.title}",
"disp_t" : "T",
"wt" : "${s.result.score}",
"cat" : "",
"cat_t" : "1",
"action" : "${s.result.clickTrackingUrl}",
"action_t" : "U"
}<#if (s.result.rank < response.resultPacket.resultsSummary.currEnd)>,</#if>
</#if>
</@s.Results>
</#if>
</@s.AfterSearchOnly>]
<#if question.inputParameterMap["callback"]?exists>)</#if>
</#escape>

Add the following code to the collection’s pre process hook script:

hook_pre_process.groovy

// Expand a partial query for collection_based auto-completion
// reads the partial_query CGI parameter and creates a disjunctive query based on the suggestions returned from padre-qs
// Imports required for access to the padre suggest service
import java.io.File;
import java.util.List;
import com.funnelback.dataapi.connector.padre.PadreConnector;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion.ActionType;
import com.funnelback.dataapi.connector.padre.suggest.Suggestion.DisplayType;

if(transaction.question.inputParameterMap['profile'].equals("auto-completion") || transaction.question.inputParameterMap['profile'].equals("auto-completion_preview")) {
  def logger = org.apache.log4j.Logger.getLogger("partial query expander")
  def q = transaction.question

  // set query completion to use qc form
  q.form = "qc"
  // Convert a partial query into a set of query terms
  // Maximum number of query terms to expand partial query to - read from collection.cfg partial_query_expansion_index parameter.
  // eg. partial_query=com might expand to query=[commerce commercial common computing]
  def partial_query_expansion_index = 5
  if ((q.collection.configuration.value(["partial_query_expansion_index"]) != null) && (q.collection.configuration.value(["partial_query_expansion_index"]).isInteger())) {
    partial_query_expansion_index = q.collection.configuration.value(["partial_query_expansion_index"])
  }

  if (q.inputParameterMap["partial_query"] != null) {
    //File searchHome = new File("/opt/funnelback") //14.2+
    File indexStem = new File(q.collection.configuration.value(["collection_root"]) + File.separator + "live" + File.separator + "idx","index")
    // NOTE: CONSTRUCTOR depends on the version of Funnelback being run
    List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem, q.collection.id) //15.16+
    // List<Suggestion> suggestions = new PadreConnector(searchHome, indexStem) //15.0-15.14
    // List<Suggestion> suggestions = new PadreConnector(indexStem) //14.0 or earlier
        .suggest(q.inputParameterMap["partial_query"])
        .suggestionCount(partial_query_expansion_index)
        .fetch();
    // build the expanded query from the list of suggestions
    def expanded_query = ""
    suggestions.each {
        expanded_query += '"'+it.key+'" '
    }
    // set the number of suggestions to the value of the configured auto-completion.show
    q.additionalParameters["num_ranks"] = [q.inputParameterMap["show"]]
    // set the query to the expanded set of query terms ORed together
    if (expanded_query != "") {
        q.query = "["+expanded_query+"]"
    }
  }
}

Add the following options to the collection configuration:

collection.cfg

auto-completion.program=../s/search.html
ui.modern.form.qc.content_type=application/javascript
#Optional setting to adjust number of suggestions that the partial query is expanded to
#partial_query_expansion_index=8

Help Center

Menu

Collection-based auto-completion

Background

Setup process