Implementer training - alternate output formats

Funnelback includes XML and JSON endpoints that return the raw Funnelback data model in these formats.

These can be useful when interacting with Funnelback, however the data model is complex and contains a lot of data (meaning the response packets can be quite large).

Funnelback also provides an endpoint that is designed to stream all the results back to the user in a single call. The all results endpoint can be used to return all matching Funnelback results in JSON or CSV format. This endpoint only returns the results section of the data model and only minimal modifications can be made to the return format.

Funnelback’s HTML endpoint utilizes Freemarker to template the results. This is traditionally used to format the search results as HTML. However, this templating is extremely flexible and can be adapted to return any text-based format - such as CSV, RSS or even custom XML/JSON formats.

This is useful if you need to export search results, or use Funnelback’s index as a data source to be consumed by another service.

Returning search results (only) as CSV or custom JSON

The all results endpoint is ideal for providing a CSV export or custom JSON of search results.

A number of parameters can be supplied to the endpoint that control the fields returned and the field labels used.

The all results endpoint returns the result level data for all the matching results to a query. This is ideal for search results export. If data model elements from outside the results element are required (e.g. faceted navigation counts, search result counts, query) then a Freemarker template will be required to create the custom output.

Tutorial: Return search results as CSV

In this exercise the all results endpoint will be configured to return the search results as CSV.

  1. Decide what fields you want to return as the columns of your CSV. Any of the fields with the search result part of the data model can be returned. For the airport finder results page we decide to return only the following fields, which are all sourced from the indexed metadata: "Name","City","Country","IATA/FAA","ICAO","Altitude"

  2. A URL will need to be constructed that defines the fields that are required. Several parameters will be required to set the fields and field names to use:

    • collection: Required. Set this to the search package ID.

    • profile: Set this to the results page ID.

    • fields: Required. Defines the fields that will be returned by the all results endpoint as a comma separated list of fields. Nested items in the results part of the data model require an xpath style value. e.g. listMetadata/author will return the author field from the listMetadata sub-element.

    • fieldnames: Optional list of field names to use. These will be presented as the column values (CSV) or element keys (JSON). If omitted then the raw values from fields will be used (e.g. listMetadata/author).

    • SF: Required if any metadata elements are being returned. Ensure this is configured to return all the metadata fields that are required for the output.

    • SM: Required if any metadata elements or the automatically generated results summary are being returned.

  3. Decide what fields need to be returned. For the airport finder results page we will return six columns: Name, City, Country, IATA/FAA, ICAO and Altitude. Open the search dashboard, change to the airports data data source and view the metadata mappings. Make a note of the metadata class names for these fields.

  4. Define the query string parameters. To return the six fields will require the following parameters:

    • collection=default~sp-airports

    • profile=airport-finder

    • fields=listMetadata/name,listMetadata/city,listMetadata/country,listMetadata/iataFaa,listMetadata/icao,listMetadata/altitude

    • fieldnames=Name,City,Country,IATA/FAA,ICAO,Altitude

    • SM=meta

    • SF=[name,city,country,iataFaa,icao,altitude]

  5. Construct a URL from this information and view the response from the all results JSON endpoint. Note that you need to URL encode your parameter values. This illustrates the custom JSON that can be returned using the all results endpoint:

    exercise return results as csv 01
  6. Access the corresponding all results CSV endpoint. When you click the link a CSV file should be downloaded to your computer. Open this file after you’ve downloaded it.

    exercise return results as csv 02
    The values of the metadata within the CSV will be surrounded with square brackets as the listMetadata element is an array/list of values. You will need to convert these values as desired. This can easily be done using macros within programs such as Microsoft Excel or Google Sheets.
  7. The file will have been saved with a file name similar to all-results.csv. The file name can be defined by adding an additional parameter, fileName, to the URL. Modify the previous URL and add &fileName=airports.csv to the URL then press enter. The file should download and this time be saved as airports.csv.

  8. Open the search dashboard and change to the airport finder results page.

  9. Edit the default template (select the simple.ftl from the edit results page templates listing on the template panel). Add a button above the search results that allows the results to be downloaded as CSV. Add the following code immediately before the <ol> tag with an ID of search-results (approx. line 505):

    <p><a class="btn btn-success" href="/s/all-results.csv?collection=${question.collection.id}&query=${question.query}&profile=${question.profile}&fields=listMetadata/name,listMetadata/city,listMetadata/country,listMetadata/iataFaa,listMetadata/icao,listMetadata/altitude&fieldnames=Name,City,Country,IATA/FAA,ICAO,Altitude&SF=[name,city,country,iataFaa,icao,altitude]&SM=meta&fileName=airports.csv">Download results as CSV</a></p>
  10. Run a search for denmark and observe that a download results as CSV button appears above the 28 search results.

    exercise return results as csv 03
  11. Click the download as csv button and the csv file will download. Open the CSV file and confirm that all the results matching the query are included in the CSV data (matching the 28 results specified by the results summary).

    exercise return results as csv 04

Using Freemarker templates to return other text-based formats

Funnelback’s HTML endpoint can be used to define a template that returns the search results in any arbitrary text-based format.

This is commonly used for returning custom JSON or XML but can also be used to return other known formats such as tab delimited data or GeoJSON.

If you are creating a template for a result format that you wish to be automatically downloaded (instead of displayed in the browser) you can set an additional HTTP response header for the template to ensure that the browser downloads the file. This header can also be used to define the filename.

If a file name isn’t set the download will save the file using the browser’s default naming.

The name of the file can be set by setting the content-disposition HTTP header. To do this edit the results page configuration and add the following:

ui.modern.form.<TEMPLATE-NAME>.headers.1=Content-Disposition: attachment; filename=<FILE-NAME>

This will tell Funnelback to send a custom HTTP header (Content-Disposition) with the <TEMPLATE-NAME>.ftl template, instructing the browser to save the file as <FILE-NAME>.

Tutorial: Return search results as GeoJSON

In this exercise a Freemarker template will be created that formats the set of search results as a GeoJSON feed.

  1. Log in to the search dashboard where you are doing your training.

    See: Training - search dashboard access information if you’re not sure how to access the training. Ignore this step if you’re treating this as a non-interactive tutorial.
  2. Change to the airport finder results page.

  3. Create a new custom template. Click the edit results page templates item on the template panel, then click the add new button.

    exercise return results as geojson 01
  4. Enter geojson.ftl as the name of the file once the new file editor loads.

    exercise return results as geojson 02
  5. Create template code to format the results as GeoJSON. Copy the code below into your template editor and hit save.

    <#ftl encoding="utf-8" output_format="JSON"/>
    <#import "/web/templates/modernui/funnelback_classic.ftl" as s/>
    <#import "/web/templates/modernui/funnelback.ftl" as fb/>
    <#compress>
    <#-- geojson.ftl
    Outputs Funnelback response in GeoJSON format.
    -->
    <#-- Read the metadata field to source the latLong value from from the map.geospatialClass collection.cfg option -->
    <#--<#assign latLong=question.currentProfileConfig.get("map.geospatialClass")/>-->
    <#-- Hard code using the latLong metadata field for this example -->
    <#assign latLong="latlong"/>
    <@s.AfterSearchOnly>
    <#-- NO RESULTS -->
    <@s.IfDefCGI name="callback">${question.inputParameters["callback"]?first!}(</@s.IfDefCGI>
    {
    <#if response.resultPacket.resultsSummary.totalMatching != 0>
    <#-- RESULTS -->
    "type": "FeatureCollection",
    "features": [
    <@s.Results>
    <#if s.result.class.simpleName != "TierBar">
    <#if s.result.listMetadata[latLong]?? && s.result.listMetadata[latLong]?first!?matches("-?\\d+\\.\\d+;-?\\d+\\.\\d+")> <#-- has geo-coord and it's formatted correctly - update to the meta class containing the geospatial coordinate -->
    <#-- EACH RESULT -->
    {
    "type": "Feature",
    "geometry": {
    "type": "Point",
    "coordinates": [${s.result.listMetadata[latLong]?first!?replace(".*\\;","","r")},${s.result.listMetadata[latLong]?first!?replace("\\;.*","","r")}]
    },
    "properties": { <#-- Fill out with all the custom metadata you wish to expose (e.g. for use in the map display -->
    "rank": "${s.result.rank?string}",
    "title": "${s.result.title!"No title"}",
    <#if s.result.date??>"date": "${s.result.date?string["dd MMM YYYY"]}",</#if>
    "summary": "${s.result.summary!}",
    "fileSize": "${s.result.fileSize!}",
    "fileType": "${s.result.fileType!}",
    "exploreLink": "${s.result.exploreLink!}",
    <#if s.result.kmFromOrigin?? && question.inputParameters["origin"]??>"kmFromOrigin": "${s.result.kmFromOrigin?string("0.###")}",</#if>
    <#-- MORE METADATA FIELDS... -->
    "listMetadata": {
    <#list s.result.listMetadata?keys as metaDataKey>
    "${metaDataKey}": "${s.result.listMetadata[metaDataKey]?join(",")}"<#if metaDataKey_has_next>,</#if>
    </#list>
    },
    "displayUrl": "${s.result.liveUrl!}",
    "cacheUrl": "${s.result.cacheUrl!}",
    "clickTrackingUrl": "${s.result.clickTrackingUrl!}"
    }
    }<#if s.result.rank &lt; response.resultPacket.resultsSummary.currEnd>,</#if>
    </#if> <#-- has geo-coord -->
    </#if>
    </@s.Results>
    ]
    </#if>
    }<@s.IfDefCGI name="callback">)</@s.IfDefCGI>
    </@s.AfterSearchOnly>
    </#compress>

    As you can see from the code above the template is bare bones and only handles the case of formatting search results after the search has run plus a minimal amount of wrapping code required by the GeoJSON format. Each result is templated as a feature element in the JSON data.

    Also observe that the template defines a JSON output format (<#ftl encoding="utf-8" output_format="JSON"/>) to ensure that only valid JSON is produced.

  6. Run a search using the geojson template - click on the search icon (eye icon third from the right) that appears in the available actions within the file listing.

    exercise return results as geojson 03
  7. A blank screen will display - Funnelback has loaded with the template by no query is supplied. Specify a query by adding &query=!showall to the end of the URL. A fairly empty JSON response is returned to the screen. The response may look like unformatted text, depending on the JSON plugin you have installed. The response is quite empty because the template requires the latlong metadata field to be returned in the response and the collection isn’t currently configured to return this.

    exercise return results as geojson 04
  8. Return to the search dashboard and add latlong to the list of summary fields set in the results page configuration. (Update the -SF parameter to include latlong). Refresh the results and observe that you are now seeing data points returned

    exercise return results as geojson 05
  9. The template is now configured to return the text in the GeoJSON format. To ensure that your browser correctly detects the JSON we should also configure the GeoJSON template to return the correct MIME type in the HTTP headers. The correct MIME type to return for JSON is application/json - Funnelback templates are returned as text/html by default (which is why the browser renders the text). Return to the administration interface and edit the profile configuration (customize panel, edit results page configuration).

    If you are returning JSON as JSONP response you should use a text/javascript mime type as the JSON is wrapped in a Javascript callback. .
    exercise return results as geojson 06
  10. Add the following setting to the profile configuration then save and then publish. This option configures Funnelback to return the text/javascript content type header when using the geojson.ftl template. We are doing this because the template is configured to support a callback function to enable JSONP.

    • Parameter key: ui.modern.form.*.content_type

    • Form name: geojson

    • Value: text/javascript

    the content type option is set per-template name at the profile level - the above option would apply to a template called geojson.ftl but only when the matching profile is set as part of the query. ui.modern.form.TEMPLATE.content_type would set the content type for a template named TEMPLATE.ftl.
  11. Rerun the search using the geojson template. This time the browser correctly detect and format the response as JSON in the browser window.

    exercise return results as geojson 08
  12. Observe that the GeoJSON response only contains 10 results - this is because it is templating what is returned by Funnelback, and that defaults to the first 10 results. In order to return more results an additional parameter needs to be supplied that tells Funnelback how many results to return. E.g. try adding &num_ranks=30 to the URL and observe that 30 results are now returned. If you wish to return all the results when accessing the template you will need to set the num_ranks value either to a number equivalent to the number of documents in your index, or link it from another template where you can read the number of results.

Extended exercise: Plot search results on a map

Use the code for the Funnelback mapping plugin to add a search-powered map to your search results page that uses a GeoJSON feed to supply the data points.