SEARCH 202 - Implement a new search

Log in to the search dashboard where you are doing your training.

Click the add new search package button. This opens the screen used to create a new search package.

When prompted, enter Library as the search package name. Observe that the search package ID is automatically generated from the name that you enter. Click the continue button to proceed to the second step of search package creation.

Step 2 of search package creation enables you to (optionally) attach existing data sources to your new search package, which you pick from the drop-down menu. Click the proceed button to skip this step as we will create our data source later.

Review the information that you have entered. If you find any mistakes in the information you have entered you can return to a previous step by clicking on the step number. Once you are satisfied with the information in the review panel, click the finish button to create your search package.

The search package management screen is displayed. Data sources and results pages can be set up from the panel located at the bottom of your page. However, for this exercise we will return to the main search dashboard before setting these up. Return to the search dashboard home page by clicking the dashboard item in the breadcrumb trail, the search dashboard item in the side navigation, or by clicking on the Funnelback logo on the menu bar.

You are returned to the search package listing. Scroll down the list of search packages and observe that the Library search package that you just created is now listed with the other search packages.

Log in to the search dashboard where you are doing your training.

Start the process to create a new data source by clicking on the create a data source button, located on the data sources tab that is displayed for the search package you created in the previous exercise.

A modal opens providing you with the option to create or attach a data source. The attach option can be used to attach an existing data source to your search package. We are not interested in using any of the existing data sources in our new Library search, so we will create a new data source. Click the create button.

The first step of data source creation requires you to choose your data source type. Select the web radio button then click the continue button.

Enter Shakespeare when prompted for a data source name. Observe that the data source ID is automatically generated from your chosen name. Click the proceed button to move to the configuration step.

Enter some basic configuration for the data source. At a minimum you need to tell the crawler what should and shouldn’t be included in the crawl.

Enter the following into the What website(s) do you want to crawl? field:

https://docs.squiz.net/training-resources/shakespeare/

Click the proceed button to review the information you have provided. If you find any errors in what you’ve provided you can click on the step number to update the information. Once you are happy with the information in the review panel click the proceed button to create your data source.

You are provided with a final option to update your data source now, or to update it later. Select the option to update now, then click the finish button to complete the data source creation steps.

The data source management screen loads. Note that the status of the data source reads queued. This means that an update of the data source has been added to the task queue, but hasn’t yet started. The status will refresh as the update progresses.

After a short time the status message should update to indicate that the update is running (or has finished).

Repeat the above steps to create a second data source with the following details:

When you’ve finished creating the data source, run an update.

Log in to the search dashboard where you are doing your training.

Locate the library search package that you created in a previous exercise.

Click on the results pages tab then click the create a results page button.

Enter Book finder as the results page name then click the continue button.

You are then provided with an option to add a thumbnail image for the results page - this image is displayed on the results page management screen and insights dashboard. We’ll skip this step now (an image is easily added later). Click the proceed button to move to the next step of the results page creation process.

Review the settings that you have entered. Once satisfied, click the finish button to create the results page.

You are redirected to the results page management screen where you can perform further setup of your new results page.

Log in to the search dashboard where you are doing your training.

locate the austen data source.

Open the manage data source screen (by clicking on the austen title, or selecting the > configuration menu item. The status should now be showing update complete, which indicates that the update completed successfully.

Access the data source configuration by clicking the edit data source configuration option in the settings section.

Update the start url by clicking on the start_url item. Update the value to https://docs.squiz.net/training-resources/austin/ (change austen to austin) then save your changes. We are intentionally using a URL containing a typo in this example so we can examine the resulting errors.

Return to the data source manage screen by clicking Austen in the breadcrumbs, then start an update by clicking on the update data source item in the update section. The update should fail almost immediately, and the status will update to show update failed (note you might need to refresh your browser screen).

Access the logs (click on browse log files in the tools section). Inspect the log files for the data source - starting with the update-training~ds-austen.log. The update-<DATA-SOURCE-ID>.log provides an overview of the update process and should give you an idea of where the update failed. Observe that there is an error returned by the crawl step. This suggests that we should investigate the crawl logs further.

As the data source update failed, the offline view will contain all the log files for the update that failed. Locate the crawler.log from the offline logs section and inspect this. The log reports that no URLs were stored.

Examine the url_errors.log which logs errors that occurred during the crawl. From this log you can see that a 404 not found error was returned when accessing http://localhost:9080/training/training-data/ds~austin/ which is the seed URL for the crawl. This explains why nothing was indexed because the start page was not found, so the crawl could not progress any further.

E https://docs.squiz.net/training-resources/austin/ [404 Not Found] [2022:09:26:03:07:54]

With this information at hand you can investigate further. In this case the reason the crawl failed was due to the seed URL being incorrectly typed. But you might visit the seed URL from your browser to investigate further.

Return to the edit data source configuration screen and correct the start URL.

Log in to the search dashboard where you are doing your training.

Locate the library search package.

Create another results page, but this time call the results page Shakespeare search.

Run a search for dream ensuring the preview option is selected and observe that pages from both the Shakespeare and Austen sites are returned.

Select edit results page configuration from the customize section. The index can be scoped by setting an additional query processor option.

Add a query processor option to scope the results page - various options can be used to scope the results page including scope, xscope, gscope1 and clive. The clive parameter is a special scoping parameter that can be applied to results page to restrict the results to only include pages from a specified data source or set of data sources. The clive parameter takes a list of data source IDs to include in the scoped set of data. Add a clive parameter to scope the results page to pages from the training~ds-shakespeare data source then click the save button, (but do not publish) the file. You can find the data source ID in the information panel at the top of the results page management screen.

-clive=training~ds-shakespeare

Rerun the search for dream against the preview version of the Shakespeare search results page and observe that the results are now restricted to only pages from the Shakespeare site.

Rerun the search for dream against the live version of the Shakespeare search results page. Observe that pages are returned from both sites - this is because the query processor options must be published for the changes to take effect on the live version of the results page.

Edit results page configuration again, and publish the query processor options to make the setting affect the live version of the results page.

Rerun the search for dream against the Shakespeare search results page. Observe that pages are now restricted to Shakespeare site pages.

Log in to the search dashboard where you are doing your training.

Navigate to the Shakespeare search results page

Select edit results page configuration from the customize section that was edited in the previous exercise. Set the results page to return five results per page and sort alphabetically then save and publish the changes.

-clive=training~ds-shakespeare -sort=title -num_ranks=5

Rerun the search for dream against the Shakespeare search results page specifying and observe that the results are sorted alphabetically by title and only 5 results are returned per page. Results are all from the Shakespeare website.

Log in to the search dashboard where you are doing your training.

Locate the airports search package.

Navigate to the manage data source screen for the airports data data source.

Edit the metadata mappings. (Settings panel, configure metadata mappings).

Modify the mapping for the <LATLONG> field to set the type as a geospatial coordinate.

Modify the mapping for the <Altitude field> to be number then save the changes.

Rebuild the index (as you have changed the metadata configuration). Reminder: Update panel, start advanced update, rebuild live index.

Run a search for !showall and inspect the JSON noting that kmFromOrigin elements now appear (due to the elements containing geospatial metadata).

Return to the HTML results and add numeric constraints to the query to return only airports that are located between 2000 ft and 3000 ft: add &lt_altitude=3000&ge_altitude=2000 to the URL observing that the number of matching results is reduced and that altitudes of the matching results are all now between 2000 and 3000.

Remove the numeric parameters and add an origin parameter to the URL: &origin=48.8588336;2.2769957 (this is the lat/long value for Paris, France). Observe that the kmFromOrigin field now contains values.

Geospatial searches can be limited to a radius measured from the origin (in km). Supply an optional maxdist parameter and set this to 500km, by adding &maxdist=500 to the URL. Note that the number of results has dropped dramatically and are all airports within 500km of Paris.

Edit the template to print out the kmFromOrigin value in the results. Add the following below the metadata (e.g. immediately before the </dl> tag at approx line 595) that is printed in the result template:

<#if s.result.kmFromOrigin??>
<dt>Distance from origin:</dt><dd>${s.result.kmFromOrigin} km</dd>
</#if>

Run the !showall search again and observe the distance is now returned in the results.

Sort the results by proximity to the origin by adding &sort=prox and observe that the kmFromOrigin values are now sorted by distance (from nearest to farthest).

Log in to the search dashboard where you are doing your training.

Click the plugins item on the navigation panel.

This opens the plugins screen, which lists all the currently available plugins. The list of available plugins will increase as new plugins are released.

Have a quick scroll through the available plugins to get a feel for the different types of plugins and how they extend the available functionality.

Type facet into the search plugins box and observe that the tiles update to only display plugins relating to faceted navigation.

Clear the search box and then choose data source from the runs on menu. Observe that the listed plugins are now filtered to include only those that can be applied to a data source.

Change the sort order to be reverse alphabetic by selecting name Z-A from the order by menu.

Log in to the search dashboard where you are doing your training.

Run a search against the silent films search results page for buster and observe that the text for each result link ends with : Free Download & Streaming : Internet Archive.

Edit the template for the silent films results page navigating to the silent films search results page, selecting edit results page templates from the template panel.

Locate the results block in the template, and specifically the variable printed inside the result link. In this case it is the ${s.result.title} variable.

Edit the ${s.result.title} variable to use the Freemarker replace function to remove the : Free Download & Streaming : Internet Archive from each title. Update the variable to ${s.result.title?replace(" : Free Download & Streaming : Internet Archive","")} then save and publish the template.

Repeat the search against the silent films search results page for buster and observe that : Free Download & Streaming : Internet Archive no longer appears at the end of each link. If the text is still appearing check your replace parameter carefully and make sure you haven’t got any incorrect quotes or dashes as the replace matches characters exactly.

The function calls can also be chained - for example adding ?upper_case to the end of ${s.result.title?replace(" : Free Download & Streaming : Internet Archive","")?upper_case} will remove : Free Download & Streaming : Internet Archive and then uppercase the remaining title text.

View the data model for the search results (update your URL to be search.json instead of search.html) and observe that the titles still show the text that you removed. This demonstrates that the change you have applied above changes the text when the template generates the HTML, but the underlying data model is unaffected.

Undo the change you made to your template (to replace the end of the title) then save your template.

Log in to the search dashboard where you are doing your training.

Navigate to the manage results page screen for the silent films search results page.

Run a search against the silent films search results page for buster and observe that the text for each result link ends with : Free Download & Streaming : Internet Archive. This is the same as you saw in the previous exercise. We’ll now configure a plugin to remove the text.

Navigate back to the search dashboard, then click the plugins item in the navigation. This opens up the plugins management screen.

The extensions screen lists all the available plugins.

Locate the clean title plugin in the list. The tile presents some background information on the plugin, and provides a link to the documentation for the plugin

Click the learn more about plugin link. This opens the plugin documentation in a new browser tab. Have a quick read of the documentation then switch back to the tab containing the configuration screen. Don’t close the documentation tab as you’ll want to refer to this to configure the plugin.

Click on the plugin tile. This opens the wizard which guides you through the steps required to configure your plugin. The view documentation button can be used to access the plugin documentation if you need more information.

The location step is where you choose where to enable the plugin. If the plugin works on both data sources and results pages you will need to make a choice where you want the plugin enabled by selecting the appropriate radio button. You are then presented with a select list, where you select the results page or data source where you would like to enable the plugin. In this exercise we will enable the plugin on the results page. Select the results page radio button, then select Silent films search from the list.

In the configuration keys step you are presented with the various fields that you can configure for the plugin you’ve selected..

Add a new parameter with parameter 1 set to title and the value set to : Free Download & Streaming : Internet Archive, then press the proceed button. This configures the plugin with a rule called title to remove the string of text. The plugin supports multiple clean title rules. You define these in the same way, but with a different identifier (parameter 1).

You will be presented with a summary outlining what you’ve configured for the plugin. Review the summary then click the finish button to commit the plugin configuration to your results page.

You will be directed to the results page management screen, where you can test out your plugin before you publish the changes. Run a search for buster (against the preview version of the results page) and observe that the titles have been cleaned.

Run the same search (against the live version of the results page) and observe that the titles still display text that you configured the plugin to clean. This is because the plugin configuration hasn’t been published.

Return to the results page management screen and publish the plugin. Scroll down to the plugins heading then select publish from the menu for the clean title plugin. Observe that the menu also provides options to disable, delete and unpublish your plugin.

Rerun the search (against the live version of the results page) and observe that the titles are now cleaned.

Run a search for shakespeare against the book finder results page. Observe the home page results result (The Complete Works of William Shakespeare, URL: https://docs.squiz.net/training-resources/shakespeare/) is returned in the search results.

Open the search dashboard and switch to the shakespeare data source.

Click the manage data source configuration files item from the settings panel, then create a new kill_exact.cfg. This configuration file removes URLs that exactly match what is listed within the file.

The editor window will load for creating a new kill_exact.cfg file. The format of this file is one URL per line. If you don’t include a protocol then http is assumed. Add the following to kill_exact.cfg then save the file:

https://docs.squiz.net/training-resources/shakespeare/

As we are making a change that affects the makeup of the index we will need to rebuild the index. Rebuild the index by running an advanced update to re-index the live view.

Repeat the search for shakespeare and observe that the Complete Works of William Shakespeare result no longer appears in the results.

Run a search for hamlet and observe that a number of results are returned, corresponding to different sections of the play. We will now define a kill pattern to remove all of the hamlet items. Observe that all the items we want to kill start with a common URL base.

Return to the search dashboard and create a new kill_partial.cfg.

Add the following to the file and then save it:

https://docs.squiz.net/training-resources/shakespeare/hamlet/

Rebuild the index then repeat the search for hamlet. Observe that all the results relating to chapters of Hamlet are now purged from the search results.

Decide what fields you want to return as the columns of your CSV. Any of the fields with the search result part of the data model can be returned. For the airport finder results page we decide to return only the following fields, which are all sourced from the indexed metadata: "Name","City","Country","IATA/FAA","ICAO","Altitude"

A URL will need to be constructed that defines the fields that are required. Several parameters will be required to set the fields and field names to use:

Decide what fields need to be returned. For the airport finder results page we will return six columns: Name, City, Country, IATA/FAA, ICAO and Altitude. Open the search dashboard, change to the airports data data source and view the metadata mappings. Make a note of the metadata class names for these fields.

Define the query string parameters. To return the six fields will require the following parameters:

Construct a URL from this information and view the response from the all results JSON endpoint. Note that you need to URL encode your parameter values. This illustrates the custom JSON that can be returned using the all results endpoint:

Access the corresponding all results CSV endpoint. When you click the link a CSV file should be downloaded to your computer. Open this file after you’ve downloaded it.

The file will have been saved with a file name similar to all-results.csv. The file name can be defined by adding an additional parameter, fileName, to the URL. Modify the previous URL and add &fileName=airports.csv to the URL then press enter. The file should download and this time be saved as airports.csv.

Open the search dashboard and change to the airport finder results page.

Edit the default template (select the simple.ftl from the edit results page templates listing on the template panel). Add a button above the search results that allows the results to be downloaded as CSV. Add the following code immediately before the <ol> tag with an ID of search-results (approx. line 505):

<p><a class="btn btn-success" href="/s/all-results.csv?collection=${question.collection.id}&query=${question.query}&profile=${question.profile}&fields=listMetadata/name,listMetadata/city,listMetadata/country,listMetadata/iataFaa,listMetadata/icao,listMetadata/altitude&fieldnames=Name,City,Country,IATA/FAA,ICAO,Altitude&SF=[name,city,country,iataFaa,icao,altitude]&SM=meta&fileName=airports.csv">Download results as CSV</a></p>

Run a search for denmark and observe that a download results as CSV button appears above the 28 search results.

Click the download as csv button and the csv file will download. Open the CSV file and confirm that all the results matching the query are included in the CSV data (matching the 28 results specified by the results summary).

Log in to the search dashboard where you are doing your training.

Switch to the Library search package management page.

Select the edit search package configuration from the settings panel, then add the following configuration options.

Run a search on the shakespeare search results page for taming of the shrew and observe the JSON or XML response now includes data beneath the extraSearches node. Within this is an element corresponding to each extra search that is run, and beneath that is a question and response element which mirrors that of the main search.

Make use of the extra search results within your template by adding an <@ExtraResults> block. The code nested within the ExtraResults block is identical to what you would use in your standard results, but the code is drawn from the extra search item within the data model. E.g. Print the title for each extra search result. Edit the default template (select edit results page templates from the template panel) and insert the following immediately after the </@s.ContextualNavigation> tag (approx. line 644) then save the template.

<@fb.ExtraResults name="silent-films">
<div class="well">
  <h3>Related videos</h3>
  <div class="row">
  <@s.Results>
    <#if s.result.class.simpleName != "TierBar">
    <div class="col-md-3">
      <a href="${s.result.liveUrl}">
        <#if s.result.listMetadata["image"]??>
            <div class="row"><img class="img-thumbnail pull-left" style="margin-right:0.5em;" src="${s.result.listMetadata["image"]?first!}" alt="${s.result.listMetadata["image"]?first!}" title="${s.result.listMetadata["image"]?first!}"/></div><div class="row" style="margin-right: 2em;"><strong>${s.result.listMetadata["t"]?first!}</strong></div>
        <#else>
            <strong>${s.result.listMetadata["t"]?first!}</strong>
        </#if>
    </a>
    </div>
    </#if>
  </@s.Results>
  </div>
 </div>
</@fb.ExtraResults>

Rerun the search and this time view the HTML results, observing the related videos appearing below the related searches panel.

Run a search for twelfth night and confirm that the related videos are related to the search for dashboard.

Run a search against the foodista results page for sugar. Observe the order of search results.

Provide maximum influence to document recency by setting cool.5=1.0. This can be added as a CGI parameter, or in the query processor options of the results page configuration. Add &cool.5=1.0 to the URL and observe the change in result ordering. Observe that despite the order changing to increase the influence of date, the results are not returned sorted by date (because there are still other factors that influence the ranking).

Open the results page management screen for the foodista results page.

Select edit results page configuration from the customize panel. Locate the query processor options setting and set -cool.5=1.0 to the query processor options, then save and publish the setting.

Rerun the first search against the foodista results page and observe that the result ordering reflects the search that was run when recency was up-weighted. This is because the cool.5 setting has been set as a default for the results page. It can still be overridden by setting cool.5 in the URL string, but will be set to 1.0 when it’s not specified elsewhere.

From the search dashboard open the library search package management screen.

Click the edit search package configuration option from the settings panel.

In this example we’ll configure our library search package to push results from the Jane Austen data source up in the rankings, and push down the Shakespeare results. Add (and save, but don’t publish) two meta.components.*.weight settings for the two different data sources:

Edit the URL and add the following parameter: &cool.21=1.0 - this sets the maximum influence that can be provided by the data source relative weightings you’ve configured. Observe the effect on the search results order.

Adjust the influence by changing the value of the cool.21 parameter and observe the effect on the results. When you’ve found a value that you’re happy with you can apply this as the default for your results page by setting it in the results page configuration.

Set the influence as a results page default by adding -cool.21=0.9 to the query_processor_options in the results page configuration.

Rerun the query, removing the cool.21 value from the URL and observe that your default setting is now applied.

Switch to the foodista results page and perform a search for pork (http://localhost:9080/s/search.html?collection=training~sp-foodista&query=pork).

Change the settings so that the maximum up-weight is provided if the search query terms appear within the tags field. Remove the influence provided by the title metadata. Set the query processor options to the following (other query processor options may have been set for previous exercises so remove them and add the following):

-stem=2 -SF=[image,title,c,originalUrl] -sco=2[k,K,t,tags] -wmeta.tags=1.0 -wmeta.t=0.0

Log in to the search dashboard and switch to the simpsons - website data source.

Click the manage data source configuration files item from the settings panel and create a qie.cfg file

Add the following URL weightings to the qie.cfg and then save the file: 0.25 provides a moderate down-weight, while 1.0 is the maximum up-weight that can be provided via QIE. Items default to having a weight of 0.5.

# down-weight reviews and upweight episodes
0.25 www.simpsoncrazy.com/reviews/
1.0  www.simpsoncrazy.com/episodes/

Re-index the data source ( From the update panel: start advanced update  re-index live view.

Switch to the simpsons results page, then run a query for homer adding cool.4=0 and then cool.4=1.0 to the URL to observe the effect of QIE when it has no influence and when it has the maximum influence. Applying the maximum influence from QIE pushes episode pages to the top of the results (and this is despite the default same site suppression being applied).

Like other ranking settings you can set this in the results page configuration once you have found an appropriate influence to apply to your QIE.