Help Center

Menu

Metadata is missing or truncated in search results

This article shows how to investigate the search results that have either missing or truncated metadata.

Search results are missing metadata

Empty or missing metadata in the search results is quite a common problem that can have a number of causes:

  • Funnelback hasn’t been configured to return the desired metadata classes by setting appropriate summary fields (-SF) and summary mode (-SM) query processor options.

  • The template is not configured to display the correct metadata classes.

  • The data source hasn’t been correctly configured to map the metadata fields to the metadata classes.

  • The metadata is not present in the source documents.

  • Metadata that is applied via external metadata is not correctly specified.

  • Any filters that generate/extract metadata are not working correctly.

Don’t forget that metadata classes are case-sensitive so check that the case of the metadata class used exactly matches what has been configured in the metadata mappings configuration.

Investigate missing metadata

This tutorial covers the different things to check when debugging missing metadata.

  1. Run a search that returns results that have metadata that is expected but missing.

  2. View the data model for the search and locate the result with the missing metadata. Check to see if the metadata is listed in the listMetadata element of the search result. If the metadata is listed and showing in the data model but not showing to the end user then the template is mis-configured.

  3. If the metadata is missing from the data model alter the data model URL to force the metadata summary mode and to return the fields that you are interested in (&SM=meta&SF=[<METADATA-FIELDS>]). Update <METADATA-FIELDS> to be a comma separated list of fields you wish to see. Again inspect listMetadata element. If the items are now present then you need to update the query processor options to set the SF value to add the missing metadata fields, and possibly also set an SM value (metadata should be returned by default for most queries unless SM is set to override the default settings). Remember, these need to be set on the results page that is accepting the queries.

  4. If a missing query processor option isn’t the cause then we need to look at the index itself. The first thing to check is the metadata mappings on the data source to verify that the metadata fields are mapped to the correct classes. If the metadata mapping is missing add the mapping then re-index the data source and this should fix the issue.

  5. If the metadata mappings are present then you’ll need to check the source data to see what Funnelback is indexing. The next step will depend on where the metadata comes from.

    1. Look at the cached version of the page and check the embedded metadata - this will show you what Funnelback indexed and will include metadata that were present in the source document and anything that was injected via filters. Also check the HTTP headers at the bottom of the cached copy.

    2. Look at the live version the page and check the embedded metadata - this will show you what is currently embedded in the page. Note: this could have changed since Funnelback indexed it, but it’s worth checking here and then comparing to the cached version of the page.

    3. Check external_metadata.cfg for the data source to see if the metadata is attached to the page this way. If external metadata is being used to add the metadata then there may be an error in the external metadata file (such as multiple matching lines), or the URL may not match correctly. Check the Step-Index.log and locate the section where the external metadata rules are loaded. If there are problems with the external metadata file then messages will be printed to the log.

    4. If the missing metadata is injected via a filter then check the filter logs for any messages relating to filtering - if the filter had any syntax errors it won’t have been loaded at all. You may also need to modify the filter to print out more information if there’s an error in the logic implemented in the filter.

Metadata is truncated in search results

Sometimes you will see metadata presented in the search results that is cut off or truncated. There are a number of different causes for this:

  • The template is truncating the metadata when it is displayed (in the Freemarker using something like the <@s.Cut> macro or via on of the Freemarker built-in functions).

  • The metadata buffer is too small causing the metadata to be truncated when it is returned from the index.

  • The maximum field length for indexed metadata is too small (controlled by the -mdsfml indexer option).

Investigate truncated metadata

This tutorial goes through the steps required to debug truncated metadata.

  1. Run a search that results in metadata getting truncated.

  2. View the data model for the search and locate the result with the truncated metadata.

  3. Check to see if the metadata is truncated in the data model - if it is complete when viewing the data model but truncated in the search results that you viewed then check the search result template as this will be truncating the metadata when it’s displayed (e.g. with a <@s.Cut> or similar).

  4. If the metadata is truncated in the data model as well, the next thing to try is to increase the size of the metadata buffer. The metadata buffer size usually only needs to be altered when you have a large amount of metadata for a results page. Add the following to the URL for the data model results &MBL=10000 and press enter. Recheck the metadata field to see if it is still truncated. If the metadata field is no longer truncated then you will need to increase the size of the metadata buffer in the query processor options for the search (either in the results page or search package configuration. Experiment with smaller values of MBL to find an appropriate value to set as large values will increase the memory required to run the search). Once an appropriate value is determined set the buffer size using the -MBL query processor option.

  5. If the metadata is still truncated even with a large metadata buffer then it is most likely that the truncation has occurred at index time (or it is truncated in the source data).

  6. Try increasing the metadata field length indexer option (-mdsfml) on the data source that contains the result. After you change the -mdsfml value you will need to rebuild the live index of the data source. The Step-Index.log will indicate if metadata has been truncated.

  7. After the re-indexing is complete recheck the searches - both at the data model level, then what is returned to the user. The metadata field should now be complete (or if it is still truncated then re-run through the steps above trying larger values of the settings).

See also