Help Center

Menu

Ignore canonical URLs in web pages

This article outlines the steps required to ignore canonical URLs that are embedded in web pages.

Details

Canonical URLs are great when used correctly - they allow a web administrator to define the URL that should be used to reference the page (especially when there may be lots of aliases that all resolve to the same page). By default, Funnelback will always use a canonical link if it is defined, replacing the URL of the page that was crawled with whatever was defined in the canonical link tag.

However, it is fairly common for the canonical link tag to be used incorrectly, with an incorrect URL being placed within the tag.

For example if every page on a website had a canonical link that was the site’s homepage then Funnelback would only end up with a single document in the index as all the other pages would be marked as duplicates.

Always try to get the website owner to fix the canonical URL at the source before ignoring the links.

The easiest method is to set an indexer option that instructs the indexer to ignore the canonical link references.

  1. Add the following to the indexer_options in collection.cfg

    indexer_options= -ignore_link_rel_canonical

  2. Update or reindex the web data source

If you need to selectively ignore canonical URLs a custom plugin will be required to remove the canonical URL tags from the specific pages.