Preparing your Matrix website for search
This section outlines the steps you should take to prepare your Squiz CMS website for search. The advice here applies to both integrating with DXP search and improving your site’s general SEO.
At a high level:
-
Ensure you have configured your
robots.txt
with appropriate rules for web robots, or are using appropriate robots meta tags within your content pages. If the search URL is part of the site’s domain then ensure this is disallowed in therobots.txt
to prevent search engines from indexing your search results pages.This will improve your search results by preventing the indexing of non-content pages. This will also potentially have positive impacts on both the time it takes to update the search index for your website, and also any bandwidth costs you might incur for your website. -
If your website has dynamically generated pages with varying URLs (like browse functions that might pass parameters to a script), or your website can be accessed using various URLs the use canonical URLs to set what the page URLs should be when indexing a particular page.
This will improve your search results by removing duplication in the search results. It will also potentially have similar positive impacts on the time taken to index and your bandwidth usage. -
If your website has content pages that are only designed to be accessed via a search function consider creating a
sitemap.xml
file that lists these unlinked pages and list the sitemap in yourrobots.txt
.
Prepare your site
-
Using robots.txt and robots meta tags to control what is indexed by search engines
Use
robots.txt
and web robots meta tags to control how web crawlers access your content. -
Controlling which parts of your pages are indexed by the DXP search
Use noindex tags to hide parts of your page from the DXP search indexer.
-
Use canonical URLs to avoid duplication in your search results, and to provide friendlier page URLs.
-
Provide alternatives for Javascript-generated content
Don’t prevent access to your website by requiring Javascript.