Step 3: Create a robots.txt file

A robots.txt file is an industry-standard file used to control a 'spider' or 'robot' crawler. You create a robots.txt file in your Matrix instance. It is used to prevent a crawler from accessing all or some of your websites.

Setting up a robots.txt file is covered in the Matrix documentation.

To find out how and where to create your robots.txt file, read Create a robots.txt file.