Block a Bot From Spidering Website
Issue: You would like to block a particular bot from crawling/indexing your site.
Solution: Edit your store's Robots.txt file.
To begin:
Log into your Shift4Shop Online Store Manager.
Using the left navigation menu, go to Marketing > SEO Tools.
Locate and click on the link labeled Edit Robots.txt File.
This page will have two distinct areas. Within the top half of the page, you will see the Robots.txt section containing your store's regular robots.txt file. It should look like this:
Sitemap: http://[store-url]/sitemap.xml
# Disallow all crawlers access to certain pages.
User-agent: *
Disallow: /checkout.asp
Disallow: /add_cart.asp
Disallow: /view_cart.asp
Disallow: /error.asp
Disallow: /shipquote.asp
Disallow: /rssfeed.asp
Disallow: /mobile/
Most bots are given a user-agent name that can be used in the Robots.txt file to block them from accessing the site. Provided you know the bot's name in question, add the following to the Robots.txt file.
# Block [name] from crawling site
User-agent: [name]
Disallow: /
Replace the [name] above with the user-agent name of the offending bot.
Click "Save" at the top right to commit your changes.