A new updated course is available at https://merchantcodex.com/course/website-optimization-for-google-shopping/
Here is a guide on how to add the correct user-agents in your robots.txt file in Shopify, for the purpose of Google Merchant Center.
First login to Shopify and edit your theme code.
If you already have added a robots.txt or you see a robots.txt file in the template section, ignore the following step.
Add a robots.txt file by going to the template section and click on add a new template
At the top of the file hit enter so you have 1 clear line, and paste the following code below and click on save.
User-agent: Googlebot
Disallow: /admin/
Allow: /
User-agent: Googlebot-image
Disallow: /admin/
Allow: /
User-agent: Storebot-Google
Disallow: /admin/
Allow: /
Hello, how can I add a template with the new interface please ?
I will soon create a new video and update the guide.
hi, you’ve referred me to this after a consultation, but it’s outdated. just as a heads up. i’ll look to figure it out, but others might need it.
Hello Jordan, thank you for the update. Yes I am planning to update the guide, while the steps are nearly the same. There is an additional step and the UI change is confusing the current guidance. I will plan this in. Thank you Jordan.
what does the disallow mean? and could you also share how to use url inspection for any product page url?
In a robots.txt file, “Disallow” means to instruct web crawlers (also known as robots or spiders) not to access or crawl specific files, directories, or sections of a website.
For URL Instpection:
Go to Google Search Console
Select your website property.
Paste your product page URL into the URL Inspection Tool (top search bar).
Press Enter.
Google will return data like:
✅ “URL is on Google”
❌ “Blocked by robots.txt”
Expand the “Coverage” section to view crawl status and robots.txt rules.
Hi Emanuel,
Why is it necessary to add Robots.txt in Shopify for Google Merchant Center?
Thank you very much.
In case Google has trouble crawling your website.