If you encounter a warning saying images cannot be crawled because of robots.txt restrictions or it says you have to wait 3 days than this article will explain the issue in more detail.
Diagnostics Tab
First lets understand the diagnostics tab, when you see errors and warnings in the diagnostics tab and you have fixed all issues. It can take up to 30 days for the issues to clear up. This is because Google’s Bot for Google Shopping does not crawl every page every day. You will notice that after 2 days your issue count will drop lower and lower. So do not panic that you don’t see it drop within the next day, wait at least 2 days and check the original error count vs the new one. If it’s dropping down you have done a good job.
Understanding the message
When you receive a message related to the robots.txt, it does not mean it is related to this file. It can also mean that you have an error in the image url or the size of the image is too large. Which might cause your server not to respond quickly enough.
This warning can be triggered by a single or multiple issues.
Checklist
Lets create a simple checklist for you to confirm that all settings on your website meet Google’s requirements.
- Make sure Google can crawl your images by allowing access in robots.txt, to know what to do please follow Googles instructions
- If you have additional images, as Google does not use them mutch at all at this current stage, is to remove them from your data feed
- Make sure your product covers at least 75% of the image size
- Optimize your images, large images can cause delays in Google crawling or your server timing out. I recommend using compression tools to make your images lighter and also make sure the pixel size is not too large
- Check you have no violations on the images such as watermarks, borders, text, promotions, etc…
- You can check when Google last crawled your product by logging in to Google Merchant > Products > List and click on your product. Under technical details you can see when it was last crawled.
If you have checked everything and there are no issues wait for a couple of re-crawls and Google will automatically resolve the issue.
Optional
You can force a re crawl quicker by changing the filename or appending a random query string.