If you encounter a warning saying images cannot be crawled because of robots.txt restrictions or it says you have to wait three days, this article will explain the issue in more detail.
Diagnostics Tab
First, let’s understand the diagnostics tab. When you see errors and warnings in the diagnostics tab, and you have fixed all issues. However, it can take up to 30 days for the problems to clear up. This is because Google’s Bot for Google Shopping does not crawl every page every day. You will notice that after two days, your issue count will drop lower and lower. So do not panic that you don’t see it fall within the next day, wait at least two days and check the original error count vs. the new one. If it’s dropping down, you have done an excellent job.
Understanding the message
When you receive a message related to the robots.txt, it does not mean it is related to this file. It can also mean that you have an error in the image URL or the image size is too large, which might cause your server not to respond quickly enough.
Single or multiple issues can trigger this warning.
Checklist
Let’s create a simple checklist for you to confirm that all settings on your website meet Google’s requirements.
- Make sure Google can crawl your images by allowing access in robots.txt. To know what to do, please follow Googles instructions.
- If you have additional images, as Google does not use them much at all at this current stage, is to remove them from your data feed
- Make sure your product covers at least 75% of the image size
- Optimize your images. Large images can cause delays in Google crawling or your server timing out. I recommend using compression tools to make your images lighter and also make sure the pixel size is not too large
- Check you have no violations on the images such as watermarks, borders, text, promotions, etc.
- You can check when Google last crawled your product by logging in to Google Merchant > Products > List and click on your product. Under technical details, you can see when it was last crawled.
If you have checked everything and there are no issues, wait for a few re-crawls, and Google will automatically resolve the issue.
Optional
You can force a recrawl quicker by changing the filename or appending a random query string.