If you are receiving a crawl error from Google Merchant for both desktop and mobile devices, than I hope this article will help you trying to solve your issue.
Covering Error Messages :
- Product pages cannot be accessed
- Product pages cannot be accessed from a mobile device
Basic Steps to Undertake
First make sure that the links are effectively working.
Go to your diagnostics panel and click on the Items tab in the right bottom corner
Click on the digit in the column Effected Items next to one of the product pages cannot be accessed
Click on the title of the effected item, now a new tab is opened with the details of that product.
Click on the item URL
If you see that the page is not found, than you have added a wrong URL which you will need to fix.
If you have no problems please continue reading this article.
Check if your page returns a 200 code
Check if your pages are returning a 200 response code, this can easily be done with an online tool named HTTP Status Code Checker.
Too many redirects
With the above tool you can check how many redirects there are for a given link. It is recommended to not exceed 3 redirect. However I recommend that you do not submit redirected links. Simply submit the final destination link.
Check if your page has invalid characters
Cross reference the URL of the actually working page versus the link present in Google Merchant. It is possible that you have added a space in the URL known as (%20). It is recommended not to submit URL’s with spaces.
Check your robots.txt
Validate if your robots.txt file is allowing Google to crawl your website. The below will say disallow: which means you are allowing Google. (I know this is very confusing and I had to research myself several article to verify this is correct.) Find out at Google’s original documentation.
User-agent: Googlebot
Disallow:
User-agent: AdsBot-Google
Disallow:
User-agent: Googlebot-Image
Disallow:
Check if your server is up and running at all times of the day.
Having a reliable server is very important, if you are a serious business owner than having a web hosting company that can guarantee that the server is up 99.9% of the time is a must. This means that the 0.01% is roughly 5 minutes of downtime per month.
I suggest that you don’t stop there, find a web hosting company that will refund you if the downtime is more than 0.01%.
Finally check your server logs
I will not cover how to check for this as moz.com has already written a great article on how to check your server logs for Google Crawlers.