If you are receiving a crawl error from Google Merchant Center for both desktop and or mobile devices, than I hope this article will help you trying to solve your issue.
If you have a few random products that are disapproved due to unavailable landing page and the volume of disapprovals does not increase, then do not worry too much, as most likely the issue is temporaritly, and it will auto resolve itself in 3 days to a week. However I will provide you with checks you can perform.
Covering Error Messages :
- Unavailable desktop landing page
- Unavailable mobile landing page
The specific crawl errors can be:
- HTTP 5xx response
- HTTP 4xx response
- Page not found (404)
Steps to Undertake
First make sure that the links are effectively working.
Google Merchant Center Classic
- Go to Products
- Click on unavailable landing page for desktop or mobile
- Select one of your products that are listed
- Click on view on my website
Google Merchant Center Next
- Go to Products
- Needs Attention
- Filter based on what needs attention
- Click on the product example
- Click on the website URL
If you see that the page is not found, than you have added a wrong URL which you will need to fix.
If you have no problems please continue reading this article.
Ensure the feed has been refreshed
Double check that your data feed is still updating Google Merchant Center. Sometimes an outdated feed can cause these issues.
You can check when the last time it was updated by viewing your product information in Google Merchant Center and checking the status area.
Check if your page returns a 200 code
Check if your pages are returning a 200 response code; this can easily be done with an online tool named HTTP Status Code Checker.
Also, ensure that the landing page works for everyone worldwide and that you are not blocking any users.
HTTP 5xx Response
Usually this can be related to DNS or server issues. Double check if your server is up and working, and that the DNS settings are pointing to the right server.
HTTP 4xx Response and 404 errors
These two errors related to pages that no longer exist, for example you have removed the products from your website, or the URL is incorrectly written.
Too many redirects
With the above tool, you can check how many redirects there are for a given link. It is recommended to be at most three redirects. However, I recommend that you do not submit redirected links. Submit the final destination link.
Check that your website’s DNS settings are correct.
Check if your page has invalid characters
Cross-reference the actual working page URL versus the link in Google Merchant. You may have added a space in the URL (%20). It is recommended not to submit URL’s with spaces.
Check your robots.txt
Validate if your robots.txt file is allowing Google to crawl your website. Find out at Google’s original documentation.
Check if your server is up and running at all times of the day.
Having a reliable server is very important; if you are a serious business owner, having a web hosting company that can guarantee that the server is up 99.9% of the time is a must. This means that the 0.01% is roughly 5 minutes of monthly downtime.
You should not stop there; find a web hosting company that will refund you if the downtime exceeds 0.01%.
Finally check your server logs
I will not cover how to check for this as moz.com has already written a great article on how to check your server logs for Google Crawlers.