Google Can’t Access Your Website? Here’s What’s Changed in Google Merchant Center

Today I want to talk about something that Barry Schwartz picked up on recently, Google has made some improvements to how it handles situations where Googlebot can’t access your website for Google Merchant Center. Barry shared a post with all the changes nicely screenshotted, so definitely go and follow him and check out his site, seroundtable.com, because there’s a lot of great information over there.

In this article I’m not going to go through every single change, go check Barry’s post for that. What I want to do is pull out the parts that actually matter for Google Merchant Center merchants, based on what I’ve seen working with clients over the years.

1. Get Your Robots.txt Right

This is something I’ve been telling clients for years, and it’s now been made explicit: you need to add specific user agents to your robots.txt file that explicitly allow Googlebot and Googlebot-Image.

So you’d have:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-Image
Disallow:

There’s one that’s missing from the official documentation that I always recommend adding as well, Storebot-Google. Not sure why it’s left out, but I’d add it in exactly the same way.

Now, if you want to block something specific, like your admin area (or wp-admin if you’re on WooCommerce), you’d do:

User-agent: Googlebot
Disallow: /admin
Allow:

I know this trips people up, why would you put “Allow:” with nothing after it, right after a “Disallow”? I get it, it confused me too when I first started working with it. But the way to think about it is: the “Disallow: /admin” line is blocking that one specific folder, and the “Allow:” line (empty) is saying “allow everything else on the site.” You’re not allowing nothing, you’re allowing the rest of the site by default. If you’d rather follow the documentation exactly and skip the “Allow” shorthand, that’s fine too, it’s up to you and how you want to manage what gets blocked.

There is a great guide for adding robots.txt here: https://merchantcodex.com/course/website-optimization-for-google-shopping/

2. Don’t Fully Trust the “It’s an Easy Fix” Messaging

The documentation says that when you get a 404 or 500 error in Google Merchant Center, there’s a clear way to get that reprocessed. In my experience, that’s not 100% accurate. Just last week, I had a case where Google would not recrawl a particular website no matter what we tried. I genuinely don’t know why, but the point is, don’t assume it’ll always go smoothly just because the documentation says so.

3. Reprocessing Your Feed, Step by Step

Here’s what to actually do:

1. Go to Google Merchant Center
2. Scroll down to Data sources
3. Find your primary feed and click Update
4. Do the same for any supplemental sources

If you’re using the Merchant API, you won’t see an Update button. In that case, you need to contact whoever manages your feed/app integration and ask them to push the data through to Merchant Center again. Who exactly you contact depends on your feed solution, so reach out to their support team.

4. Triggering a Recrawl

Once you’ve made your fixes, go to Diagnostics and use the Request Website Check option. I actually made a full video walking through this exact process, how to fix “Product page unavailable” errors and 500 internal errors in Google Merchant Center. It’s only about 2 minutes long, so worth a watch if you want to see it step by step.

Quick heads up though, I’ve had at least one account where the “Request a website check” option simply wasn’t available. In that case, it took close to 2 days for around 1,000 products to get reprocessed, and there was nothing I could do to speed it up. That’s just how it works internally on Google’s end sometimes.

5. How Long Should You Actually Expect to Wait?

The good news: Google is now more specific about timing. It used to say “a couple of days”, now it states 12 to 48 hours for the system to reflect changes once accessibility is restored.

But from real experience, this varies a lot. I’ve had accounts where clicking “fetch now” resolved the issue in under 30 minutes. I’ve had other clients with around 1,000 disapproved products (also due to server issues) where it took noticeably longer, because the products couldn’t all be processed at once. So take the 12–48 hour window as a general guide, not a guarantee.