Should you block AI crawlers? Block the right one, or you vanish from ChatGPT

Somewhere in your hosting dashboard there may be a toggle that says something like “Block AI bots.” It sounds responsible. Protect your content, keep the robots out, done.

Flip the wrong version of it and you’ve just told ChatGPT you don’t want to be recommended. Nobody warns you about that part, and nothing on your site looks any different afterwards.

The short answer

For a business that sells a service, you almost always want AI search crawlers allowed. Blocking AI training crawlers is a reasonable choice and costs you very little. The mistake is treating “AI bots” as one thing. They aren’t. Different bots do different jobs, and you can block one without touching the others.

If you only take one thing from this post: training bots learn from your content, search bots send you customers.

Not all AI bots do the same job

OpenAI is the clearest example because it documents its crawlers separately, and states that “each setting is independent of the others”:

• GPTBot collects content that may be used to train future models. Block it and your pages shouldn’t end up in training data. It doesn’t affect whether you show up in ChatGPT search.
• OAI-SearchBot is what surfaces websites in ChatGPT’s search answers. Block it and, in OpenAI’s words, your site won’t be shown in those answers.
• ChatGPT-User visits a page when a real person asks ChatGPT something that needs it, live. It’s triggered by users, not crawling on a schedule, and OpenAI notes robots.txt rules may not apply to it.

Google works the same way with different names. Normal Googlebot handles Search. A separate token, Google-Extended, controls whether your content is used to train Gemini and to ground answers in the Gemini apps. Google is explicit that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.”

Other AI companies increasingly split their bots along the same training / search / user lines. Before blocking anything, read that company’s own bot documentation. It takes five minutes.

The cost of blocking the wrong one

Picture an HVAC company that read a scary headline about AI scraping and added a blanket block for every AI user agent it could find. Its Google rankings don’t move, so it assumes it got away clean.

Meanwhile, someone asks ChatGPT, “who services heat pumps near me and has good reviews?” ChatGPT goes looking. It’s been told to stay out of that site, so it recommends the three competitors it can read.

No warning shows up anywhere. There’s no Search Console report for answers you weren’t in. It’s the same silent failure as AI crawlers that can’t read your JavaScript, except you did it to yourself.

When blocking training actually makes sense

Here’s the honest part: blocking training crawlers is a legitimate business decision, not paranoia. It makes most sense if:

• Your content is the product. Publishers, course creators, paid research, original photography. If people pay for what’s on the page, there’s no reason to hand it to a model for free.
• You’ve invested heavily in original material and don’t want it reworded back at your competitors’ customers.
• You just don’t want to. That’s a valid answer too.

For a plumber, a dental clinic or a roofing company, the calculation is different. Your service pages exist to be found and quoted. The upside of blocking GPTBot is mostly symbolic, and the downside of accidentally blocking search bots alongside it is very real. We’d leave training alone or block it narrowly. Never broadly.

Check what your host already decided for you

This is the bit most owners miss: you may not have chosen any of this.

Cloudflare, which sits in front of a huge share of small-business sites, announced in July 2025 that it was changing its default to block AI crawlers. Its settings have kept changing since then. Per Cloudflare’s current documentation, new domains from September 15, 2026 get Training and Agent bots blocked on pages that display ads, while Search stays allowed. Bots that do both search and training get blocked by any setting that blocks training, including the old “Block AI bots” toggle.

That last part matters. If someone flipped that toggle on your site a year ago, it may now be catching more than it used to.

So check two places:

1. Your robots.txt. Open yoursite.com/robots.txt. Look for Disallow: / under user agents like GPTBot, OAI-SearchBot, Google-Extended, or a wildcard rule that shuts everything out.
2. Your CDN or hosting dashboard. Look for AI bot, crawler or bot-management settings. A robots.txt that welcomes everyone means nothing if the firewall in front of it is turning bots away.

What we recommend for most service businesses

A sensible starting point, in robots.txt terms:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

That keeps you in ChatGPT’s search answers while opting out of training. Swap the GPTBot rule to Allow if you’re happy to be in training data, which is a perfectly fine choice for most local businesses. Be more careful with Google-Extended: it also covers grounding in the Gemini apps, so blocking it is a trade-off, not a free win.

Then confirm it worked. Run the 30-minute ChatGPT check a few days later. OpenAI says robots.txt changes can take around 24 hours to show up in search. While you’re at it, an llms.txt file is a good way to tell the bots you do let in what your business actually does.

The bottom line

“Block AI” isn’t a privacy setting. It’s a visibility setting with a privacy label on it. Decide on training and search separately, write down what you chose, and check your host isn’t overruling you.

This is the kind of plumbing we check on every SEO engagement, because the most expensive SEO problem is the one that makes you invisible without telling you. If you’d like a second pair of eyes on your robots.txt and CDN settings, send us your URL. If everything’s already set up right, we’ll tell you that and you won’t owe us a thing.