We were blocking ChatGPT.

We make software that checks whether AI tools like ChatGPT, Gemini, and Perplexity recommend your brand. This week we pointed it at our own website.

It failed the audit. Not on a technicality. Our site was refusing to talk to AI crawlers at all.

What we found

Requests from OAI-SearchBot, Claude-SearchBot, and PerplexityBot got HTTP 403 before they reached a single page. Those are answer-access crawlers, not the separate training-only bots. You can test this on your own site in one line:

curl -s -o /dev/null -w "%{http_code}" \
  -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4)" https://yoursite.com

Ours said 403. Cloudflare's separate bot block was denying answer-access requests before they reached the site. Its managed robots.txt also contained training and combined-use controls we had never chosen ourselves:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The robots rules opted out of several training crawlers, while the edge block stopped the answer-access crawlers too. Those are different choices and we had made neither deliberately.

How it happened

Nobody on our side decided to hide from AI. Cloudflare offers protection against AI crawlers, and it is very easy to have it on without noticing. For a publisher who sells content, blocking training crawlers can be a reasonable default. Blocking answer-access crawlers is a separate decision and can quietly remove a company from AI search and retrieval.

That is the uncomfortable part: the setting does not announce itself. Your site keeps working. Humans see nothing. You only find out if you fetch your site the way an AI crawler does.

Why this matters

AI tools recommend brands they can read. If a crawler gets a 403, your site contributes nothing to the answer. Whoever gets quoted instead is a competitor whose site loads fine. There is no error message and no dashboard warning you about the customers who asked an AI and got someone else's name.

What we changed

The fix took an afternoon. In order:

• Turned off the AI crawler blocking in Cloudflare and removed the managed robots.txt.
• Published our own robots.txt that names the AI crawlers and welcomes them, including training crawlers. We want to be in the models, not just in their search results.
• Added a sitemap, canonicals, and llms.txt so crawlers know what exists and what it means.
• Added Organization and product structured data. We had been recommending schema to customers while our own site had none.

Our agent readiness score went from failing during the block to 93 right after unblocking, and 100 once the structured data shipped.

Check your own site

Three things, five minutes:

• Run the curl above with OAI-SearchBot, Claude-SearchBot, and PerplexityBot user agents. Anything other than 200 needs explaining.
• Read yoursite.com/robots.txt and look for AI crawler names you did not put there.
• If you are behind Cloudflare or another CDN, find the AI bot setting and make sure it says what you actually want.

Or run the audit we run on ourselves. The first one is free, and it checks all of this plus whether AI tools actually recommend you today.