We were blocking ChatGPT.

We make software that checks whether AI tools like ChatGPT, Gemini, and Perplexity recommend your brand. This week we pointed it at our own website.

It failed the audit. Not on a technicality. Our site was refusing to talk to AI crawlers at all.

What we found

Requests from GPTBot, ClaudeBot, and PerplexityBot got HTTP 403 before they reached a single page. You can test this on your own site in one line:

curl -s -o /dev/null -w "%{http_code}" \
  -A "Mozilla/5.0 (compatible; GPTBot/1.2)" https://yoursite.com

Ours said 403. And our robots.txt, which we had never written, said this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Every major AI crawler, told to leave.

How it happened

Nobody on our side decided to hide from AI. Cloudflare offers protection against AI crawlers, and it is very easy to have it on without noticing. For a publisher who sells content, blocking AI training is a reasonable default. For a company that wants AI tools to know it exists and recommend it, it is quietly fatal.

That is the uncomfortable part: the setting does not announce itself. Your site keeps working. Humans see nothing. You only find out if you fetch your site the way an AI crawler does.

Why this matters

AI tools recommend brands they can read. If a crawler gets a 403, your site contributes nothing to the answer. Whoever gets quoted instead is a competitor whose site loads fine. There is no error message and no dashboard warning you about the customers who asked an AI and got someone else's name.

What we changed

The fix took an afternoon. In order:

• Turned off the AI crawler blocking in Cloudflare and removed the managed robots.txt.
• Published our own robots.txt that names the AI crawlers and welcomes them, including training crawlers. We want to be in the models, not just in their search results.
• Added a sitemap, canonicals, and llms.txt so crawlers know what exists and what it means.
• Added Organization and product structured data. We had been recommending schema to customers while our own site had none.

Our agent readiness score went from failing during the block to 93 right after unblocking, and 100 once the structured data shipped.

Check your own site

Three things, five minutes:

• Run the curl above with GPTBot, ClaudeBot, and PerplexityBot user agents. Anything other than 200 needs explaining.
• Read yoursite.com/robots.txt and look for AI crawler names you did not put there.
• If you are behind Cloudflare or another CDN, find the AI bot setting and make sure it says what you actually want.

Or run the audit we run on ourselves. The first one is free, and it checks all of this plus whether AI tools actually recommend you today.