Hi Iwan,

Google isn't the only bot visiting your site anymore. ChatGPT, Claude, and Perplexity all send their own bots to crawl your pages before they'll ever cite or recommend you. If those bots are getting blocked, and on many sites they are, you're invisible to AI search before anyone even judges your content.

If AI tools can't see your content, they can't recommend it, no matter how good it is. Most site owners have never checked whether that's already happening to them.

Here is how to check and fix it:

1. Audit your robots.txt for AI-specific blocks

Many CMS platforms and security plugins block AI crawlers by default, without ever telling you.

  • Check your robots.txt for explicit disallow rules on GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.

  • If you don't recognize a rule blocking one of these, it likely came from a plugin update, not a deliberate choice.

  • Test your key pages directly against each bot's user agent using a tool like Screaming Frog's custom user agent feature.

2. Separate your AI-visibility strategy from your Google strategy

Blocking Google-Extended stops your content from training future Gemini models, but does not stop citations in AI Overviews, which pull from the live index.

  • Decide deliberately whether you want your content used for AI model training versus AI search citations. These are different toggles.

  • If you want citations but not training data usage, block Google-Extended specifically and leave Googlebot untouched.

  • Document this decision so it doesn't get silently reversed by a future plugin or developer.

3. Check your CDN and firewall settings, not just robots.txt

Robots.txt is a suggestion. Firewalls and bot-protection services enforce actual blocks that never show up in a standard SEO audit.

  • Review your CDN or WAF (Cloudflare, Sucuri, etc.) for bot-management rules that silently block AI user agents.

  • Check for JavaScript challenges or CAPTCHA walls that stop non-browser crawlers, including AI bots, from rendering your page.

  • Whitelist verified AI crawler IP ranges the same way you'd whitelist Googlebot's.

4. Recheck after every major platform update

A single WordPress security plugin update can silently add new bot blocks overnight.

  • Set a recurring quarterly check of your robots.txt and firewall bot rules.

  • Compare your current file against a saved baseline to catch unauthorized changes fast.

  • Treat AI crawler access like a core technical SEO metric, not a one-time setup task.

Click here to get a free audit of your site's AI crawler access.

To your continued success,

Matt Diggity