Free Tool · No Sign-Up
Free AI Crawler Access Checker
Enter your domain. We read your robots.txt and tell you exactly which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more — are blocked from indexing your pages.
This checks 1 thing. HellSEO's full audit scores 18 items across SEO, AEO, and GEO — including backlink authority and score history this page doesn't cover.
See Full ScoreHow It Works
We fetch your site's robots.txt and parse every User-agent / Disallow group in it — the same rules search engines and AI crawlers read before touching your site.
For each known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others), we check whether that specific user-agent — or a wildcard * covering everyone — is blocked with a full Disallow: /.
This only reads your robots.txt; it doesn't crawl your pages or store anything. Free, and it runs in a couple of seconds.
Example Result
This robots.txt keeps AI training crawlers out while still letting an AI search crawler fetch pages. The checker reads rules like these and reports, bot by bot, what is allowed.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /GPTBot blocked
OAI-SearchBot allowed
ClaudeBot blocked
Google-Extended blocked
PerplexityBot allowed (no rule, so the default applies)Common mistakes
- A User-agent: * group with Disallow: / blocks every AI crawler that has no rule of its own.
- Blocking GPTBot does not block OAI-SearchBot. They are separate user-agent tokens with different jobs (OpenAI documents both).
- Blocking Googlebot removes a site from Google Search. Google-Extended is the separate control for Gemini training and grounding.
- robots.txt is a request that well-behaved crawlers follow, not access control. Private content needs authentication.
- Staging rules often leak into production. Re-check the live file after each deploy.
Who Uses This
SaaS & product sites
Confirm ChatGPT and Perplexity can actually read your docs and pricing pages before assuming AI traffic will find you.
Publishers & blogs
Decide deliberately whether to allow AI training crawlers (CCBot, Bytespider) instead of inheriting a default block from your CMS.
Agencies
Add this to a client onboarding checklist — a blocked crawler is a one-line fix that's easy to miss.
Frequently Asked Questions
What happens if I block an AI crawler?
That bot can no longer read your pages, so your content can't appear in that engine's answers — even if you rank #1 on Google. Many CMS default robots.txt files block these bots without you knowing.
Does blocking GPTBot also block ChatGPT search?
No. OpenAI uses separate user-agents — GPTBot for training data, OAI-SearchBot for live ChatGPT search results. You can block one and allow the other.
Will this affect my Google ranking?
No. Google's classic crawler (Googlebot) is separate from Google-Extended, which only controls Gemini training and grounding. Blocking Google-Extended doesn't affect normal search ranking.
How do I unblock a bot?
Remove or edit the matching User-agent / Disallow block in your robots.txt file, then re-check here to confirm.
Want the full picture?
This tool checks one thing. A full HellSEO audit scores your site across SEO, AEO, and GEO — and gives you ready-to-paste fixes.
Run a Full Audit