GEO
We Checked 70 Major Sites: Who Blocks AI Crawlers in 2026?
Published 2026-10-21
Everyone has an opinion about whether to block AI crawlers. We wanted data instead. On 2026-10-21 we read the robots.txt, llms.txt and homepage of 70 well-known sites and counted what they actually do.
How we measured
We picked 70 well-known sites across news, SaaS, e-commerce, reference, SEO tools, travel and developer infrastructure. For each one a script fetched /robots.txt, /llms.txt and the homepage HTML on 2026-10-21. 62 sites returned a readable robots.txt; 8 did not (cnn.com, walmart.com, homedepot.com, stackoverflow.com, adidas.com, udemy.com, khanacademy.org, arstechnica.com), usually because they refuse automated requests. Those 8 are excluded from the robots.txt counts below.
A crawler counts as blocked when the file has a group for that exact user-agent token with Disallow: /. Partial means that group disallows some paths. No rule means there is no group for that token, so the crawler falls back to the User-agent: * group, which we did not evaluate. That makes "blocked" a floor: a site could still block a bot through a wildcard rule.
Which AI crawlers do big sites block?
Counts out of the 62 sites with a readable robots.txt:
| Crawler | Blocked | Partial | Explicitly allowed | No rule |
|---|---|---|---|---|
| GPTBot | 15 | 8 | 3 | 36 |
| OAI-SearchBot | 6 | 11 | 1 | 44 |
| ChatGPT-User | 9 | 9 | 2 | 42 |
| ClaudeBot | 20 | 5 | 1 | 36 |
| PerplexityBot | 13 | 6 | 3 | 40 |
| Google-Extended | 12 | 5 | 2 | 43 |
| CCBot | 17 | 3 | 1 | 41 |
| Googlebot | 0 | 11 | 2 | 49 |
ClaudeBot (20 sites) and CCBot (17) are blocked most often. GPTBot is blocked by 15 sites. Googlebot is blocked site-wide by none, which is what you would expect: nobody on this list wants out of Google Search.
Training and search are treated as different decisions
9 sites block GPTBot but not OAI-SearchBot: techcrunch.com, canva.com, ebay.com, medium.com, healthline.com, webmd.com, airbnb.com, tripadvisor.com and linkedin.com. Only 6 sites block both. No site blocked OAI-SearchBot alone. That matches OpenAI's own split: GPTBot is for training, OAI-SearchBot is for surfacing sites in ChatGPT search (see GPTBot vs OAI-SearchBot).
On the Google side, 12 sites block Google-Extended: nytimes.com, bbc.com, bloomberg.com, techcrunch.com, amazon.com, investopedia.com, nerdwallet.com, tripadvisor.com, yelp.com, linkedin.com, wired.com and theverge.com. Four of them (investopedia.com, nerdwallet.com, wired.com, theverge.com) block Google-Extended without blocking GPTBot. See Google-Extended vs Googlebot for what that token does and does not control.
33 of the 62 sites (53%) have no AI-specific block for any of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended or CCBot. Open is still the majority position.
llms.txt adoption
19 of 70 sites (27%) served a real llms.txt: github.com, stripe.com, shopify.com, hubspot.com, atlassian.com, slack.com, dropbox.com, mailchimp.com, wix.com, squarespace.com, wordpress.com, etsy.com, target.com, semrush.com, yoast.com, vercel.com, netlify.com, cloudflare.com and coursera.org. None of the news publishers in the sample had one. We counted a file only if it returned HTTP 200, began with a Markdown heading and was not an HTML page. Whether any AI system reads these files is still unconfirmed (see llms.txt in 2026).
Question headings and FAQ schema on homepages
Of the homepages that had at least 4 H2/H3 headings in their HTML (41 sites), the median share of headings phrased as a question was 3%. 18 of the 41 had no question heading at all, and none reached 30%. Only 2 of 49 homepages we could read (squarespace.com and coursera.org) carried FAQPage structured data. Homepages are a weak test of this idea since the question format fits article and help pages better, and our script only reads server-rendered HTML. For what we know about the format itself, see Do Question-Based Headings Help AI Cite You?
What this does and does not tell you
- It shows what large brands decided, not whether blocking works or helps them.
- Sites with legal or licensing reasons to restrict training (publishers) behave differently from SaaS companies that want to be cited.
- The sample is hand-picked and lopsided toward large US-based brands. Small business sites were not measured.
- Wildcard rules, per-path rules and server-side bot blocking are not captured by this method.
- It is one snapshot, taken 2026-10-21. Robots.txt files change.
To check your own rules for these crawlers, use the AI Crawler Checker.
Sources
Official documentation this post relies on (links checked 2026-10-21).
Frequently Asked Questions
How many of the 70 sites block GPTBot?
15 of the 62 sites whose robots.txt we could read (24%) have a rule that disallows GPTBot for the whole site, and another 8 restrict it partially. 36 have no GPTBot-specific rule at all.
Do sites that block GPTBot also block OAI-SearchBot?
Mostly not. Of the 15 sites that block GPTBot, only 6 also block OAI-SearchBot. 9 block training but leave ChatGPT's search crawler alone. No site in the sample blocked OAI-SearchBot without also blocking GPTBot.
How common is llms.txt?
19 of the 70 sites (27%) served a file at /llms.txt. They were almost all developer-tool, SaaS, commerce and infrastructure companies. None of the news publishers we sampled had one. A file being present does not mean any AI system reads it.
Is this a statistically valid sample?
No. It is 70 well-known sites we picked by hand, not a random sample of the web, and it skews toward large US and global brands. Treat it as a snapshot of how big sites behave, not as a measurement of all sites.
Related Free Tools
Related Posts
GPTBot vs OAI-SearchBot: What Each One Actually Does
Google-Extended vs Googlebot: Why They're Not the Same
llms.txt in 2026: Does It Actually Do Anything for SEO?