GEO
Google-Extended vs Googlebot: Why They're Not the Same
Published 2026-09-28 · Updated 2026-10-21
Googlebot and Google-Extended are two different user-agent tokens Google reads in robots.txt: Googlebot controls whether your site shows up in Google Search, and Google-Extended controls whether your content is used for Gemini training and grounding. Disallowing one has no effect on the other.
What does Googlebot control?
Googlebot is the crawler behind ordinary Google Search. If Googlebot can fetch a page, that page becomes eligible for indexing and ranking in regular search results. Disallowing Googlebot in robots.txt — or a site-wide Disallow: / under User-agent: * that catches it too — removes a site from Google Search entirely.
What does Google-Extended control instead?
Google-Extended is a separate, narrower token. It doesn't affect Search indexing at all — it only controls whether Google can use a site's content to train Gemini models and for grounding in Gemini Apps and Vertex AI. A site can disallow Google-Extended while Googlebot keeps crawling and ranking it normally.
Why do people mix these up?
Both tokens start with "Google," both live in the same robots.txt file, and both relate to "Google using my content." The mistake usually goes one of two directions:
- Blocking Googlebot thinking it only stops AI training — this actually removes the site from Google Search results entirely, a far bigger consequence than intended.
- Leaving Google-Extended allowed by default, unaware it exists — the site's content keeps feeding Gemini training and grounding even if the owner would have preferred to opt out.
How should you configure both, if you want AI visibility?
For a site that wants maximum AI visibility — being eligible for AI Overviews (which depends on Googlebot and Search indexing) and being usable as grounding context for Gemini — the answer is to leave both allowed. For a site that wants Search traffic but not to feed AI training, the split configuration looks like this:
| Goal | Googlebot | Google-Extended |
|---|---|---|
| Rank in Google Search, be eligible for AI Overviews, allow Gemini training | Allow | Allow |
| Rank in Google Search, opt out of AI training | Allow | Disallow |
| Opt out of Google entirely | Disallow | Irrelevant — already excluded |
Check which of these two tokens your current robots.txt actually sets with the free AI Crawler Checker — most sites have never explicitly set Google-Extended at all, which defaults to allowed.
What we measured: Google-Extended in the wild
Of 62 well-known sites with a readable robots.txt, 12 block Google-Extended (nytimes.com, bbc.com, amazon.com and yelp.com among them) and none block Googlebot site-wide. Four of the 12 block Google-Extended but not GPTBot, so the two decisions are made independently. Details are in our 70-site crawler study.
Sources
Official documentation this post relies on (links checked 2026-10-21).
Frequently Asked Questions
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls whether Google can use your content for Gemini model training and for grounding in Gemini Apps and Vertex AI. Googlebot, which controls Search indexing and ranking, is a completely separate user-agent and is unaffected.
Does blocking Googlebot stop me from appearing in AI Overviews?
Yes, indirectly. Google says a page must be indexed and eligible to appear in Search with a snippet to be shown as a supporting link in AI Overviews, so blocking Googlebot closes that path. Google-Extended is documented as a control for Gemini training and grounding, not for AI Overviews.
Can I allow Search but opt out of AI training?
Yes. Allow Googlebot as normal and add a separate Disallow rule for Google-Extended. This keeps your pages indexable and rankable in Google Search while excluding them from the training and grounding data used for Gemini.
Is Google-Extended the same thing as an AI crawler block?
It's one specific AI-use control among several — GPTBot, ClaudeBot, and PerplexityBot each have their own separate user-agent tokens, so blocking Google-Extended has no effect on those other companies' crawlers.
Related Free Tools
Related Posts
GPTBot vs OAI-SearchBot: What Each One Actually Does
llms.txt in 2026: Does It Actually Do Anything for SEO?