LLMentions
AI crawlers · Googlebot

Googlebot: Google's search engine crawler

Googlebot is Google's search engine crawler. What it does, whether to allow it, robots.txt rules and how to verify real Googlebot traffic.

↓ Markdown for agents
Vendor
Google
Type
Search engine crawler
Runs JavaScript
Yes
IP ranges published
Yes

Googlebot is Google's search crawler. It indexes pages for Google Search, including AI Overviews and AI Mode.

Should you allow Googlebot?

Allow it. Googlebot powers Google Search, including AI Overviews and AI Mode. Blocking it removes you from that search engine, including AI Overviews and AI Mode.

robots.txt

To allow Googlebot everywhere:

text
User-agent: Googlebot
Allow: /

To block it:

text
User-agent: Googlebot
Disallow: /

Bots follow the most specific User-agent group that matches them. If you have a User-agent: Googlebot group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.

Identifying Googlebot

Googlebot identifies itself with Googlebot in its User-Agent header. It renders JavaScript, but rendering is deferred and costly, so server-rendered content is still read faster and more reliably.

Google publishes the IP ranges Googlebot crawls from at developers.google.com/static/crawling/ipranges/common-crawlers.json. Anyone can fake a user agent, so a hit claiming to be Googlebot from outside those ranges is spoofed. LLMentions' bots_report checks this for you.

Check your site

terminal
llmentions audit example.com            # is Googlebot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log   # did Googlebot really visit, and is it genuine?

Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=Googlebot.