# Googlebot

Googlebot is Google's search crawler. It indexes pages for Google Search, including AI Overviews and AI Mode.

## Should you allow Googlebot?

**Allow it.** Googlebot powers Google Search, including AI Overviews and AI Mode. Blocking it removes you from that search engine, including AI Overviews and AI Mode.

## robots.txt

To allow Googlebot everywhere:

```text
User-agent: Googlebot
Allow: /
```

To block it:

```text
User-agent: Googlebot
Disallow: /
```

Bots follow the most specific `User-agent` group that matches them. If you have a `User-agent: Googlebot` group, the rules under `User-agent: *` no longer apply to it. Matching on the token is case-insensitive.

## Identifying Googlebot

Googlebot identifies itself with `Googlebot` in its `User-Agent` header. It renders JavaScript, but rendering is deferred and costly, so server-rendered content is still read faster and more reliably.

Google publishes the IP ranges Googlebot crawls from at [developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json). Anyone can fake a user agent, so a hit claiming to be Googlebot from outside those ranges is spoofed. LLMentions' `bots_report` checks this for you.

## Check your site

```bash
llmentions audit example.com            # is Googlebot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log   # did Googlebot really visit, and is it genuine?
```

Or identify any user agent over REST (free): `GET https://api.llmentions.com/v1/bots/lookup?ua=Googlebot`.

---
https://llmentions.com/bots/googlebot/
