AI crawlers · GPTBot
GPTBot: OpenAI's AI training crawler
GPTBot is OpenAI's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real GPTBot traffic.
↓ Markdown for agents- Vendor
- OpenAI
- Type
- AI training crawler
- Runs JavaScript
- No
- IP ranges published
- Yes
GPTBot is OpenAI's crawler for collecting training data for OpenAI's models. It reads public pages so future models know about them. It does not fetch pages to answer a specific question.
Should you allow GPTBot?
Your choice. Blocking GPTBot keeps your content out of future training data. It does not remove you from AI search answers, which use separate bots (OAI-SearchBot and ChatGPT-User). Being in training data can still help models know your brand when they answer without searching.
robots.txt
To allow GPTBot everywhere:
User-agent: GPTBot
Allow: /To block it:
User-agent: GPTBot
Disallow: /Bots follow the most specific User-agent group that matches them. If you have a User-agent: GPTBot group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.
Identifying GPTBot
GPTBot identifies itself with GPTBot in its User-Agent header. It does not run JavaScript: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.
OpenAI publishes the IP ranges GPTBot crawls from at openai.com/gptbot.json. Anyone can fake a user agent, so a hit claiming to be GPTBot from outside those ranges is spoofed. LLMentions' bots_report checks this for you.
Check your site
llmentions audit example.com # is GPTBot allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log # did GPTBot really visit, and is it genuine?Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=GPTBot.
Other OpenAI bots
- OAI-SearchBot: AI search indexer
- ChatGPT-User: live AI user fetch