AI crawlers · ChatGPT-User
ChatGPT-User: OpenAI's live AI user fetch
ChatGPT-User is OpenAI's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real ChatGPT-User traffic.
↓ Markdown for agents- Vendor
- OpenAI
- Type
- Live AI user fetch
- Runs JavaScript
- No
- IP ranges published
- Yes
ChatGPT-User fetches a page live when a ChatGPT user's question needs it, for example when the user pastes a link or the assistant decides to read a source. A visit from it is a strong signal that your page is being used in a real answer.
Should you allow ChatGPT-User?
Allow it. ChatGPT-User acts for a person who is asking about your page right now. Blocking it means the assistant answers without your page, often citing someone else instead.
robots.txt
To allow ChatGPT-User everywhere:
User-agent: ChatGPT-User
Allow: /To block it:
User-agent: ChatGPT-User
Disallow: /Bots follow the most specific User-agent group that matches them. If you have a User-agent: ChatGPT-User group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.
Identifying ChatGPT-User
ChatGPT-User identifies itself with ChatGPT-User in its User-Agent header. It does not run JavaScript: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.
OpenAI publishes the IP ranges ChatGPT-User crawls from at openai.com/chatgpt-user.json. Anyone can fake a user agent, so a hit claiming to be ChatGPT-User from outside those ranges is spoofed. LLMentions' bots_report checks this for you.
Check your site
llmentions audit example.com # is ChatGPT-User allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log # did ChatGPT-User really visit, and is it genuine?Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=ChatGPT-User.
Other OpenAI bots
- GPTBot: AI training crawler
- OAI-SearchBot: AI search indexer