LLMentions
AI crawlers

AI crawler directory

Every AI and search crawler that matters for AI visibility: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more. What each does, whether to allow it, and robots.txt rules.

↓ Markdown for agents

Which bots should you let in? It depends on what each one is for. AI search indexers and live user fetches decide whether AI answers can cite you, so blocking them removes you from answers. Training crawlers are a separate choice. None of the AI bots run JavaScript.

Check your own site with llmentions audit example.com, and see which bots really visit with llmentions bots access.log. Both are in the site audit guide.

User fetch

Training