# AI crawler directory

Every AI and search crawler that matters for AI visibility: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and more. What each does, whether to allow it, and robots.txt rules.

Which bots should you let in? It depends on what each one is for. **AI search indexers** and **live user fetches** decide whether AI answers can cite you, so blocking them removes you from answers. **Training crawlers** are a separate choice. None of the AI bots run JavaScript.

Check your own site with `llmentions audit example.com`, and see which bots really visit with `llmentions bots access.log`. Both are in the [site audit guide](/docs/site-audit/).

## Search

- [Googlebot](/bots/googlebot/): Googlebot is Google's search engine crawler. What it does, whether to allow it, robots.txt rules and how to verify real Googlebot traffic.
- [Bingbot](/bots/bingbot/): Bingbot is Microsoft's search engine crawler. What it does, whether to allow it, robots.txt rules and how to verify real Bingbot traffic.
- [Applebot](/bots/applebot/): Applebot is Apple's search engine crawler. What it does, whether to allow it, robots.txt rules and how to verify real Applebot traffic.

## AI search

- [OAI-SearchBot](/bots/oai-searchbot/): OAI-SearchBot is OpenAI's AI search indexer. What it does, whether to allow it, robots.txt rules and how to verify real OAI-SearchBot traffic.
- [Claude-SearchBot](/bots/claude-searchbot/): Claude-SearchBot is Anthropic's AI search indexer. What it does, whether to allow it, robots.txt rules and how to verify real Claude-SearchBot traffic.
- [PerplexityBot](/bots/perplexitybot/): PerplexityBot is Perplexity's AI search indexer. What it does, whether to allow it, robots.txt rules and how to verify real PerplexityBot traffic.

## User fetch

- [ChatGPT-User](/bots/chatgpt-user/): ChatGPT-User is OpenAI's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real ChatGPT-User traffic.
- [Claude-User](/bots/claude-user/): Claude-User is Anthropic's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real Claude-User traffic.
- [Perplexity-User](/bots/perplexity-user/): Perplexity-User is Perplexity's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real Perplexity-User traffic.
- [DuckAssistBot](/bots/duckassistbot/): DuckAssistBot is DuckDuckGo's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real DuckAssistBot traffic.
- [MistralAI-User](/bots/mistralai-user/): MistralAI-User is Mistral's live AI user fetch. What it does, whether to allow it, robots.txt rules and how to verify real MistralAI-User traffic.

## Training

- [GPTBot](/bots/gptbot/): GPTBot is OpenAI's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real GPTBot traffic.
- [ClaudeBot](/bots/claudebot/): ClaudeBot is Anthropic's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real ClaudeBot traffic.
- [Meta-ExternalAgent](/bots/meta-externalagent/): Meta-ExternalAgent is Meta's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real Meta-ExternalAgent traffic.
- [Amazonbot](/bots/amazonbot/): Amazonbot is Amazon's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real Amazonbot traffic.
- [Bytespider](/bots/bytespider/): Bytespider is ByteDance's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real Bytespider traffic.
- [CCBot](/bots/ccbot/): CCBot is Common Crawl's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real CCBot traffic.
