AI crawlers · Meta-ExternalAgent
Meta-ExternalAgent: Meta's AI training crawler
Meta-ExternalAgent is Meta's AI training crawler. What it does, whether to allow it, robots.txt rules and how to verify real Meta-ExternalAgent traffic.
↓ Markdown for agents- Vendor
- Meta
- Type
- AI training crawler
- Runs JavaScript
- No
- IP ranges published
- No
Meta-ExternalAgent is Meta's crawler for collecting training data for Meta's AI models. It reads public pages so future models know about them. It does not fetch pages to answer a specific question.
Should you allow Meta-ExternalAgent?
Your choice. Blocking Meta-ExternalAgent keeps your content out of future training data. It does not remove you from AI search answers, which use separate bots. Being in training data can still help models know your brand when they answer without searching.
robots.txt
To allow Meta-ExternalAgent everywhere:
User-agent: Meta-ExternalAgent
Allow: /To block it:
User-agent: Meta-ExternalAgent
Disallow: /Bots follow the most specific User-agent group that matches them. If you have a User-agent: Meta-ExternalAgent group, the rules under User-agent: * no longer apply to it. Matching on the token is case-insensitive.
Identifying Meta-ExternalAgent
Meta-ExternalAgent identifies itself with Meta-ExternalAgent in its User-Agent header. It does not run JavaScript: it reads the raw HTML only. Content your site adds with JavaScript is invisible to it.
Meta doesn't publish a machine-readable IP range list for Meta-ExternalAgent that we verify against, so treat the user agent alone as unverified. bots_report marks these hits as unverifiable rather than genuine.
Check your site
llmentions audit example.com # is Meta-ExternalAgent allowed by robots.txt, and what does it see?
llmentions bots /var/log/nginx/access.log # did Meta-ExternalAgent really visit, and is it genuine?Or identify any user agent over REST (free): GET https://api.llmentions.com/v1/bots/lookup?ua=Meta-ExternalAgent.