LLMentions
Audit checks · no_robots_txt

No robots.txt

No robots.txt: why it keeps you out of AI answers and how to fix it. Audit check no_robots_txt (notice).

↓ Markdown for agents
Check id
no_robots_txt
Severity
notice
Tool
site_audit

Without robots.txt, bots fall back to defaults and you lose the easiest place to list your sitemap and state your AI policy.

How to fix it

Publish /robots.txt, even if it allows everything, and list your sitemap in it.

How LLMentions checks it

site_audit crawls your site the way AI bots do (raw HTML, no JavaScript), reads robots.txt for every known AI and search bot, and renders a few pages in a real browser for comparison. This check reports as no_robots_txt with severity notice.

terminal
llmentions audit example.com

The result lists every issue with the pages affected, most severe first, and what changed since your last audit. See the site audit guide.