LLMentions
Audit checks · blocked_for_ai_search

robots.txt blocks AI search bots from this page

robots.txt blocks AI search bots from this page: why it keeps you out of AI answers and how to fix it. Audit check blocked_for_ai_search (warning).

↓ Markdown for agents
Check id
blocked_for_ai_search
Severity
warning
Tool
site_audit

A path-level Disallow can hide your best pages (docs, pricing, comparisons) from AI search bots even when the rest of the site is open.

How to fix it

Unblock the path for AI search and user-fetch bots if it should be cited.

How LLMentions checks it

site_audit crawls your site the way AI bots do (raw HTML, no JavaScript), reads robots.txt for every known AI and search bot, and renders a few pages in a real browser for comparison. This check reports as blocked_for_ai_search with severity warning, which fails CI with --fail-on warning.

terminal
llmentions audit example.com

The result lists every issue with the pages affected, most severe first, and what changed since your last audit. See the site audit guide.