Docs · Site audit
Site audit
Check whether AI answer engines can read and cite your site. Crawl as AI bots see it, check robots.txt per bot, find JavaScript-only content, and verify real bot traffic from your logs.
↓ Markdown for agentsAn engine can only cite what it can read. Most sites that are invisible in AI answers have a mechanical cause: robots.txt blocks the bots that build AI search indexes, a CDN rule challenges them, or the main content only appears after JavaScript runs, which AI crawlers don't do.
site_audit finds those causes and tells you how to fix each one. bots_report then confirms, from your own server logs, which bots actually visit.
Run an audit
llmentions audit acme.comOr with MCP (site_audit) or REST:
curl -s https://api.llmentions.com/v1/site/audit \
-H "authorization: Bearer $LLMENTIONS_API_KEY" \
-H 'content-type: application/json' \
-d '{"url": "https://acme.com", "max_pages": 100, "render": 5}'What the audit does:
- Reads robots.txt for every known AI and search bot, and reports which ones are allowed or blocked: training bots, AI search indexers and live user-fetch bots are listed separately, because blocking each has different consequences.
- Fetches your homepage as an AI bot and as a browser and compares the two, to catch CDN and WAF rules that challenge AI user agents.
- Crawls up to
max_pagespages (default 25, at most 100) without JavaScript, the way AI crawlers see them, and records the text, title, headings, canonical, structured data and response time. - Renders up to
renderpages in a real browser (default 3, at most 10; homepage and thinnest pages first) and compares the text, to find content that only exists after JavaScript. - Lists issues most-severe first, each with the pages affected and a fix.
Audits above 25 pages run in the background; poll audit_get (or llmentions audits acme.com). Every audit is stored, and each new one reports what changed since the previous one: new issues, fixed issues and AI-bot access changes.
Example output
acme.com · 48 pages · 5 rendered · 2.1 s
AI access OAI-SearchBot ✓ ChatGPT-User ✓ PerplexityBot ✗ Claude-SearchBot ✓ GPTBot ✗ (training)
ERROR ai_search_blocked robots.txt blocks an AI search or answer bot PerplexityBot
→ allow search_index and user_fetch bots; blocking them keeps you out of AI answers
ERROR content_needs_js Most of the content only appears after JavaScript runs 12 pages
→ server-render (SSR/SSG) the main content
WARNING slow_response Slow server response (over 800 ms) 3 pages
NOTICE no_structured_data No JSON-LD structured data 48 pages
since last audit: 1 new, 2 fixed, 0 changed issue(s); 0 AI-bot access change(s)Every check, what it means and how to fix it is listed in the audit checks reference.
Price
$0.0002 per page crawled and $0.002 per page rendered. A 100-page audit with 5 rendered pages costs $0.03. Test keys audit for free.
Fail CI when your site stops being readable
llmentions audit https://preview-123.acme.dev --fail-on regression--fail-on takes error, warning or regression, and exits 1 when it trips. There is a ready-made GitHub Action in CLI → In CI.
Verify real bot traffic
Your robots.txt says who may crawl. Your logs say who actually did. Send a raw access log (up to 20 MB) to bots_report:
llmentions bots /var/log/nginx/access.logYou get hits, status codes and top pages per bot, the live AI fetches (a proxy for real-time citations), and IP verification: a hit claiming to be GPTBot from outside OpenAI's published ranges is flagged as spoofed. bots_report is free.
The AI crawler directory lists every bot we recognise, what it's for, and whether you should allow it.