Pricing

Pay per usage

SEO Audit: On-Page, Redirects & AI Crawler Access

Audits pages for on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Ai-Q Labs

Actor stats

Bookmarked

Total users

Monthly active users

4 days ago

Last modified

The four checks other auditors skip

1. Which AI crawlers your robots.txt blocks. Twenty-six of them, by name — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended and the rest — each with the rule that matched. Blocking them may be exactly what you want; not knowing you block them is not.

2. Whether you publish /llms.txt. The emerging convention for telling AI assistants which pages matter. Checked properly: a site that answers 200 with its normal HTML template for any unknown path does not count as having one.

3. Redirects that quietly throw the path away. After a migration, old URLs often 301 to the front page. Every status checker reports 301 → 200 OK and calls it healthy. Google calls it a soft 404 and drops the page. This Actor follows each hop by hand and says so.

4. noindex sent in the X-Robots-Tag header. It is the only standard way to noindex a PDF, and a checker that only parses HTML never sees it. Read from the response headers, so it is caught on files as well as pages.

Everything else it checks

Indexability — HTTP status, full redirect chain, redirect loops, HTTPS downgrades, temporary redirects used for permanent moves, robots.txt disallow, meta robots, nofollow, nosnippet/noarchive, soft 404 detection.

On-page — title and its length, meta description and its length, H1 count, heading outline and skipped levels, canonical (missing, off-site, pointing at the home page), html lang, viewport, word count that counts CJK correctly, internal and external link counts, images without alt text with samples.

International — hreflang entries, invalid language tags, missing self-reference, missing x-default.

Structured and social — every schema.org @type found in JSON-LD including nested and @graph entries, JSON-LD blocks that fail to parse, Open Graph completeness, Twitter Card tags.

Each finding carries a severity (error / warning / notice), a sentence explaining what it means, and often the evidence that triggered it. The score starts at 100 and subtracts by severity, so it is a summary of the findings rather than a number invented separately from them.

Input

{
  "urls": [
    "https://example.com/",
    "https://example.com/pricing",
    "example.com/blog/post"
  ],
  "checkAiCrawlers": true,
  "respectRobotsTxt": true,
  "onlyIssues": false,
  "maxConcurrency": 8
}

Bare hosts work. One entry may hold several URLs separated by newlines, commas or spaces, so a pasted column from a spreadsheet is a valid input.

Output

One row per URL. Three views are provided: Overview (score and findings), Indexing & redirects, and AI crawler access. Totals, the average score and an issue breakdown are written to the key-value store as SUMMARY.

{
  "url": "https://example.com/old-guide",
  "finalUrl": "https://example.com/",
  "status": 200,
  "score": 76,
  "indexable": true,
  "errors": 1,
  "warnings": 1,
  "redirectHops": 1,
  "redirectChain": ["301 https://example.com/"],
  "title": "Example",
  "canonical": "https://example.com/",
  "canonicalIsSelf": false,
  "aiCrawlersBlocked": ["GPTBot", "CCBot"],
  "llmsTxt": false,
  "issues": [
    {
      "code": "redirect_to_homepage",
      "severity": "error",
      "message": "The redirect throws the path away and lands on the home page…"
    }
  ]
}

Manners

One robots.txt and one llms.txt request per host, whatever the URL count. The page itself is fetched once, and at most 400 KB of it is read. Pages that robots.txt disallows are reported with the rule that blocked them and not downloaded. No JavaScript is executed and nothing is crawled beyond the URLs you supply.

Limits worth knowing

Findings come from the HTML as served. A site that builds its <head> in the browser will look emptier here than it does to a rendering crawler.
hreflang is checked per page. Whether the other side of the pair points back cannot be judged from one document.
The score is a convenience. Read the issue list.

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Audits any URL for classic SEO health and for AI/LLM discoverability - AI crawler access in robots.txt, llms.txt, structured data, metadata - in a single pass.

Ai-Q Labs

Technical SEO & AI Crawler Audit

taroyamada/technical-seo-portfolio-regression-report

Audit public pages you are authorized to test for robots, noindex, sitemap, canonical, and AI crawler access changes. Deliver source-linked issues and portfolio reports.

naoki anzai

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

David

Robots Indexability Audit

toronto_777/robots-indexability-audit

Audit public robots.txt, sitemap declarations, homepage robots directives, and crawler allow/block signals for SEO and AI visibility checks.

Steven Feng

AI Crawler Readiness Auditor

nectared_screen/ai-crawler-readiness-auditor

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

jaehan byun

LLMs.txt / AI Readiness Auditor

civicdataworks/llms-txt-ai-readiness-auditor

Audit websites for AI crawler/readiness basics: llms.txt, sitemap, robots rules, titles, metadata, and AI crawler directives.

Rowan Mercer

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Ken Yao

Robots.txt Auditor & Sitemap Finder

andok/robotstxt-auditor

Scan robots.txt files in bulk to extract sitemap URLs and verify crawler directives for technical SEO compliance.

Andok

Robots-Noindex Conflict Checker

junipr/robots-noindex-conflict-checker

Detect conflicts between robots.txt blocking and page-level meta/X-Robots noindex signals.

junipr

Website SEO, Accessibility & AI Readiness Auditor

feeng/website-seo-ai-readiness-auditor

🔎 Crawl websites and turn technical SEO, accessibility, security, performance, broken-link, structured-data, robots.txt, llms.txt, and AI crawler signals into prioritized 0–100 audit scores and client-ready reports.