Technical SEO & AI Crawler Audit

The default initialRunMode: "baseline_only" stores the first portfolio baseline with zero rows and zero charge. Later unchanged monitor runs also emit zero rows and zero charge. Use emit_backfill only when you explicitly want the current observations and report delivered as billable rows.

Site owners, SEO agencies, and content teams use this actor to audit public indexability and AI crawler access signals. Provide public URLs and optional AI crawler user-agent names. The actor returns source-linked policy observations, indexability issues, reports, and export rows.

Start with a low-cost paid trial

Use this capped input to verify the source and buyer-facing output before buying a full report. It disables the report and export events and caps the run at $0.00. This authorization-gated starter is a free safe demo; confirm authorization and choose a paid output before a live audit. A zero-row run has zero event charge.

{
  "urls": [
    "https://example.com/"
  ],
  "aiCrawlerUserAgents": [
    "GPTBot",
    "ClaudeBot"
  ],
  "maxPages": 3,
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": false,
  "emitPageRows": false,
  "emitRawRows": false,
  "generateReport": true,
  "emitExport": false,
  "emitUnchanged": false,
  "initialRunMode": "baseline_only",
  "maxChargeUsd": 0,
  "dryRun": true
}

When the result fits your workflow, use the full-report input below to generate the full seo-portfolio-report output ($3.00) and seo-agency-export export ($2.00). The cap covers the selected report/export path; unchanged monitoring runs still produce zero rows and zero event charge.

{
  "urls": [
    "https://your-authorized-domain.example/"
  ],
  "aiCrawlerUserAgents": [
    "GPTBot",
    "ClaudeBot"
  ],
  "maxPages": 3,
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": false,
  "emitPageRows": false,
  "emitRawRows": false,
  "generateReport": true,
  "emitExport": true,
  "emitUnchanged": false,
  "initialRunMode": "emit_backfill",
  "maxChargeUsd": 5.27,
  "dryRun": false
}

Store Quickstart

Run with dryRun=false and public URLs that you own or are allowed to audit.

{
  "urls": ["https://example.com/?siteQaCanary=indexability-ai-crawler-v1"],
  "aiCrawlerUserAgents": ["GPTBot", "Google-Extended", "PerplexityBot", "ClaudeBot"],
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": true,
  "generateReport": true,
  "emitUnchanged": false,
  "dryRun": false
}

Run the next report

• seo-portfolio-report: $3.000 per portfolio report.
• seo-agency-export: $2.000 per generated export.
• Measure Core Web Vitals, Lighthouse scores, axe-core findings, and score regressions with Website Lighthouse & WCAG Regression Report.

Input Examples

Audit one page and origin policies

{
  "urls": ["https://example.com/blog/launch"],
  "aiCrawlerUserAgents": ["GPTBot", "ClaudeBot"],
  "checkRobotsTxt": true,
  "checkLlmsTxt": true,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Batch audit a site section

{
  "urls": [
    "https://example.com/",
    "https://example.com/pricing",
    "https://example.com/docs"
  ],
  "maxPages": 25,
  "emitPageRows": false,
  "generateReport": true,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Generate a handoff export

{
  "urls": ["https://example.com/landing-page"],
  "aiCrawlerUserAgents": ["GPTBot", "Google-Extended", "PerplexityBot"],
  "emitExport": true,
  "emitUnchanged": false,
  "authorizedUseConfirmed": true,
  "dryRun": false
}

Sample Output

{
  "actorName": "technical-seo-portfolio-regression-report",
  "rowType": "indexability_issue",
  "billingEventName": "seo-regression-detected",
  "issueType": "ai_crawler_disallowed_by_robots",
  "severity": "high",
  "sourceUrl": "https://example.com/?siteQaCanary=indexability-ai-crawler-v1"
}

Output Fields

• rowType: ai_crawler_policy_observation, indexability_issue, ai_crawler_indexability_report, or indexability_export.
• billingEventName: PAY_PER_EVENT event name used for the row.
• sourceUrl: public URL or policy file that supports the row.
• issueType: detected source-linked issue when applicable.
• blockedUserAgents: crawler names with broad robots.txt blocks when detected.

Pricing And No-Change Runs

• seo-page-audited: $0.012 per audited public page.
• seo-regression-detected: $0.080 per source-linked regression.
• seo-portfolio-report: $3.000 per portfolio report.
• seo-agency-export: $2.000 per generated export.

When emitUnchanged=false, repeated unchanged runs emit zero dataset rows and zero charges after state is saved.

maxChargeUsd is an optional positive per-run cap. The actor estimates every output row from the configured PAY_PER_EVENT unit prices before calling pushData; if the estimate exceeds the cap, it fails before charging. 0 preserves the existing no-explicit-cap behavior. Delivery journal entries record pending, charged, failed, and uncertain attempts so a partial retry can deduplicate confirmed charges and refuses to retry an uncertain charge.

Compliance Guardrails

• Public pages, robots.txt, and llms.txt only.
• No login, paywall, CAPTCHA, private session, credentialed API, or bypass behavior.
• Non-dry runs require authorizedUseConfirmed=true; use this only for sites you own, manage, or are allowed to audit.
• This is an unofficial audit tool and is not affiliated with any crawler, search engine, or AI provider.
• No ranking guarantee, AI citation guarantee, legal conclusion, or compliance certification.

Bundle Paths

• seo-portfolio-report: $3.000 per portfolio report.
• Pair with Site QA Content Report Scraper for page content issue reports.
• Pair with Site QA Broken Link Report Scraper for link health reports.

See Also

• Site QA Content Report Scraper for content QA issue reports.
• Site QA Broken Link Report Scraper for broken link reports.