Robots Noindex Conflict Checker

Compare supplied robots.txt rules with page-level noindex evidence and identify blocked URLs whose noindex directive may not be observed.

Inputs

• inputItems
• maxPages
• includeReport
• maxChargeUsd

{
  "inputItems": [
    {
      "url": "https://en.wikipedia.org/wiki/Special:UserLogin",
      "robotsUrl": "https://en.wikipedia.org/robots.txt",
      "statusCode": 200,
      "robotsTxt": "User-agent: *\nDisallow: /w/\nDisallow: /wiki/Special:",
      "pageHeaders": {
        "content-type": "text/html; charset=UTF-8"
      },
      "html": "<meta name=\"robots\" content=\"noindex,nofollow,max-image-preview:standard\">"
    }
  ],
  "maxPages": 1,
  "includeReport": true,
  "maxChargeUsd": 1
}

The bundled exact input combines the public Wikipedia robots rule for /wiki/Special: with the public noindex meta directive observed on Special:UserLogin at 2026-07-12T05:17:40Z.

Dataset

The actor emits structured, actor-specific evidence rows. Empty source input produces no dataset rows.

Billing

This actor uses pay-per-event billing with platform usage included in the configured event prices:

• actor-start
• page-audited
• rule-parsed
• finding-emitted
• audit-report-generated

Every dataset row and report artifact is charge-gated. A zero-dollar cap stops before work with no paid output.

Use Cases

• Find a robots path-prefix conflict
• Check an X-Robots-Tag conflict
• Check a meta robots conflict
• Verify a page has no robots/noindex conflict
• Export robots/noindex conflict evidence

Limitations

The actor evaluates supplied robots and page observations and does not crawl URLs. Disallow matching uses path prefixes for the supplied user-agent context and does not implement every robots exclusion extension.