# AI Crawler Policy Auditor (`junipr/ai-crawler-policy-auditor`) Actor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

- **URL**: https://apify.com/junipr/ai-crawler-policy-auditor.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.90 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Crawler Policy Auditor

### Store Positioning

**Store title:** AI Crawler Policy Auditor

**Short description:** Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

**SEO title:** AI Crawler Policy Auditor — technical SEO, web, and domain audit

**SEO description:** Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness. Use it to find crawlability, indexability, security, metadata, and page-quality issues with evidence-backed rows and audit reports.

**Categories:** SEO\_TOOLS

**Keywords:** ai, crawler, policy, auditor, web/domain audit

### Fixed-Inclusive PPE Pricing

This actor uses pay-per-event pricing. Event prices include Apify platform usage; users are not expected to pay a separate platform-usage pass-through charge for the configured pricing model.

- Tier: W1 — Web/domain audit
- Primary event: `page-audited` at $0.00490 base
- Default max charge: $10.00
- Store discounts: FREE/BRONZE base, SILVER discounted, GOLD deepest approved discount

Event set:

- `actor-start`: base $0.00500, GOLD $0.00400. Ai Crawler Policy Auditor: charged when actor start is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `page-audited`: base $0.00490, GOLD $0.00392. Ai Crawler Policy Auditor: charged when page audited is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `rule-parsed`: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when rule parsed is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `finding-emitted`: base $0.00372, GOLD $0.00298. Ai Crawler Policy Auditor: charged when finding emitted is completed. The price includes Apify platform usage; no separate usage pass-through is intended.
- `audit-report-generated`: base $0.05000, GOLD $0.04000. Ai Crawler Policy Auditor: charged when audit report generated is completed. The price includes Apify platform usage; no separate usage pass-through is intended.

### Public Task Concepts

- Audit AI Crawler Policy controls with capped supplied targets
- Find high-priority AI Crawler Policy issues before release
- Validate AI Crawler Policy evidence from supplied pages
- Prioritize AI Crawler Policy fixes with severity and proof
- Export AI Crawler Policy QA rows for client review

Audit robots.txt, llms.txt, X-Robots-Tag, and meta robots signals for AI crawler access, blocking, and site-readiness.

### What This Actor Does

AI Crawler Policy Auditor checks whether major AI and search crawler user agents are allowed, blocked, partially blocked, or unmanaged for a target path. It parses supplied robots.txt snapshots by default, can compare previous and current policies, detects page-level robots restrictions such as `noindex` and `noai`, and writes one structured dataset row per audited target.

The default run is local-first: it uses an inline fixture, makes no network calls, and writes summary diagnostics to the key-value store. Optional live robots fetching is available only when `fetchUrls` is explicitly set to `true`, with strict target, timeout, and byte caps.

### What This Actor Does Not Do

- It does not guarantee search ranking, AI citation, traffic, revenue, or compliance outcomes.
- It does not provide legal, medical, or financial advice.
- It does not collect sensitive personal data.
- It does not crawl full websites or render pages in a browser.
- It does not fetch live URLs unless `fetchUrls` is explicitly enabled.

### Operational Status

- Package status: local-green.
- Cloud status: cloud-deferred until the main actor publication workflow pushes and verifies the actor remotely.
- Default input: inline robots fixture with `fetchUrls: false`.
- Store icon workflow: actor-local `icon.png` is included; shared landing icon manifests and live Console upload are intentionally left to the main agent.

### Best Use Cases

- Confirm whether GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, and similar agents can access key pages.
- Audit an intentional AI crawler opt-out policy before a launch or client delivery.
- Catch robots.txt changes that newly block or unblock AI crawler agents.
- Detect mixed policy signals where robots.txt allows crawlers but page-level tags say `noindex`, `noai`, or `noimageai`.
- Produce a concise KVS report for SEO, content, platform, or AI search readiness QA.

### Why Use This Actor

| Option | Strength | Weakness | Best fit |
| --- | --- | --- | --- |
| Manual robots.txt review | Fast for one site | Easy to miss wildcard behavior, path specificity, and changed blocked agents | One-off spot checks |
| Generic SEO crawler | Broad page crawl | Usually not focused on AI crawler user-agent policy or supplied examples testing | Full-site SEO diagnostics |
| AI Crawler Policy Auditor | Focused AI crawler access decisions, deterministic defaults, KVS report, PPE billing controls | Does not render pages or crawl deep paths | AI crawler policy QA and repeatable robots snapshots |

### How To Use

Run the actor with the default input for a zero-config local policy audit. To audit your own policy without network calls, paste robots.txt content into `targets[].robotsTxt` and keep `fetchUrls` set to `false`.

```json
{
  "targets": [
    {
      "sourceId": "homepage-policy",
      "sourceUrl": "https://example.com",
      "targetPath": "/",
      "robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nDisallow: /\nSitemap: https://example.com/sitemap.xml",
      "llmsTxt": "# AI crawler guidance\nAllowed for documentation indexing.",
      "headers": {
        "x-robots-tag": "index, follow"
      },
      "html": "<meta name=\"robots\" content=\"index, follow\">"
    }
  ],
  "expectedAccess": "allow",
  "fetchUrls": false,
  "includeReport": true,
  "maxTargets": 10
}
```

#### AI Readiness Check

Use `expectedAccess: "allow"` when the goal is discoverability or AI search readiness. Blocked AI crawler agents become warnings or failures depending on scope.

#### AI Opt-Out Check

Use `expectedAccess: "block"` when the goal is to verify an opt-out policy. The actor flags agents that remain allowed, partially blocked, or unmanaged.

#### Inventory-Only Review

Use `expectedAccess: "neutral"` when you only want a policy inventory. Mixed policies are still surfaced, but the actor avoids assuming whether blocking is good or bad.

### Input Configuration

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `targets` | array | Inline fixture | Robots, llms.txt, header, and HTML snapshots to audit. |
| `urls` | string array | `[]` | Optional site URLs used only when `fetchUrls` is true. |
| `fetchUrls` | boolean | `false` | Fetch `/robots.txt` and `/llms.txt` with timeout and byte caps. |
| `expectedAccess` | string | `allow` | `allow`, `block`, or `neutral`. |
| `auditedAgents` | string array | Major AI and search crawlers | User-agent tokens to evaluate against robots.txt. |
| `comparePrevious` | boolean | `true` | Compare `previousRobotsTxt` when supplied. |
| `requireLlmsTxt` | boolean | `false` | Flag targets without llms.txt content. |
| `includeReport` | boolean | `true` | Write JSON and Markdown report artifacts to KVS. |
| `maxTargets` | integer | `25` | Maximum targets to audit; hard cap is `250`. |
| `fetchTimeoutMs` | integer | `10000` | Timeout for optional live fetches. |
| `maxRobotsBytes` | integer | `200000` | Maximum bytes to read from fetched robots.txt. |
| `maxChargeUsd` | number | `10` | Hard PPE cap; stops before work or withholds uncharged reports when the next event would exceed it. |
| `debug` | boolean | `false` | Enable debug logs. |

### Output Format

Each dataset row represents one audited target. Important fields include:

- `status`, `severity`, and `aiCrawlerReadinessScore`: Overall result.
- `expectedAccess` and `policySummary`: Goal and plain-English outcome.
- `allowedAgents`, `blockedAgents`, `partiallyBlockedAgents`, and `unknownAgents`: Per-agent access buckets.
- `agentPolicies`: Detailed matched robots group and rule for each audited user agent.
- `hasRobotsTxt`, `robotsGroupCount`, `hasSitemap`, `hasLlmsTxt`, and byte counts: Policy inventory.
- `xRobotsTag`, `metaRobotsDirectives`, `hasNoindexSignal`, and `hasNoaiSignal`: Page-level policy signals.
- `changedRobotsPolicy`, `addedBlockedAgents`, and `removedBlockedAgents`: Previous/current diff results.
- `issues` and `recommendation`: Structured findings and next action.

The key-value store also contains:

- `AI_CRAWLER_POLICY_RESULTS.json`: Full result array.
- `AI_CRAWLER_POLICY_SUMMARY.json`: Run-level counts, scores, blocked agents, and top issues.
- `AI_CRAWLER_POLICY_REPORT.md`: Human-readable report.

### Example Output

```json
{
  "sourceId": "sample-ai-ready-site",
  "sourceUrl": "https://example.com",
  "targetPath": "/",
  "status": "pass",
  "severity": "none",
  "aiCrawlerReadinessScore": 100,
  "expectedAccess": "allow",
  "policySummary": "Audited crawlers are allowed for the target path.",
  "allowedAgents": ["Bingbot", "Bytespider", "CCBot", "ChatGPT-User", "Claude-User", "ClaudeBot", "GPTBot", "Google-Extended", "GoogleOther", "Googlebot", "Meta-ExternalAgent", "OAI-SearchBot", "PerplexityBot", "anthropic-ai"],
  "blockedAgents": [],
  "unknownAgents": [],
  "hasRobotsTxt": true,
  "hasSitemap": true,
  "hasLlmsTxt": true,
  "hasNoindexSignal": false,
  "hasNoaiSignal": false,
  "issueCount": 0,
  "recommendation": "AI crawler policy signals are aligned for the supplied target."
}
```

### Integration Examples

#### Node.js

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('junipr/ai-crawler-policy-auditor').call({
  targets: [{ sourceId: 'site', robotsTxt: 'User-agent: *\nAllow: /' }],
  expectedAccess: 'allow',
  fetchUrls: false
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].blockedAgents);
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient(token="YOUR_APIFY_TOKEN")
run = client.actor("junipr/ai-crawler-policy-auditor").call(run_input={
    "targets": [{"sourceId": "site", "robotsTxt": "User-agent: *\nAllow: /"}],
    "expectedAccess": "allow",
    "fetchUrls": False,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items[0]["recommendation"])
```

### Cost-Control Tips

- Keep `fetchUrls` false for deterministic CI, QA, and client snapshot reviews.
- Use `maxTargets` between `1` and `10` for initial live checks.
- Prefer inline `robotsTxt` snapshots for repeatable regression tests.
- Disable `includeReport` if you only need dataset rows.
- Use `expectedAccess: "neutral"` for inventory-only reports where blocking may be intentional.

### Public Task Examples

This package includes five prepared task concepts:

- AI crawler allow-list check.
- AI crawler block audit.
- Mixed AI crawler policy review.
- Page-level noai signal scan.
- Optional live robots fetch.

### FAQ

#### Does this actor make network calls by default?

No. The default input is fully inline and `fetchUrls` defaults to `false`.

#### Can it fetch live robots.txt files?

Yes. Set `fetchUrls` to `true` and provide `urls`. The actor fetches `/robots.txt` and optionally `/llms.txt` with strict caps.

#### Does blocking an AI crawler always mean the site is wrong?

No. That depends on your policy goal. Use `expectedAccess: "block"` for opt-out audits or `neutral` for inventory-only checks.

#### Does this actor replace legal review?

No. It reports technical policy signals and does not provide legal advice.

#### Are diagnostics billed as dataset rows?

### Troubleshooting

- `robots-txt-missing`: Supply inline robots content or enable live fetching for authorized URLs.
- `ai-crawlers-blocked`: Review blocked agents against your AI search or opt-out goal.
- `mixed-policy-signals`: Check for conflicts between robots.txt, meta robots, X-Robots-Tag, and selected expected access.
- `robots-policy-changed`: Review the previous/current diff before treating the current policy as intentional.

### Limitations

- Robots matching is designed for practical policy QA, not a full crawler implementation.
- Live fetching can fail due auth, redirects, firewalls, DNS, or network conditions.
- The actor audits a target path, not every URL on a site.
- Page-level signals are parsed from supplied HTML and headers; the actor does not render JavaScript.

### Source And Safety Notes

Audit only URLs and policy snapshots you are authorized to inspect. Avoid uploading sensitive internal policy content unless you are allowed to process it in Apify.

### Changelog

- `1.0.0`: Initial package build with local-first robots auditing, optional capped fetches, previous/current diffing, KVS reports, PPE billing, examples, and fixture tests.

# Actor input Schema

## `targets` (type: `array`):

Robots.txt and page-signal snapshots to audit. Defaults to a local fixture that makes no network calls.

## `urls` (type: `array`):

Optional site URLs to fetch only when fetchUrls is true. The actor fetches /robots.txt and /llms.txt with strict byte and timeout caps.

## `fetchUrls` (type: `boolean`):

Fetch robots.txt and llms.txt over HTTP. Default is false to keep zero-config runs local, deterministic, and cheap.

## `expectedAccess` (type: `string`):

Use allow for AI crawler readiness checks, block for opt-out audits, or neutral for inventory-only reporting.

## `auditedAgents` (type: `array`):

Crawler user-agent tokens to evaluate against robots.txt.

## `comparePrevious` (type: `boolean`):

Compare previousRobotsTxt against current robotsTxt when a previous snapshot is supplied.

## `requireLlmsTxt` (type: `boolean`):

Flag targets that do not include llms.txt content.

## `maxItems` (type: `integer`):

Alias for Max Targets/Maximum Records used by the Junipr actor baseline. When both are provided, actor-specific limits still apply.

## `includeReport` (type: `boolean`):

Write summary JSON, full result JSON, and Markdown report artifacts to the key-value store.

## `maxTargets` (type: `integer`):

Maximum robots or URL targets to audit.

## `fetchTimeoutMs` (type: `integer`):

HTTP timeout for optional robots.txt and llms.txt fetches.

## `maxRobotsBytes` (type: `integer`):

Maximum bytes to read from a fetched robots.txt file.

## `maxChargeUsd` (type: `number`):

Hard local spending cap for actor-start, per-item, and report events. The actor stops before work or withholds uncharged output when the next event would exceed this amount.

## `dryRun` (type: `boolean`):

Validate input and write a dry-run summary without PPE charges or dataset output.

## `debug` (type: `boolean`):

Enable debug logs.

## Actor input object example

```json
{
  "targets": [
    {
      "sourceId": "sample-ai-ready-site",
      "sourceUrl": "https://example.com",
      "targetPath": "/",
      "robotsTxt": "User-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ChatGPT-User\nAllow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: Claude-User\nAllow: /\n\nUser-agent: anthropic-ai\nAllow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Google-Extended\nAllow: /\n\nUser-agent: GoogleOther\nAllow: /\n\nUser-agent: CCBot\nAllow: /\n\nUser-agent: Bytespider\nAllow: /\n\nUser-agent: Meta-ExternalAgent\nAllow: /\n\nSitemap: https://example.com/sitemap.xml",
      "llmsTxt": "# Example AI crawler guidance\n\nThis fixture represents a site that allows major AI crawlers and exposes a sitemap.",
      "headers": {
        "x-robots-tag": "index, follow"
      },
      "html": "<!doctype html><html><head><title>Example</title><meta name=\"robots\" content=\"index, follow\"></head><body>Example</body></html>"
    }
  ],
  "urls": [],
  "fetchUrls": false,
  "expectedAccess": "allow",
  "auditedAgents": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot",
    "ClaudeBot",
    "Claude-User",
    "anthropic-ai",
    "PerplexityBot",
    "Google-Extended",
    "GoogleOther",
    "CCBot",
    "Bytespider",
    "Meta-ExternalAgent",
    "Googlebot",
    "Bingbot"
  ],
  "comparePrevious": true,
  "requireLlmsTxt": false,
  "maxItems": 25,
  "includeReport": true,
  "maxTargets": 25,
  "fetchTimeoutMs": 10000,
  "maxRobotsBytes": 200000,
  "maxChargeUsd": 10,
  "dryRun": false,
  "debug": false
}
```

# Actor output Schema

## `results` (type: `string`):

Each dataset row is one audited robots or site policy target with crawler access decisions, findings, and recommendations.

## `resultsJson` (type: `string`):

Full JSON array of audit results.

## `summaryJson` (type: `string`):

Run-level counts, scores, blocked agents, and top issues.

## `summaryReport` (type: `string`):

Markdown report describing AI crawler policy audit results.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("junipr/ai-crawler-policy-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("junipr/ai-crawler-policy-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call junipr/ai-crawler-policy-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=junipr/ai-crawler-policy-auditor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/G5wXmAAc0fFGPHtVl/builds/80vNFOXNuAAgowY5A/openapi.json