# SEO Audit: On-Page, Redirects & AI Crawler Access (`aiqlabs/seo-audit-tool`) Actor

Audits pages for on-page and technical SEO, and adds four checks general auditors skip: AI crawler access in robots.txt, llms.txt, redirects that drop the path, and noindex sent via X-Robots-Tag.

- **URL**: https://apify.com/aiqlabs/seo-audit-tool.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## SEO Audit: On-Page, Redirects & AI Crawler Access

Give it a list of URLs. Get one row per page with the on-page and technical SEO
findings, a score, and an explicit list of what to fix — plus four checks that
general SEO auditors leave out.

Free. No API key. No browser, so a few hundred URLs cost almost nothing.

### The four checks other auditors skip

**1. Which AI crawlers your robots.txt blocks.** Twenty-six of them, by name —
GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Applebot-Extended and
the rest — each with the rule that matched. Blocking them may be exactly what
you want; not knowing you block them is not.

**2. Whether you publish /llms.txt.** The emerging convention for telling AI
assistants which pages matter. Checked properly: a site that answers 200 with
its normal HTML template for any unknown path does not count as having one.

**3. Redirects that quietly throw the path away.** After a migration, old URLs
often 301 to the front page. Every status checker reports `301 → 200 OK` and
calls it healthy. Google calls it a soft 404 and drops the page. This Actor
follows each hop by hand and says so.

**4. noindex sent in the X-Robots-Tag header.** It is the only standard way to
noindex a PDF, and a checker that only parses HTML never sees it. Read from the
response headers, so it is caught on files as well as pages.

### Everything else it checks

**Indexability** — HTTP status, full redirect chain, redirect loops, HTTPS
downgrades, temporary redirects used for permanent moves, robots.txt disallow,
meta robots, nofollow, nosnippet/noarchive, soft 404 detection.

**On-page** — title and its length, meta description and its length, H1 count,
heading outline and skipped levels, canonical (missing, off-site, pointing at
the home page), `html lang`, viewport, word count that counts CJK correctly,
internal and external link counts, images without alt text with samples.

**International** — hreflang entries, invalid language tags, missing
self-reference, missing `x-default`.

**Structured and social** — every schema.org `@type` found in JSON-LD including
nested and `@graph` entries, JSON-LD blocks that fail to parse, Open Graph
completeness, Twitter Card tags.

Each finding carries a severity (error / warning / notice), a sentence
explaining what it means, and often the evidence that triggered it. The score
starts at 100 and subtracts by severity, so it is a summary of the findings
rather than a number invented separately from them.

### Input

```json
{
  "urls": [
    "https://example.com/",
    "https://example.com/pricing",
    "example.com/blog/post"
  ],
  "checkAiCrawlers": true,
  "respectRobotsTxt": true,
  "onlyIssues": false,
  "maxConcurrency": 8
}
```

Bare hosts work. One entry may hold several URLs separated by newlines, commas
or spaces, so a pasted column from a spreadsheet is a valid input.

### Output

One row per URL. Three views are provided: **Overview** (score and findings),
**Indexing & redirects**, and **AI crawler access**. Totals, the average score
and an issue breakdown are written to the key-value store as `SUMMARY`.

```json
{
  "url": "https://example.com/old-guide",
  "finalUrl": "https://example.com/",
  "status": 200,
  "score": 76,
  "indexable": true,
  "errors": 1,
  "warnings": 1,
  "redirectHops": 1,
  "redirectChain": ["301 https://example.com/"],
  "title": "Example",
  "canonical": "https://example.com/",
  "canonicalIsSelf": false,
  "aiCrawlersBlocked": ["GPTBot", "CCBot"],
  "llmsTxt": false,
  "issues": [
    {
      "code": "redirect_to_homepage",
      "severity": "error",
      "message": "The redirect throws the path away and lands on the home page…"
    }
  ]
}
```

### Manners

One robots.txt and one llms.txt request per host, whatever the URL count. The
page itself is fetched once, and at most 400 KB of it is read. Pages that
robots.txt disallows are reported with the rule that blocked them and **not**
downloaded. No JavaScript is executed and nothing is crawled beyond the URLs
you supply.

### Limits worth knowing

- Findings come from the HTML as served. A site that builds its `<head>` in the
  browser will look emptier here than it does to a rendering crawler.
- hreflang is checked per page. Whether the other side of the pair points back
  cannot be judged from one document.
- The score is a convenience. Read the issue list.

# Actor input Schema

## `urls` (type: `array`):

Pages to check. Full URLs (https://example.com/pricing) or bare hosts (example.com) both work, and one entry may hold several URLs separated by newlines, commas or spaces.

## `checkAiCrawlers` (type: `boolean`):

Read robots.txt for each host and report which of 26 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others) are blocked, plus whether the site publishes /llms.txt. One extra request per host.

## `respectRobotsTxt` (type: `boolean`):

Skip downloading pages that robots.txt disallows for Googlebot. The URL is still reported, with the rule that blocked it, so a page hidden by mistake is visible in the output.

## `onlyIssues` (type: `boolean`):

Leave out pages that produced no error and no warning. Useful when auditing a large site and only the problems matter.

## `maxConcurrency` (type: `integer`):

How many pages to fetch at the same time. Lower this for small servers.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one response before giving up on it.

## `maxRedirects` (type: `integer`):

How many redirects to follow before reporting the chain as unterminated. Every hop is kept in the output.

## `maxUrls` (type: `integer`):

Hard cap on how many URLs are audited in one run, counted after duplicates are removed.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com",
    "https://www.wordpress.org"
  ],
  "checkAiCrawlers": true,
  "respectRobotsTxt": true,
  "onlyIssues": false,
  "maxConcurrency": 8,
  "requestTimeoutSecs": 20,
  "maxRedirects": 8,
  "maxUrls": 5000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com",
        "https://www.wordpress.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/seo-audit-tool").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://apify.com",
        "https://www.wordpress.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/seo-audit-tool").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com",
    "https://www.wordpress.org"
  ]
}' |
apify call aiqlabs/seo-audit-tool --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/seo-audit-tool",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/ZjU3YtyaWgqpNriNj/builds/w1vpQ0IrZo54ELyYY/openapi.json