# Sitemap Checker & Broken Link Finder (`aiqlabs/sitemap-checker`) Actor

Find every sitemap a site publishes, expand sitemap indexes, and audit each URL: broken links, redirect chains, URLs blocked by the site's own robots.txt, XML errors, duplicates and invalid lastmod. One row per URL. Respects robots.txt.

- **URL**: https://apify.com/aiqlabs/sitemap-checker.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Checker & Broken Link Finder

Give it a domain. It finds every sitemap the site publishes, expands the sitemap
index, and returns **one row per URL** — with the broken links, the redirects,
the XML mistakes, and the URLs the site's own `robots.txt` blocks.

Most sitemap actors stop at extracting the list of URLs. Extracting is the easy
half. This one **checks** them.

### What it finds

| Issue | Severity | Why it costs you |
|---|---|---|
| `blocked_by_robots` | error | The URL is submitted for indexing **and** blocked from crawling. Google reports this and drops the page. |
| `broken_link` / `server_error` | error | A 4xx or 5xx URL in your sitemap. Wasted crawl budget, and a bad signal about the whole file. |
| `unreachable` | error | The URL does not respond at all. |
| `invalid_url` / `missing_loc` | error | A relative path, a `ftp://` scheme, or a `<url>` with no `<loc>`. |
| `unencoded_space` | error | A raw space in a URL. Some parsers stop reading the file at that point. |
| `redirect` / `redirect_chain` | warning | The URL 301s somewhere else. Sitemaps should list the destination, not the old address. |
| `sitemap_redirected` | warning | The sitemap file itself redirects — often the apex-to-www hop nobody noticed. |
| `cross_host` | warning | The URL is on a host this sitemap is not authorised to cover. |
| `insecure_http` | warning | An `http://` URL inside an `https://` sitemap. |
| `duplicate` | warning | The same URL listed twice, across all of the site's sitemaps. |
| `bad_lastmod` / `lastmod_in_future` | warning | Not a W3C datetime, an impossible date such as `2026-02-31`, or a date that has not happened yet. |
| `bad_changefreq` / `bad_priority` | warning | Values outside what the protocol allows. |
| `has_fragment` / `url_too_long` | warning | `#anchor` fragments are dropped; URLs over 2048 characters are truncated. |
| `too_many_urls` | error | Over 50,000 URLs in one file, so the file is ignored. |
| `sitemap_parse_error` / `sitemap_http_error` | error | The file is malformed, or it does not load. |
| `no_sitemap_found` | error | `robots.txt` names none and the common paths do not answer. |

#### The one that pays for itself

A URL that sits in your sitemap while your `robots.txt` blocks it is a
contradiction: you are asking Google to index a page you will not let it read.
Nothing inside the sitemap reveals it, because the conflict lives in a different
file.

It is not a rare mistake. On one well-known SaaS site in our live test suite,
**4,008 of 6,278 sitemap URLs were blocked by that site's own `robots.txt`** —
two `Disallow:` lines, quietly cancelling most of the sitemap.

Every blocked URL comes back with the exact rule responsible, e.g.
`Disallow: /offers$`, so the fix is one line either way.

### How it finds your sitemaps

1. Read `robots.txt` and use every `Sitemap:` line it declares — including
   sitemaps hosted on another domain, which the protocol allows.
2. If it declares none, try the common paths: `/sitemap.xml`,
   `/sitemap_index.xml`, `/sitemap-index.xml`, `/sitemap.xml.gz`,
   `/wp-sitemap.xml` and two more, keeping the first that really is a sitemap.
   A 200 response carrying an HTML error page does not count.
3. Expand every sitemap index, recursively, up to the depth you set.
4. Handle `.xml.gz` by looking at the bytes, not the headers — gzipped sitemaps
   are usually served without `content-encoding`, and plenty of readers get this
   wrong. Plain-text sitemaps work too.

### Input

| Field | Default | Notes |
|---|---|---|
| `domains` | — | Bare domains or full URLs. One entry may hold several, separated by spaces, commas or newlines, so a pasted spreadsheet column works. |
| `checkUrlStatus` | `true` | Request every URL and report status, hops and destination. Turn off for an XML-only audit, which is much faster. |
| `maxUrlsToCheck` | `300` | Cap on the broken-link requests per site. URLs past the cap are still validated and returned. |
| `robotsAgent` | `Googlebot` | Which crawler the `robots.txt` rules are read for. |
| `maxUrlsPerSite` | `5000` | Row cap per site. |
| `maxSitemaps` | `50` | How many sitemap files to open. |
| `maxDepth` | `3` | How deep to follow nested indexes. |
| `onlyIssues` | `false` | Drop the clean rows and keep the fix list. |
| `siteConcurrency` / `urlConcurrency` | `4` / `8` | Sites in parallel, and requests in flight per site. |
| `requestTimeoutSecs` | `20` | Per request. |

### Output

One row per URL, so the dataset drops straight into a spreadsheet:

```json
{
  "site": "example.com",
  "url": "https://example.com/offers/spring",
  "sitemap": "https://example.com/sitemap.xml",
  "severity": "error",
  "issues": ["blocked_by_robots"],
  "issueDetails": ["robots.txt blocks Googlebot with \"Disallow: /offers/*\""],
  "httpStatus": 200,
  "reachable": true,
  "finalUrl": null,
  "redirectCount": 0,
  "blockedByRobots": true,
  "robotsRule": "Disallow: /offers/*",
  "lastmod": "2026-07-01",
  "changefreq": "weekly",
  "priority": "0.8",
  "images": 2,
  "videos": 0,
  "alternates": 3,
  "sitemapKind": "urlset",
  "checkedAt": "2026-07-29T09:00:00.000Z"
}
```

Filter on `severity` for the fix list, or set `onlyIssues` and get it directly.

A per-site rollup — sitemaps found, URL and issue counts, and every sitemap file
with its size, entry count and problems — is written to the key-value store as
`SUMMARY`.

### Being a good guest

- Sitemaps are published to be read; this actor reads them and nothing else.
- It **does not crawl**. It never follows links out of a page, and it never
  renders JavaScript.
- With `checkUrlStatus` on it sends one `HEAD` per URL (falling back to `GET`
  only when the server rejects `HEAD`), never downloading page bodies, and never
  more than `maxUrlsToCheck` per site.
- Concurrency is capped and configurable, so you can dial it down for a fragile
  host.

### Limits worth knowing

- `robots.txt` rules are only applied to URLs on the host that served them. A
  sitemap covering several hosts is checked against the one robots file we can
  legitimately attribute.
- `lastmod` is validated for format and plausibility, not for truthfulness.
- The URL check reports what the server returns. A page that renders an error
  message with a 200 status is counted as reachable, because it is.
- Sites behind bot protection may answer the sitemap request with a challenge
  page. Those are reported as `no_sitemap_found` rather than being guessed at.

### Tests

```bash
npm test            # 20 unit tests, no network
npm run test:live   # 26 assertions against real sites
```

The live suite deliberately pins awkward real-world cases: a gzipped sitemap, an
apex-to-www redirect, a sitemap cross-submitted from a CDN host, a `/sitemap.xml`
that answers 200 with an HTML page, a domain with no sitemap at all, and a site
that blocks thousands of its own sitemap URLs.

# Actor input Schema

## `domains` (type: `array`):

Sites to audit. Full URLs are accepted and reduced to the domain. One entry may hold several sites separated by spaces, commas or newlines, so pasting a spreadsheet column works.

## `checkUrlStatus` (type: `boolean`):

Request every listed URL and report its HTTP status, redirect hops and final destination. Turn off to only validate the XML, which is much faster.

## `maxUrlsToCheck` (type: `integer`):

Upper bound on the broken-link requests sent to one site. URLs beyond this are still validated and returned, just without an HTTP status.

## `urlConcurrency` (type: `integer`):

How many URLs of the same site to request at once. Lower this if the site rate-limits.

## `robotsAgent` (type: `string`):

Which crawler the robots.txt rules are read for. Googlebot is the one that decides whether your pages get indexed.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many rows for one site. Protects the run from a sitemap with hundreds of thousands of entries.

## `maxSitemaps` (type: `integer`):

How many sitemap files to open, including the ones listed inside a sitemap index.

## `maxDepth` (type: `integer`):

How deep to follow nested sitemap indexes. The protocol allows one level; more than that is unusual.

## `onlyIssues` (type: `boolean`):

Drop the clean rows and keep the errors and warnings. Useful when the sitemap is large and you only want the fix list.

## `siteConcurrency` (type: `integer`):

How many websites to audit in parallel.

## `requestTimeoutSecs` (type: `integer`):

Give up on a single request after this many seconds.

## Actor input object example

```json
{
  "domains": [
    "apify.com"
  ],
  "checkUrlStatus": true,
  "maxUrlsToCheck": 300,
  "urlConcurrency": 8,
  "robotsAgent": "Googlebot",
  "maxUrlsPerSite": 5000,
  "maxSitemaps": 50,
  "maxDepth": 3,
  "onlyIssues": false,
  "siteConcurrency": 4,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/sitemap-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/sitemap-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com"
  ]
}' |
apify call aiqlabs/sitemap-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/sitemap-checker",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qtfbyYhb0blnZdqIF/builds/2XlVriZoDrwZsOxGg/openapi.json