# Sitemap Content Extractor (`darknezz/sitemap-content-extractor`) Actor

Crawl any website sitemap.xml and extract structured content from each page. Full-text extraction, metadata, headings, and word counts for SEO audits and content inventories.

- **URL**: https://apify.com/darknezz/sitemap-content-extractor.md
- **Developed by:** [Oaida Adrian](https://apify.com/darknezz) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 site analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Content Extractor — Crawl a Sitemap, Extract Clean Content

Point this Actor at any website's `sitemap.xml` and it crawls every listed URL and returns **clean full-text content plus metadata** for each page — title, meta description, keywords, H1 headings, word count, and last-modified date. Perfect for SEO audits, content inventories, site migrations, and building AI training corpora from documentation sites.

No browser automation to configure, no page-by-page URL lists to maintain — the sitemap *is* the input.

### Why this Actor

- **Sitemap index aware** — handles both plain sitemaps (`<urlset>`) and sitemap indexes (`<sitemapindex>`), recursively following every child sitemap.
- **Gzip support** — reads `.xml.gz` sitemaps transparently.
- **Clean extraction** — uses trafilatura to strip nav/ads/boilerplate and return just the article text.
- **Precise scoping** — include/exclude URL regex patterns so you crawl only `/blog/` or skip `/tag/` pages.
- **Structured output** — one dataset item per page, ready for search indexing, embeddings, or a content spreadsheet.

### How it works

Give it a `sitemapUrl`. The Actor fetches and parses the sitemap (following index files and gzip automatically), applies your include/exclude filters, then visits up to `maxUrls` pages and extracts clean content and metadata from each. Set `extractContent: false` to inventory URLs and metadata only (faster, no page fetches).

### Input

```json
{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "maxUrls": 50,
  "extractContent": true,
  "includePatterns": ["/blog/"],
  "excludePatterns": ["/tag/", "/author/"],
  "proxyConfiguration": { "useApifyProxy": true }
}
```

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `sitemapUrl` | string | Yes | — | URL to `sitemap.xml` or a sitemap index |
| `maxUrls` | integer | No | 50 | Maximum URLs to process |
| `extractContent` | boolean | No | true | Fetch each page and extract full text |
| `includePatterns` | array | No | \[] | Only process URLs matching these regex patterns |
| `excludePatterns` | array | No | \[] | Skip URLs matching these regex patterns |
| `proxyConfiguration` | object | No | Apify Proxy | Proxy settings for page fetches |

### Output

One item per page:

```json
{
  "url": "https://apify.com/blog/web-scraping-guide",
  "title": "The Complete Web Scraping Guide",
  "content": "Web scraping is the process of ...",
  "wordCount": 2184,
  "metaDescription": "Learn web scraping from scratch...",
  "metaKeywords": "web scraping, crawling",
  "h1Headings": ["The Complete Web Scraping Guide"],
  "lastmod": "2026-06-30",
  "extractedAt": "2026-07-18T09:12:44Z"
}
```

### Use cases

- 🔍 **SEO audits** — inventory every indexable page, spot missing titles/meta descriptions, and measure content depth by word count.
- 🚚 **Site migrations** — pull all content from a legacy site into a structured dataset before rebuilding.
- 🤖 **AI training data** — collect clean text from documentation and blog sitemaps to feed LLM fine-tuning or RAG.
- 📚 **Documentation indexing** — build a searchable index of a docs site from its sitemap in one run.
- 🕵️ **Competitor analysis** — map a competitor's content coverage and structure across their whole site.

### Run it from the API

```bash
curl -X POST "https://api.apify.com/v2/acts/darknezz~sitemap-content-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sitemapUrl":"https://apify.com/sitemap.xml","maxUrls":100,"includePatterns":["/blog/"]}'
```

**Scheduling:** attach an Apify Schedule (e.g. daily) to re-crawl a sitemap and keep a content inventory or search index continuously fresh — the `lastmod` field lets you detect which pages changed.

### Pricing

Pay per event: a small fee per extracted page (`page-extracted`), plus Apify's standard platform events. You pay only for the pages you actually crawl — cap spend with `maxUrls`.

### FAQ

**Does it follow sitemap index files?** Yes — nested `<sitemapindex>` files are followed recursively, so a single index URL crawls the whole site.

**Can I crawl only part of a site?** Yes — use `includePatterns` / `excludePatterns` with regex to scope to specific sections (e.g. only `/blog/`, skip `/tag/`).

**What if a site has no sitemap?** This Actor requires a sitemap URL. For arbitrary link-following crawls, use a general web crawler instead.

**Do I need a proxy?** Most sitemaps and pages fetch fine over Apify Proxy (the default). Sites with heavy anti-bot protection may need residential proxies.

**How do I get metadata without full text?** Set `extractContent: false` — you still get title, meta tags, and `lastmod` per URL, much faster.

# Actor input Schema

## `sitemapUrl` (type: `string`):

URL to sitemap.xml or sitemap index file.

## `maxUrls` (type: `integer`):

Maximum URLs to process.

## `extractContent` (type: `boolean`):

Visit each page and extract full text content.

## `includePatterns` (type: `array`):

Only process URLs matching these regex patterns.

## `excludePatterns` (type: `array`):

Skip URLs matching these regex patterns.

## `proxyConfiguration` (type: `object`):

Proxy settings for scraping.

## Actor input object example

```json
{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "maxUrls": 50,
  "extractContent": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `url` (type: `string`):

No description

## `lastmod` (type: `string`):

No description

## `title` (type: `string`):

No description

## `metaDescription` (type: `string`):

No description

## `content` (type: `string`):

No description

## `wordCount` (type: `string`):

No description

## `extractedAt` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("darknezz/sitemap-content-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sitemapUrl": "https://apify.com/sitemap.xml",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("darknezz/sitemap-content-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrl": "https://apify.com/sitemap.xml",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call darknezz/sitemap-content-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=darknezz/sitemap-content-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/hj9j22bTb99RFXhvu/builds/D7pHKVLNfHfz26V6Q/openapi.json