# Sitemap URL Extractor (`ninhothedev/sitemap-url-extractor`) Actor

$0.5/1K 🔥 Fast sitemap URL extractor! Every URL from any site's sitemap.xml & robots.txt — with lastmod & priority. JSON, CSV, Excel or API in seconds. Paste sites & pull thousands of URLs for SEO & crawling ⚡

- **URL**: https://apify.com/ninhothedev/sitemap-url-extractor.md
- **Developed by:** [ninhothedev](https://apify.com/ninhothedev) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor 🗺️

**Extract every URL from any website's `sitemap.xml` and `robots.txt` — fast, cheap, and proxy-free.** Give this Apify Actor a list of domains and it discovers all their sitemaps, recurses through sitemap indexes, unpacks `.gz` sitemaps, and returns a clean, de-duplicated list of every URL together with `lastmod`, `changefreq`, and `priority`.

Perfect for **SEO audits, site crawling, site migrations, and content inventories** — no browser, no residential proxies, runs on the cheapest compute.

***

### ✨ What it does

For every site you provide, the actor:

1. Fetches `/robots.txt` and reads every `Sitemap:` directive.
2. Also probes the common `/sitemap.xml` and `/sitemap_index.xml` paths.
3. For each sitemap: downloads it (Chrome-impersonating fetch with a plain HTTP fallback), and gunzips it automatically when it is `.gz` or gzip-encoded.
4. Recurses `<sitemapindex>` files into their child sitemaps (depth-capped to avoid runaway crawls).
5. Emits **one dataset row per URL**, de-duplicated by URL.

### 📤 Output

One item per discovered URL (JSON):

```json
{
  "source_site": "https://apify.com",
  "url": "https://apify.com/store",
  "lastmod": "2024-05-01",
  "changefreq": "daily",
  "priority": 0.8,
  "from_sitemap": "https://apify.com/sitemap/pages.xml",
  "scraped_at": "2026-07-03T08:00:00+00:00"
}
```

Every field is nullable — sitemaps vary wildly in how much metadata they include, and this actor never guesses.

### ⚙️ Input

| Field | Type | Description |
|-------|------|-------------|
| `siteUrls` | array | Website URLs (or bare domains) to extract from. Default: `apify.com`, `python.org`. |
| `maxItems` | integer | Cap on total URLs extracted across all sites (default `1000`, max `50000`). |

```json
{
  "siteUrls": ["https://www.apify.com", "https://www.python.org"],
  "maxItems": 1000
}
```

### 💰 Pricing

Roughly **$0.50 per 1,000 URLs extracted**, plus a tiny per-input overhead. Because this actor is datacenter-friendly (no residential proxies, no headless browser, 512 MB RAM), it is one of the cheapest ways to pull a full URL inventory of a site. A typical mid-size site (a few thousand URLs) costs a few cents.

### 🎯 Use cases

- **SEO audits** — get the full indexable-URL set a site is advertising to search engines, then compare against what is actually indexed.
- **Crawling seed lists** — feed the URL list straight into a crawler instead of link-discovery crawling from scratch.
- **Site migrations** — snapshot every URL (with `lastmod`/`priority`) before and after a replatform to catch dropped pages and broken redirects.
- **Content audits** — inventory every page, spot stale content via `lastmod`, and prioritize by the site's own `priority` hints.

### 🆚 Why this actor

| | Sitemap URL Extractor | Full website crawler |
|---|---|---|
| Speed | Instant (reads the sitemap) | Slow (follows every link) |
| Cost | ~$0.5 / 1k URLs | Much higher |
| Proxies | None needed | Often residential |
| Coverage | Everything the site publishes | Only what it links |

If a site publishes a sitemap, this is the fastest and cheapest way to enumerate it.

### 🔗 Related actors by ninhothedev

- [SEO On-Page Auditor](https://apify.com/ninhothedev/seo-onpage-auditor)
- [Security Headers Checker](https://apify.com/ninhothedev/security-headers-checker)
- [Website Tech Stack Detector](https://apify.com/ninhothedev/website-tech-stack-detector)
- [Wayback Machine Scraper](https://apify.com/ninhothedev/wayback-machine-scraper)

### 🛟 Support

Questions, edge cases, or a feature request? Open an issue on the actor's Issues tab and it will be handled quickly.

***

**Keywords:** sitemap extractor, sitemap.xml parser, robots.txt sitemap, URL extractor, SEO crawler, site URL inventory, sitemap scraper, extract all URLs from website, sitemap index crawler, content audit tool.

# Actor input Schema

## `siteUrls` (type: `array`):

List of website URLs (or bare domains) to extract sitemap URLs from. For each site the actor reads robots.txt plus /sitemap.xml and /sitemap\_index.xml, then recurses any sitemap indexes.

## `maxItems` (type: `integer`):

Maximum number of URLs to extract across all input sites. Use this to cap cost and runtime on very large sites.

## Actor input object example

```json
{
  "siteUrls": [
    "https://www.apify.com",
    "https://www.python.org"
  ],
  "maxItems": 1000
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrls": [
        "https://www.apify.com",
        "https://www.python.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ninhothedev/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrls": [
        "https://www.apify.com",
        "https://www.python.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("ninhothedev/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrls": [
    "https://www.apify.com",
    "https://www.python.org"
  ]
}' |
apify call ninhothedev/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=ninhothedev/sitemap-url-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FhJPEiCv76LJS7eAY/builds/QhT2O2opOXx31g3Az/openapi.json