# Sitemap Extractor (`apify/sitemap-extractor`) Actor
This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.
- **URL**: https://apify.com/apify/sitemap-extractor.md
- **Developed by:** [Apify](https://apify.com/apify) (Apify)
- **Categories:** Developer tools, SEO tools, Open source
- **Stats:** 166 total users, 43 monthly users, 45.9% runs succeeded, 5 bookmarks
- **User rating**: 1.65 out of 5 stars
## Pricing
from $0.10 / 1,000 results
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Sitemap Extractor
This Actor is designed to bridge the gap between discovery and crawling. By traversing a website's `sitemap.xml` structure, it compiles a comprehensive list of all published pages and verifies their status before you commit resources to a full-scale scrape.
### Features
- **Recursive Sitemap Discovery:** Automatically detects and traverses nested sitemaps (sitemap indexes).
- **Efficiency:** Uses HTTP HEAD requests for URL validation, which are significantly faster and consume less bandwidth than full GET requests.
- **Proxy Support:** Integrated with Apify Proxy to prevent rate limiting or blocking during the discovery phase.
- **Detailed Output:** Provides the final URL and the corresponding HTTP status code.
### How it Works
1. **Input:** You provide one or more "Start URLs" pointing to the domain name root, sitemaps or sitemap indexes.
2. **Extraction:** The Actor parses the XML, extracting both page URLs and links to further sitemaps.
3. **Validation:** For every page URL found, the Actor performs a status check.
4. **Deduplication:** The crawler uses unique keys to ensure that even if a URL appears in multiple sitemaps, it is only checked once.
### Usage
This Actor is ideal for:
- **Pre-crawling filter:** Generating a "clean" list of URLs for actors like _Website Content Crawler_ or _Web Scraper_.
- **SEO Audits:** Quickly identifying 404 Not Found or 500 Server Error pages listed in your sitemap.
- **Site Mapping:** Getting a high-level overview of a site's architecture.
### Configuration
| Field | Description |
| :---------------------: | :-------------------------------------------------------------: |
| **Start URLs** | Just a domain name or a list of sitemap XML URLs to start from. |
| **Proxy configuration** | Settings for Apify Proxies. |
# Actor input Schema
## `startUrls` (type: `array`):
A static list of domains to scrape.
For details, see the Start URLs section in the README.
## `proxyConfiguration` (type: `object`):
Specifies proxy servers that will be used by the scraper in order to hide its origin.
For details, see Proxy configuration in README.
## `proxyRotation` (type: `string`):
This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.
## `maxCrawlingDepth` (type: `integer`):
Specifies how many sitemap levels the crawler will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers.
If set to 1, will process only the root sitemap. If set to 0, there is no limit.
## `maxRequestRetries` (type: `integer`):
The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by the Page function.
If set to 0, the page will be considered failed right after the first error.
## Actor input object example
```json
{
"startUrls": [
{
"url": "https://docs.apify.com"
}
],
"proxyConfiguration": {
"useApifyProxy": true
},
"proxyRotation": "RECOMMENDED",
"maxCrawlingDepth": 0,
"maxRequestRetries": 3
}
````
# Actor output Schema
## `results` (type: `string`):
Dataset items with discovered URLs and crawl status.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
{
"url": "https://docs.apify.com"
}
],
"proxyConfiguration": {
"useApifyProxy": true
}
};
// Run the Actor and wait for it to finish
const run = await client.actor("apify/sitemap-extractor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"startUrls": [{ "url": "https://docs.apify.com" }],
"proxyConfiguration": { "useApifyProxy": True },
}
# Run the Actor and wait for it to finish
run = client.actor("apify/sitemap-extractor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
{
"url": "https://docs.apify.com"
}
],
"proxyConfiguration": {
"useApifyProxy": true
}
}' |
apify call apify/sitemap-extractor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=apify/sitemap-extractor",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Sitemap Extractor",
"description": "This Apify Actor extracts all URLs from a website's sitemaps and checks their status codes via lightweight HTTP requests. It provides a clean list of valid links, acting as an ideal pre-processor to ensure your larger crawling projects target only active URLs.",
"version": "0.1",
"x-build-id": "dAlDkitAryhKuiust"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/apify~sitemap-extractor/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-apify-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/apify~sitemap-extractor/runs": {
"post": {
"operationId": "runs-sync-apify-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/apify~sitemap-extractor/run-sync": {
"post": {
"operationId": "run-sync-apify-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"startUrls",
"proxyConfiguration"
],
"properties": {
"startUrls": {
"title": "Start URLs",
"type": "array",
"description": "A static list of domains to scrape.
For details, see the Start URLs section in the README.",
"items": {
"type": "object",
"required": [
"url"
],
"properties": {
"url": {
"type": "string",
"title": "URL of a web page",
"format": "uri"
}
}
}
},
"proxyConfiguration": {
"title": "Proxy configuration",
"type": "object",
"description": "Specifies proxy servers that will be used by the scraper in order to hide its origin.
For details, see Proxy configuration in README.",
"default": {
"useApifyProxy": true
}
},
"proxyRotation": {
"title": "Proxy rotation",
"enum": [
"RECOMMENDED",
"PER_REQUEST",
"UNTIL_FAILURE"
],
"type": "string",
"description": "This property indicates the strategy of proxy rotation and can only be used in conjunction with Apify Proxy. The recommended setting automatically picks the best proxies from your available pool and rotates them evenly, discarding proxies that become blocked or unresponsive. If this strategy does not work for you for any reason, you may configure the scraper to either use a new proxy for each request, or to use one proxy as long as possible, until the proxy fails. IMPORTANT: This setting will only use your available Apify Proxy pool, so if you don't have enough proxies for a given task, no rotation setting will produce satisfactory results.",
"default": "RECOMMENDED"
},
"maxCrawlingDepth": {
"title": "Max crawling depth",
"minimum": 0,
"type": "integer",
"description": "Specifies how many sitemap levels the crawler will descend. This value is a safeguard against infinite crawling depths for misconfigured scrapers.
If set to 1, will process only the root sitemap. If set to 0, there is no limit.",
"default": 0
},
"maxRequestRetries": {
"title": "Max request retries",
"minimum": 0,
"type": "integer",
"description": "The maximum number of times the scraper will retry to load each web page on error, in case of a page load error or an exception thrown by the Page function.
If set to 0, the page will be considered failed right after the first error.",
"default": 3
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```