# Sitemap URL Finder (`thescrapelab/sitemap-target-url-extractor`) Actor

Find and export URLs from any website’s robots.txt and sitemaps. Enter a domain or website URL, optionally filter matching URLs by text, and get clean dataset rows with the URL, domain, path, source sitemap, and match details.

- **URL**: https://apify.com/thescrapelab/sitemap-target-url-extractor.md
- **Developed by:** [Inus Grobler](https://apify.com/thescrapelab) (community)
- **Categories:** Developer tools, SEO tools, Automation
- **Stats:** 5 total users, 4 monthly users, 97.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.05 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Finder

Sitemap URL Finder extracts URLs from website sitemaps and robots.txt for SEO teams, data teams, QA teams, and crawler builders who need a clean URL inventory before a larger crawl.

Enter one or more domains or website URLs. The Actor checks `robots.txt`, discovers sitemap files, follows sitemap indexes, reads XML, plain-text, and gzip sitemaps, removes duplicate URLs, and saves ready-to-export rows in the dataset.

### Use Cases

- Build a URL inventory for SEO audits, migrations, QA checks, or crawl planning.
- Find product, category, blog, documentation, listing, or support URLs before scraping page details.
- Export sitemap URLs with the source sitemap attached for downstream workflows.
- Filter sitemap results to one section, such as `/products/`, `/blog/`, `/docs/`, or `/store/`.
- Prepare URL lists for content crawlers, monitoring, enrichment, RAG ingestion, or lead workflows.

### What Data You Get

Each dataset row contains:

- `url`: URL found in a sitemap.
- `domain`: hostname of the found URL.
- `path`: path part of the URL.
- `sourceUrl`: sitemap or robots-discovered file where the URL was found.
- `sourceDomain`: hostname of the source sitemap.
- `filterType`: `all`, `contains`, or `regex`.
- `filterValue`: text or regular expression used for matching.
- `matchedRegex`: regular expression used, when provided through the API.
- `lastmod`: optional last modified value from the sitemap entry.
- `changefreq`: optional change frequency value from the sitemap entry.
- `priority`: optional priority value from the sitemap entry.
- `matchedAt`: UTC timestamp when the URL was saved.

The run output also links to a summary record with processed sitemap counts, discovered URL counts, saved result count, filter details, and failed request count.

### Input

Use `websites` for normal runs. You can enter domains, homepages, or site sections; the Actor normalizes each value to the website origin and discovers common sitemap locations automatically.

```json
{
  "websites": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "includeUrlText": "/platform/",
  "maxResults": 25
}
```

#### Main Settings

- `websites`: Website homepages or domains to scan. The Actor checks `robots.txt` and `/sitemap.xml` for each website.
- `includeUrlText`: Optional text that found URLs must contain. Leave it empty to save every sitemap URL.
- `maxResults`: Maximum number of URL rows to save.

#### Optional API Settings

- `maxRequestsPerCrawl`: Safety cap for robots.txt and sitemap files fetched in one run.
- `targetUrlRegex`: API-only regular expression filter. It takes precedence over `includeUrlText`.
- `websiteUrls` and `startUrls`: Legacy/API aliases for existing integrations.

### Example Output

```json
{
  "url": "https://docs.apify.com/platform/actors",
  "domain": "docs.apify.com",
  "path": "/platform/actors",
  "sourceUrl": "https://docs.apify.com/sitemap_base.xml",
  "sourceDomain": "docs.apify.com",
  "lastmod": "2026-06-10",
  "changefreq": "weekly",
  "priority": "0.8",
  "filterType": "contains",
  "filterValue": "/platform/",
  "matchedAt": "2026-06-11T13:55:51.865Z"
}
```

### How To Run

1. Open the Actor in Apify Console.
2. Add one or more websites in the Input tab.
3. Optionally set `URL contains` if you only want one site section.
4. Set `Max results` to control dataset size and cost.
5. Start the run and open the Dataset tab when it finishes.

Results are pushed to the dataset while the Actor runs, so partial results can still be useful if a long run is stopped or times out.

### Exporting Results

After a run, open the Dataset tab and export results as JSON, CSV, Excel, XML, RSS, or HTML. API users can read the default dataset from the run response.

### Python API Example

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")

run = client.actor("thescrapelab/sitemap-target-url-extractor").call(run_input={
    "websites": [{"url": "https://docs.apify.com"}],
    "includeUrlText": "/platform/",
    "maxResults": 25,
})

if run is None:
    raise RuntimeError("Actor run failed")

items = client.dataset(run["defaultDatasetId"]).list_items().items

for item in items:
    print(item["url"], item["sourceUrl"])
```

### Limits And Caveats

- The Actor reads sitemap files; it does not crawl every HTML page to discover links.
- Some websites do not publish complete or valid sitemaps.
- Password-protected, blocked, or private sitemaps may return no results.
- Very large sites can contain many sitemap indexes. Use `maxResults` and `maxRequestsPerCrawl` to keep runs predictable.
- Sitemap metadata is included only when the website provides it. Image, video, and alternate-language sitemap extensions are not currently included.

### Troubleshooting

**No results were found.** The website may not publish sitemap URLs, or your `URL contains` filter may be too narrow. Try leaving the filter empty.

**The run finished quickly with failed request counts.** Some sitemap URLs returned permanent errors such as 404. The Actor skips those instead of wasting retries.

**The output has fewer rows than expected.** Check `maxResults`, `maxRequestsPerCrawl`, and any filter value. Also confirm the website sitemap actually lists the URLs you expect.

**The run is slow on a large website.** Keep the default 256 MB memory for most runs, raise `maxResults` gradually, and use `maxRequestsPerCrawl` to keep very large sitemap indexes predictable.

### Pricing

The recommended pricing model is pay per result with a very small Actor start event. This keeps small tests inexpensive and makes larger runs scale with the number of useful URLs returned. Platform usage is low because the Actor uses lightweight HTTP requests instead of a browser and defaults to the 256 MB memory tier.

### FAQ

#### Can this extract all URLs from a sitemap?

Yes. Leave `URL contains` empty and set `Max results` high enough for the website.

#### Can it find sitemap URLs from robots.txt?

Yes. The Actor checks `robots.txt`, follows sitemap directives, and also tries `/sitemap.xml`.

#### Can it parse sitemap indexes?

Yes. It follows nested sitemap indexes until the request or result limits are reached.

#### Does it support gzip sitemaps?

Yes. Gzip-compressed sitemap responses are decompressed before parsing.

#### Can I filter only product or blog URLs?

Yes. Use `URL contains` with a path fragment such as `/products/`, `/blog/`, `/category/`, or `/docs/`.

#### Is this a full website crawler?

No. It extracts URLs listed in sitemaps. Use a full web crawler if you need to discover links from page HTML.

# Actor input Schema

## `websites` (type: `array`):

Website homepages or domains to scan. The Actor automatically checks robots.txt and sitemap.xml for each website.

## `includeUrlText` (type: `string`):

Optional text that found URLs must contain. Leave empty to save every sitemap URL.

## `maxResults` (type: `integer`):

Maximum number of URLs to save. Very large values create large datasets and higher run costs.

## `maxRequestsPerCrawl` (type: `integer`):

Advanced safety cap for robots.txt and sitemap files fetched during one run. Lower it to control cost on very large sites.

## `targetUrlRegex` (type: `string`):

Advanced API filter. If set, only URLs matching this regular expression are saved. This takes precedence over URL contains.

## `websiteUrls` (type: `array`):

Legacy API alias for websites. Use websites in new Console runs.

## `startUrls` (type: `array`):

Legacy/API input for direct robots.txt, sitemap.xml, sitemap index, or plain-text sitemap URLs.

## Actor input object example

```json
{
  "websites": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxResults": 25,
  "maxRequestsPerCrawl": 25,
  "websiteUrls": [],
  "startUrls": []
}
```

# Actor output Schema

## `summary` (type: `string`):

URL to a key-value store record with crawl counts, filter details, and saved match totals.

## `results` (type: `string`):

URL to the default dataset containing useful URL rows with domain, path, source sitemap, and filter details.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        {
            "url": "https://docs.apify.com"
        }
    ],
    "includeUrlText": "",
    "maxResults": 25,
    "maxRequestsPerCrawl": 25,
    "targetUrlRegex": "",
    "websiteUrls": [],
    "startUrls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("thescrapelab/sitemap-target-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websites": [{ "url": "https://docs.apify.com" }],
    "includeUrlText": "",
    "maxResults": 25,
    "maxRequestsPerCrawl": 25,
    "targetUrlRegex": "",
    "websiteUrls": [],
    "startUrls": [],
}

# Run the Actor and wait for it to finish
run = client.actor("thescrapelab/sitemap-target-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "includeUrlText": "",
  "maxResults": 25,
  "maxRequestsPerCrawl": 25,
  "targetUrlRegex": "",
  "websiteUrls": [],
  "startUrls": []
}' |
apify call thescrapelab/sitemap-target-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=thescrapelab/sitemap-target-url-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HVZhEd51Qxs0TAw6F/builds/0XrbhHh7yFYeNdtZQ/openapi.json