Sitemap URL Extractor 🗺️

Extract every URL from any website's sitemap.xml and robots.txt — fast, cheap, and proxy-free. Give this Apify Actor a list of domains and it discovers all their sitemaps, recurses through sitemap indexes, unpacks .gz sitemaps, and returns a clean, de-duplicated list of every URL together with lastmod, changefreq, and priority.

Perfect for SEO audits, site crawling, site migrations, and content inventories — no browser, no residential proxies, runs on the cheapest compute.

---

✨ What it does

For every site you provide, the actor:

1. Fetches /robots.txt and reads every Sitemap: directive.
2. Also probes the common /sitemap.xml and /sitemap_index.xml paths.
3. For each sitemap: downloads it (Chrome-impersonating fetch with a plain HTTP fallback), and gunzips it automatically when it is .gz or gzip-encoded.
4. Recurses <sitemapindex> files into their child sitemaps (depth-capped to avoid runaway crawls).
5. Emits one dataset row per URL, de-duplicated by URL.

📤 Output

One item per discovered URL (JSON):

{
  "source_site": "https://apify.com",
  "url": "https://apify.com/store",
  "lastmod": "2024-05-01",
  "changefreq": "daily",
  "priority": 0.8,
  "from_sitemap": "https://apify.com/sitemap/pages.xml",
  "scraped_at": "2026-07-03T08:00:00+00:00"
}

Every field is nullable — sitemaps vary wildly in how much metadata they include, and this actor never guesses.

⚙️ Input

Field	Type	Description
siteUrls	array	Website URLs (or bare domains) to extract from. Default: apify.com, python.org.
maxItems	integer	Cap on total URLs extracted across all sites (default 1000, max 50000).

{
  "siteUrls": ["https://www.apify.com", "https://www.python.org"],
  "maxItems": 1000
}

💰 Pricing

Roughly $0.50 per 1,000 URLs extracted, plus a tiny per-input overhead. Because this actor is datacenter-friendly (no residential proxies, no headless browser, 512 MB RAM), it is one of the cheapest ways to pull a full URL inventory of a site. A typical mid-size site (a few thousand URLs) costs a few cents.

🎯 Use cases

• SEO audits — get the full indexable-URL set a site is advertising to search engines, then compare against what is actually indexed.
• Crawling seed lists — feed the URL list straight into a crawler instead of link-discovery crawling from scratch.
• Site migrations — snapshot every URL (with lastmod/priority) before and after a replatform to catch dropped pages and broken redirects.
• Content audits — inventory every page, spot stale content via lastmod, and prioritize by the site's own priority hints.

🆚 Why this actor

	Sitemap URL Extractor	Full website crawler
Speed	Instant (reads the sitemap)	Slow (follows every link)
Cost	~$0.5 / 1k URLs	Much higher
Proxies	None needed	Often residential
Coverage	Everything the site publishes	Only what it links

If a site publishes a sitemap, this is the fastest and cheapest way to enumerate it.

🔗 Related actors by ninhothedev

• SEO On-Page Auditor
• Security Headers Checker
• Website Tech Stack Detector
• Wayback Machine Scraper

🛟 Support

Questions, edge cases, or a feature request? Open an issue on the actor's Issues tab and it will be handled quickly.

---

Keywords: sitemap extractor, sitemap.xml parser, robots.txt sitemap, URL extractor, SEO crawler, site URL inventory, sitemap scraper, extract all URLs from website, sitemap index crawler, content audit tool.