Sitemap Checker & Broken Link Finder

Give it a domain. It finds every sitemap the site publishes, expands the sitemap index, and returns one row per URL — with the broken links, the redirects, the XML mistakes, and the URLs the site's own robots.txt blocks.

Most sitemap actors stop at extracting the list of URLs. Extracting is the easy half. This one checks them.

What it finds

Issue	Severity	Why it costs you
blocked_by_robots	error	The URL is submitted for indexing and blocked from crawling. Google reports this and drops the page.
broken_link / server_error	error	A 4xx or 5xx URL in your sitemap. Wasted crawl budget, and a bad signal about the whole file.
unreachable	error	The URL does not respond at all.
invalid_url / missing_loc	error	A relative path, a ftp:// scheme, or a <url> with no <loc>.
unencoded_space	error	A raw space in a URL. Some parsers stop reading the file at that point.
redirect / redirect_chain	warning	The URL 301s somewhere else. Sitemaps should list the destination, not the old address.
sitemap_redirected	warning	The sitemap file itself redirects — often the apex-to-www hop nobody noticed.
cross_host	warning	The URL is on a host this sitemap is not authorised to cover.
insecure_http	warning	An http:// URL inside an https:// sitemap.
duplicate	warning	The same URL listed twice, across all of the site's sitemaps.
bad_lastmod / lastmod_in_future	warning	Not a W3C datetime, an impossible date such as 2026-02-31, or a date that has not happened yet.
bad_changefreq / bad_priority	warning	Values outside what the protocol allows.
has_fragment / url_too_long	warning	#anchor fragments are dropped; URLs over 2048 characters are truncated.
too_many_urls	error	Over 50,000 URLs in one file, so the file is ignored.
sitemap_parse_error / sitemap_http_error	error	The file is malformed, or it does not load.
no_sitemap_found	error	robots.txt names none and the common paths do not answer.

The one that pays for itself

A URL that sits in your sitemap while your robots.txt blocks it is a contradiction: you are asking Google to index a page you will not let it read. Nothing inside the sitemap reveals it, because the conflict lives in a different file.

It is not a rare mistake. On one well-known SaaS site in our live test suite, 4,008 of 6,278 sitemap URLs were blocked by that site's own robots.txt — two Disallow: lines, quietly cancelling most of the sitemap.

Every blocked URL comes back with the exact rule responsible, e.g. Disallow: /offers$, so the fix is one line either way.

How it finds your sitemaps

1. Read robots.txt and use every Sitemap: line it declares — including sitemaps hosted on another domain, which the protocol allows.
2. If it declares none, try the common paths: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap.xml.gz, /wp-sitemap.xml and two more, keeping the first that really is a sitemap. A 200 response carrying an HTML error page does not count.
3. Expand every sitemap index, recursively, up to the depth you set.
4. Handle .xml.gz by looking at the bytes, not the headers — gzipped sitemaps are usually served without content-encoding, and plenty of readers get this wrong. Plain-text sitemaps work too.

Input

Field	Default	Notes
domains	—	Bare domains or full URLs. One entry may hold several, separated by spaces, commas or newlines, so a pasted spreadsheet column works.
checkUrlStatus	true	Request every URL and report status, hops and destination. Turn off for an XML-only audit, which is much faster.
maxUrlsToCheck	300	Cap on the broken-link requests per site. URLs past the cap are still validated and returned.
robotsAgent	Googlebot	Which crawler the robots.txt rules are read for.
maxUrlsPerSite	5000	Row cap per site.
maxSitemaps	50	How many sitemap files to open.
maxDepth	3	How deep to follow nested indexes.
onlyIssues	false	Drop the clean rows and keep the fix list.
siteConcurrency / urlConcurrency	4 / 8	Sites in parallel, and requests in flight per site.
requestTimeoutSecs	20	Per request.

Output

One row per URL, so the dataset drops straight into a spreadsheet:

{
  "site": "example.com",
  "url": "https://example.com/offers/spring",
  "sitemap": "https://example.com/sitemap.xml",
  "severity": "error",
  "issues": ["blocked_by_robots"],
  "issueDetails": ["robots.txt blocks Googlebot with \"Disallow: /offers/*\""],
  "httpStatus": 200,
  "reachable": true,
  "finalUrl": null,
  "redirectCount": 0,
  "blockedByRobots": true,
  "robotsRule": "Disallow: /offers/*",
  "lastmod": "2026-07-01",
  "changefreq": "weekly",
  "priority": "0.8",
  "images": 2,
  "videos": 0,
  "alternates": 3,
  "sitemapKind": "urlset",
  "checkedAt": "2026-07-29T09:00:00.000Z"
}

Filter on severity for the fix list, or set onlyIssues and get it directly.

A per-site rollup — sitemaps found, URL and issue counts, and every sitemap file with its size, entry count and problems — is written to the key-value store as SUMMARY.

Being a good guest

• Sitemaps are published to be read; this actor reads them and nothing else.
• It does not crawl. It never follows links out of a page, and it never renders JavaScript.
• With checkUrlStatus on it sends one HEAD per URL (falling back to GET only when the server rejects HEAD), never downloading page bodies, and never more than maxUrlsToCheck per site.
• Concurrency is capped and configurable, so you can dial it down for a fragile host.

Limits worth knowing

• robots.txt rules are only applied to URLs on the host that served them. A sitemap covering several hosts is checked against the one robots file we can legitimately attribute.
• lastmod is validated for format and plausibility, not for truthfulness.
• The URL check reports what the server returns. A page that renders an error message with a 200 status is counted as reachable, because it is.
• Sites behind bot protection may answer the sitemap request with a challenge page. Those are reported as no_sitemap_found rather than being guessed at.

Tests

npm test            # 20 unit tests, no network
npm run test:live   # 26 assertions against real sites

The live suite deliberately pins awkward real-world cases: a gzipped sitemap, an apex-to-www redirect, a sitemap cross-submitted from a CDN host, a /sitemap.xml that answers 200 with an HTML page, a domain with no sitemap at all, and a site that blocks thousands of its own sitemap URLs.