# Wayback Machine URL Extractor - Archived URLs (`logiover/wayback-machine-url-extractor`) Actor Extract every archived URL of any domain from the Internet Archive's Wayback Machine (CDX API). Recover lost or old pages, build redirect maps and run OSINT, with date and status filters. No API key, export to CSV or JSON. - **URL**: https://apify.com/logiover/wayback-machine-url-extractor.md - **Developed by:** [Logiover](https://apify.com/logiover) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 21 total users, 6 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## 🕰️ Wayback Machine URL Extractor — Archived & Historical URLs via the CDX API **Recover every historical URL a website has ever published — straight from the Internet Archive's Wayback Machine.** This **Wayback Machine URL extractor** queries the public **CDX API** to pull the full **archived URL** inventory for any domain — including pages that were deleted, renamed, or lost in a migration. Feed in one domain and get back up to tens of thousands of unique URLs, each with its capture date, archived HTTP status, MIME type, content digest, and a direct Wayback snapshot link. ![Apify Actor](https://img.shields.io/badge/Apify-Actor-00A67E?logo=apify\&logoColor=white) ![No API key](https://img.shields.io/badge/No%20API%20key-required-2ea44f) ![Pay per result](https://img.shields.io/badge/Pricing-Pay%20per%20result-1C7ED6) ![Category](https://img.shields.io/badge/Category-SEO%20%7C%20OSINT-8B5CF6) ![Export](https://img.shields.io/badge/Export-JSON%20%7C%20CSV%20%7C%20Excel-F59E0B) Point it at a single domain and it pulls the entire historical URL footprint automatically — no writing raw CDX queries by hand, no pagination logic, no rate-limit headaches. Whether you're rebuilding a redirect map after a site move, recovering deleted content, doing OSINT on a target's history, or reclaiming lost backlinks, this is the **Internet Archive URL extractor** that does it at scale, one clean row per archived URL, with **no API key and no login**. > ### 🏆 Why this Wayback Machine scraper? > > full historical URL inventory back to 1996 · official Internet Archive **CDX API** · date-range & HTTP-status filters · subdomain / host / prefix matching · direct `web.archive.org` snapshot links · streamed pagination for 100k+ URL domains · **no API key / no login** · export to JSON / CSV / Excel *** ### ✨ What this Actor does - **🕰️ Full historical URL inventory** — pulls every unique URL the Wayback Machine has on record for a domain, going back to 1996. - **🔑 No API key required** — uses the open Internet Archive **CDX API**; no auth, no token, no login. - **🌐 Subdomain, host & prefix matching** — capture a host plus all subdomains and paths, narrow to a single exact host, or match a specific path prefix via `matchType`. - **📅 Date-range filtering** — restrict to snapshots captured between two dates with `fromDate` / `toDate` (`YYYYMMDD`). - **✅ Status-code filtering** — keep only `200 OK` captures and drop dead or redirected ones with `filterStatus`. - **🔗 Direct snapshot links** — every row includes a ready-to-open `web.archive.org/web/...` URL for the exact archived capture. - **🌊 Streamed pagination** — pages through massive result sets with the CDX `resumeKey` mechanism, so memory stays flat even on 100k+ URL domains. - **🔢 Result caps** — set `maxResults` per domain, or `0` for unlimited. - **📋 Multiple domains per run** — process a whole list of domains in one go. - **📤 Export-ready** — JSON, CSV, and Excel output via the Apify Dataset or REST API. ### 🚀 Quick start (3 steps) 1. **Configure** — add one or more domains to **Domains** (e.g. `nasa.gov`, `bbc.com`). Optionally pick a `matchType`, set a date range, filter by status code, or raise `maxResults`. 2. **Run** — click **Start**. The Actor builds the CDX query, pages through the archive, and streams archived URLs into the dataset. 3. **Get your data** — export the archived URL list as JSON, CSV, or Excel from the **Output** / **Dataset** tab, or open any row's `snapshotUrl` to view the archived page. ### 📥 Input Give the Actor at least one domain; every other field is an optional filter. Here are three ready-to-run scenarios: **Scenario A — Full historical inventory of a domain** ```json { "domains": ["nasa.gov"], "matchType": "subdomains", "maxResults": 0, "proxyConfiguration": { "useApifyProxy": true } } ``` **Scenario B — Redirect map: only live (200) pages from a date window** ```json { "domains": ["example.com"], "matchType": "subdomains", "fromDate": "20100101", "toDate": "20201231", "filterStatus": "200", "maxResults": 5000 } ``` **Scenario C — Several domains, exact host only** ```json { "domains": ["bbc.com", "cnn.com"], "matchType": "host", "maxResults": 10000 } ``` | Field | Type | Description | |-------|------|-------------| | `domains` | array | **Required.** One or more domains or full URLs (e.g. `nasa.gov`, `bbc.com`, `https://example.com`). Scheme, `www.`, and paths are normalized automatically — no trailing wildcards. | | `matchType` | enum | How the domain is matched: `subdomains` (host + all subdomains + paths, broadest — default), `host` (exact host only), `domain` (exact domain), `prefix` (URLs starting with a path prefix). | | `fromDate` | string | Optional `YYYYMMDD` lower bound on capture date (e.g. `20100101`). Empty = no lower bound. | | `toDate` | string | Optional `YYYYMMDD` upper bound on capture date (e.g. `20201231`). Empty = no upper bound. | | `filterStatus` | string | Optional — only return captures with this HTTP status (e.g. `200` to exclude dead/redirected captures). | | `maxResults` | integer | Max unique URLs per domain. `0` = unlimited (large sites can yield hundreds of thousands). Default `5000`. | | `proxyConfiguration` | object | Apify Proxy settings, recommended to avoid rate limiting from the Internet Archive. Defaults to Apify Proxy. | ### 📤 Output Every run produces an **Archived URLs** dataset — one row per unique archived URL, deduplicated so each URL appears once rather than once per capture. Here is a trimmed, realistic sample (these are exactly the fields the Actor emits): ```json [ { "domain": "nasa.gov", "url": "http://www.nasa.gov/mission_pages/station/main/index.html", "timestamp": "20120114043915", "capturedAt": "2012-01-14T04:39:15.000Z", "statusCode": "200", "mimeType": "text/html", "digest": "AB23CD45EF67GH89IJ01KL23MN45OP67", "snapshotUrl": "https://web.archive.org/web/20120114043915/http://www.nasa.gov/mission_pages/station/main/index.html" }, { "domain": "nasa.gov", "url": "http://www.nasa.gov/topics/earth/features/index.html", "timestamp": "20150803121044", "capturedAt": "2015-08-03T12:10:44.000Z", "statusCode": "301", "mimeType": "text/html", "digest": "ZZ98YY76XX54WW32VV10UU98TT76SS54", "snapshotUrl": "https://web.archive.org/web/20150803121044/http://www.nasa.gov/topics/earth/features/index.html" } ] ```
📋 Full field reference (click to expand) | Field | Description | |-------|-------------| | `domain` | The normalized domain this URL belongs to | | `url` | The original archived URL | | `timestamp` | Raw 14-digit Wayback capture timestamp (`YYYYMMDDhhmmss`) | | `capturedAt` | ISO 8601 form of the capture timestamp | | `statusCode` | HTTP status the archive recorded for that capture (e.g. `200`, `301`, `404`, or `-`) | | `mimeType` | Content type recorded at capture time (e.g. `text/html`) | | `digest` | Wayback content digest (used internally for de-duplication) | | `snapshotUrl` | Direct link to the archived snapshot on `web.archive.org` | > Some `statusCode` values are `-`: the Wayback index occasionally records captures without a stored status (e.g. revisit records). Those rows are still valid archived URLs.
### 🔍 How it works 1. Each domain you provide is normalized — scheme, `www.`, paths, and wildcards are reduced to a bare host. 2. A **CDX API** query is built from your `matchType`, date range, and status filter, requesting the `original`, `timestamp`, `statuscode`, `mimetype`, and `digest` fields with `collapse=urlkey` so each URL appears **once** instead of returning every capture of it. 3. Results are paged with the CDX `showResumeKey` / `resumeKey` mechanism, and each page is pushed to the dataset in a batch — so even domains with hundreds of thousands of archived URLs stream out without exhausting memory. 4. For every row, a direct `snapshotUrl` is constructed in the `https://web.archive.org/web//` form so you can open the exact archived page. 5. Slow responses, `5xx`, and `429` errors are retried with exponential backoff on a fresh proxy IP — the CDX index can be slow, so retries keep large runs reliable. ### 💡 Use cases - **🔁 SEO migration & redirect maps** — recover lost/old URLs after a site move and rebuild a complete **301 redirect map** so you don't lose link equity. - **📄 Content recovery** — find and restore blog posts, product pages, or docs that were deleted but still live in the archive. - **🕵️ OSINT & research** — enumerate a target domain's historical footprint: old endpoints, removed pages, and forgotten subdomains. - **🔗 Link reclamation** — surface old URLs that still earn backlinks, then redirect them to reclaim the link value. - **🗂️ Finding old endpoints** — reveal admin paths, legacy APIs, and orphaned pages that no longer appear on the live site. - **🏛️ Web archaeology & competitive research** — reconstruct how a competitor's URL structure and content changed across years of snapshots, and build URL/MIME/capture-history datasets for analysis. ### 👥 Who uses it SEO specialists & consultants · web-migration engineers · security researchers & OSINT analysts · penetration testers · digital archivists · content & growth teams · journalists · data scientists building historical web datasets ### 💰 Pricing This Actor uses Apify's **pay-per-result** model — you pay for the archived URLs it returns plus the underlying platform compute, with no monthly subscription. The Internet Archive CDX API itself is free and needs no key; you only pay for the Apify run. Use `maxResults`, date ranges, and status filters to keep large-domain runs lean. See the **Pricing** tab on the Actor's Apify page for exact, up-to-date figures. ### ❓ Frequently Asked Questions #### Is this a Wayback Machine / CDX API alternative? It's a ready-made front end for the **Internet Archive CDX API**. Instead of hand-crafting CDX queries, handling `resumeKey` pagination, and de-duplicating captures yourself, you send a domain and get a clean, structured, export-ready dataset of archived URLs. #### Can I use it without an API key or login? Yes. The Internet Archive CDX API is public and requires **no API key and no login**. You only pay for the Apify platform usage of the run itself. #### How do I get all URLs of a website from the Wayback Machine? Add the domain to **Domains**, leave `matchType` on `subdomains`, set `maxResults` to `0` for everything, and run it. The Actor queries the CDX API and returns one row per unique archived URL. #### Can I find old or deleted pages of a domain? Yes — that's the core use case. The Wayback Machine keeps URLs even after they're removed from the live site, so deleted blog posts, retired product pages, and old endpoints all appear in the results with a `snapshotUrl` to view them. #### How do I export archived URLs to CSV or JSON? Run the Actor, then download the dataset as **CSV, JSON, or Excel** (or pull it via the REST API). Every archived URL is one row, so it drops straight into a spreadsheet or pipeline. #### Is it legal to extract archived URLs? The Actor only reads **publicly available** archive data from the Internet Archive — the same snapshots anyone can browse on web.archive.org. It performs no login and accesses no private data. You remain responsible for using the recovered URLs in line with applicable law and the source's terms. #### How much data can it return? Up to tens of thousands per domain — set `maxResults` to `0` for unlimited. Results stream to the dataset in pages via the CDX `resumeKey`, so even 100k+ URL domains run without memory issues. #### Can I filter by date or HTTP status? Yes — set `fromDate` / `toDate` (`YYYYMMDD`) to restrict to a capture window, and `filterStatus` (e.g. `200`) to keep only captures with a specific HTTP status. #### How do I build a 301 redirect map after a site migration? Extract all archived URLs for the domain with `filterStatus` set to `200`, then map each recovered old URL to its new destination to rebuild link equity. #### Why are some `statusCode` values `-`? The Wayback index sometimes records captures without a stored status code (e.g. revisit records). Those rows are still valid archived URLs. ### 🔗 More website & lead-gen tools by logiover | Actor | What it does | |-------|--------------| | [Sitemap to URL Crawler](https://apify.com/logiover/sitemap-to-url-crawler) | Extract all URLs from any website's sitemap.xml | | [Website Link Graph Crawler](https://apify.com/logiover/website-link-graph-crawler) | Crawl internal links and map a site's full link graph | | [Subdomain Finder](https://apify.com/logiover/subdomain-finder) | Enumerate a domain's subdomains | | [Website SEO Audit Crawler](https://apify.com/logiover/website-seo-audit-crawler) | Full on-page SEO audit across an entire site | | [URL to Markdown](https://apify.com/logiover/url-to-markdown) | Convert any page to clean, RAG-ready Markdown | | [Bulk URL Status Checker](https://apify.com/logiover/bulk-url-status-checker) | Check HTTP status codes for a list of URLs in bulk | | [Broken Link Checker](https://apify.com/logiover/broken-link-checker) | Crawl a site and find dead links with HTTP status codes | | [Certificate Transparency Monitor](https://apify.com/logiover/certificate-transparency-monitor) | Discover hosts/subdomains from CT logs | | [Bulk WHOIS & RDAP Lookup](https://apify.com/logiover/bulk-whois-rdap-lookup) | Bulk domain ownership & registration data | | [Website Contact Scraper](https://apify.com/logiover/website-contact-scraper) | Extract emails, phones, and socials from websites | | [Website Tech Stack Detector](https://apify.com/logiover/website-tech-stack-detector) | Detect the technologies a site is built with | | [Website Change Monitor](https://apify.com/logiover/website-change-monitor) | Detect and track changes to any web page | 👉 Browse all **[logiover scrapers on Apify Store](https://apify.com/logiover)** — 180+ actors across real estate, jobs, crypto, social media & B2B data. ### ⏰ Scheduling & integration - **Schedule** — use Apify **Schedules** to re-run the extractor periodically and track how a domain's archived footprint grows over time. - **Export** — download any run as JSON, CSV, or Excel, or query the dataset via the REST API. - **Automate** — pipe results into **Make**, **n8n**, or **Zapier** to feed recovered URLs into a redirect-map builder, status checker, or Google Sheet. - **API & webhooks** — start runs and pull results with the Apify API, and attach **webhooks** so downstream steps fire automatically when a run finishes. ### ⭐ Support & feedback Hit an issue or want a new filter exposed? Open a ticket on the Actor's **Issues** tab and include the domain, the `matchType` you used, and the volume you expected. If this Actor saves you time, a ★★★★★ review on its Apify Store page is hugely appreciated and helps others find it. ### ⚖️ Legal This Actor reads only **publicly available** archive data served by the Internet Archive's public CDX API — the same snapshots any visitor can browse on web.archive.org. It performs no login and accesses no private data. You are responsible for using the recovered URLs and snapshots in compliance with applicable laws, the Internet Archive's terms, and (where relevant) GDPR. *** ### 📝 Changelog #### 2026-08-01 - Completed the August 2026 full health check: verified empty/programmatic default, Console UI default, and two source-informed alternative inputs on Apify. - Confirmed successful live execution, non-empty structured output, dataset-field/type integrity, and logical sample quality within the 5-minute quality window. - Fixed the run Output link from `{{links.apiDefaultDatasetUrl}}` to `{{links.apiDefaultDatasetUrl}}/items` so the results table opens the dataset items endpoint. - Declared 8 dataset fields from typed live cloud samples so the output contract is no longer an empty placeholder. - Declared 8 nullable dataset fields from typed live cloud samples so the output contract is no longer an empty placeholder or brittle to sparse modes. #### 2026-08-01 — Health-check remediation - August 2026 monthly health-check remediation is in progress; this build contains fixes verified from empty/default, Console-default, targeted live probes, or field-level semantic review. - Fixed the run Output link from `{{links.apiDefaultDatasetUrl}}` to `{{links.apiDefaultDatasetUrl}}/items` so the results table opens the dataset items endpoint. - The final four-input matrix verdict will be appended after post-build cloud revalidation. #### 2026-07-06 - ✨ README overhaul: richer output sample, ready-to-run example scenarios, cross-promo links, and clearer quick-start. #### 2026-07-01 - Maintenance pass: re-verified end-to-end on live data and confirmed successful runs within the 5-minute quality window on the default input. - Sharpened Store metadata (SEO title & description) and expanded the FAQ with high-intent, long-tail questions for easier discovery in Google and Apify Store search. - Added ready-to-run example tasks that cover common real-world use cases. #### 2026-06-15 - Initial release — extract archived URLs from the Wayback Machine CDX API with date/status filters, CSV/JSON export, no API key. # Actor input Schema ## `domains` (type: `array`): One or more domains (or full URLs) to extract archived URLs for. Examples: nasa.gov, bbc.com, https://example.com. Do not include trailing wildcards. Leave empty to run a sensible default domain. ## `matchType` (type: `string`): How the domain is matched against the archive. 'subdomains' = the host and all of its subdomains and paths (broadest, default). 'host' = only the exact host (no subdomains). 'domain' = exact host only. 'prefix' = URLs that start with the given path prefix. ## `fromDate` (type: `string`): Optional. Only return snapshots captured on or after this date. Format: YYYYMMDD (e.g. 20100101). Leave empty for no lower bound. ## `toDate` (type: `string`): Optional. Only return snapshots captured on or before this date. Format: YYYYMMDD (e.g. 20201231). Leave empty for no upper bound. ## `filterStatus` (type: `string`): Optional. Only return URLs whose archived snapshot returned this HTTP status code (e.g. 200 to exclude dead/redirected captures). Choose 'All statuses' for no filter. ## `maxResults` (type: `integer`): Maximum number of unique URLs to extract per domain. Use 0 for unlimited (warning: large sites can yield hundreds of thousands of URLs). ## `proxyConfiguration` (type: `object`): Recommended to avoid rate limiting from the Internet Archive. ## Actor input object example ```json { "domains": [ "nasa.gov" ], "matchType": "subdomains", "filterStatus": "", "maxResults": 5000, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): The dataset containing every archived URL with its capture timestamp, HTTP status, MIME type and Wayback snapshot link. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "nasa.gov" ] }; // Run the Actor and wait for it to finish const run = await client.actor("logiover/wayback-machine-url-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["nasa.gov"] } # Run the Actor and wait for it to finish run = client.actor("logiover/wayback-machine-url-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "nasa.gov" ] }' | apify call logiover/wayback-machine-url-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=logiover/wayback-machine-url-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/acts/wVUfXvYdnQCrkDGFR/builds/EK3R3mxZh9dkbknk6/openapi.json