Wayback Machine & Internet Archive Scraper

Search the Internet Archive catalog or export Wayback Machine snapshot-index rows from the public CDX API. Results are normalized for research, monitoring, digital humanities, AI/data pipelines, and archival discovery.

No cookies, login credentials, browser automation, or proxy configuration are required.

Modes

migrationAudit — missing-URL recovery for site migrations

Compare the historical Wayback URL inventory with a current URL list or sitemap. The output is a review queue of archived HTML URLs that are absent from the current site inventory, with a replay URL and a concrete REVIEW_FOR_REDIRECT_OR_CONTENT_RECOVERY action. This is intended for SEO migrations, redesign QA, and lost-content recovery—not as proof that a redirect is automatically appropriate.

{
  "mode": "migrationAudit",
  "url": "https://example.com/*",
  "currentSitemapUrl": "https://example.com/sitemap.xml",
  "currentUrls": ["https://example.com/landing-page"],
  "fromTimestamp": "2018",
  "maxItems": 500
}

The mode reads only successful archived text/html CDX records, collapses by canonical URL key, normalizes fragments/trailing slashes, rejects malformed crawler-artifact URLs longer than 2,048 characters, follows at most 20 sitemap documents, scans at most 50,000 historical rows, deduplicates stable historical URLs, and enforces the same global 10,000-result cap. At least one currentUrls entry or currentSitemapUrl is required.

catalog (default)

The original behavior is unchanged: search Archive.org's advanced-search catalog and optionally enrich records with item metadata and downloadable-file inventories. Omitting mode still selects this mode. items is accepted as a legacy alias for catalog.

{
  "query": "collection:nasa",
  "mediaType": "texts",
  "yearFrom": 2010,
  "yearTo": 2026,
  "sortBy": "downloads desc",
  "maxItems": 100,
  "includeMetadata": true,
  "includeFiles": true,
  "fileLimit": 25
}

snapshots

Queries https://web.archive.org/cdx/search/cdx and emits one dataset row per indexed capture. This mode reads only the CDX snapshot index. It does not request, scrape, or return archived page bodies.

{
  "mode": "snapshots",
  "urls": [
    "https://example.com/",
    "https://example.org/news/*"
  ],
  "fromTimestamp": "20200101",
  "toTimestamp": "20241231235959",
  "statusCodes": [200, 301],
  "mimeTypes": ["text/html", "application/json"],
  "collapse": "digest",
  "maxItems": 5000
}

Use url for one URL, urls for a list, or both. Inputs are deduplicated. Only absolute http:// and https:// URLs are accepted, with at most 100 input URLs. CDX patterns such as a trailing * are passed to the API.

maxItems is a global output cap across every URL and page, not a per-URL limit. The hard cap is 10,000 rows. CDX requests use bounded timeouts, four attempts for transient transport/429/5xx failures, a page size of at most 1,000, resume-key pagination, duplicate suppression, and a pagination safety limit.

Wayback snapshot output

{
  "timestamp": "20240102030405",
  "datetime": "2024-01-02T03:04:05Z",
  "originalUrl": "https://example.com/",
  "archiveUrl": "https://web.archive.org/web/20240102030405/https://example.com/",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "ABCDEF123456",
  "length": 12345,
  "sourceUrl": "https://example.com/",
  "observedAt": "2026-07-27T12:00:00Z"
}

• timestamp: 14-digit CDX capture timestamp in UTC
• datetime: the same timestamp as ISO 8601 UTC
• originalUrl: URL recorded by the index
• archiveUrl: constructed Wayback replay URL; its presence does not guarantee replay access
• statusCode, mimeType, digest, length: values reported by CDX; some historical rows may contain missing/null values
• sourceUrl: input URL or pattern that produced the row
• observedAt: time this Actor observed the index row

Results are available through the Apify dataset as JSON, CSV, Excel, XML, RSS, or via API and integrations.

Snapshot filters

Field	Description
url / urls	One URL or up to 100 total absolute HTTP(S) URLs/CDX patterns
fromTimestamp / toTimestamp	Inclusive timestamp prefixes, from YYYY through YYYYMMDDhhmmss
statusCodes	Up to 20 HTTP status codes; alternatives are combined into one CDX regex filter
mimeTypes	Up to 20 simple MIME values or wildcard patterns
collapse	None, digest, urlkey, or timestamp grouping at year/month/day/hour/minute/second precision
maxItems	Global dataset-row cap, 1–10,000

collapse: "digest" is useful when repeated crawls stored identical content. Collapse is performed by CDX before rows reach the Actor; the Actor additionally removes exact duplicate snapshot rows across pages and overlapping input URLs.

Catalog output

Catalog rows can contain:

• Internet Archive identifier and canonical item URL
• title, creator, date, year, description, and media type
• downloads, collections, subjects, and languages
• license, public date, and added date
• optional full metadata
• optional downloadable-file inventory with format, size, checksums, and direct URL

Plain-text and Internet Archive advanced-search clauses work in query, including collection:, creator:, subject:, and boolean operators.

Catalog field	Description
query	Required in catalog/items mode; plain text or advanced-search query
mediaType	Optional texts, movies, audio, software, image, or web filter
yearFrom / yearTo	Optional inclusive year range
sortBy	Most downloaded, newest, oldest, or title A-Z
maxItems	Maximum records, 1–10,000
includeMetadata	Fetch the complete metadata object for every record
includeFiles	Include downloadable file records and checksums
fileLimit	Maximum file entries included per item

Verified Wayback cloud sample

A verified Apify cloud run exported five CDX snapshot rows and registered five apify-default-dataset-item charged events:

• Run: 9jEYZjDMVrtP8nKeW
• Dataset: NdS3869CAO8pNPHMy
• View the Wayback snapshot sample and field guide
• Download sample CSV
• Download sample JSON

The sample used https://example.com/, calendar year 2024, HTTP 200, text/html, collapse=digest, and maxItems=5. Dataset rows retain capture provenance but do not include archived page bodies.

Source limitations and responsible use

The CDX endpoint is a public Internet Archive service, but availability, response format, rate limits, filtering behavior, and retention are controlled by Internet Archive and can change without notice. It can return 429 or temporary server errors. Large/wildcard queries may be slow or restricted.

CDX is an index, not proof that replay content is currently available. Rows may lag crawls, be incomplete, contain historical crawler metadata, disappear, or be blocked from replay because of exclusions, rights-holder requests, robots policies, legal restrictions, or operational issues. statusCode and mimeType describe the archived capture record—not the current live URL. digest and length should not be treated as independently verified content facts. The Actor reports what the public endpoint returned and records observedAt for provenance.

Archive.org records and Wayback captures can have different rights and access conditions. An index row, replay URL, or downloadable URL does not grant permission to reuse or redistribute content. Check source terms, licenses, privacy obligations, and applicable law before use. Avoid using broad URL patterns to collect unnecessary personal data.

This Actor is an independent data tool and is not affiliated with or endorsed by Internet Archive.

Cost and performance

Catalog search uses a paginated public endpoint. Enabling catalog metadata/files adds one request per item. Snapshot mode uses CDX index pages only and never downloads archived bodies. Start with a small maxItems and narrow URL/time filters before running broad wildcard queries.