# Wayback Site Migration Audit & Archive Scraper (`immense_insulator/internet-archive-data-extractor`) Actor

Export Wayback snapshots, find historical URLs missing after site migrations, or search Archive.org metadata. Structured JSON/CSV with no login, browser, or proxy.

- **URL**: https://apify.com/immense\_insulator/internet-archive-data-extractor.md
- **Developed by:** [xinyao a](https://apify.com/immense_insulator) (community)
- **Categories:** Developer tools, Education, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 dataset items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine & Internet Archive Scraper

Search the Internet Archive catalog **or** export Wayback Machine snapshot-index rows from the public CDX API. Results are normalized for research, monitoring, digital humanities, AI/data pipelines, and archival discovery.

No cookies, login credentials, browser automation, or proxy configuration are required.

### Modes

#### `migrationAudit` — missing-URL recovery for site migrations

Compare the historical Wayback URL inventory with a current URL list or sitemap. The output is a review queue of archived HTML URLs that are absent from the current site inventory, with a replay URL and a concrete `REVIEW_FOR_REDIRECT_OR_CONTENT_RECOVERY` action. This is intended for SEO migrations, redesign QA, and lost-content recovery—not as proof that a redirect is automatically appropriate.

```json
{
  "mode": "migrationAudit",
  "url": "https://example.com/*",
  "currentSitemapUrl": "https://example.com/sitemap.xml",
  "currentUrls": ["https://example.com/landing-page"],
  "fromTimestamp": "2018",
  "maxItems": 500
}
```

The mode reads only successful archived `text/html` CDX records, collapses by canonical URL key, normalizes fragments/trailing slashes, rejects malformed crawler-artifact URLs longer than 2,048 characters, follows at most 20 sitemap documents, scans at most 50,000 historical rows, deduplicates stable historical URLs, and enforces the same global 10,000-result cap. At least one `currentUrls` entry or `currentSitemapUrl` is required.

#### `catalog` (default)

The original behavior is unchanged: search Archive.org's advanced-search catalog and optionally enrich records with item metadata and downloadable-file inventories. Omitting `mode` still selects this mode. `items` is accepted as a legacy alias for `catalog`.

```json
{
  "query": "collection:nasa",
  "mediaType": "texts",
  "yearFrom": 2010,
  "yearTo": 2026,
  "sortBy": "downloads desc",
  "maxItems": 100,
  "includeMetadata": true,
  "includeFiles": true,
  "fileLimit": 25
}
```

#### `snapshots`

Queries `https://web.archive.org/cdx/search/cdx` and emits one dataset row per indexed capture. This mode reads **only the CDX snapshot index**. It does not request, scrape, or return archived page bodies.

```json
{
  "mode": "snapshots",
  "urls": [
    "https://example.com/",
    "https://example.org/news/*"
  ],
  "fromTimestamp": "20200101",
  "toTimestamp": "20241231235959",
  "statusCodes": [200, 301],
  "mimeTypes": ["text/html", "application/json"],
  "collapse": "digest",
  "maxItems": 5000
}
```

Use `url` for one URL, `urls` for a list, or both. Inputs are deduplicated. Only absolute `http://` and `https://` URLs are accepted, with at most 100 input URLs. CDX patterns such as a trailing `*` are passed to the API.

`maxItems` is a global output cap across every URL and page, not a per-URL limit. The hard cap is 10,000 rows. CDX requests use bounded timeouts, four attempts for transient transport/429/5xx failures, a page size of at most 1,000, resume-key pagination, duplicate suppression, and a pagination safety limit.

### Wayback snapshot output

```json
{
  "timestamp": "20240102030405",
  "datetime": "2024-01-02T03:04:05Z",
  "originalUrl": "https://example.com/",
  "archiveUrl": "https://web.archive.org/web/20240102030405/https://example.com/",
  "statusCode": 200,
  "mimeType": "text/html",
  "digest": "ABCDEF123456",
  "length": 12345,
  "sourceUrl": "https://example.com/",
  "observedAt": "2026-07-27T12:00:00Z"
}
```

- `timestamp`: 14-digit CDX capture timestamp in UTC
- `datetime`: the same timestamp as ISO 8601 UTC
- `originalUrl`: URL recorded by the index
- `archiveUrl`: constructed Wayback replay URL; its presence does not guarantee replay access
- `statusCode`, `mimeType`, `digest`, `length`: values reported by CDX; some historical rows may contain missing/null values
- `sourceUrl`: input URL or pattern that produced the row
- `observedAt`: time this Actor observed the index row

Results are available through the Apify dataset as JSON, CSV, Excel, XML, RSS, or via API and integrations.

### Snapshot filters

| Field | Description |
|---|---|
| `url` / `urls` | One URL or up to 100 total absolute HTTP(S) URLs/CDX patterns |
| `fromTimestamp` / `toTimestamp` | Inclusive timestamp prefixes, from `YYYY` through `YYYYMMDDhhmmss` |
| `statusCodes` | Up to 20 HTTP status codes; alternatives are combined into one CDX regex filter |
| `mimeTypes` | Up to 20 simple MIME values or wildcard patterns |
| `collapse` | None, `digest`, `urlkey`, or timestamp grouping at year/month/day/hour/minute/second precision |
| `maxItems` | Global dataset-row cap, 1–10,000 |

`collapse: "digest"` is useful when repeated crawls stored identical content. Collapse is performed by CDX before rows reach the Actor; the Actor additionally removes exact duplicate snapshot rows across pages and overlapping input URLs.

### Catalog output

Catalog rows can contain:

- Internet Archive identifier and canonical item URL
- title, creator, date, year, description, and media type
- downloads, collections, subjects, and languages
- license, public date, and added date
- optional full metadata
- optional downloadable-file inventory with format, size, checksums, and direct URL

Plain-text and Internet Archive advanced-search clauses work in `query`, including `collection:`, `creator:`, `subject:`, and boolean operators.

| Catalog field | Description |
|---|---|
| `query` | Required in catalog/items mode; plain text or advanced-search query |
| `mediaType` | Optional texts, movies, audio, software, image, or web filter |
| `yearFrom` / `yearTo` | Optional inclusive year range |
| `sortBy` | Most downloaded, newest, oldest, or title A-Z |
| `maxItems` | Maximum records, 1–10,000 |
| `includeMetadata` | Fetch the complete metadata object for every record |
| `includeFiles` | Include downloadable file records and checksums |
| `fileLimit` | Maximum file entries included per item |

### Verified Wayback cloud sample

A verified Apify cloud run exported five CDX snapshot rows and registered five `apify-default-dataset-item` charged events:

- Run: `9jEYZjDMVrtP8nKeW`
- Dataset: `NdS3869CAO8pNPHMy`
- [View the Wayback snapshot sample and field guide](https://tools.axinyao.com/wayback-machine-snapshot-history-api)
- [Download sample CSV](https://tools.axinyao.com/static/data-products/internet-archive/wayback-sample.csv)
- [Download sample JSON](https://tools.axinyao.com/static/data-products/internet-archive/wayback-sample.json)

The sample used `https://example.com/`, calendar year 2024, HTTP 200, `text/html`, `collapse=digest`, and `maxItems=5`. Dataset rows retain capture provenance but do not include archived page bodies.

### Source limitations and responsible use

The CDX endpoint is a public Internet Archive service, but availability, response format, rate limits, filtering behavior, and retention are controlled by Internet Archive and can change without notice. It can return `429` or temporary server errors. Large/wildcard queries may be slow or restricted.

CDX is an index, not proof that replay content is currently available. Rows may lag crawls, be incomplete, contain historical crawler metadata, disappear, or be blocked from replay because of exclusions, rights-holder requests, robots policies, legal restrictions, or operational issues. `statusCode` and `mimeType` describe the archived capture record—not the current live URL. `digest` and `length` should not be treated as independently verified content facts. The Actor reports what the public endpoint returned and records `observedAt` for provenance.

Archive.org records and Wayback captures can have different rights and access conditions. An index row, replay URL, or downloadable URL does not grant permission to reuse or redistribute content. Check source terms, licenses, privacy obligations, and applicable law before use. Avoid using broad URL patterns to collect unnecessary personal data.

This Actor is an independent data tool and is not affiliated with or endorsed by Internet Archive.

### Cost and performance

Catalog search uses a paginated public endpoint. Enabling catalog metadata/files adds one request per item. Snapshot mode uses CDX index pages only and never downloads archived bodies. Start with a small `maxItems` and narrow URL/time filters before running broad wildcard queries.

# Actor input Schema

## `mode` (type: `string`):

catalog searches Archive.org items; snapshots returns CDX rows; migrationAudit finds archived HTML URLs absent from currentUrls or a current sitemap; items aliases catalog.

## `query` (type: `string`):

Required in catalog/items mode. Internet Archive advanced-search query; plain text and clauses such as mediatype:texts are supported.

## `url` (type: `string`):

A single http/https URL or CDX URL pattern for snapshots mode, for example https://example.com/\*.

## `urls` (type: `array`):

Optional list of http/https URLs or CDX URL patterns. Combined with url, deduplicated, and limited to 100 entries.

## `currentUrls` (type: `array`):

migrationAudit only. Current absolute URLs; archived URLs absent from this inventory become review candidates. Combined with currentSitemapUrl.

## `currentSitemapUrl` (type: `string`):

migrationAudit only. Optional absolute sitemap or sitemap-index URL. At most 20 sitemap documents and 10,000 current URLs are read.

## `fromTimestamp` (type: `string`):

Optional inclusive CDX timestamp prefix, from YYYY through YYYYMMDDhhmmss (digits only).

## `toTimestamp` (type: `string`):

Optional inclusive CDX timestamp prefix, from YYYY through YYYYMMDDhhmmss (digits only).

## `statusCodes` (type: `array`):

Optional HTTP status codes to include, such as \[200, 301]. Up to 20 values.

## `mimeTypes` (type: `array`):

Optional CDX MIME types to include, such as text/html or application/json. Up to 20 values.

## `collapse` (type: `string`):

Optional server-side CDX collapse. digest removes repeated content; urlkey groups canonical URLs; timestamp:N groups by timestamp prefix length.

## `mediaType` (type: `string`):

Optional catalog mediatype filter.

## `yearFrom` (type: `integer`):

Optional earliest catalog year (inclusive).

## `yearTo` (type: `integer`):

Optional latest catalog year (inclusive).

## `sortBy` (type: `string`):

Choose how matching catalog records are ordered before export.

## `maxItems` (type: `integer`):

Global maximum dataset rows across all URLs or catalog pages. Hard limit: 10,000.

## `includeMetadata` (type: `boolean`):

Catalog mode only. Fetch each item's full metadata record.

## `includeFiles` (type: `boolean`):

Catalog mode only. Include file names, formats, sizes, checksums, and download URLs. Enables full metadata.

## `fileLimit` (type: `integer`):

Catalog mode only. Maximum file entries returned for each item.

## Actor input object example

```json
{
  "mode": "catalog",
  "query": "collection:opensource_movies",
  "collapse": "",
  "mediaType": "",
  "sortBy": "downloads desc",
  "maxItems": 100,
  "includeMetadata": false,
  "includeFiles": false,
  "fileLimit": 100
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("immense_insulator/internet-archive-data-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("immense_insulator/internet-archive-data-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call immense_insulator/internet-archive-data-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=immense_insulator/internet-archive-data-extractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/mARRK4VgChQIGHwki/builds/RhP9RKQll0qqIIKPg/openapi.json