# Internet Archive Scraper β€” Wayback + Search + Metadata \[$1.5πŸ’°] (`memo23/internet-archive-scraper`) Actor Internet Archive all-in-one scraper β€” paste a URL or domain for Wayback snapshot history, a keyword for library search, or an item ID for full metadata + file lists. Date filters, availability checks, flat JSON rows. Pure HTTP via open APIs, no browser, no proxies. $1.50/1k rows. - **URL**: https://apify.com/memo23/internet-archive-scraper.md - **Developed by:** [Muhamed Didovic](https://apify.com/memo23) (community) - **Categories:** SEO tools, Agents, AI - **Stats:** 17 total users, 15 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.50 / 1,000 archive rows This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** β€” the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** β€” [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** β€” the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** β€” the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** β€” the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Internet Archive Scraper β€” Wayback Machine + Library Search + Item Metadata

Internet Archive Scraper logo

The fastest way to turn archive.org into structured data. Paste a URL or domain for its full **Wayback Machine snapshot history**, a keyword to search the **Internet Archive library** (books, movies, software, music), or an item ID for **complete metadata + file lists** β€” every query is auto-classified, no mode switches, no configuration. ### Why Use This Scraper? - βœ… Four data sources in one actor β€” Wayback snapshots, library search, item metadata, closest-snapshot lookups - βœ… Paste anything: URLs, bare domains, keywords, item IDs β€” each line auto-routed to the right API - βœ… Date-range filters for both snapshot history and library search - βœ… Flat JSON rows ready for CSV / spreadsheet export β€” no nested-array digging - βœ… No proxies needed and none charged β€” archive.org's APIs are open, so runs are fast and cheap - βœ… Duplicate-snapshot collapsing for clean domain-history timelines ### Overview The Internet Archive Scraper is built for SEO specialists, researchers, digital archivists, and developers who need structured data from archive.org without stitching together four different APIs. The output is a stream of flat rows, and the row shape follows the query type: a domain query yields **snapshot rows**, a keyword yields **search-result rows**, an item ID yields one **item row**, and a `snapshotOf:` lookup yields one **availability row**. All rows carry a `type` discriminator and echo the originating `query`, so mixed runs stay easy to split downstream. ### Supported Inputs #### Query types (auto-detected per line) | Query type | Pattern | Example | Rows produced | |---|---|---|---| | Domain history | bare domain | `example.com` | one row per Wayback snapshot of the whole host | | URL history | full URL | `https://example.com/page` | one row per snapshot of that exact URL | | Path prefix | trailing `/*` | `example.com/blog/*` | snapshots of everything under the path | | Wayback URL | `web.archive.org/web/...` | `https://web.archive.org/web/2020/https://example.com` | snapshot history of the embedded URL | | Library search | free text | `nasa apollo mission` | one row per matching archive.org item | | Item metadata | item URL or `item:` prefix | `https://archive.org/details/InformationM` or `item:InformationM` | one metadata row with the full file list | | Closest snapshot | `snapshotOf:URL[@YYYYMMDD]` | `snapshotOf:example.com@20200601` | one availability row with the nearest archived copy | #### Copy-pasteable input ```json { "queries": [ "example.com", "nasa apollo mission", "item:InformationM", "snapshotOf:example.com@20200601" ], "maxItems": 200 } ``` #### Unsupported inputs - ❌ Content that was excluded from the Wayback Machine by the site owner (robots-excluded domains return no snapshots) - ❌ Borrow-only / lending-library book content (metadata is returned, but not restricted file downloads) - ❌ Full-text passages from inside books or videos β€” rows describe items and files, not their inner content ### Use Cases | Audience | Use case | |---|---| | **SEO teams** | Audit a domain's history before buying it; recover content from expired domains; track competitor site changes over the years | | **Researchers** | Bulk-export library holdings on a topic β€” books, films, recordings β€” with dates, collections, and download counts | | **Legal / compliance teams** | Document what a page said on a specific date with a citable snapshot URL | | **Digital archivists** | Inventory item collections with full metadata and file-level detail (formats, sizes, checksums) | | **Developers** | Feed snapshot timelines or library metadata into apps without writing four API clients | ### How It Works ![How the Internet Archive Scraper works](https://raw.githubusercontent.com/muhamed-didovic/muhamed-didovic.github.io/main/assets/how-it-works-internet-archive.png) 1. **Input** β€” add any mix of URLs, domains, keywords, item IDs, or `snapshotOf:` lookups, one per line 2. **Classify** β€” each line is auto-routed: URL/domain β†’ Wayback CDX index, free text β†’ library search, item ID β†’ metadata API, `snapshotOf:` β†’ availability API 3. **Fetch** β€” the actor calls archive.org's open APIs over plain HTTP with polite concurrency, retries, and back-off 4. **Filter** β€” date ranges, mediatype, sort order, and duplicate-collapsing are applied server-side where the API supports it 5. **Output** β€” flat, typed JSON rows land in the dataset, exportable as JSON, CSV, Excel, or XML ### Input Configuration #### Input fields | Field | Type | Required | Notes | |---|---|---|---| | `queries` | `array` | yes | One query per line; each auto-classified (see Supported Inputs) | | `from` | `string` (date) | optional | Range start; snapshot timestamps in CDX mode, `publicdate` in search mode. Defaults to `1996-01-01` | | `to` | `string` (date) | optional | Range end; empty = up to now. Also the target date for `snapshotOf:` without `@date` | | `mediatype` | `string` enum | optional | Search mode only: `texts`, `movies`, `audio`, `image`, `software`, `data`, `web`, `collection`, `etree` | | `collapseSnapshots` | `boolean` | optional | CDX mode only: collapse consecutive identical-content snapshots. Default `false` | | `searchSort` | `string` enum | optional | Search mode only: `downloads desc/asc`, `publicdate desc/asc`, `date desc/asc`, `titleSorter asc`. Empty = relevance | | `maxItems` | `integer` | optional | Global cap on rows across all queries. Default `100` | | `maxConcurrency` | `integer` | optional | Parallel queries, default `2`, hard cap `5` β€” archive.org is a nonprofit, be kind | #### Common scenarios **1. Full snapshot history of a domain, deduplicated** ```json { "queries": ["example.com"], "collapseSnapshots": true, "maxItems": 1000 } ``` **2. Most-downloaded NASA images added since 2020** ```json { "queries": ["nasa"], "mediatype": "image", "from": "2020-01-01", "searchSort": "downloads desc", "maxItems": 500 } ``` **3. What did a page look like on a given date?** ```json { "queries": ["snapshotOf:cnn.com@20080915"] } ``` ### Output Overview Every row is flat (no nested objects except small arrays like `collections`) and carries: - **`type`** β€” `snapshot`, `searchResult`, `item`, or `availability` - **`query`** β€” the input line that produced the row, for easy grouping - **Snapshot rows** β€” timestamp, ISO date, original URL, ready-to-open `snapshotUrl`, MIME type, HTTP status, content digest, size - **Search-result rows** β€” identifier, item URL, title, mediatype, creator, dates, downloads, size, collections, subjects, plus `totalFound` for the whole query - **Item rows** β€” full metadata (title, description, publisher, license, collections) plus file count and per-file download URLs, formats, sizes, and MD5 checksums - **Availability rows** β€” whether an archived copy exists near the requested date, and the closest snapshot's URL, timestamp, and status ### Output Samples #### Domain query β†’ snapshot row ```jsonc { "type": "snapshot", "query": "example.com", "urlkey": "com,example)/", "timestamp": "20020120142510", "snapshotDate": "2002-01-20T14:25:10Z", "originalUrl": "http://example.com:80/", "snapshotUrl": "https://web.archive.org/web/20020120142510/http://example.com:80/", "mimeType": "text/html", "statusCode": 200, "digest": "HT2DYGA5UKZCPBSFVCV3JOBXGW2G5UUA", "contentLength": 1792, "scrapedAt": "2026-07-17T10:01:03.171Z" } ``` #### Keyword query β†’ search-result row ```jsonc { "type": "searchResult", "query": "nasa", "identifier": "prgd14higzbtwbvzrdeclelzpz0xjxbaagvlkcaj", "itemUrl": "https://archive.org/details/prgd14higzbtwbvzrdeclelzpz0xjxbaagvlkcaj", "title": "Small Steps, Giant Leaps: Episode 25, NASA Technology Transfer", "mediatype": "audio", "creator": "Small Steps, Giant Leaps", "date": "2019-12-10T00:00:00Z", "publicDate": "2023-01-16T20:23:02Z", "downloads": 1, "itemSizeBytes": 18627727, "collections": ["podcasts_mirror_bobarchives", "podcasts_mirror"], "subjects": ["Podcast", "APPEL", "NASA", "projectmanagement"], "totalFound": 78788, "scrapedAt": "2026-07-17T08:37:45.951Z" } ``` #### Item query β†’ metadata row (trimmed) ```jsonc { "type": "item", "query": "item:InformationM", "identifier": "InformationM", "itemUrl": "https://archive.org/details/InformationM", "title": "Information Machine, The", "mediatype": "movies", "description": "Applies graphic sensitivity to medium in cartoon form, and traces the history of storing and analyzing information…", "publisher": "Eames (Charles And Ray)", "date": "1958", "licenseUrl": "http://creativecommons.org/licenses/publicdomain/", "collections": ["prelinger"], "itemSizeBytes": 684174158 /* + filesCount and files[] with per-file name, format, size, md5, downloadUrl */ } ``` #### `snapshotOf:` query β†’ availability row ```jsonc { "type": "availability", "query": "snapshotOf:example.com@20200601", "url": "example.com", "requestedTimestamp": "20200601", "available": true, "snapshotUrl": "https://web.archive.org/web/20200601235352/https://example.com/", "snapshotTimestamp": "20200601235352", "snapshotStatus": "200" } ``` ### Key Output Fields #### Common - `type`, `query`, `scrapedAt` #### Snapshot (CDX) - `timestamp`, `snapshotDate`, `originalUrl`, `snapshotUrl` - `mimeType`, `statusCode`, `digest`, `contentLength` #### Search result - `identifier`, `itemUrl`, `title`, `mediatype`, `creator`, `description` - `date`, `year`, `publicDate`, `addedDate` - `downloads`, `itemSizeBytes`, `collections[]`, `subjects[]`, `totalFound` #### Item metadata - `identifier`, `title`, `description`, `publisher`, `licenseUrl`, `language` - `filesCount`, `files[]` β€” per-file `name`, `format`, `size`, `md5`, `downloadUrl` #### Availability - `available`, `requestedTimestamp`, `snapshotUrl`, `snapshotTimestamp`, `snapshotStatus` ### FAQ #### Do I need proxies? No. The actor uses archive.org's open, unauthenticated APIs β€” there is no anti-bot wall, so nothing about proxies appears in the input and no proxy costs are added to your runs. #### How does the actor know which mode to use? Each query line is classified by shape: anything URL- or domain-like goes to the Wayback CDX index, `item:` prefixes and archive.org item URLs go to the metadata API, `snapshotOf:` goes to the availability API, and everything else is a library search. You can mix all four in one run. #### Why did my `snapshotOf:` lookup return `available: false` for a page I know is archived? Archive.org's availability endpoint is occasionally flaky β€” the identical query can return empty once and succeed minutes later, both with HTTP 200. Re-run the query, or use the domain/URL history mode (CDX) instead, which is reliable and returns every snapshot rather than just the closest one. #### How many results can a library search return? Standard pagination is capped by archive.org at around 10,000 results per query. For bigger harvests, split the query with `from`/`to` date ranges. `totalFound` on each row tells you the full match count. #### What does `maxItems` limit exactly? It is a global budget across all queries in the run, not per query. If one domain's snapshot history fills the cap, later queries in the list are skipped β€” the run log shows how many queries were processed. #### Can I download the actual files or page content? Rows include ready-to-use URLs (`snapshotUrl` for archived pages, per-file `downloadUrl` for item files), but the actor deliberately does not download file contents β€” it delivers the structured index so you fetch only what you need. #### Can I scrape borrow-only books or private collections? No. The actor reads public metadata and public snapshot indexes only. Restricted-access content (lending-library books, darked items) is out of scope. #### Why do some snapshot rows have `statusCode: null`? Those are `warc/revisit` records β€” the Wayback crawler noted the content was unchanged rather than storing a new copy. The `digest` still identifies the content version. ### Support - For issues or feature requests, please use the **Issues** tab on the actor's Apify Console page. - Author's website: - Email: ### Additional Services - Need deep pagination beyond 10k results, extra CDX filters (status code, MIME type), or scheduled domain-history monitoring? Drop a line β€” custom builds are quick on this codebase. - Custom output shapes, private builds, or direct API access (no Apify fee, just a usage fee): ### Explore More Scrapers If you found this useful, you might also like: - [**Trustpilot Scraper**](https://apify.com/memo23/trustpilot-scraper-ppe) β€” company reviews at scale for reputation research - [**Glassdoor Scraper**](https://apify.com/memo23/glassdoor-scraper-ppr) β€” employer reviews, salaries, and interview data - [**VRBO Scraper**](https://apify.com/memo23/vrbo-scraper) β€” vacation-rental properties with calendar, rates, and full review history Full list at [apify.com/memo23](https://apify.com/memo23). *** ### ⚠️ Disclaimer This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by the Internet Archive (a 501(c)(3) non-profit) or any of its projects, including the Wayback Machine. All trademarks mentioned are the property of their respective owners. The scraper accesses only publicly available data via archive.org's open APIs β€” no authenticated endpoints, no restricted-access collections, and no content behind the archive.org login. Users are responsible for ensuring their use complies with archive.org's Terms of Use, the licenses attached to individual items, applicable data-protection law (GDPR, CCPA, etc.), and any contractual obligations of their own organization. Please keep concurrency low β€” the Internet Archive is a nonprofit library. *** ### SEO Keywords internet archive scraper, archive.org scraper, wayback machine scraper, scrape wayback machine, wayback machine API, archive.org API, CDX API scraper, website history scraper, domain history checker, snapshot history export, expired domain research, old website content recovery, archive.org search export, internet archive metadata, digital library data, web archive data extraction, SEO domain audit, historical website data, Apify internet archive, wayback snapshots CSV # Actor input Schema ## `queries` (type: `array`): One query per line. Examples: `example.com` (snapshot history of a domain), `https://example.com/page` (history of one URL), `nasa` (library search), `https://archive.org/details/InformationM` or `item:InformationM` (item metadata + files), `snapshotOf:example.com@20200601` (closest snapshot to a date). ## `archives` (type: `array`): Which web archives to pull snapshot history from. **Wayback Machine** (archive.org) is the default and gives the richest metadata. **archive.today** is a separate mirror that often has pages Wayback missed. **Memento aggregator** (TimeTravel) returns snapshots from 20+ member archives at once. Multiple = merged, each row tagged with its `archive`. Applies to URL/domain queries only (library search & item lookups are archive.org). Tip: enable a proxy in the Proxy section for archive.today / Memento β€” those mirrors rate-limit per IP. ## `from` (type: `string`): Start of the date range. Applies to snapshot timestamps in CDX mode and to `publicdate` in search mode. Defaults to 1996-01-01 (the Wayback Machine's first crawls), i.e. no effective lower bound. ## `to` (type: `string`): End of the date range. Applies to snapshot timestamps in CDX mode and to `publicdate` in search mode. Also used as the default target date for `snapshotOf:` lookups without an explicit `@date`. Leave empty for 'up to now'. ## `mediatype` (type: `string`): Restrict library-search results to one archive.org mediatype. Ignored in CDX / metadata / availability modes. ## `collapseSnapshots` (type: `boolean`): When enabled, consecutive Wayback snapshots with identical content (same digest) are collapsed into one row. Ignored in other modes. ## `searchSort` (type: `string`): Sort for library-search results. Default (empty) is relevance. ## `maxItems` (type: `integer`): Hard cap on total dataset rows across all queries. Always enforced β€” CDX requests carry an explicit limit so large domains stay fast. ## `maxConcurrency` (type: `integer`): How many queries are processed in parallel. archive.org is a nonprofit β€” keep this low (hard-capped at 5). ## `proxy` (type: `object`): Optional proxy. Default off. When enabled, retries rotate a fresh IP to ride out per-IP rate limits from archive.org or aggressive archive mirrors. Apify Residential is recommended when you do enable it. ## Actor input object example ```json { "queries": [ "example.com", "nasa", "item:InformationM", "snapshotOf:example.com@20200601" ], "archives": [ "waybackmachine" ], "from": "1996-01-01", "collapseSnapshots": false, "maxItems": 100, "maxConcurrency": 2 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "example.com", "nasa", "item:InformationM", "snapshotOf:example.com@20200601" ], "from": "1996-01-01" }; // Run the Actor and wait for it to finish const run = await client.actor("memo23/internet-archive-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": [ "example.com", "nasa", "item:InformationM", "snapshotOf:example.com@20200601", ], "from": "1996-01-01", } # Run the Actor and wait for it to finish run = client.actor("memo23/internet-archive-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "example.com", "nasa", "item:InformationM", "snapshotOf:example.com@20200601" ], "from": "1996-01-01" }' | apify call memo23/internet-archive-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=memo23/internet-archive-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/9J9RcEthCpKX3xDx3/builds/x4oDQZIiRhlJEm4aW/openapi.json