# Dead Link Checker & Wayback URL Recovery (`aiqlabs/dead-link-checker`) Actor

Pulls every URL the Internet Archive captured for a site, checks each one against the live site today, and suggests where the dead ones should redirect. Catches the two losses other tools call healthy: 200s that say "not found", and 301s that dump every old path on the home page.

- **URL**: https://apify.com/aiqlabs/dead-link-checker.md
- **Developed by:** [Ai-Q Labs](https://apify.com/aiqlabs) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wayback Machine URL Recovery: Dead URLs & Redirects

Every other Wayback Machine tool hands you a list of URLs the Internet Archive
captured and stops there. The list is the easy half. The work is finding out which of
those URLs are broken **today** and deciding where each one should point.

This Actor does both. It reads the archive's index for your site, requests every
archived URL against the live site, and for each one that is gone it proposes a
destination taken from the site's current sitemap — with the reason it chose that
destination and how confident it is.

### The two losses that look healthy

A redirect audit that only reads status codes will tell you a migration went fine when
it did not. Two failures answer `200` or `301` and are still treated by Google as
missing pages:

| What you see | What it is |
|---|---|
| `301` → `200 OK` | The redirect **discarded the path** and landed on the home page. Every old URL now points at the same place, so none of them carry anything. |
| `200 OK` | The page renders "not found" while returning success. A **soft 404**. |

Both are reported here as losses, with the reason spelled out. This is the whole point
of running the check against the live site rather than reading the archive alone.

Verified on a real site while this was written: `docs.apify.com/academy/tools` and
`docs.apify.com/academy/concepts/querying-css-selectors` both return `301` to
`docs.apify.com/`, which answers `200`. A status checker reports two healthy
redirects. They are two lost pages.

### What it will not do

**It never suggests the home page.** When nothing on the current site matches, the row
says so and the destination is left empty. Sending unmatched URLs to the home page is
the most common migration mistake there is, and it converts a visible 404 into an
invisible one — the link equity is lost either way, but now nothing reports it.

**It never redirects a page that still works.** A page marked `noindex` is invisible to
search but perfectly readable to a person. It is reported as a finding and given no
destination, because moving it would destroy something that is working.

### How a destination is chosen

Four rules, strongest first. The first one that matches wins, so a weak signal can
never outrank a strong one.

1. **The same path, on the current host or scheme.** The `http`→`https` and
   `www`→apex moves that produce most dead URLs. Confidence 0.99.
2. **The same page name, elsewhere on the site.** A reorganised section:
   `/2019/06/widget-guide` → `/guides/widget-guide`. Confidence 0.9–0.95.
   A *generic* name (`pricing`, `about`, `api`, `login`, …) only counts when the
   surrounding path agrees too — otherwise `/someuser/some-actor/pricing` would be
   sent to the company's own price list, which is a real mistake this rule caught.
3. **A similar page name.** Scored on the words the two names share, with at least
   two words in common required. One shared word is a coincidence on any large site.
   Confidence 0.5–0.85.
4. **The section the page belonged to.** Weaker than the page itself, and labelled as
   such, but a genuine destination. Confidence 0.4.

### Output

One row per archived URL:

| Field | |
|---|---|
| `url` | the URL as the archive recorded it |
| `verdict` | `live`, `redirected`, `redirect_to_home`, `soft_404`, `noindex`, `gone`, `blocked`, `server_error`, `unreachable` |
| `lost` | true when the page is gone and a redirect is the right answer |
| `reason` | why, in plain English |
| `currentStatus`, `finalUrl`, `redirectCount`, `redirectChain` | what the live site does now |
| `currentTitle` | the title of whatever is served today |
| `archivedCaptures`, `archivedFirstSeen`, `archivedLastSeen` | how long the archive saw it |
| `waybackUrl` | the last capture, ready to open |
| `suggestedRedirect`, `suggestionBasis`, `suggestionConfidence` | where it should point, and why |
| `issues` | short codes for filtering |

Two key-value records are written as well:

- **`SUMMARY`** — counts per site, including `silentLosses`: the losses that answer 200
  or 301.
- **`REDIRECT_MAP`** — the `from` → `to` pairs alone, ready to paste into a server
  config or a spreadsheet.

### Input

| | |
|---|---|
| **Sites to recover** | `example.com`, or `https://example.com/blog/` with match set to prefix |
| **Archive match** | `domain` (host and subdomains), `host`, `prefix`, `exact` |
| **Archived on or after / before** | optional `yyyyMMdd` range |
| **Archived URLs to check per site** | each costs one request to the live site |
| **Suggest a destination** | off gives you the status check alone |
| **Similarity needed for a name match** | how strict rule 3 is |
| **Only output lost URLs** | makes the dataset the worklist |
| **Respect robots.txt** | on by default; skipped URLs are reported as skipped |

### What it costs the sites involved

One request per archived URL, plus `robots.txt` and the sitemaps once per site. No
crawling, no JavaScript, no browser. Only the first 200 KB of each page is read, which
is far more than enough to see a title, an `h1` and a `noindex`.

### Limits worth knowing

- **The current-page list comes from the site's sitemap.** No sitemap means no
  suggestions, and the run says so rather than inventing them. A sitemap that is out of
  date will produce destinations that are out of date.
- **The archive is not a complete record.** It holds what a crawler happened to reach.
  A page it never captured cannot be recovered here.
- **Only successful HTML captures count.** A URL the archive only ever saw as a 404 is
  not evidence that a page existed.
- **URLs over 2,048 characters are ignored** and counted separately. Those come from
  broken publishing software writing article text into the path; they were never real
  pages and checking them only wastes requests.
- **Confidence is a ranking, not a probability.** Rule 1 is close to certain. Rule 4 is
  a reasonable guess. Read the `suggestionBasis` before applying a map wholesale.

# Actor input Schema

## `domains` (type: `array`):

One host per line (example.com), or a URL prefix to limit the search to part of a site (https://example.com/blog/). A prefix needs "Archive match" set to prefix.

## `matchType` (type: `string`):

domain covers the host and every subdomain, host covers only the exact host, prefix covers one path subtree, exact covers a single URL.

## `archivedFrom` (type: `string`):

Optional yyyyMMdd, for example 20180101. Leave empty to include the whole archive.

## `archivedTo` (type: `string`):

Optional yyyyMMdd, for example 20231231. Leave empty to include the whole archive.

## `maxArchivedUrls` (type: `integer`):

Each one costs a request to the live site, so this is the main control over how long a run takes.

## `suggestRedirects` (type: `boolean`):

Reads the site's current sitemap and matches each dead URL against it. Turn off to get the status check alone.

## `minSimilarity` (type: `number`):

How much of a page name has to match before a rewritten name counts as the same page. Higher is stricter. Exact paths and identical names are matched regardless of this setting.

## `onlyLost` (type: `boolean`):

Skip the archived URLs that still work, so the dataset is the worklist.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site disallows and report them as skipped.

## `robotsAgent` (type: `string`):

Which robots.txt group applies. Googlebot is the useful answer for SEO work.

## `maxConcurrency` (type: `integer`):

How many archived URLs are checked at once. Lower this if the site you are auditing is small or slow.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one response before recording the URL as unreachable.

## Actor input object example

```json
{
  "domains": [
    "docs.apify.com"
  ],
  "matchType": "host",
  "archivedFrom": "",
  "archivedTo": "",
  "maxArchivedUrls": 200,
  "suggestRedirects": true,
  "minSimilarity": 0.5,
  "onlyLost": false,
  "respectRobotsTxt": true,
  "robotsAgent": "Googlebot",
  "maxConcurrency": 8,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `redirectMap` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "docs.apify.com"
    ],
    "matchType": "host",
    "maxArchivedUrls": 200
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiqlabs/dead-link-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": ["docs.apify.com"],
    "matchType": "host",
    "maxArchivedUrls": 200,
}

# Run the Actor and wait for it to finish
run = client.actor("aiqlabs/dead-link-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "docs.apify.com"
  ],
  "matchType": "host",
  "maxArchivedUrls": 200
}' |
apify call aiqlabs/dead-link-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=aiqlabs/dead-link-checker",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0k240LUhCUupuNEI8/builds/f01sMGHBmo3ZHideI/openapi.json