Wayback Machine URL Recovery: Dead URLs & Redirects

Every other Wayback Machine tool hands you a list of URLs the Internet Archive captured and stops there. The list is the easy half. The work is finding out which of those URLs are broken today and deciding where each one should point.

This Actor does both. It reads the archive's index for your site, requests every archived URL against the live site, and for each one that is gone it proposes a destination taken from the site's current sitemap — with the reason it chose that destination and how confident it is.

The two losses that look healthy

A redirect audit that only reads status codes will tell you a migration went fine when it did not. Two failures answer 200 or 301 and are still treated by Google as missing pages:

What you see	What it is
301 → 200 OK	The redirect discarded the path and landed on the home page. Every old URL now points at the same place, so none of them carry anything.
200 OK	The page renders "not found" while returning success. A soft 404.

Both are reported here as losses, with the reason spelled out. This is the whole point of running the check against the live site rather than reading the archive alone.

Verified on a real site while this was written: docs.apify.com/academy/tools and docs.apify.com/academy/concepts/querying-css-selectors both return 301 to docs.apify.com/, which answers 200. A status checker reports two healthy redirects. They are two lost pages.

What it will not do

It never suggests the home page. When nothing on the current site matches, the row says so and the destination is left empty. Sending unmatched URLs to the home page is the most common migration mistake there is, and it converts a visible 404 into an invisible one — the link equity is lost either way, but now nothing reports it.

It never redirects a page that still works. A page marked noindex is invisible to search but perfectly readable to a person. It is reported as a finding and given no destination, because moving it would destroy something that is working.

How a destination is chosen

Four rules, strongest first. The first one that matches wins, so a weak signal can never outrank a strong one.

1. The same path, on the current host or scheme. The http→https and www→apex moves that produce most dead URLs. Confidence 0.99.
2. The same page name, elsewhere on the site. A reorganised section: /2019/06/widget-guide → /guides/widget-guide. Confidence 0.9–0.95. A generic name (pricing, about, api, login, …) only counts when the surrounding path agrees too — otherwise /someuser/some-actor/pricing would be sent to the company's own price list, which is a real mistake this rule caught.
3. A similar page name. Scored on the words the two names share, with at least two words in common required. One shared word is a coincidence on any large site. Confidence 0.5–0.85.
4. The section the page belonged to. Weaker than the page itself, and labelled as such, but a genuine destination. Confidence 0.4.

Output

One row per archived URL:

Field
url	the URL as the archive recorded it
verdict	live, redirected, redirect_to_home, soft_404, noindex, gone, blocked, server_error, unreachable
lost	true when the page is gone and a redirect is the right answer
reason	why, in plain English
currentStatus, finalUrl, redirectCount, redirectChain	what the live site does now
currentTitle	the title of whatever is served today
archivedCaptures, archivedFirstSeen, archivedLastSeen	how long the archive saw it
waybackUrl	the last capture, ready to open
suggestedRedirect, suggestionBasis, suggestionConfidence	where it should point, and why
issues	short codes for filtering

Two key-value records are written as well:

• SUMMARY — counts per site, including silentLosses: the losses that answer 200 or 301.
• REDIRECT_MAP — the from → to pairs alone, ready to paste into a server config or a spreadsheet.

Input

Sites to recover	example.com, or https://example.com/blog/ with match set to prefix
Archive match	domain (host and subdomains), host, prefix, exact
Archived on or after / before	optional yyyyMMdd range
Archived URLs to check per site	each costs one request to the live site
Suggest a destination	off gives you the status check alone
Similarity needed for a name match	how strict rule 3 is
Only output lost URLs	makes the dataset the worklist
Respect robots.txt	on by default; skipped URLs are reported as skipped

What it costs the sites involved

One request per archived URL, plus robots.txt and the sitemaps once per site. No crawling, no JavaScript, no browser. Only the first 200 KB of each page is read, which is far more than enough to see a title, an h1 and a noindex.

Limits worth knowing

• The current-page list comes from the site's sitemap. No sitemap means no suggestions, and the run says so rather than inventing them. A sitemap that is out of date will produce destinations that are out of date.
• The archive is not a complete record. It holds what a crawler happened to reach. A page it never captured cannot be recovered here.
• Only successful HTML captures count. A URL the archive only ever saw as a 404 is not evidence that a page existed.
• URLs over 2,048 characters are ignored and counted separately. Those come from broken publishing software writing article text into the path; they were never real pages and checking them only wastes requests.
• Confidence is a ranking, not a probability. Rule 1 is close to certain. Rule 4 is a reasonable guess. Read the suggestionBasis before applying a map wholesale.