Dataset Deduplicator & Cleaner

🔗 Part of the Apify actors collection — actors that chain: scrape → clean → use.

Deduplicate a dataset, merge duplicate records and clean scraped data in one deterministic run. Point it at any Apify dataset — or paste JSON straight in — and get back deduplicated, type-consistent records plus a data quality report. No scraping, no external APIs, nothing to maintain.

$0.0005 per output record ($0.50 per 1 000), billed only for what survives deduplication.

Quick start: paste a JSON array into Inline data (or give a Source dataset ID), then Run. The prefilled example works as-is.

What you get back, and how fast

Your own fields, minus the duplicates. This is a passthrough cleaner: the output records carry the same keys your input had, with values normalized and duplicate rows removed. Nothing is renamed and nothing is invented. Beside the dataset it writes a quality report to the key-value store under QUALITY_REPORT: input and output counts, duplicates removed, dedup mode, per-field fill rates, and any warnings (truncation, fuzzy degrade).

Measured on our last front-door check: a small batch in 3.5 seconds. There is no browser and no network fetch here — the work is proportional to record count, and exact dedup is linear where fuzzy matching is not.

$0.50 per 1 000 output records ($0.0005 each), billed only for what survives deduplication.

---

What can this dataset cleaner do?

• Deduplicate — exact, normalized or fuzzy (near-duplicate) matching.
• Merge duplicate records — when duplicates are found, keep the most complete one rather than an arbitrary survivor.
• Clean scraped data by field type — emails, phones, URLs, whitespace, HTML, empty values.
• Report quality — input/output counts, duplicates removed and per-field fill rates, saved as QUALITY_REPORT in the key-value store.
• Chain after any scraper — pass the previous run's dataset ID and it picks up where that Actor left off, without copying the data through your machine.

Who is it for?

Role	What it solves
Scraper users	Clean and de-dupe an Actor's output before it reaches a database or spreadsheet.
Data / ops teams	Standardise messy exports — leads, catalogues, contacts — without writing scripts.
Analysts	De-dupe and normalise lists, then check fill rates at a glance.
AI pipelines	Remove near-duplicates before embedding or fine-tuning, where duplicates are expensive noise.

How do I deduplicate a dataset?

Set dedupMode, and dedupKeys to the fields that identify a duplicate:

{ "sourceDatasetId": "abc123XYZ", "dedupMode": "normalized", "dedupKeys": ["email"], "keepStrategy": "most_complete" }

dedupMode	Description
none	Do not deduplicate — clean only.
exact	Byte-identical values.
normalized	Ignores case, whitespace and punctuation — catches "John Smith" vs "john smith!". (default)
fuzzy	Also merges near-duplicates by string similarity (similarityThreshold, 0.5–0.99). Meant for a few thousand rows; on very large or low-diversity inputs it automatically degrades to normalized and says so in the run status, so a run never hangs.

Leave dedupKeys empty to compare the whole record. When duplicates are found, keepStrategy decides which survives:

• most_complete — the record with the most filled fields (default)
• first / last — the first or last occurrence

A guard worth knowing about: if none of your dedupKeys exists in any record, the run fails with the field names echoed back, instead of silently treating every record as identical and collapsing your dataset to one row. If fewer than half the records carry a key, it warns.

How do I clean scraped data?

Fields are cleaned by type. With smartFields on (default), emails, phones and URLs are detected by field name; everything else is treated as text.

Option	What it does
cleanEmails	Lowercase and trim values that look like a valid email. Invalid ones are left as text, not dropped.
cleanPhones	Compact phone-like fields to digits, keeping a leading +/00 country prefix.
cleanUrls	Lowercase the scheme and host, drop a trailing slash.
stripHtml	Remove HTML tags and unescape entities (& → &).
emptyToNull	Turn empty or blank values into null for consistency. (default on)
dropEmptyFields	Remove empty fields entirely instead of keeping them as null.
case	Force plain-text fields to lower / upper / title.
coerceTypes	Convert numeric and true/false strings into real numbers and booleans.

Whitespace collapse, end-trimming and control-character stripping are always applied to text.

⬇️ Input

Field	Type	Default	Description
sourceDatasetId	string	—	ID of an Apify dataset to clean. Use this to chain after a scraper.
data	array	—	Inline JSON records, as an alternative to sourceDatasetId.
dedupMode	string	normalized	none, exact, normalized or fuzzy.
dedupKeys	array	—	Fields identifying a duplicate. Empty = compare the whole record.
similarityThreshold	number	0.9	Fuzzy match threshold, 0.5–0.99. Only used when dedupMode is fuzzy.
keepStrategy	string	most_complete	Which duplicate survives: most_complete, first or last.
smartFields	boolean	true	Detect emails, phones and URLs by field name.
cleanEmails / cleanPhones / cleanUrls	boolean	true	Per-type cleaning, see above.
stripHtml	boolean	false	Remove HTML tags and unescape entities.
coerceTypes	boolean	false	Convert numeric and boolean strings into real types.
emptyToNull	boolean	true	Turn empty values into null.
dropEmptyFields	boolean	false	Drop empty fields instead of keeping them as null.
case	string	—	Force text fields to lower, upper or title.
includeReport	boolean	true	Write QUALITY_REPORT to the key-value store.
maxItems	integer	0 (no limit)	Stop after this many output records. The cheapest way to try the Actor — you are billed per output record, so maxItems: 10 costs half a cent.

Provide either sourceDatasetId or data.

{
  "data": [
    { "name": "  Acme Corp  ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", "notes": "<p>Top client</p>" },
    { "name": "Acme Corp",     "email": "sales@acme.com", "phone": "+15551234567",     "website": "http://acme.com",  "notes": "Top client" },
    { "name": "Globex Inc",    "email": "info@globex.io", "phone": "",                 "website": "https://globex.io/", "notes": "" }
  ],
  "dedupMode": "normalized",
  "dedupKeys": ["name"],
  "stripHtml": true
}

⬆️ Output — what you get back

🧾 Cleaned records (the dataset)

Deduplicated on name, HTML stripped, empty values turned to null:

[
  { "name": "Acme Corp",  "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com",  "notes": "Top client" },
  { "name": "Globex Inc", "email": "info@globex.io", "phone": null,           "website": "https://globex.io", "notes": null }
]

Three records in, two out, one duplicate merged — and you are charged for the two, not the three.

📊 Data quality report (the key-value store)

Saved under QUALITY_REPORT when includeReport is on. It is not written to the dataset and not charged, so your output stays clean:

{
  "inputRecords": 3,
  "outputRecords": 2,
  "duplicatesRemoved": 1,
  "dedupMode": "normalized",
  "fieldValuesCleaned": 7,
  "fieldCount": 5,
  "fields": {
    "email":   { "filled": 2, "empty": 0, "fillRate": 1.0 },
    "phone":   { "filled": 1, "empty": 1, "fillRate": 0.5 },
    "website": { "filled": 2, "empty": 0, "fillRate": 1.0 },
    "notes":   { "filled": 1, "empty": 1, "fillRate": 0.5 }
  }
}

Fill rates are the fastest way to spot a scraper that quietly stopped populating a field.

What happens when something fails?

Situation	What the Actor does
Source dataset cannot be read	Fails with the underlying error.
Input contains no object records	Fails with Input contains no object (dict) records.
dedupKeys match no field in any record	Fails, echoing the key names — rather than collapsing the whole dataset into one row. Field names are case-sensitive.
Fewer than half the records carry a dedup key	Succeeds, but warns that dedup may be incomplete.
Input larger than 50 000 records	Processes the first 50 000 and reports the truncation in both the status and the quality report. Never silently.
fuzzy too expensive for this input	Degrades to normalized and says so, instead of hanging.
Writing a record keeps failing	After 5 consecutive failures the run aborts rather than burning runtime, reporting how many were delivered and skipped.
Input had records, output has none	The run fails — a run that eats data and returns nothing is not a success.

How much does it cost?

• $0.0005 per output record — $0.50 per 1 000, billed only for records written after deduplication. Removing 40% of your rows removes 40% of the bill.
• An empty run costs nothing.
• The quality report is free — it goes to the key-value store, not the dataset.
• Priced roughly six times below our scrapers on purpose: this Actor processes data you have already paid to collect, so it should not cost the same again.

FAQ

How do I chain it after a scraper?

Run the scraper, take its dataset ID from the run, and pass it as sourceDatasetId. The data never travels through your machine. In an Apify Task or Schedule you can wire the two together so every scrape is cleaned automatically.

Can I use it with the Apify API?

Yes. POST /v2/acts/eszetael_lab~dataset-deduplicator-cleaner/runs with your input as the JSON body. Standard API, CLI and client libraries all work.

Can I use it through an MCP server?

Yes. It is exposed through Apify's Actors MCP server, so an AI agent can call it as a tool — and it is enabled for agentic payments, so an agent can run and pay for it without a human in the loop.

Is the output deterministic?

Yes. The same input and the same configuration always produce the same output, in the same order. There is no model, no sampling and no external service in the path.

How large a dataset can it handle?

Up to 50 000 records per run for exact and normalized. Larger source datasets are paginated in full and the run tells you if the cap truncated the input. fuzzy is meant for a few thousand rows and degrades gracefully above that.

Does it send my data anywhere?

No. No third-party APIs, no browser, no network calls beyond Apify's own storage. That is also why it cannot break when someone else's service changes.

Your feedback

Found a case where dedup merges something it should not, or leaves something it should? Open an issue on the Actor's Issues tab. Real failure reports are worth more to us than feature requests.

Related actors

Three tools built to chain into each other — scrape, then clean, then use. This one is the clean step; either of the others can feed it a dataset ID directly.

• Python Web Scraper — scrape or crawl any website with your own Python page function, then pipe the dataset here.
• Bluesky Scraper — posts, profiles, threads and full-text search over the AT Protocol, no login required.

All three are on pay-per-result pricing, and an empty run costs nothing in every one of them.

---

Licensed under the Apache License 2.0.

Keywords: deduplicate dataset, remove duplicates, merge duplicate records, clean scraped data, data cleaning, fuzzy matching, data quality report, normalize emails, dataset cleaner.