# Dataset Deduplicator & Cleaner (`eszetael_lab/dataset-deduplicator-cleaner`) Actor Deduplicate a dataset: remove duplicates and merge duplicate records with exact, normalized or fuzzy matching. Clean scraped data — emails, phones, URLs, HTML — and get a data quality report. Runs on an Apify dataset ID or inline JSON, so it chains after any scraper. - **URL**: https://apify.com/eszetael\_lab/dataset-deduplicator-cleaner.md - **Developed by:** [Radosław Szal](https://apify.com/eszetael_lab) (community) - **Categories:** Developer tools, AI, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.35 / 1,000 cleaned records This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Dataset Deduplicator & Cleaner > 🔗 Part of the **[Apify actors collection](https://github.com/Eszetael/apify-actors)** — actors that chain: scrape → clean → use. **Deduplicate a dataset, merge duplicate records and clean scraped data in one deterministic run.** Point it at any Apify dataset — or paste JSON straight in — and get back deduplicated, type-consistent records plus a **data quality report**. No scraping, no external APIs, nothing to maintain. **$0.0005 per output record** ($0.50 per 1 000), billed only for what survives deduplication. **Quick start:** paste a JSON array into **Inline data** (or give a **Source dataset ID**), then Run. The prefilled example works as-is. ### What you get back, and how fast **Your own fields, minus the duplicates.** This is a passthrough cleaner: the output records carry the same keys your input had, with values normalized and duplicate rows removed. Nothing is renamed and nothing is invented. Beside the dataset it writes a **quality report** to the key-value store under `QUALITY_REPORT`: input and output counts, duplicates removed, dedup mode, per-field fill rates, and any warnings (truncation, fuzzy degrade). Measured on our last front-door check: **a small batch in 3.5 seconds**. There is no browser and no network fetch here — the work is proportional to record count, and exact dedup is linear where fuzzy matching is not. **$0.50 per 1 000 output records** ($0.0005 each), billed only for what survives deduplication. *** ### What can this dataset cleaner do? - **Deduplicate** — exact, normalized or fuzzy (near-duplicate) matching. - **Merge duplicate records** — when duplicates are found, keep the most complete one rather than an arbitrary survivor. - **Clean scraped data by field type** — emails, phones, URLs, whitespace, HTML, empty values. - **Report quality** — input/output counts, duplicates removed and per-field fill rates, saved as `QUALITY_REPORT` in the key-value store. - **Chain after any scraper** — pass the previous run's dataset ID and it picks up where that Actor left off, without copying the data through your machine. ### Who is it for? | Role | What it solves | |------|----------------| | **Scraper users** | Clean and de-dupe an Actor's output before it reaches a database or spreadsheet. | | **Data / ops teams** | Standardise messy exports — leads, catalogues, contacts — without writing scripts. | | **Analysts** | De-dupe and normalise lists, then check fill rates at a glance. | | **AI pipelines** | Remove near-duplicates before embedding or fine-tuning, where duplicates are expensive noise. | ### How do I deduplicate a dataset? Set `dedupMode`, and `dedupKeys` to the fields that identify a duplicate: ```json { "sourceDatasetId": "abc123XYZ", "dedupMode": "normalized", "dedupKeys": ["email"], "keepStrategy": "most_complete" } ``` | `dedupMode` | Description | |------|-------------| | `none` | Do not deduplicate — clean only. | | `exact` | Byte-identical values. | | `normalized` | Ignores case, whitespace and punctuation — catches `"John Smith"` vs `"john smith!"`. **(default)** | | `fuzzy` | Also merges near-duplicates by string similarity (`similarityThreshold`, 0.5–0.99). Meant for a few thousand rows; on very large or low-diversity inputs it automatically degrades to `normalized` and says so in the run status, so a run never hangs. | Leave `dedupKeys` empty to compare the whole record. When duplicates are found, `keepStrategy` decides which survives: - `most_complete` — the record with the most filled fields *(default)* - `first` / `last` — the first or last occurrence **A guard worth knowing about:** if none of your `dedupKeys` exists in any record, the run **fails** with the field names echoed back, instead of silently treating every record as identical and collapsing your dataset to one row. If fewer than half the records carry a key, it warns. ### How do I clean scraped data? Fields are cleaned by type. With `smartFields` on (default), emails, phones and URLs are detected by field name; everything else is treated as text. | Option | What it does | |--------|--------------| | `cleanEmails` | Lowercase and trim values that look like a valid email. Invalid ones are left as text, **not dropped**. | | `cleanPhones` | Compact phone-like fields to digits, keeping a leading `+`/`00` country prefix. | | `cleanUrls` | Lowercase the scheme and host, drop a trailing slash. | | `stripHtml` | Remove HTML tags and unescape entities (`&` → `&`). | | `emptyToNull` | Turn empty or blank values into `null` for consistency. *(default on)* | | `dropEmptyFields` | Remove empty fields entirely instead of keeping them as `null`. | | `case` | Force plain-text fields to `lower` / `upper` / `title`. | | `coerceTypes` | Convert numeric and `true`/`false` strings into real numbers and booleans. | Whitespace collapse, end-trimming and control-character stripping are always applied to text. ### ⬇️ Input | Field | Type | Default | Description | |---|---|---|---| | `sourceDatasetId` | string | — | ID of an Apify dataset to clean. Use this to chain after a scraper. | | `data` | array | — | Inline JSON records, as an alternative to `sourceDatasetId`. | | `dedupMode` | string | `normalized` | `none`, `exact`, `normalized` or `fuzzy`. | | `dedupKeys` | array | — | Fields identifying a duplicate. Empty = compare the whole record. | | `similarityThreshold` | number | 0.9 | Fuzzy match threshold, 0.5–0.99. Only used when `dedupMode` is `fuzzy`. | | `keepStrategy` | string | `most_complete` | Which duplicate survives: `most_complete`, `first` or `last`. | | `smartFields` | boolean | `true` | Detect emails, phones and URLs by field name. | | `cleanEmails` / `cleanPhones` / `cleanUrls` | boolean | `true` | Per-type cleaning, see above. | | `stripHtml` | boolean | `false` | Remove HTML tags and unescape entities. | | `coerceTypes` | boolean | `false` | Convert numeric and boolean strings into real types. | | `emptyToNull` | boolean | `true` | Turn empty values into `null`. | | `dropEmptyFields` | boolean | `false` | Drop empty fields instead of keeping them as `null`. | | `case` | string | — | Force text fields to `lower`, `upper` or `title`. | | `includeReport` | boolean | `true` | Write `QUALITY_REPORT` to the key-value store. | | **`maxItems`** | integer | 0 (no limit) | Stop after this many output records. **The cheapest way to try the Actor** — you are billed per output record, so `maxItems: 10` costs half a cent. | Provide **either** `sourceDatasetId` **or** `data`. ```json { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", "notes": "

Top client

" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "http://acme.com", "notes": "Top client" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "", "website": "https://globex.io/", "notes": "" } ], "dedupMode": "normalized", "dedupKeys": ["name"], "stripHtml": true } ``` ### ⬆️ Output — what you get back #### 🧾 Cleaned records (the dataset) Deduplicated on `name`, HTML stripped, empty values turned to `null`: ```json [ { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com", "notes": "Top client" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": null, "website": "https://globex.io", "notes": null } ] ``` Three records in, two out, one duplicate merged — and you are charged for the two, not the three. #### 📊 Data quality report (the key-value store) Saved under `QUALITY_REPORT` when `includeReport` is on. It is **not** written to the dataset and **not** charged, so your output stays clean: ```json { "inputRecords": 3, "outputRecords": 2, "duplicatesRemoved": 1, "dedupMode": "normalized", "fieldValuesCleaned": 7, "fieldCount": 5, "fields": { "email": { "filled": 2, "empty": 0, "fillRate": 1.0 }, "phone": { "filled": 1, "empty": 1, "fillRate": 0.5 }, "website": { "filled": 2, "empty": 0, "fillRate": 1.0 }, "notes": { "filled": 1, "empty": 1, "fillRate": 0.5 } } } ``` Fill rates are the fastest way to spot a scraper that quietly stopped populating a field. ### What happens when something fails? | Situation | What the Actor does | |---|---| | Source dataset cannot be read | Fails with the underlying error. | | Input contains no object records | Fails with `Input contains no object (dict) records.` | | `dedupKeys` match no field in any record | **Fails**, echoing the key names — rather than collapsing the whole dataset into one row. Field names are case-sensitive. | | Fewer than half the records carry a dedup key | Succeeds, but warns that dedup may be incomplete. | | Input larger than 50 000 records | Processes the first 50 000 and reports the truncation in both the status and the quality report. Never silently. | | `fuzzy` too expensive for this input | Degrades to `normalized` and says so, instead of hanging. | | Writing a record keeps failing | After 5 consecutive failures the run **aborts** rather than burning runtime, reporting how many were delivered and skipped. | | **Input had records, output has none** | The run **fails** — a run that eats data and returns nothing is not a success. | ### How much does it cost? - **$0.0005 per output record** — $0.50 per 1 000, billed only for records written **after** deduplication. Removing 40% of your rows removes 40% of the bill. - **An empty run costs nothing.** - The quality report is free — it goes to the key-value store, not the dataset. - Priced roughly six times below our scrapers on purpose: this Actor processes data you have **already paid to collect**, so it should not cost the same again. ### FAQ #### How do I chain it after a scraper? Run the scraper, take its dataset ID from the run, and pass it as `sourceDatasetId`. The data never travels through your machine. In an Apify **Task** or **Schedule** you can wire the two together so every scrape is cleaned automatically. #### Can I use it with the Apify API? Yes. `POST /v2/acts/eszetael_lab~dataset-deduplicator-cleaner/runs` with your input as the JSON body. Standard API, CLI and client libraries all work. #### Can I use it through an MCP server? Yes. It is exposed through Apify's Actors MCP server, so an AI agent can call it as a tool — and it is enabled for **agentic payments**, so an agent can run and pay for it without a human in the loop. #### Is the output deterministic? Yes. The same input and the same configuration always produce the same output, in the same order. There is no model, no sampling and no external service in the path. #### How large a dataset can it handle? Up to **50 000 records** per run for `exact` and `normalized`. Larger source datasets are paginated in full and the run tells you if the cap truncated the input. `fuzzy` is meant for a few thousand rows and degrades gracefully above that. #### Does it send my data anywhere? No. No third-party APIs, no browser, no network calls beyond Apify's own storage. That is also why it cannot break when someone else's service changes. #### Your feedback Found a case where dedup merges something it should not, or leaves something it should? Open an issue on the Actor's **Issues** tab. Real failure reports are worth more to us than feature requests. ### Related actors Three tools built to chain into each other — scrape, then clean, then use. This one is the **clean** step; either of the others can feed it a dataset ID directly. - **[Python Web Scraper](https://apify.com/eszetael_lab/reliable-playwright-scraper)** — scrape or crawl any website with your own Python page function, then pipe the dataset here. - **[Bluesky Scraper](https://apify.com/eszetael_lab/bluesky-scraper)** — posts, profiles, threads and full-text search over the AT Protocol, no login required. All three are on pay-per-result pricing, and an empty run costs nothing in every one of them. *** Licensed under the Apache License 2.0. *Keywords: deduplicate dataset, remove duplicates, merge duplicate records, clean scraped data, data cleaning, fuzzy matching, data quality report, normalize emails, dataset cleaner.* # Actor input Schema ## `sourceDatasetId` (type: `string`): Pick an existing Apify dataset to clean — for example the output of another Actor. Use this OR paste records into 'Inline data' below. Declaring it as a resource is what lets this Actor run with limited permissions: it may read the dataset you point at, and nothing else on your account. ## `data` (type: `array`): Records to clean as a JSON array of objects. Use this instead of a source dataset ID for ad-hoc or piped data. ## `dedupMode` (type: `string`): How to detect duplicates. Exact = byte-identical. Normalized = ignores case/whitespace/punctuation. Fuzzy = also merges near-duplicates by similarity. None = only clean, keep every row. ## `dedupKeys` (type: `array`): Field names that identify a duplicate (e.g. email, url). Leave empty to compare the whole record. ## `similarityThreshold` (type: `number`): For fuzzy mode only: 0.5–0.99. Higher = stricter (fewer merges). 0.9 is a good start. ## `keepStrategy` (type: `string`): When duplicates are found, which record survives. ## `smartFields` (type: `boolean`): Auto-detect emails, phones and URLs by field name and clean them accordingly. ## `cleanEmails` (type: `boolean`): Lowercase and trim values in email-like fields (only if they look like a valid address). ## `cleanPhones` (type: `boolean`): Normalize phone-like fields to a compact digits (+country) form. ## `cleanUrls` (type: `boolean`): Lowercase the scheme and host and drop trailing slashes in URL-like fields. ## `stripHtml` (type: `boolean`): Remove HTML tags and unescape entities from text fields. ## `coerceTypes` (type: `boolean`): Convert numeric and true/false strings into real numbers and booleans (AI-ready output). ## `emptyToNull` (type: `boolean`): Turn empty strings and blank values into null for consistency. ## `dropEmptyFields` (type: `boolean`): Remove empty fields from each record entirely instead of keeping them as null. ## `case` (type: `string`): Optionally force the case of plain text fields (emails/phones/URLs are handled separately). ## `includeReport` (type: `boolean`): Save a data-quality report (counts, duplicates removed, per-field fill rates) to the key-value store as QUALITY\_REPORT. ## `maxItems` (type: `integer`): Cap on delivered records (0 = no cap, up to the safety limit of 50000). You are billed per OUTPUT record ($0.0005), so a 5000-record dataset costs about $2.50 — and deduplication lowers it. Leave at 0 unless you want a hard cost ceiling: a cap silently truncates your data. ## Actor input object example ```json { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ], "dedupMode": "normalized", "dedupKeys": [ "email" ], "similarityThreshold": 0.9, "keepStrategy": "most_complete", "smartFields": true, "cleanEmails": true, "cleanPhones": true, "cleanUrls": true, "stripHtml": false, "coerceTypes": false, "emptyToNull": true, "dropEmptyFields": false, "case": "", "includeReport": true, "maxItems": 0 } ``` # Actor output Schema ## `cleanRecords` (type: `string`): The deduplicated, cleaned dataset. Fields mirror your input records. ## `qualityReport` (type: `string`): JSON report under the QUALITY\_REPORT key: input/output counts, duplicates removed, dedup mode, per-field fill rates, and any warnings (truncation, fuzzy degrade). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ], "dedupMode": "normalized", "dedupKeys": [ "email" ], "similarityThreshold": 0.9, "keepStrategy": "most_complete", "includeReport": true }; // Run the Actor and wait for it to finish const run = await client.actor("eszetael_lab/dataset-deduplicator-cleaner").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/", }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com", }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/", }, ], "dedupMode": "normalized", "dedupKeys": ["email"], "similarityThreshold": 0.9, "keepStrategy": "most_complete", "includeReport": True, } # Run the Actor and wait for it to finish run = client.actor("eszetael_lab/dataset-deduplicator-cleaner").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "data": [ { "name": " Acme Corp ", "email": "SALES@ACME.COM", "phone": "+1 555 123 4567", "website": "https://acme.com/" }, { "name": "Acme Corp", "email": "sales@acme.com", "phone": "+15551234567", "website": "https://acme.com" }, { "name": "Globex Inc", "email": "info@globex.io", "phone": "+1 555 987 6543", "website": "https://globex.io/" } ], "dedupMode": "normalized", "dedupKeys": [ "email" ], "similarityThreshold": 0.9, "keepStrategy": "most_complete", "includeReport": true }' | apify call eszetael_lab/dataset-deduplicator-cleaner --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=eszetael_lab/dataset-deduplicator-cleaner", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/JiyEPlaHstuFTOfLT/builds/k4rapfuxv6GxCuGrZ/openapi.json