# Superclean Dedupe (`superlativetech/superclean-dedupe`) Actor

Deduplicate lead records using fuzzy matching. Exact email/phone matching, Jaro-Winkler for names, token-set ratio for companies. Smart field auto-detection. Input: array of record objects. Output: clustered records with isCanonical, matchScore, matchReasons. Batch or instant pair API.

- **URL**: https://apify.com/superlativetech/superclean-dedupe.md
- **Developed by:** [Superlative](https://apify.com/superlativetech) (community)
- **Categories:** Automation, Lead generation, Open source
- **Stats:** 3 total users, 1 monthly users, 100.0% runs succeeded, 2 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Superclean Dedupe

Deduplicate lead records using fuzzy matching. Find and merge duplicates across company names, person names, emails, phone numbers, and more.

### What does Superclean Dedupe do?

Superclean Dedupe finds duplicate records in your lead data using multi-field fuzzy matching — not just exact string comparison.

- **Exact matching** — Normalized email, phone, and domain matching catches obvious duplicates
- **Fuzzy name matching** — Jaro-Winkler algorithm matches "Jim Smith" with "James Smith", "McDonald" with "Mcdonald"
- **Fuzzy company matching** — Token-set ratio matches "ACME Corp" with "Acme Corporation", handles word-order differences
- **Smart auto-detection** — Automatically detects which fields to match on (email, name, company, phone, domain, etc.)
- **Transitive clustering** — If A matches B and B matches C, all three are grouped together
- **Canonical selection** — Picks the most complete record as the "winner" in each duplicate cluster
- **[Instant pair comparison](#standby-mode-instant-api)** — Sub-second comparison of two records via Standby HTTP API

### Use with AI Agents

Available as an MCP tool via the [Apify MCP Server](https://mcp.apify.com).

| Property | Value |
| --- | --- |
| Actor ID | `superlativetech/superclean-dedupe` |
| Standby URL | `https://superlativetech--superclean-dedupe.apify.actor` |
| Input | `records` (object\[]) or `a` + `b` (objects, Standby pair comparison) |
| Output | `{id, record, clusterId, clusterSize, isCanonical, duplicateOf, matchScore, matchReasons, confidence}` per record |
| Pricing | Free — you only pay standard Apify platform usage |
| Idempotent | Yes — same input always produces same output |

#### Output schema

```json
{ "id": 1, "record": {"name": "Jim Smith", "email": "jim@acme.com"}, "clusterId": "c_1", "clusterSize": 2, "isCanonical": true, "duplicateOf": null, "matchScore": null, "matchReasons": [], "confidence": 1 }
```

#### Pipeline composability

This actor works in lead enrichment pipelines:

1. **Scrape** → 2. **Clean** (Superclean actors) → 3. **Dedupe** (this actor) → 4. **Enrich** (DNS/WHOIS) → 5. **Score** (ICP Scorer)

**Pro tip:** Clean names and companies with Superclean actors *first*, then dedupe. Normalized values produce much better match results.

#### Standby (instant pair comparison)

```
GET https://superlativetech--superclean-dedupe.apify.actor?token=TOKEN&a={"name":"Jim Smith","email":"jim@acme.com"}&b={"name":"James Smith","email":"jim@acme.com"}
```

### What else can Superclean do?

If you're cleaning lead data, you might also need:

- **[Superclean Company Names](https://apify.com/superlativetech/superclean-company-names?fpr=8e9l1)** — Clean messy company names for cold emails and CRM
- **[Superclean Person Names](https://apify.com/superlativetech/superclean-person-names?fpr=8e9l1)** — Clean person names for cold email personalization
- **[Superclean Emails](https://apify.com/superlativetech/superclean-emails?fpr=8e9l1)** — Validate emails, fix typos, detect disposable providers
- **[Superclean Job Titles](https://apify.com/superlativetech/superclean-job-titles?fpr=8e9l1)** — Normalize job titles for lead scoring and personalization
- **[Superclean Phone Numbers](https://apify.com/superlativetech/superclean-phone-numbers?fpr=8e9l1)** — Format and validate phone numbers
- **[Superclean URLs](https://apify.com/superlativetech/superclean-urls?fpr=8e9l1)** — Clean and normalize URLs from lead data
- **[Superclean Places](https://apify.com/superlativetech/superclean-places?fpr=8e9l1)** — Parse and normalize location data from lead exports
- **[Superlead ICP Scorer](https://apify.com/superlativetech/superlead-icp-scorer?fpr=8e9l1)** — Score leads against your Ideal Customer Profile with AI

### Why deduplicate lead data?

Your lead data comes from many sources with duplicate problems:

- Same person scraped from LinkedIn, Apollo, and a website — three records
- "ACME Corp" from one source, "Acme Corporation, Inc." from another
- "jim@acme.com" and "jim+linkedin@acme.com" — same person, different emails
- CRM imports from multiple campaigns with overlapping prospects

Duplicate records mean:

- **Wasted outreach** — Same prospect gets the same email twice
- **Inflated metrics** — Pipeline looks bigger than it is
- **CRM pollution** — Duplicate contacts create confusion for sales reps
- **Higher costs** — Enrichment and scoring costs multiply with duplicates

### How to use Superclean Dedupe

1. Paste your lead records as JSON into the input field
2. Click **Start** and download your deduplicated results
3. Filter to `isCanonical === true` to get the deduplicated set

Each record is compared against potential matches using multi-field fuzzy scoring, then grouped into clusters of duplicates.

### How matching works

#### Smart field auto-detection

If you don't specify `matchFields`, the actor automatically detects fields from your records:

| Field | Method | Weight | Notes |
| --- | --- | --- | --- |
| `email` | Exact (normalized) | 1.0 | Strips plus addressing, Gmail dots |
| `phone` | Exact (digits only) | 0.9 | Strips all formatting |
| `name` / `fullName` | Jaro-Winkler | 0.8 | Best for short strings |
| `firstName` / `lastName` | Jaro-Winkler | 0.5 each | Combined when both present |
| `company` / `companyName` | Token-set ratio | 0.7 | Handles word-order differences |
| `domain` / `website` | Exact (extracted) | 0.7 | Ignores free email domains |
| `city` | Exact | 0.3 | Supplementary signal |
| `title` / `jobTitle` | Token-set ratio | 0.3 | Weak signal, helps disambiguation |

#### Custom match fields

Override auto-detection with explicit field configuration:

```json
{
  "matchFields": [
    { "field": "email", "method": "exact", "weight": 1.0 },
    { "field": "name", "method": "jaro-winkler", "weight": 0.8 },
    { "field": "company", "method": "token-set", "weight": 0.6 }
  ]
}
```

Available methods: `exact`, `jaro-winkler`, `levenshtein`, `token-set`

#### Threshold

The `threshold` parameter (default 0.85) controls how similar two records must be to be considered duplicates:

- **0.95** — Very strict. Only near-identical records merge. Minimizes false positives.
- **0.85** — Balanced default. Catches most real duplicates with few false positives.
- **0.70** — Aggressive. Catches more duplicates but may over-merge.
- **0.50** — Very aggressive. Only use for exploration or when you'll manually review.

When in doubt, start strict (0.90) and lower if you're missing duplicates.

### How many records can you deduplicate?

The actor handles up to 100,000 records per run. Multi-pass blocking avoids the O(n^2) comparison problem — for 10,000 records, only ~100K-300K comparisons are needed instead of 50 million.

For best performance, clean your data with Superclean actors first. Normalized names and companies produce dramatically better match results.

### How much will it cost you?

This Actor is **free to use** — there are no per-result charges. You only pay for the Apify platform compute your runs consume, which is typically negligible.

### Input parameters

| Parameter | Type | Description |
| --- | --- | --- |
| `records` | array | Array of record objects to deduplicate |
| `record` | object | Single record — API shorthand. If both `record` and `records` are provided, `record` is prepended to the list |
| `threshold` | number | Similarity threshold for dedup (0-1, default 0.85) |
| `matchFields` | array | Optional field configuration (auto-detected if omitted) |

#### Input example

```json
{
  "records": [
    { "name": "Jim Smith", "email": "jim@acme.com", "company": "ACME Corp" },
    { "name": "James Smith", "email": "jim@acme.com", "company": "Acme Corporation" },
    { "name": "Jane Doe", "email": "jane@example.com", "company": "Example Inc" },
    { "name": "J. Doe", "email": "jane@example.com", "company": "Example" },
    { "name": "Bob Wilson", "email": "bob@other.com", "company": "Other LLC" }
  ]
}
```

### During the Actor run

The Actor processes records in four stages:

1. **Auto-detect** — Scan record keys and select matching fields/methods
2. **Block** — Multi-pass blocking groups candidate pairs (avoids comparing every pair)
3. **Score** — Weighted multi-field similarity scoring for each candidate pair
4. **Cluster** — Union-Find groups transitive matches; select canonical records

You'll see progress logs as records are processed. Results are available in real-time.

### Output format

Results are saved to the default dataset. Each input record appears in the output, annotated with its cluster membership.

#### Output example

```json
[
  {
    "id": 1,
    "record": { "name": "Jim Smith", "email": "jim@acme.com", "company": "ACME Corp" },
    "clusterId": "c_1",
    "clusterSize": 2,
    "isCanonical": true,
    "duplicateOf": null,
    "matchScore": null,
    "matchReasons": [],
    "confidence": 1
  },
  {
    "id": 2,
    "record": { "name": "James Smith", "email": "jim@acme.com", "company": "Acme Corporation" },
    "clusterId": "c_1",
    "clusterSize": 2,
    "isCanonical": false,
    "duplicateOf": 1,
    "matchScore": 0.92,
    "matchReasons": ["email_exact", "name_jaro-winkler:0.87", "company_token-set:0.91"],
    "confidence": 0.92
  },
  {
    "id": 5,
    "record": { "name": "Bob Wilson", "email": "bob@other.com", "company": "Other LLC" },
    "clusterId": "c_3",
    "clusterSize": 1,
    "isCanonical": true,
    "duplicateOf": null,
    "matchScore": null,
    "matchReasons": [],
    "confidence": 1
  }
]
```

| Field | Description |
| --- | --- |
| `id` | Row number (1-based, matches input order) |
| `record` | Original input record object |
| `clusterId` | Cluster identifier — records in the same cluster are duplicates |
| `clusterSize` | Number of records in this cluster (1 = unique, >1 = has duplicates) |
| `isCanonical` | True if this is the "best" (most complete) record in the cluster |
| `duplicateOf` | ID of the canonical record this is a duplicate of (null if canonical) |
| `matchScore` | Composite similarity score against the canonical record (0-1) |
| `matchReasons` | Which fields matched and their individual scores |
| `confidence` | Overall confidence in the duplicate match (0-1) |

#### How to get the deduplicated set

Filter the output to `isCanonical === true`:

```javascript
const deduped = results.filter(r => r.isCanonical);
```

### Standby mode (instant pair comparison)

Standby mode provides instant comparison of two records without running a full batch. Ideal for real-time dedup in agent workflows, Clay enrichment steps, and form validation.

#### Standby URL

```
https://superlativetech--superclean-dedupe.apify.actor?token=YOUR_API_TOKEN
```

#### Compare two records

```bash
curl "https://superlativetech--superclean-dedupe.apify.actor?token=TOKEN&a=%7B%22name%22%3A%22Jim+Smith%22%2C%22email%22%3A%22jim%40acme.com%22%7D&b=%7B%22name%22%3A%22James+Smith%22%2C%22email%22%3A%22jim%40acme.com%22%7D"
```

#### Query parameters

| Parameter | Required | Description |
| --- | --- | --- |
| `a` | Yes | First record as URL-encoded JSON |
| `b` | Yes | Second record as URL-encoded JSON |
| `threshold` | No | Similarity threshold (default 0.85) |

#### Response format

```json
{
  "isDuplicate": true,
  "score": 0.92,
  "reasons": ["email_exact", "name_jaro-winkler:0.87"],
  "confidence": 0.92,
  "fieldsUsed": ["email", "name", "company"]
}
```

### Limitations

- Does not perform cross-run dedup (each run is independent)
- Fuzzy matching works best on English text; non-Latin scripts may produce lower accuracy
- Maximum ~100,000 records per run (256 MB memory constraint)
- Auto-detection requires standard field names (email, name, company, phone, etc.)
- Custom field names need explicit `matchFields` configuration

### Integrations

Superclean Dedupe works with any tool that can call Apify Actors:

- **Clay** — Add as a dedup step after enrichment
- **Make** — Use the Apify module to run the Actor
- **Zapier** — Trigger runs and retrieve results automatically
- **n8n** — Self-hosted workflow automation

### Using Superclean Dedupe with the Apify API

**Node.js:**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('superlativetech/superclean-dedupe').call({
  records: [
    { name: 'Jim Smith', email: 'jim@acme.com', company: 'ACME Corp' },
    { name: 'James Smith', email: 'jim@acme.com', company: 'Acme Corporation' },
    { name: 'Bob Wilson', email: 'bob@other.com', company: 'Other LLC' },
  ]
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const deduped = items.filter(r => r.isCanonical);
console.log(`${items.length} records → ${deduped.length} unique`);
```

**Python:**

```python
from apify_client import ApifyClient

client = ApifyClient('YOUR_API_TOKEN')

run = client.actor('superlativetech/superclean-dedupe').call(run_input={
    'records': [
        {'name': 'Jim Smith', 'email': 'jim@acme.com', 'company': 'ACME Corp'},
        {'name': 'James Smith', 'email': 'jim@acme.com', 'company': 'Acme Corporation'},
        {'name': 'Bob Wilson', 'email': 'bob@other.com', 'company': 'Other LLC'},
    ]
})

items = client.dataset(run['defaultDatasetId']).list_items().items
deduped = [r for r in items if r['isCanonical']]
print(f"{len(items)} records → {len(deduped)} unique")
```

Check out the [Apify API reference](https://docs.apify.com/api) for full details, or click the **API** tab above for more code examples.

### Your feedback

We're always improving Superclean Actors. If you have feature requests, find a bug, or need help with a specific use case, please open an issue in the Actor's Issues tab.

### Leave a review

If Superclean Dedupe saves you time or improves your lead data, please leave a review. Your feedback helps other users discover the tool and helps us understand what's working well.

***

Built by Superlative

# Actor input Schema

## `records` (type: `array`):

Array of record objects to deduplicate. Each record should have fields like name, email, company, phone, etc.

## `record` (type: `object`):

Single record object — API shorthand. If both record and records are provided, record is prepended to the list.

## `threshold` (type: `number`):

Similarity score threshold for considering two records as duplicates (0-1). Higher = stricter (fewer false merges). Lower = more aggressive (catches more duplicates but may over-merge).

## `matchFields` (type: `array`):

Optional. Specify which fields to match on and how. If omitted, fields are auto-detected from record keys. Each item: {field, method, weight}. Methods: exact, jaro-winkler, levenshtein, token-set.

## Actor input object example

```json
{
  "records": [
    {
      "name": "Jim Smith",
      "email": "jim@acme.com",
      "company": "ACME Corp"
    },
    {
      "name": "James Smith",
      "email": "jim@acme.com",
      "company": "Acme Corporation"
    },
    {
      "name": "Jane Doe",
      "email": "jane@example.com",
      "company": "Example Inc"
    },
    {
      "name": "J. Doe",
      "email": "jane@example.com",
      "company": "Example"
    },
    {
      "name": "Bob Wilson",
      "email": "bob@other.com",
      "company": "Other LLC"
    }
  ],
  "threshold": 0.85
}
```

# Actor output Schema

## `allResults` (type: `string`):

All records annotated with cluster IDs, canonical flags, match scores, and reasons

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "name": "Jim Smith",
            "email": "jim@acme.com",
            "company": "ACME Corp"
        },
        {
            "name": "James Smith",
            "email": "jim@acme.com",
            "company": "Acme Corporation"
        },
        {
            "name": "Jane Doe",
            "email": "jane@example.com",
            "company": "Example Inc"
        },
        {
            "name": "J. Doe",
            "email": "jane@example.com",
            "company": "Example"
        },
        {
            "name": "Bob Wilson",
            "email": "bob@other.com",
            "company": "Other LLC"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("superlativetech/superclean-dedupe").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [
        {
            "name": "Jim Smith",
            "email": "jim@acme.com",
            "company": "ACME Corp",
        },
        {
            "name": "James Smith",
            "email": "jim@acme.com",
            "company": "Acme Corporation",
        },
        {
            "name": "Jane Doe",
            "email": "jane@example.com",
            "company": "Example Inc",
        },
        {
            "name": "J. Doe",
            "email": "jane@example.com",
            "company": "Example",
        },
        {
            "name": "Bob Wilson",
            "email": "bob@other.com",
            "company": "Other LLC",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("superlativetech/superclean-dedupe").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "records": [
    {
      "name": "Jim Smith",
      "email": "jim@acme.com",
      "company": "ACME Corp"
    },
    {
      "name": "James Smith",
      "email": "jim@acme.com",
      "company": "Acme Corporation"
    },
    {
      "name": "Jane Doe",
      "email": "jane@example.com",
      "company": "Example Inc"
    },
    {
      "name": "J. Doe",
      "email": "jane@example.com",
      "company": "Example"
    },
    {
      "name": "Bob Wilson",
      "email": "bob@other.com",
      "company": "Other LLC"
    }
  ]
}' |
apify call superlativetech/superclean-dedupe --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=superlativetech/superclean-dedupe",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/llvky7GnCZ4UWokCg/builds/OfXSsu3PjYRLqnuAS/openapi.json