# JSON Dataset Cleaner and Deduplicator (`rodrgds/dataset-cleaner`) Actor

Clean JSON datasets, remove empty rows, deduplicate by any field, validate emails, and prepare scraper output for CRMs, analysis, or AI workflows.

- **URL**: https://apify.com/rodrgds/dataset-cleaner.md
- **Developed by:** [Rodrigo Dias](https://apify.com/rodrgds) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 cleaned rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Dataset Cleaner

Normalize messy CRM exports, scraper output, and lead lists before they move into analytics, enrichment, email tools, or AI pipelines.

This Actor takes an array of JSON rows, standardizes common fields, rejects rows that fail required checks, deduplicates after normalization, and writes an audit-friendly dataset containing cleaned rows, rejected rows, and one summary row.

### What It Cleans

- Trims, collapses, and removes control characters from selected text columns or all string columns
- Normalizes email addresses by trimming and lowercasing, then validates with a pragmatic format check
- Parses phone numbers with `libphonenumber-js`, using a configurable default country for local numbers
- Outputs valid phone numbers in E.164 format and adds national format, country, and phone type metadata when available
- Normalizes websites, URLs, and domains by adding `https://` when needed and lowercasing hostnames
- Converts unambiguous dates to `YYYY-MM-DD`
- Converts safe numeric strings such as `$1,234.50` to numbers
- Removes empty rows and deduplicates by one or more normalized fields
- Adds row-level changes, warnings, errors, and summary counts

Phone validation runs locally with the bundled library. No external API, paid provider, or API key is required.

### Example Input

```json
{
  "data": [
    {
      "name": "  Alice   Adams  ",
      "email": " ALICE@EXAMPLE.COM ",
      "phone": "(415) 555-2671",
      "website": "Example.COM/contact",
      "revenue": "$1,234.50"
    },
    {
      "name": "Alice Adams",
      "email": "alice@example.com",
      "phone": "+1 415 555 2671",
      "website": "https://example.com/contact",
      "revenue": "1234.50"
    },
    {
      "name": "Broken Lead",
      "email": "broken@example.com",
      "phone": "not a phone",
      "website": "broken.example"
    }
  ],
  "dedupKeys": ["email"],
  "phoneColumns": ["phone"],
  "numberColumns": ["revenue"],
  "requireValidPhone": true,
  "outputRejectedRows": true
}
```

### Example Output

```json
[
  {
    "__rowType": "cleaned",
    "name": "Alice Adams",
    "email": "alice@example.com",
    "phone": "+14155552671",
    "website": "https://example.com/contact",
    "revenue": 1234.5,
    "phone_national": "(415) 555-2671",
    "phone_country": "US",
    "phone_type": "FIXED_LINE_OR_MOBILE",
    "__changes": [
      { "field": "name", "operation": "text-clean", "before": "  Alice   Adams  ", "after": "Alice Adams" },
      { "field": "email", "operation": "email-normalize", "before": "ALICE@EXAMPLE.COM", "after": "alice@example.com" },
      { "field": "phone", "operation": "phone-normalize", "before": "(415) 555-2671", "after": "+14155552671" },
      { "field": "website", "operation": "url-normalize", "before": "Example.COM/contact", "after": "https://example.com/contact" },
      { "field": "revenue", "operation": "number-normalize", "before": "$1,234.50", "after": 1234.5 }
    ],
    "__warnings": []
  },
  {
    "__rowType": "rejected",
    "__errors": ["Duplicate row by email"],
    "__warnings": [],
    "__originalRow": {
      "name": "Alice Adams",
      "email": "alice@example.com",
      "phone": "+1 415 555 2671",
      "website": "https://example.com/contact",
      "revenue": "1234.50"
    }
  },
  {
    "__rowType": "rejected",
    "__errors": ["Invalid phone in phone"],
    "__warnings": [],
    "__originalRow": {
      "name": "Broken Lead",
      "email": "broken@example.com",
      "phone": "not a phone",
      "website": "broken.example"
    },
    "name": "Broken Lead",
    "email": "broken@example.com",
    "phone": "not a phone",
    "website": "https://broken.example/"
  },
  {
    "__rowType": "summary",
    "originalRows": 3,
    "cleanedRows": 1,
    "rejectedRows": 2,
    "duplicateRows": 1,
    "invalidPhones": 1,
    "changedFields": 10
  }
]
```

### Important Options

- `textColumns`: specific fields to trim and clean. Leave empty to clean all string fields.
- `emailColumns`: specific email fields. Leave empty to auto-detect field names containing `email`.
- `phoneColumns`: specific phone fields. Leave empty to auto-detect `phone`, `mobile`, and `tel` fields.
- `defaultPhoneCountry`: ISO-2 country code used when a phone number has no `+` country prefix. Defaults to `US`.
- `requireValidEmail` and `requireValidPhone`: reject bad rows when enabled. Otherwise, keep the row and add warnings.
- `urlColumns`: specific URL/domain fields. Leave empty to auto-detect `url`, `website`, and `domain` fields.
- `dateColumns` and `numberColumns`: explicit fields to normalize as dates or numbers.
- `dedupKeys`: one or more fields used together for duplicate detection after normalization.
- `outputRejectedRows`: include rejected rows in the dataset with errors and original input.
- `includeCleaningMetadata`: include `__changes` and `__warnings` on cleaned rows.

### Output Rows

Every output item has a `__rowType`:

- `cleaned`: accepted normalized row
- `rejected`: dropped row with `__errors`, optional `__warnings`, and `__originalRow`
- `summary`: final row with counts for input rows, cleaned rows, rejected rows, duplicates, warnings, and invalid field types

### Common Use Cases

- Clean CRM imports before sending leads to HubSpot, Salesforce, Pipedrive, Airtable, or outreach tools
- Normalize scraper output before exporting CSVs or loading a warehouse
- QA lead generation results by rejecting invalid phone numbers or emails
- Prepare consistent structured data for dashboards, enrichment, embeddings, and AI workflows

# Actor input Schema

## `data` (type: `array`):

Array of JSON objects to clean. Each object is treated as one row.

## `textColumns` (type: `array`):

String fields to clean. Leave empty to clean all string fields.

## `trimWhitespace` (type: `boolean`):

Remove leading and trailing whitespace from selected text fields.

## `collapseWhitespace` (type: `boolean`):

Replace repeated whitespace inside text fields with a single space.

## `removeControlCharacters` (type: `boolean`):

Remove non-printing control characters from text fields.

## `emailColumns` (type: `array`):

Email fields to normalize and validate. Leave empty to auto-detect fields containing email.

## `normalizeEmails` (type: `boolean`):

Trim and lowercase email addresses, then validate their shape.

## `requireValidEmail` (type: `boolean`):

Reject rows with invalid email values instead of keeping them with warnings.

## `phoneColumns` (type: `array`):

Phone fields to normalize and validate. Leave empty to auto-detect phone, mobile, and tel fields.

## `defaultPhoneCountry` (type: `string`):

ISO-2 country code used when phone numbers do not include a country prefix.

## `normalizePhones` (type: `boolean`):

Parse phone numbers, output E.164, and add national format, country, and type metadata when available.

## `requireValidPhone` (type: `boolean`):

Reject rows with invalid phone values instead of keeping them with warnings.

## `urlColumns` (type: `array`):

URL, website, or domain fields to normalize. Leave empty to auto-detect url, website, and domain fields.

## `normalizeUrls` (type: `boolean`):

Trim URLs, add https:// when missing, lowercase hostnames, and normalize domain-only fields to hostnames.

## `dateColumns` (type: `array`):

Date fields to normalize to YYYY-MM-DD when the date is unambiguous.

## `numberColumns` (type: `array`):

Number fields to normalize by removing common currency symbols and thousands separators when safe.

## `dedupKeys` (type: `array`):

Fields used together as the deduplication key after normalization.

## `removeEmpty` (type: `boolean`):

Reject rows where all fields are blank, null, or missing.

## `outputRejectedRows` (type: `boolean`):

Write rejected rows to the dataset with errors and original row data.

## `includeCleaningMetadata` (type: `boolean`):

Add \_\_changes and \_\_warnings to cleaned rows.

## Actor input object example

```json
{
  "data": [
    {
      "name": "  Alice Adams  ",
      "email": " ALICE@EXAMPLE.COM ",
      "phone": "(415) 555-2671",
      "website": "Example.com/contact"
    }
  ],
  "textColumns": [],
  "trimWhitespace": true,
  "collapseWhitespace": true,
  "removeControlCharacters": true,
  "emailColumns": [],
  "normalizeEmails": true,
  "requireValidEmail": false,
  "phoneColumns": [],
  "defaultPhoneCountry": "US",
  "normalizePhones": true,
  "requireValidPhone": false,
  "urlColumns": [],
  "normalizeUrls": true,
  "dateColumns": [],
  "numberColumns": [],
  "dedupKeys": [],
  "removeEmpty": true,
  "outputRejectedRows": true,
  "includeCleaningMetadata": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "data": [
        {
            "name": "  Alice Adams  ",
            "email": " ALICE@EXAMPLE.COM ",
            "phone": "(415) 555-2671",
            "website": "Example.com/contact"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("rodrgds/dataset-cleaner").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "data": [{
            "name": "  Alice Adams  ",
            "email": " ALICE@EXAMPLE.COM ",
            "phone": "(415) 555-2671",
            "website": "Example.com/contact",
        }] }

# Run the Actor and wait for it to finish
run = client.actor("rodrgds/dataset-cleaner").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "data": [
    {
      "name": "  Alice Adams  ",
      "email": " ALICE@EXAMPLE.COM ",
      "phone": "(415) 555-2671",
      "website": "Example.com/contact"
    }
  ]
}' |
apify call rodrgds/dataset-cleaner --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=rodrgds/dataset-cleaner",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dXIlblYokxdFNJil3/builds/U1Eavdcwc19Fcwhtd/openapi.json