# CSV Deduper Normalizer (`junipr/csv-deduper-normalizer`) Actor

Deduplicate and normalize CSV-style rows. Clean whitespace, casing, domains, URLs, and emails, then output kept and duplicate rows plus clean CSV/JSON files.

- **URL**: https://apify.com/junipr/csv-deduper-normalizer.md
- **Developed by:** [junipr](https://apify.com/junipr) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.90 / 1,000 row checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## CSV Deduper Normalizer

Normalize CSV-style rows, detect duplicate keys and invalid values, and export clean JSON and CSV files for downstream imports.

### What It Cleans

- Leading, trailing, and repeated whitespace.
- Domain names, including protocol, path, query, and `www` removal.
- URLs, including hostname casing, fragments, trailing slashes, and common tracking parameters.
- Email casing and basic email syntax.
- Configured lowercase and title-case fields.
- Duplicate rows based on one or more normalized key fields.
- Missing key fields and duplicate CSV header names.

The CSV parser handles quoted delimiters, escaped quotes, embedded newlines, CRLF input, and UTF-8 byte-order marks. Structured row objects are also accepted.

### Input

Provide one source: `csvText`, `rows`, or a public `csvUrl` with `fetchUrls` enabled.

| Field | Purpose |
| --- | --- |
| `csvText` | CSV content with a header row. |
| `csvUrl` | Public CSV URL fetched only when `fetchUrls` is true and `csvText` is empty. |
| `rows` | Structured row objects used instead of CSV text. |
| `delimiter` | Single-character delimiter; comma by default. |
| `keyFields` | Normalized fields combined into the duplicate key. |
| `domainFields` | Fields normalized as domains. |
| `urlFields` | Fields normalized as URLs. |
| `emailFields` | Fields lowercased and checked for basic email syntax. |
| `lowercaseFields` | Additional fields converted to lowercase. |
| `titleCaseFields` | Fields converted to title case. |
| `trimWhitespace` / `collapseWhitespace` | String whitespace controls. |
| `includeDuplicates` | Include duplicate rows in the dataset; clean files always contain kept rows only. |
| `maxItems` | Row cap, from 1 through 50,000. |
| `maxCsvBytes` | Retrieved CSV size cap, up to 5,000,000 bytes. |
| `includeReport` | Write result, summary, clean-file, and Markdown report records. |

```json
{
  "csvText": "company,domain,url,email\n ACME INC ,https://www.Example.com/path,https://Example.com/pricing/?utm_source=ad,Sales@Example.com\nAcme Inc,example.com,example.com/pricing,sales@example.com",
  "keyFields": ["domain", "email"],
  "domainFields": ["domain"],
  "urlFields": ["url"],
  "emailFields": ["email"],
  "titleCaseFields": ["company"],
  "includeDuplicates": true,
  "maxItems": 100,
  "includeReport": true
}
```

### Dataset Rows

Each row describes one kept, duplicate, invalid, or diagnostic result. Important fields include:

- `rowIndex`, `status`, `action`, `dedupeKey`, and `duplicateOfRowIndex`.
- `originalRow`, `normalizedRow`, and `changedFields`.
- `issueCount` and `issues`, with codes such as `duplicate-row`, `missing-key-field`, and `invalid-email`.
- `sourceType`, `sourceId`, `processedAt`, and `recommendation`.
- `diagnosticCode` and `sourceError` for blocked retrieval or missing data rows.

Row statuses are `clean`, `changed`, `duplicate`, and `invalid`. A duplicate points to the zero-based index of the first matching normalized key.

### Stored Artifacts

- Dataset: emitted row results after the `includeDuplicates` filter.
- `CSV_DEDUPER_NORMALIZER_RESULTS.json`: the same emitted result rows.
- `CSV_DEDUPER_NORMALIZER_SUMMARY.json`: full processed, kept, duplicate, changed, invalid, issue, diagnostic, and charge-limit counts.
- `CSV_DEDUPER_NORMALIZER_CLEAN.csv`: normalized kept rows.
- `CSV_DEDUPER_NORMALIZER_CLEAN.json`: normalized kept rows as JSON.
- `CSV_DEDUPER_NORMALIZER_REPORT.md`: readable QA report.

### Usage Charges

The actor uses pay per event. Platform usage is included in these event prices.

| Event | Price | When charged |
| --- | ---: | --- |
| `actor-start` | $0.00500 | Once before paid work starts. |
| `row-checked` | $0.00390 | Before each non-diagnostic dataset row is emitted. |
| `issue-detected` | $0.00372 | Once per issue on a row, before that row is emitted. |
| `qa-report-generated` | $0.05000 | Before result, summary, clean-file, and report records are written. |

If Apify reports that a charge was limited or only partly accepted, the actor stops before the corresponding paid output. Set `includeReport` to false when KVS artifacts are not needed, and set `includeDuplicates` to false when duplicate dataset rows are not needed.

### Public Tasks

Five capped task inputs cover domain and email deduplication, URL cleanup, company-name casing, missing-key QA, and clean-file export. Each task supplies concrete data and bounded row counts.

### Retrieval Safety

- Only public `http` and `https` CSV URLs are accepted.
- Credentials in URLs and redirects are rejected.
- Loopback, link-local, and private network destinations are blocked unless `allowPrivateUrls` is explicitly enabled for controlled testing.
- Response length, retained CSV bytes, row count, and request duration are bounded.
- Fetch failures and empty inputs produce explicit diagnostic rows rather than fabricated data.

### Limitations

- Deduplication is exact after configured normalization; fuzzy entity matching is not performed.
- Email checks validate syntax only and do not prove mailbox ownership or deliverability.
- Domain and URL normalization does not verify that a host exists.
- Inputs larger than the configured byte or row cap must be split by the caller.

# Actor input Schema

## `csvText` (type: `string`):

CSV data with a header row.

## `csvUrl` (type: `string`):

CSV URL fetched only when enabled and csvText is empty.

## `rows` (type: `array`):

Structured rows used instead of CSV text.

## `delimiter` (type: `string`):

Single-character CSV delimiter.

## `keyFields` (type: `array`):

Fields combined into the duplicate key.

## `domainFields` (type: `array`):

Field names whose values should be normalized as domains.

## `urlFields` (type: `array`):

Field names whose values should be normalized as URLs.

## `emailFields` (type: `array`):

Field names whose values should be validated as email addresses.

## `lowercaseFields` (type: `array`):

Field names whose string values should be lowercased.

## `titleCaseFields` (type: `array`):

Field names whose string values should be converted to title case.

## `trimWhitespace` (type: `boolean`):

Trim leading and trailing whitespace from string fields.

## `collapseWhitespace` (type: `boolean`):

Collapse repeated internal whitespace in normalized string fields.

## `includeDuplicates` (type: `boolean`):

Include duplicate rows in the dataset output instead of summary-only counts.

## `fetchUrls` (type: `boolean`):

Fetch csvUrl when csvText and inline rows are empty.

## `fetchTimeoutMs` (type: `integer`):

Maximum time to wait when fetching a public CSV URL.

## `allowPrivateUrls` (type: `boolean`):

Allow fetching private or local network CSV URLs.

## `maxCsvBytes` (type: `integer`):

Maximum number of CSV bytes to read from text or fetched content.

## `maxItems` (type: `integer`):

Maximum number of CSV or inline rows to process.

## `includeReport` (type: `boolean`):

Write cleaned CSV, duplicate summary, and Markdown report artifacts.

## `dryRun` (type: `boolean`):

Validate and summarize input without writing paid dataset rows.

## `debug` (type: `boolean`):

Enable additional diagnostic log output.

## Actor input object example

```json
{
  "csvText": "",
  "csvUrl": "",
  "rows": [],
  "delimiter": ",",
  "keyFields": [
    "domain"
  ],
  "domainFields": [
    "domain"
  ],
  "urlFields": [
    "url"
  ],
  "emailFields": [
    "email"
  ],
  "lowercaseFields": [],
  "titleCaseFields": [],
  "trimWhitespace": true,
  "collapseWhitespace": true,
  "includeDuplicates": true,
  "fetchUrls": false,
  "fetchTimeoutMs": 10000,
  "allowPrivateUrls": false,
  "maxCsvBytes": 500000,
  "maxItems": 100,
  "includeReport": true,
  "dryRun": false,
  "debug": false
}
```

# Actor output Schema

## `results` (type: `string`):

Normalize, validate, deduplicate, and export CSV-style rows.

## `artifact1` (type: `string`):

CSV\_DEDUPER\_NORMALIZER\_RESULTS.json

## `artifact2` (type: `string`):

CSV\_DEDUPER\_NORMALIZER\_SUMMARY.json

## `artifact3` (type: `string`):

CSV\_DEDUPER\_NORMALIZER\_REPORT.md

## `artifact4` (type: `string`):

CSV\_DEDUPER\_NORMALIZER\_CLEAN.csv

## `artifact5` (type: `string`):

CSV\_DEDUPER\_NORMALIZER\_CLEAN.json

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("junipr/csv-deduper-normalizer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("junipr/csv-deduper-normalizer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call junipr/csv-deduper-normalizer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=junipr/csv-deduper-normalizer",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wax9cbzIiYUiHgLza/builds/rEZDTnE2fPJOKwbR9/openapi.json