# Duplications Checker (`lukaskrivka/duplications-checker`) Actor Check your dataset for duplications. Accept only the highest quality data! - **URL**: https://apify.com/lukaskrivka/duplications-checker.md - **Developed by:** [Lukรกลก Kล™ivka](https://apify.com/lukaskrivka) (community) - **Categories:** Automation, Open source - **Stats:** 248 total users, 1 monthly users, 100.0% runs succeeded, 12 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### Duplications Checker - [Overview](#overview) - [How it works](#how-it-works) - [Input](#input) - [preCheckFunction](#preCheckFunction) - [Report](#report) - [Epilogue](#epilogue) #### Overview Duplications Checker is an [Apify actor](https://apify.com/actors) that helps you find duplicates in your datasets or JSON array. - Loads data from Apify [Dataset](https://apify.com/docs/storage#dataset), [Key Value store](https://apify.com/docs/storage#key-value-store) or an arbitrary JSON and checks each item against all others for duplicate field. - The check takes seconds to a maximum of a few minutes for larger datasets. - Produces a report so you know exactly how many problems are there and which items contained them. - It is very useful to append this actor as a [webhook](https://apify.com/docs/webhooks). You can easily chain another actor after this one to [send an email](https://apify.com/apify/send-mail) or [add a report to your Google Sheets](https://apify.com/lukaskrivka/google-sheets) to name just a few examples. Check [Apify Store](https://apify.com/store) for more. #### How it works - Loads data in batches into memory (Key Value store or raw data are loaded all at once). - Each item in the batch is scanned for the provided field. Actor keeps track of previous occurences and count duplicates. - A [report](#reports) is created after the whole run and saved as `OUTPUT` to the default Key Value store. - Between each batch, the state of the actor is saved so it doesn't have to repeat the work after restart(migration). #### Input This actor expects a JSON object as an input. You can also set it up in a visual UI editor on Apify. You can find examples in the Input and Example Run tabs of the actor page in Apify Store. All the input fields (regardless of section) are top level fields. **Main input fields** - `datasetId` <[string](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)> Id of dataset where the data are located. If you need to use other input types like Key value store or raw JSON, use `keyValueStoreRecord` or `rawData` **You have specify this, `keyValueStoreRecord` or `rawData` but only one of them** - `checkOnlyCleanItems` <[boolean](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> Only clean dataset items will be loaded and use for duplications checking if `datasetId` option is provided. **Default: false** - `fields` <[array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array)> List of fields in each item that will be checked for duplicates. Each field must not be nested and it should contain only simple value (string or number). It is also possible to use option `field` to pass only single <[string](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)> value (due to backward compatibility). You can prepare your data with [preCheckFunction](#preCheckFunction). **Required** - `preCheckFunction` <[stringified function](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Functions)> Stringified javascipt function that can apply arbitrary transformation to the input data before the check. See [preCheckFunction](#preCheckFunction) section. **Optional** - `minDuplications` <[number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> Minimum occurences to be included in the report. **Default: 2** **Show options** - `showIndexes`: <[boolean](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> Indexes of the duplicate items will be shown in the OUTPUT report. Set to false if you don't need them. **Default: true** - `showItems`: <[boolean](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> Duplicate items will be pushed to a dataset. Set to false if you don't need them. **Default: true** - `showMissing`: <[boolean](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> Items where the values for the `field` is missing or is `null` or `''` will be included in the report **Default: true** **Dataset pagination options** - `limit`: <[number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> How many items will be checked. **Default: all** - `offset`: <[number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> From which item the checking will start. Use with `limit` to check specific items. **Default: 0** - `batchSize`: <[number](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#Number_type)> You can change number of loaded and processed items in each batch. This is only needed to be changed if you have really huge items. **Default: 1000** **Other data sources** - `keyValueStoreRecord` <[string](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures#String_type)> ID and record key if you want to load from KV store. Format is `{keyValueStoreId}+{recordKey}`, e.g. `s5NJ77qFv8b4osiGR+MY-KEY`. **You have specify this, `datasetId` or `rawData` but only one of them** - `rawData` <[array](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array)> Array of objects to be checked. **You have specify this, `keyValueStoreRecord` or `datasetId` but only one of them***. #### preCheckFunction `preCheckFunction` is useful to transform the input data before the actual check. Its main usefulness is to ensure that the field you are checking is a top level (not nested) field and that the value of that field is a simple value like number or string (*The decision to not allow deep equality check for nested structures was made for simplicity and performance reasons*). So for example, let's say you have an item with a nested field `images`: ```` \[{ "url": "https://www.bloomingdales.com/shop/product/lauren-ralph-lauren-ruffled-georgette-dress?ID=3493626\&CategoryID=1005206", "images": \[ { "src": "https://images.bloomingdalesassets.com/is/image/BLM/products/9/optimized/10317399\_fpx.tif", "cloudPath": "" } ], ... // more fields that you are not interested in }] ``` If you want to check the first image URL for duplications and keep the item `url` for a reference, you can easily transform the whole data with simple `preCheckFunction`: ``` (data) => data.map((item) => ({ url: item.url, imageUrl: item.images\[0].src })) ``` Now, set `field` in input to `imageUrl` and all will work nicely. #### Report At the end of the actor run, the report is saved to the default Key Value store as an `OUTPUT`. Also, if `showItems` is `true`, it will push duplicate items to the dataset. By default, the report will include all information but you can opt-out if you set any of `showIndexes`, `showItems`, `showMissing` to `false`. Report is an object where every `field` value that appeared at least twice (which means it was duplicate) is inluced as a key. For each of them, report contains `count` (minimum is 2), `originalIndexes` (which are indexes of items in your original dataset or after `preCheckFunction`) and `outputIndexes` (only present when `showItems` is enabled). The indexes should help you navigate the duplicates in your data. ##### OUTPUT example ``` { "https://images.bloomingdalesassets.com/is/image/BLM/products/4/optimized/9153524\_fpx.tif": { "count": 2, "originalIndexes": \[ 166, 202 ], "outputIndexes": \[ 0, 1 ] }, "https://images.bloomingdalesassets.com/is/image/BLM/products/9/optimized/9832349\_fpx.tif": { "count": 2, "originalIndexes": \[ 1001, 1002 ], "outputIndexes": \[ 2, 3 ] } } ```` The items are intentionally not included in the OUTPUT report to reduce its size. Instead they are pushed to the default dataset and you can locate them with `outputIndexes`. If you need to connect the OUTPUT with the dataset for deeper analysis, you can find the items with the help of indexes. #### Checking more fields The first version of the actor had the option to check more fields at once but it produced very complicated output and the implementation was too convoluted so I decided to abandon the idea for simplicity. In case you want to check more fields, simply run it once for each field. Since the actor consumption is pretty low, it is not a big deal. **More info coming soon!** #### Epilogue If you find any problem or would like to add a new feature, please create an issue on the [Github repo](https://github.com/metalwarrior665/duplications-checker). Thanks everybody for using it and giving any feedback! # Actor input Schema ## `datasetId` (type: `string`): Id of dataset where the data are located. If you need to use other input types like Key value store or raw JSON, look at `Other data sources` ## `checkOnlyCleanItems` (type: `boolean`): Only clean dataset items will be loaded and use for duplications checking if `datasetId` option is provided. ## `fields` (type: `array`): List of fields in each item that will be checked for duplicates. Each given field must not be nested and it should contain only simple value (string or number). You can prepare your data with preCheckFunction. ## `preCheckFunction` (type: `string`): You can specify which fields should display in the debug OUTPUT to identify bad items. By default it shows all fields which may make it unnecessary big. ## `minDuplications` (type: `integer`): Minimum occurences to be included in the report. Defaults to 2 ## `showIndexes` (type: `boolean`): Indexes of the duplicate items will be shown in the OUTPUT report. Set to false if you don't need them. ## `showItems` (type: `boolean`): Duplicate items will be pushed to a dataset. Set to false if you don't need them. ## `showMissing` (type: `boolean`): Items where the values for the `field` is missing or is `null` or `''` will be included in the report. ## `limit` (type: `integer`): How many items will be checked. Default is all ## `offset` (type: `integer`): From which item the checking will start. Use with limit to check specific items. ## `batchSize` (type: `integer`): You can change number of loaded and processed items in each batch. This is only needed if you have really huge items. ## `keyValueStoreRecord` (type: `string`): ID and record key if you want to load from KV store. Format is `{keyValueStoreId}+{recordKey}`, e.g. `s5NJ77qFv8b4osiGR+MY-KEY` ## `rawData` (type: `array`): Raw JSON array you want to check. ## Actor input object example ```json { "datasetId": "s5NJ77qFv8b4osiGR", "checkOnlyCleanItems": false, "fields": [], "preCheckFunction": "(data) => data.map((item) => ({ url: item.url, imageSrc: item.images && item.images[0] ? item.images[0].src : null }))", "minDuplications": 2, "showIndexes": true, "showItems": true, "showMissing": true, "limit": 1000, "offset": 20000, "batchSize": 5000, "keyValueStoreRecord": "s5NJ77qFv8b4osiGR+MY-KEY", "rawData": [ { "url": "https://example.com", "price": 20 }, { "url": "https://google.com", "price": 30 } ] } ```` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("lukaskrivka/duplications-checker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("lukaskrivka/duplications-checker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call lukaskrivka/duplications-checker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=lukaskrivka/duplications-checker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Duplications Checker", "description": "Check your dataset for duplications. Accept only the highest quality data!", "version": "0.1", "x-build-id": "DvYXmY5xAeWGd52zH" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/lukaskrivka~duplications-checker/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-lukaskrivka-duplications-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/lukaskrivka~duplications-checker/runs": { "post": { "operationId": "runs-sync-lukaskrivka-duplications-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/lukaskrivka~duplications-checker/run-sync": { "post": { "operationId": "run-sync-lukaskrivka-duplications-checker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "fields" ], "properties": { "datasetId": { "title": "Dataset ID", "type": "string", "description": "Id of dataset where the data are located. If you need to use other input types like Key value store or raw JSON, look at `Other data sources`" }, "checkOnlyCleanItems": { "title": "Check only clean dataset items", "type": "boolean", "description": "Only clean dataset items will be loaded and use for duplications checking if `datasetId` option is provided.", "default": false }, "fields": { "title": "Fields", "type": "array", "description": "List of fields in each item that will be checked for duplicates. Each given field must not be nested and it should contain only simple value (string or number). You can prepare your data with preCheckFunction.", "default": [], "items": { "type": "string" } }, "preCheckFunction": { "title": "Pre-check function", "type": "string", "description": "You can specify which fields should display in the debug OUTPUT to identify bad items. By default it shows all fields which may make it unnecessary big." }, "minDuplications": { "title": "Minimum duplications", "minimum": 2, "type": "integer", "description": "Minimum occurences to be included in the report. Defaults to 2", "default": 2 }, "showIndexes": { "title": "Show indexes", "type": "boolean", "description": "Indexes of the duplicate items will be shown in the OUTPUT report. Set to false if you don't need them.", "default": true }, "showItems": { "title": "Show items", "type": "boolean", "description": "Duplicate items will be pushed to a dataset. Set to false if you don't need them.", "default": true }, "showMissing": { "title": "Show missing fields", "type": "boolean", "description": "Items where the values for the `field` is missing or is `null` or `''` will be included in the report.", "default": true }, "limit": { "title": "Limit", "minimum": 1, "type": "integer", "description": "How many items will be checked. Default is all" }, "offset": { "title": "Offset", "minimum": 1, "type": "integer", "description": "From which item the checking will start. Use with limit to check specific items." }, "batchSize": { "title": "Batch Size", "minimum": 1, "type": "integer", "description": "You can change number of loaded and processed items in each batch. This is only needed if you have really huge items.", "default": 1000 }, "keyValueStoreRecord": { "title": "Key value store Record", "type": "string", "description": "ID and record key if you want to load from KV store. Format is `{keyValueStoreId}+{recordKey}`, e.g. `s5NJ77qFv8b4osiGR+MY-KEY`" }, "rawData": { "title": "Raw Data", "type": "array", "description": "Raw JSON array you want to check." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```