# Encrypted Data Integration (`sovanza.inc/encrypted-data-integration`) Actor

Encrypted Data Integration encrypts sensitive Apify data before export or automation. It supports selected fields, full records, or full payload encryption with AES-GCM, manifests, fingerprints, and dataset or key-value store output.

- **URL**: https://apify.com/sovanza.inc/encrypted-data-integration.md
- **Developed by:** [Sovanza](https://apify.com/sovanza.inc) (community)
- **Categories:** Developer tools, Automation, Other
- **Stats:** 4 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $4.00 / 1,000 encrypted records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Encrypted Data Integration

Encrypt structured records inside Apify **before** export, sync, or handoff. Load data from a **dataset**, **key-value store**, inline **JSON records**, or pasted **JSON/CSV**; apply **AES-256-GCM** (or Fernet) encryption; write ciphertext to a **dataset**, **KV bundle**, or **both**. Built for PII-safe pipelines, compliance-oriented workflows, and downstream automation without leaking plaintext into logs or exports.

### Quick start

1. Open the Actor in Apify Console and click **Start** (defaults work for a demo run).
2. Leave **Passphrase** and **Source JSON records** empty to use built-in demo data (testing only).
3. Open the run **dataset** — encrypted rows (run totals are in KV store record `RUN_SUMMARY.json`).

Replace the demo passphrase before any production workload.

### Usage

#### Console

1. Set **Source mode** (`json_records` for quick tests, `dataset` to encrypt another Actor’s output).
2. Choose **Encryption scope** and list **Fields to encrypt** for field-level mode.
3. Enter a strong **Passphrase** (secret input) or **Raw key (base64)**.
4. Enable **Remove plaintext after encryption** when exporting sensitive columns.
5. Run and export results as JSON/CSV from the dataset, or read the KV bundle when **Output mode** is `kv_store` or `both`.

#### API / scheduler

Pass the same JSON as the input schema. Secret fields (`passphrase`, `sourceJsonRecords`, etc.) are encrypted at rest on Apify. Chain this Actor after scrapers via **Scheduler** or `apify-client` `actor().call()`.

#### Example input

```json
{
  "sourceMode": "json_records",
  "sourceJsonRecords": [
    { "id": "1", "email": "user@company.com", "notes": "Confidential memo" }
  ],
  "encryptionScope": "selected_fields",
  "fieldsToEncrypt": ["email", "notes"],
  "preserveFields": ["id"],
  "removePlaintextAfterEncryption": true,
  "algorithm": "aes_gcm",
  "keyMode": "passphrase",
  "passphrase": "USE_A_ROTATED_SECRET_FROM_A_VAULT",
  "keyDerivation": "pbkdf2_sha256",
  "iterations": 200000,
  "outputMode": "dataset",
  "includeManifest": true,
  "includeHashFingerprint": true
}
```

#### Encrypt from another dataset

```json
{
  "sourceMode": "dataset",
  "sourceDatasetId": "YOUR_SOURCE_DATASET_ID",
  "maxItems": 250,
  "encryptionScope": "selected_fields",
  "fieldsToEncrypt": ["email", "phone"],
  "preserveFields": ["id", "name"],
  "algorithm": "aes_gcm",
  "keyMode": "passphrase",
  "passphrase": "USE_A_ROTATED_SECRET",
  "outputMode": "dataset"
}
```

Copy the **Source dataset ID** from the upstream Actor run’s default dataset URL in Apify Console.

##### Apify Console (health check & quality score)

After deploying build **0.6+**:

1. Leave **Passphrase** and **Source JSON records** empty — the Actor applies built-in demo data with **1000 KDF iterations** (finishes in under 1 minute).
2. Prefilled input uses `sourceMode: json_records`, `maxItems: 10`, `maxConcurrency: 2`, `iterations: 1000`.
3. Re-run **Try actor** with default prefilled input — expect encrypted demo rows in the dataset.

For production, set a strong **passphrase** (secret input) and raise **iterations** to `200000` or higher.

##### Authentication & sensitive input

Fields that hold credentials or record payloads use Apify **secret input** (`isSecret: true`):

- **`sourceJsonRecords`** — inline JSON records when `sourceMode=json_records`
- **`sourceJsonText`** — pasted JSON/CSV when `sourceMode=json_text`
- **`passphrase`** — encryption passphrase when `keyMode=passphrase`
- **`rawKeyBase64`** — raw symmetric key when `keyMode=raw_key_base64`
- **`deterministicFingerprintSalt`** — optional fingerprint salt (not used for encryption)

Secret values are encrypted at rest and are **not** written into dataset rows or logs when `redactLogs` is enabled.

### Input

| Group | Main fields |
|-------|-------------|
| **Data source** | `sourceMode`, `sourceDatasetId`, `sourceKvStoreKey`, `sourceJsonRecords`, `sourceJsonText`, `maxItems` |
| **Encryption** | `encryptionScope`, `fieldsToEncrypt`, `preserveFields`, `removePlaintextAfterEncryption`, `algorithm`, `keyMode` |
| **Secrets** | `passphrase`, `rawKeyBase64`, `deterministicFingerprintSalt` (`isSecret`) |
| **Output** | `outputMode`, `outputKvStoreKey`, `includeManifest`, `includeHashFingerprint` |
| **Performance** | `chunkSize`, `maxConcurrency` |

Full schema: `INPUT_SCHEMA.json`.

#### Encryption modes

- **`selected_fields`** — Encrypt listed fields (dotted paths supported, e.g. `contact.email`).
- **`full_record`** — One ciphertext blob per JSON object.
- **`full_payload`** — One ciphertext blob for the entire batch.

### Output

Each successful run writes to the **default dataset** (unless configured otherwise):

| Row type | Meaning |
|----------|---------|
| Encrypted record | Plain object with `*_encrypted` fields and optional `manifest` / `fingerprints` |
| `type: "__error__"` | Per-record or configuration diagnostics (no secrets) |
| **KV `RUN_SUMMARY.json`** | Run totals: `processedRecords`, `failedRecords`, `encryptedFieldsCount` |

Example encrypted field:

```json
{
  "recordId": "1",
  "id": "1",
  "email_encrypted": {
    "algorithm": "aes_gcm",
    "version": "1",
    "nonce": "BASE64_NONCE",
    "salt": "BASE64_SALT",
    "kdf": "pbkdf2_sha256",
    "iterations": 200000,
    "ciphertext": "BASE64_CIPHERTEXT"
  },
  "manifest": {
    "cryptoVersion": "1",
    "encryptionScope": "selected_fields",
    "encryptedFields": ["email"]
  }
}
```

When `outputMode` is `kv_store` or `both`, a JSON bundle is stored under `outputKvStoreKey` (default `ENCRYPTED_OUTPUT`).

### Pricing

This Actor supports **Pay per event (PPE)** on Apify Store:

| Event | When charged |
|-------|----------------|
| `apify-actor-start` | Each run start (platform-managed; optional free compute window) |
| `record-encrypted` | Each encrypted output row pushed to the dataset (primary value event) |

Configure events and **Bronze / Silver / Gold discounts** in Console → **Publication → Monetization**. See `PUBLISHING.md` for a step-by-step checklist to reach a **100/100** quality score.

Without PPE, the Actor still runs under standard platform usage billing.

### Security

- **AES-256-GCM** authenticated encryption; passphrases derived with **PBKDF2** or **scrypt**.
- Sensitive inputs use Apify **`isSecret`** encryption at rest.
- **`redactLogs`** and **`removePlaintextAfterEncryption`** reduce accidental exposure.
- **Fingerprints** are SHA-256 hashes for matching — **not** confidentiality.
- Rotate passphrases and restrict dataset access in production.

### FAQ

| Question | Answer |
|----------|--------|
| First run with empty secrets? | Demo records + demo passphrase (fast KDF). Not for production. |
| CRM / Salesforce integration? | Encrypt here; sync ciphertext with your own exporters. |
| Missing fields? | `__error__` rows unless `failOnMissingFields` is true. |
| KV input vs output? | `sourceKvStoreKey` and `outputKvStoreKey` must differ when both are used. |

### Local development

```bash
cd encrypted-data-integration
python -m pip install -r requirements.txt
python scripts/roundtrip_validation.py
```

Add `INPUT.json` for full local runs via `python main.py`.

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

### Publishing

See [PUBLISHING.md](PUBLISHING.md) for Console steps (PPE, SEO, categories, limited permissions, Store discounts).

# Actor input Schema

## `sourceMode` (type: `string`):

Where input records are loaded from: another Actor dataset, a key-value store item, inline JSON records, or pasted JSON/CSV text.

## `sourceDatasetId` (type: `string`):

Dataset ID to read when sourceMode=dataset (from the dataset URL in Apify Console).

## `sourceDatasetName` (type: `string`):

Optional dataset name instead of sourceDatasetId when sourceMode=dataset.

## `sourceKvStoreKey` (type: `string`):

Record name inside this run's default key-value store when sourceMode=kv\_store (not an encryption key). Value must be a JSON array or object.

## `sourceJsonRecords` (type: `string`):

Sensitive: encrypted in Apify, hidden from logs. JSON array of records when sourceMode=json\_records. Example: \[{"id":"1","email":"user@example.com"}]. Leave empty on first run to use built-in demo records for testing.

## `sourceJsonText` (type: `string`):

Sensitive: encrypted in Apify, hidden from logs. Raw JSON array/object or CSV when sourceMode=json\_text.

## `maxItems` (type: `integer`):

Maximum number of input records to process.

## `encryptionScope` (type: `string`):

Choose whether to encrypt selected fields, each full record, or the full exported payload.

## `fieldsToEncrypt` (type: `array`):

List of field names or dotted paths to encrypt when encryptionScope=selected\_fields.

## `preserveFields` (type: `array`):

Fields to always leave plaintext for indexing or operational visibility.

## `removePlaintextAfterEncryption` (type: `boolean`):

Whether to remove original plaintext field after encrypted version is created.

## `outputEncryptedFieldSuffix` (type: `string`):

Suffix for field-level encrypted outputs.

## `algorithm` (type: `string`):

Preferred encryption implementation. AES-GCM is the secure default.

## `keyMode` (type: `string`):

How encryption key material is supplied. This selects the mode only; it does not store a secret.

## `passphrase` (type: `string`):

Sensitive: encrypted in Apify, hidden from logs. Secret passphrase for key derivation when keyMode=passphrase. Leave empty on first run to use a built-in demo passphrase for testing only.

## `rawKeyBase64` (type: `string`):

Sensitive: encrypted in Apify, hidden from logs. Raw symmetric key in base64 when keyMode=raw\_key\_base64.

## `keyDerivation` (type: `string`):

Key derivation function when keyMode=passphrase. Not a secret value.

## `iterations` (type: `integer`):

KDF iterations for PBKDF2 or cost-like tuning for scrypt. Prefill uses 1000 for fast Apify health checks; use 200000+ in production.

## `includeManifest` (type: `boolean`):

Include encryption metadata per record or payload.

## `includeHashFingerprint` (type: `boolean`):

Include deterministic SHA-256 fingerprints for configured fields or records. Fingerprints are not encryption.

## `fingerprintFields` (type: `array`):

Optional fields to fingerprint before encryption.

## `includeRecordId` (type: `boolean`):

Include a normalized recordId field in the encrypted output when available.

## `recordIdField` (type: `string`):

Field name or dotted path used to derive recordId in output rows.

## `outputMode` (type: `string`):

Where to write encrypted output: default run dataset, a key-value store bundle, or both.

## `outputKvStoreKey` (type: `string`):

Record name for encrypted output in the default key-value store (not an encryption key).

## `chunkSize` (type: `integer`):

Number of records to process per batch during loading and encryption.

## `maxConcurrency` (type: `integer`):

Maximum number of records processed concurrently within each batch.

## `failOnMissingFields` (type: `boolean`):

If true, stop the run when a configured field to encrypt is missing.

## `failOnInvalidRecords` (type: `boolean`):

If true, stop the run when a record is not a valid JSON object for the chosen mode.

## `includeDebugFields` (type: `boolean`):

Include safe debug metadata such as which fields were encrypted, but never plaintext or secrets.

## `redactLogs` (type: `boolean`):

Force strict redaction of sensitive values in logs.

## `deterministicFingerprintSalt` (type: `string`):

Sensitive: encrypted in Apify, hidden from logs. Optional salt for SHA-256 fingerprints only (not used for encryption).

## Actor input object example

```json
{
  "sourceMode": "json_records",
  "maxItems": 10,
  "encryptionScope": "selected_fields",
  "fieldsToEncrypt": [
    "email",
    "phone",
    "contact.email"
  ],
  "preserveFields": [
    "id"
  ],
  "removePlaintextAfterEncryption": true,
  "outputEncryptedFieldSuffix": "_encrypted",
  "algorithm": "aes_gcm",
  "keyMode": "passphrase",
  "keyDerivation": "pbkdf2_sha256",
  "iterations": 1000,
  "includeManifest": true,
  "includeHashFingerprint": true,
  "fingerprintFields": [],
  "includeRecordId": true,
  "recordIdField": "id",
  "outputMode": "dataset",
  "outputKvStoreKey": "ENCRYPTED_OUTPUT",
  "chunkSize": 100,
  "maxConcurrency": 2,
  "failOnMissingFields": false,
  "failOnInvalidRecords": false,
  "includeDebugFields": false,
  "redactLogs": true
}
```

# Actor output Schema

## `records` (type: `string`):

Encrypted rows and optional **error** rows in the default dataset. Run summary is stored in KV record RUN\_SUMMARY.json.

## `kvBundle` (type: `string`):

Combined encrypted output when outputMode is kv\_store or both (default key ENCRYPTED\_OUTPUT).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceMode": "json_records",
    "sourceJsonRecords": "",
    "sourceJsonText": "",
    "maxItems": 10,
    "fieldsToEncrypt": [
        "email",
        "phone"
    ],
    "preserveFields": [
        "id"
    ],
    "passphrase": "",
    "rawKeyBase64": "",
    "iterations": 1000,
    "maxConcurrency": 2,
    "deterministicFingerprintSalt": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("sovanza.inc/encrypted-data-integration").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sourceMode": "json_records",
    "sourceJsonRecords": "",
    "sourceJsonText": "",
    "maxItems": 10,
    "fieldsToEncrypt": [
        "email",
        "phone",
    ],
    "preserveFields": ["id"],
    "passphrase": "",
    "rawKeyBase64": "",
    "iterations": 1000,
    "maxConcurrency": 2,
    "deterministicFingerprintSalt": "",
}

# Run the Actor and wait for it to finish
run = client.actor("sovanza.inc/encrypted-data-integration").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceMode": "json_records",
  "sourceJsonRecords": "",
  "sourceJsonText": "",
  "maxItems": 10,
  "fieldsToEncrypt": [
    "email",
    "phone"
  ],
  "preserveFields": [
    "id"
  ],
  "passphrase": "",
  "rawKeyBase64": "",
  "iterations": 1000,
  "maxConcurrency": 2,
  "deterministicFingerprintSalt": ""
}' |
apify call sovanza.inc/encrypted-data-integration --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=sovanza.inc/encrypted-data-integration",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w1TvmxcOp2djOe4ds/builds/vW7ZrKy3lxScsrgVM/openapi.json