# Rag Vector Store Writer (`labrat011/rag-vector-store-writer`) Actor Apify Actor that writes embedding vectors to Pinecone or Qdrant vector databases. Chains directly with RAG Embedding Generator output or accepts raw vectors with metadata. Handles batching, retries, collection creation, metadata mapping, and ID generation. Bring your own vector DB API key. - **URL**: https://apify.com/labrat011/rag-vector-store-writer.md - **Developed by:** [mick\_](https://apify.com/labrat011) (community) - **Categories:** AI, Agents, Automation - **Stats:** 4 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.40 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## RAG Vector Store Writer Apify Actor that writes embedding vectors to Pinecone or Qdrant vector databases. Chains directly with RAG Embedding Generator output or accepts raw vectors with metadata. Handles batching, retries, collection creation, metadata mapping, and ID generation. Bring your own vector DB API key. MCP-ready for AI agent integration. ### Features - Two vector database providers: Pinecone and Qdrant Cloud - Two input modes: dataset chaining (from RAG Embedding Generator) or raw vector JSON - Pinecone: resolves index host via control plane, batched upserts (max 1000/batch), namespace support - Qdrant: auto-creates collection if missing (with configurable distance metric), batched upserts - Metadata pass-through: all non-embedding fields from input become metadata/payload in the vector DB - Configurable vector ID field (default: `chunk_id` from RAG Content Chunker), UUID fallback - Exponential backoff retry on rate limits and transient failures (3 attempts) - API key marked `isSecret` -- never logged, never stored, never included in output - Hardcoded API URLs and URL pattern validation to prevent SSRF attacks - Input validation and sanitization (provider whitelist, index name regex, batch size limits) ### Requirements - Python 3.11+ - Apify platform account (for running as Actor) - Pinecone or Qdrant Cloud API key and an existing index/cluster Install dependencies: ```bash pip install -r requirements.txt ```` ### Configuration #### Actor Inputs Defined in `.actor/INPUT_SCHEMA.json`: - `api_key` (string, required) -- your Pinecone or Qdrant API key. Marked `isSecret` - `provider` (string, optional) -- `"pinecone"` (default) or `"qdrant"` - `index_name` (string, required) -- Pinecone index name or Qdrant collection name. Qdrant collections are auto-created if they don't exist - `environment` (string, optional) -- Qdrant Cloud cluster URL (e.g., `https://xyz.us-east-1.aws.cloud.qdrant.io:6333`). Required for Qdrant - `namespace` (string, optional) -- Pinecone namespace. Leave empty for default namespace. Ignored for Qdrant - `distance_metric` (string, optional) -- Distance metric for Qdrant collection creation: `"Cosine"` (default), `"Dot"`, or `"Euclid"`. Only used when auto-creating a new collection. Ignored for Pinecone - `dataset_id` (string, optional) -- Apify dataset ID from RAG Embedding Generator. Items must have an `embedding` field. Takes priority over `vectors` - `vectors` (array, optional) -- Direct vector input as JSON array. Each item needs an `embedding` field - `batch_size` (integer, optional) -- Vectors per upsert request. Default: 100. Pinecone max: 1000, Qdrant max: 500 - `id_field` (string, optional) -- Field to use as vector ID. Default: `"chunk_id"`. Falls back to UUID if missing At least one of `dataset_id` or `vectors` must be provided, plus `api_key` and `index_name`. ### Usage #### Local (CLI) ```bash APIFY_TOKEN=your-token apify run ``` #### Pinecone -- Dataset Chaining ```json { "api_key": "pcsk_your-pinecone-key", "provider": "pinecone", "index_name": "my-rag-index", "namespace": "docs", "dataset_id": "abc123XYZ", "batch_size": 200 } ``` #### Qdrant -- Dataset Chaining ```json { "api_key": "your-qdrant-api-key", "provider": "qdrant", "index_name": "my-collection", "environment": "https://xyz-example.us-east-1.aws.cloud.qdrant.io:6333", "distance_metric": "Cosine", "dataset_id": "abc123XYZ" } ``` #### Raw Vectors (Direct Input) ```json { "api_key": "pcsk_your-pinecone-key", "provider": "pinecone", "index_name": "my-index", "vectors": [ { "chunk_id": "doc1-chunk0", "embedding": [0.0123, -0.0456, 0.0789], "source_url": "https://example.com/page", "page_title": "Example Page" }, { "chunk_id": "doc1-chunk1", "embedding": [0.0321, -0.0654, 0.0987], "source_url": "https://example.com/page", "page_title": "Example Page" } ] } ``` #### Example Output A summary item is pushed to the default dataset: ```json { "_summary": true, "provider": "pinecone", "total_vectors_upserted": 42, "total_batches": 1, "processing_time": 2.341, "index_name": "my-rag-index", "namespace": "docs", "billing": { "total_vectors": 42, "amount": 0.0168, "rate_per_vector": 0.0004 } } ``` ### Pipeline Position This actor fills the vector storage step in a standard RAG pipeline: ``` Crawl (Website Content Crawler, 101K+ users) -> Clean (optional preprocessing) -> Chunk (RAG Content Chunker) -> Embed (RAG Embedding Generator) -> Store (this actor) ``` #### Chaining with RAG Embedding Generator 1. Run RAG Content Chunker on your text or crawler output 2. Run RAG Embedding Generator with the chunker's output dataset ID 3. Copy the embedding generator's output dataset ID 4. Pass it as `dataset_id` to this actor 5. This actor reads each embedding, skips `_summary` rows, and upserts vectors with all metadata preserved The full pipeline: crawl -> chunk -> embed -> store. Three dataset IDs passed between four actors. ### Architecture - `src/agent/main.py` -- Actor entry point, input routing (dataset/vectors), dataset loading, provider dispatch - `src/agent/validation.py` -- Input validation, provider whitelist, index name regex, Qdrant URL pattern check (SSRF prevention), batch size limits - `src/agent/writers/pinecone.py` -- Pinecone writer: host resolution via control plane, batched upserts, retry, metadata mapping - `src/agent/writers/qdrant.py` -- Qdrant writer: collection auto-creation, batched upserts, retry, payload mapping - `src/agent/pricing.py` -- PPE billing calculator ($0.0004/vector) - `skill.md` -- Machine-readable skill contract for agent discovery ### Security - **API key handling**: Marked `isSecret` in input schema, validated for presence only, never logged or stored, stripped from error messages via `_sanitize_error()` - **SSRF prevention**: Pinecone host resolved via hardcoded control plane URL (`api.pinecone.io`). Qdrant cluster URLs validated against strict `cloud.qdrant.io` pattern - **Provider whitelist**: Only `pinecone` and `qdrant` accepted - **Input sanitization**: Control characters stripped, index names and dataset IDs regex-validated, batch sizes bounded - **Error safety**: All error messages pass through sanitization to prevent API key leakage - **No data retention**: Vectors exist only in memory during the run ### Pricing Pay-Per-Event (PPE): **$0.0004 per vector** ($0.40 per 1,000 vectors). This is the actor's platform fee only. You also pay the vector database provider (Pinecone or Qdrant) directly via your own account. | Content Size | Approx. Vectors | Actor Fee | Notes | |-------------|----------------|-----------|-------| | Single blog post | 10-20 | $0.004-$0.008 | Typical small doc | | 10-page website | 50-100 | $0.02-$0.04 | Standard site | | 100-page docs site | 500-1,000 | $0.20-$0.40 | Documentation portal | | Large knowledge base | 5,000-10,000 | $2.00-$4.00 | Enterprise scale | ### Troubleshooting - **"API key is required"**: Provide your Pinecone or Qdrant API key in the `api_key` field - **"Invalid provider"**: Must be `"pinecone"` or `"qdrant"` - **"Index/collection name is required"**: Provide the name of your Pinecone index or Qdrant collection - **"Invalid Qdrant cluster URL"**: Must match `https://{cluster}.cloud.qdrant.io:6333` pattern - **"No input provided"**: Supply either `dataset_id` or `vectors` - **"API key is invalid or expired"**: Your provider key was rejected. Verify it in your Pinecone/Qdrant dashboard - **"Could not resolve host for Pinecone index"**: The index name doesn't exist in your Pinecone project - **"Failed after 3 attempts"**: Transient API error. Try again, or reduce `batch_size` - **"No items with valid 'embedding' arrays"**: Dataset items must have an `embedding` field with a non-empty float array ### License See `LICENSE` file for details. *** ### MCP Integration This actor works as an MCP tool through Apify's hosted MCP server. No custom server needed. - **Endpoint:** `https://mcp.apify.com?tools=labrat011/rag-vector-store-writer` - **Auth:** `Authorization: Bearer ` - **Transport:** Streamable HTTP - **Works with:** Claude Desktop, Cursor, VS Code, Windsurf, Warp, Gemini CLI **Example MCP config (Claude Desktop / Cursor):** ```json { "mcpServers": { "rag-vector-store-writer": { "url": "https://mcp.apify.com?tools=labrat011/rag-vector-store-writer", "headers": { "Authorization": "Bearer " } } } } ``` AI agents can use this actor to write embedding vectors to Pinecone or Qdrant, build vector indexes, and populate RAG knowledge bases -- all as a callable MCP tool. # Actor input Schema ## `api_key` (type: `string`): Your Pinecone or Qdrant API key. Never logged or stored beyond this run. ## `provider` (type: `string`): Which vector database to write to. ## `index_name` (type: `string`): Pinecone: the index name (used to resolve the host). Qdrant: the collection name. Will be auto-created for Qdrant if it doesn't exist. ## `environment` (type: `string`): Qdrant Cloud only: your full cluster URL (e.g., 'https://xyz-example.us-east-1.aws.cloud.qdrant.io:6333'). Not needed for Pinecone. ## `namespace` (type: `string`): Pinecone namespace to write vectors into. Leave empty for the default namespace. Ignored for Qdrant. ## `distance_metric` (type: `string`): Distance metric for Qdrant collection creation. Only used when auto-creating a new collection. Ignored for Pinecone. ## `dataset_id` (type: `string`): ID of an Apify dataset containing embedding vectors (e.g., output from RAG Embedding Generator). Each item should have an 'embedding' field. Takes priority over 'vectors' input. ## `vectors` (type: `array`): Direct vector input as a JSON array. Each item: {"id": "optional-id", "embedding": \[0.1, ...], "metadata": {"key": "value"}}. Use dataset\_id for chaining with Embedding Generator instead. ## `batch_size` (type: `integer`): Number of vectors to upsert per API request. Pinecone max: 1000 (recommended: 100-200 for 1536d vectors). Qdrant: no hard limit (recommended: 100). ## `id_field` (type: `string`): Which field from dataset items to use as the vector ID. Default: 'chunk\_id' (from RAG Content Chunker). Falls back to UUID generation if the field is missing. ## Actor input object example ```json { "provider": "pinecone", "namespace": "", "distance_metric": "Cosine", "batch_size": 100, "id_field": "chunk_id" } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("labrat011/rag-vector-store-writer").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("labrat011/rag-vector-store-writer").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call labrat011/rag-vector-store-writer --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=labrat011/rag-vector-store-writer", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Rag Vector Store Writer", "description": "Apify Actor that writes embedding vectors to Pinecone or Qdrant vector databases. Chains directly with RAG Embedding Generator output or accepts raw vectors with metadata. Handles batching, retries, collection creation, metadata mapping, and ID generation. Bring your own vector DB API key.", "version": "0.0", "x-build-id": "9Rry7wqnjSHPjo0gS" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/labrat011~rag-vector-store-writer/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-labrat011-rag-vector-store-writer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/labrat011~rag-vector-store-writer/runs": { "post": { "operationId": "runs-sync-labrat011-rag-vector-store-writer", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/labrat011~rag-vector-store-writer/run-sync": { "post": { "operationId": "run-sync-labrat011-rag-vector-store-writer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "api_key": { "title": "Vector DB API Key", "type": "string", "description": "Your Pinecone or Qdrant API key. Never logged or stored beyond this run." }, "provider": { "title": "Vector Database Provider", "enum": [ "pinecone", "qdrant" ], "type": "string", "description": "Which vector database to write to.", "default": "pinecone" }, "index_name": { "title": "Index / Collection Name", "type": "string", "description": "Pinecone: the index name (used to resolve the host). Qdrant: the collection name. Will be auto-created for Qdrant if it doesn't exist." }, "environment": { "title": "Qdrant Cluster URL", "type": "string", "description": "Qdrant Cloud only: your full cluster URL (e.g., 'https://xyz-example.us-east-1.aws.cloud.qdrant.io:6333'). Not needed for Pinecone." }, "namespace": { "title": "Namespace (Pinecone only)", "type": "string", "description": "Pinecone namespace to write vectors into. Leave empty for the default namespace. Ignored for Qdrant.", "default": "" }, "distance_metric": { "title": "Distance Metric (Qdrant only)", "enum": [ "Cosine", "Dot", "Euclid" ], "type": "string", "description": "Distance metric for Qdrant collection creation. Only used when auto-creating a new collection. Ignored for Pinecone.", "default": "Cosine" }, "dataset_id": { "title": "Apify Dataset ID (from Embedding Generator)", "type": "string", "description": "ID of an Apify dataset containing embedding vectors (e.g., output from RAG Embedding Generator). Each item should have an 'embedding' field. Takes priority over 'vectors' input." }, "vectors": { "title": "Raw Vectors (JSON)", "type": "array", "description": "Direct vector input as a JSON array. Each item: {\"id\": \"optional-id\", \"embedding\": [0.1, ...], \"metadata\": {\"key\": \"value\"}}. Use dataset_id for chaining with Embedding Generator instead." }, "batch_size": { "title": "Batch Size", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Number of vectors to upsert per API request. Pinecone max: 1000 (recommended: 100-200 for 1536d vectors). Qdrant: no hard limit (recommended: 100).", "default": 100 }, "id_field": { "title": "ID Field", "type": "string", "description": "Which field from dataset items to use as the vector ID. Default: 'chunk_id' (from RAG Content Chunker). Falls back to UUID generation if the field is missing.", "default": "chunk_id" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```