# Pinecone Integration (`apify/pinecone-integration`) Actor This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case. - **URL**: https://apify.com/apify/pinecone-integration.md - **Developed by:** [Apify](https://apify.com/apify) (Apify) - **Categories:** AI, Integrations, Open source - **Stats:** 555 total users, 33 monthly users, 98.7% runs succeeded, 39 bookmarks - **User rating**: 3.21 out of 5 stars ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Pinecone integration [![Pinecone integration](https://apify.com/actor-badge?actor=apify/pinecone-integration)](https://apify.com/apify/pinecone-integration) The Apify Pinecone integration transfers selected data from Apify Actors to a [Pinecone](https://www.pinecone.io) database. It processes the data, optionally splits it into chunks, computes embeddings, and saves them to Pinecone. This integration supports incremental updates, updating only the data that has changed. This approach reduces unnecessary embedding computation and storage operations, making it suitable for search and retrieval augmented generation (RAG) use cases. 💡 **Note**: This Actor is meant to be used together with other Actors' integration sections. For instance, if you are using the [Website Content Crawler](https://apify.com/apify/website-content-crawler), you can activate Pinecone integration to save web data as vectors to Pinecone. For more information how to leverage vector stores in Apify platform, see detailed blog post [What Pinecone is and why you should use it with your LLMs](https://blog.apify.com/what-is-pinecone-why-use-it-with-llms/). For setup instructions in the Apify Console, see the [Pinecone integration guide](https://docs.apify.com/platform/integrations/pinecone). #### What is Pinecone vector database? [Pinecone](https://www.pinecone.io/) is a cloud-native vector database designed to manage high-dimensional vector data efficiently. Its core functionality is based on the Approximate Nearest Neighbor (ANN) search, which quickly identifies and ranks matches within large datasets. The serverless architecture of Pinecone enables rapid development and deployment of powerful applications. Pinecone supports clients in Python, Java, Node.js, and Golang. ### 📋 How does the Apify-Pinecone integration work? Apify Pinecone integration computes text embeddings and store them in Pinecone. It uses [LangChain](https://www.langchain.com/) to compute embeddings and interact with [Pinecone](https://www.pinecone.io/). 1. Retrieve a dataset as output from an Actor 2. _[Optional]_ Split text data into chunks using `langchain`'s `RecursiveCharacterTextSplitter` (enable/disable using `performChunking` and specify `chunkSize`, `chunkOverlap`) 3. _[Optional]_ Update only changed data (select `dataUpdatesStrategy`) 4. Compute embeddings, e.g. using `OpenAI` or `Cohere` (specify `embeddingsProvider` and `embeddingsConfig`) 5. Save data into the database ![Apify-pinecone-integration](https://raw.githubusercontent.com/apify/actor-vector-database-integrations/master/docs/Apify-pinecone-integration-readme.png) ### ✅ Before you start To utilize this integration, ensure you have: - Created or existing `Pinecone` database. You need to know `indexName` and `apiKey`. - An account to compute embeddings using one of the providers, e.g., OpenAI or Cohere. ### 👉 Examples The configuration consists of three parts: Pinecone, embeddings provider, and data. Ensure that the vector size of your embeddings aligns with the configuration of your Pinecone index. For instance, if you're using the `text-embedding-3-small` model from `OpenAI`, it generates vectors of size `1536`. This means your Pinecone index should also be configured to accommodate vectors of the same size, `1536` in this case. For detailed input information refer to the [Input page](https://apify.com/apify/pinecone-integration/input-schema). ##### Database: Pinecone ```json { "pineconeApiKey": "YOUR-PINECONE-API-KEY", "pineconeIndexName": "apify" } ```` ##### Embeddings provider: OpenAI ```json { "embeddingsProvider": "OpenAI", "embeddingsApiKey": "YOUR-OPENAI-API-KEY", "embeddingsConfig": {"model": "text-embedding-3-large"} } ``` ##### Embeddings provider: Cohere ```json { "embeddingsApiKey": "YOUR-COHERE-API-KEY", "embeddingsProvider": "Cohere", "embeddingsConfig": {"model": "embed-multilingual-v3.0"} } ``` #### Save data from Website Content Crawler to Pinecone Data is transferred in the form of a dataset from [Website Content Crawler](https://apify.com/apify/website-content-crawler), which provides a dataset with the following output fields (truncated for brevity): ```json { "url": "https://www.apify.com", "text": "Apify is a platform that enables developers to build, run, and share automation tasks.", "metadata": {"title": "Apify"} } ``` This dataset is then processed by the Pinecone integration. In the integration settings you need to specify which fields you want to save to Pinecone, e.g., `["text"]` and which of them should be used as metadata, e.g., `{"title": "metadata.title"}`. Without any other configuration, the data is saved to Pinecone as is. ```json { "datasetFields": ["text"], "metadataDatasetFields": {"title": "metadata.title"} } ``` #### Create chunks from Website Content Crawler data and save them to the database Assume that the text data from the [Website Content Crawler](https://apify.com/apify/website-content-crawler) is too long to compute embeddings. Therefore, we need to divide the data into smaller pieces called chunks. We can leverage LangChain's `RecursiveCharacterTextSplitter` to split the text into chunks and save them into a database. The parameters `chunkSize` and `chunkOverlap` are important. The settings depend on your use case where a proper chunking helps optimize retrieval and ensures accurate responses. ```json { "datasetFields": ["text"], "metadataDatasetFields": {"title": "metadata.title"}, "performChunking": true, "chunkSize": 1000, "chunkOverlap": 0 } ``` #### Configure update strategy To control how the integration updates data in the database, use the `dataUpdatesStrategy` parameter. This parameter allows you to choose between different update strategies based on your use case, such as adding new data, upserting records, or incrementally updating records based on changes (deltas). Below are the available strategies and explanations for when to use each: - **Add data (`add`)**: - Appends new data to the database without checking for duplicates or updating existing records. - Suitable for cases where deduplication or updates are unnecessary, and the data simply needs to be added. - For example, you might use this strategy to continually append data from independent crawls without regard for overlaps. - **Upsert data (`upsert`)**: - Delete existing records in the database if they match a key or identifier and inserts new records. - Ideal when you want to maintain accurate and up-to-date data while avoiding duplication. - For instance, this is useful in cases where unique items (such as user profiles or documents) need to be managed, ensuring the database reflects the latest changes. - Check the `dataUpdatesPrimaryDatasetFields` parameter to specify which fields are used to uniquely identify each dataset item. - **Delta updates (`deltaUpdates`)**: - Incrementally updates records by identifying differences (deltas) between the new dataset and the existing database records. - Ensures only new or modified records are processed, leaving unchanged records untouched. This minimizes unnecessary database operations and improves efficiency. - This is the most efficient strategy when integrating data that evolves over time, such as website content or recurring crawls. - Check the `dataUpdatesPrimaryDatasetFields` parameter to specify which fields are used to uniquely identify each dataset item. #### Incrementally update database from the Website Content Crawler To incrementally update data from the [Website Content Crawler](https://apify.com/apify/website-content-crawler) to database, configure the integration to update only the changed or new data. This is controlled by the `dataUpdatesStrategy` setting. This way, the integration minimizes unnecessary updates and ensures that only new or modified data is processed. A checksum is computed for each dataset item (together with all metadata) and stored in the database alongside the vectors. When the data is re-crawled, the checksum is recomputed and compared with the stored checksum. If the checksum is different, the old data (including vectors) is deleted and new data is saved. Otherwise, only the `last_seen_at` metadata field is updated to indicate when the data was last seen. ##### Provide unique identifier for each dataset item To incrementally update the data, you need to be able to uniquely identify each dataset item. The variable `dataUpdatesPrimaryDatasetFields` specifies which fields are used to uniquely identify each dataset item and helps track content changes across different crawls. For instance, when working with the Website Content Crawler, you can use the URL as a unique identifier. ```json { "dataUpdatesStrategy": "deltaUpdates", "dataUpdatePrimaryDatasetFields": ["url"] } ``` Additionally, you can use the `usePineconeIdPrefix` parameter to optimize delta updates by creating a Pinecone-specific ID prefix in the database instead of storing metadata. The prefix is auto generated using the format: `item_id#chunk_id`, which results in more efficient updates. ```json { "dataUpdatesStrategy": "deltaUpdates", "dataUpdatePrimaryDatasetFields": ["url"], "usePineconeIdPrefix": true } ``` To fully maximize the potential of incremental data updates, it is recommended to start with an empty database. While it is possible to use this feature with an existing database, records that were not originally saved using a prefix or metadata will not be updated. #### Delete outdated (expired) data The integration can delete data from the database that hasn't been crawled for a specified period, which is useful when data becomes outdated, such as when a page is removed from a website. The deletion feature can be enabled or disabled using the `deleteExpiredObjects` setting. For each crawl, the `last_seen_at` metadata field is created or updated. This field records the most recent time the data object was crawled. The `expiredObjectDeletionPeriodDays` setting is used to control number of days since the last crawl, after which the data object is considered expired. If a database object has not been seen for more than the `expiredObjectDeletionPeriodDays`, it will be deleted automatically. The specific value of `expiredObjectDeletionPeriodDays` depends on your use case. - If a website is crawled daily, `expiredObjectDeletionPeriodDays` can be set to 7. - If you crawl weekly, it can be set to 30. To disable this feature, set `deleteExpiredObjects` to `false`. ```json { "deleteExpiredObjects": true, "expiredObjectDeletionPeriodDays": 30 } ``` 💡 If you are using multiple Actors to update the same database, ensure that all Actors crawl the data at the same frequency. Otherwise, data crawled by one Actor might expire due to inconsistent crawling schedules. ### 💾 Outputs This integration will save the selected fields from your Actor to Pinecone and store the chunked data in the Apify dataset. ### 🔢 Example configuration ##### Full Input Example for Website Content Crawler Actor with Pinecone integration ```json { "pineconeApiKey": "YOUR-PINECONE-API-KEY", "pineconeIndexName": "apify", "embeddingsApiKey": "YOUR-OPENAI-API-KEY", "embeddingsConfig": { "model": "text-embedding-3-small" }, "embeddingsProvider": "OpenAI", "datasetFields": [ "text" ], "dataUpdatesStrategy": "deltaUpdates", "dataUpdatePrimaryDatasetFields": ["url"], "expiredObjectDeletionPeriodDays": 7, "performChunking": true, "chunkSize": 2000, "chunkOverlap": 200 } ``` ##### Pinecone ```json { "pineconeApiKey": "YOUR-PINECONE-API-KEY", "pineconeIndexName": "apify" } ``` ##### OpenAI embeddings ```json { "embeddingsApiKey": "YOUR-OPENAI-API-KEY", "embeddingsProvider": "OpenAI", "embeddingsConfig": {"model": "text-embedding-3-large"} } ``` ##### Cohere embeddings ```json { "embeddingsApiKey": "YOUR-COHERE-API-KEY", "embeddingsProvider": "Cohere", "embeddingsConfig": {"model": "embed-multilingual-v3.0"} } ``` # Actor input Schema ## `pineconeApiKey` (type: `string`): Pinecone API KEY ## `pineconeIndexName` (type: `string`): Name of the Pinecone index where the data will be stored ## `pineconeIndexNamespace` (type: `string`): Name of the Pinecone index namespace (partition the records in an index) ## `embeddingsProvider` (type: `string`): Choose the embeddings provider to use for generating embeddings ## `embeddingsConfig` (type: `object`): Configure the parameters for the LangChain embedding class. Key points to consider: 1. Typically, you only need to specify the model name. For example, for OpenAI, set the model name as {"model": "text-embedding-3-small"}. 2. It's required to ensure that the vector size of your embeddings matches the size of embeddings in the database. 3. Here are examples of embedding models: - [OpenAI](https://platform.openai.com/docs/guides/embeddings): `text-embedding-3-small`, `text-embedding-3-large`, etc. - [Cohere](https://docs.cohere.com/docs/cohere-embed): `embed-english-v3.0`, `embed-multilingual-light-v3.0`, etc. 4. For more details about other parameters, refer to the [LangChain documentation](https://python.langchain.com/docs/integrations/text_embedding/). ## `embeddingsApiKey` (type: `string`): Value of the API KEY for the embeddings provider (if required). For example for OpenAI it is OPENAI\_API\_KEY, for Cohere it is COHERE\_API\_KEY) ## `datasetFields` (type: `array`): This array specifies the dataset fields to be selected and stored in the vector store. Only the fields listed here will be included in the vector store. For instance, when using the Website Content Crawler, you might choose to include fields such as `text`, `url`, and `metadata.title` in the vector store. ## `metadataDatasetFields` (type: `object`): A list of dataset fields which should be selected from the dataset and stored as metadata in the vector stores. For example, when using the Website Content Crawler, you might want to store `url` in metadata. In this case, use `metadataDatasetFields parameter as follows {"url": "url"}` ## `metadataObject` (type: `object`): This object allows you to store custom metadata for every item in the vector store. For example, if you want to store the `domain` as metadata, use the `metadataObject` like this: {"domain": "apify.com"}. ## `datasetId` (type: `string`): Dataset ID (when running standalone without integration) ## `dataUpdatesStrategy` (type: `string`): Choose the update strategy for the integration. The update strategy determines how the integration updates the data in the database. The available options are: - **Add data** (`add`): - Always adds new records to the database. - No checks for existing records or updates are performed. - Useful when appending data without concern for duplicates. - **Upsert data** (`upsert`): - Updates existing records if they match a key or identifier. - Inserts new records into the database if they don't already exist. - Ideal for ensuring the database contains the most up-to-date data, avoiding duplicates. - **Update changed data based on deltas** (`deltaUpdates`): - Performs incremental updates by identifying differences (deltas) between the new dataset and the existing records. - Only adds new records and updates those that have changed. - Unchanged records are left untouched. - Maximizes efficiency by reducing unnecessary updates. Select the strategy that best fits your use case. ## `dataUpdatesPrimaryDatasetFields` (type: `array`): This array contains fields that are used to uniquely identify dataset items, which helps to handle content changes across different runs. For instance, in a web content crawling scenario, the `url` field could serve as a unique identifier for each item. ## `enableDeltaUpdates` (type: `boolean`): When set to true, this setting enables incremental updates for objects in the database by comparing the changes (deltas) between the crawled dataset items and the existing objects, uniquely identified by the `datasetKeysToItemId` field. The integration will only add new objects and update those that have changed, reducing unnecessary updates. The `datasetFields`, `metadataDatasetFields`, and `metadataObject` fields are used to determine the changes. ## `deltaUpdatesPrimaryDatasetFields` (type: `array`): This array contains fields that are used to uniquely identify dataset items, which helps to handle content changes across different runs. For instance, in a web content crawling scenario, the `url` field could serve as a unique identifier for each item. ## `deleteExpiredObjects` (type: `boolean`): When set to true, delete objects from the database that have not been crawled for a specified period. ## `expiredObjectDeletionPeriodDays` (type: `integer`): This setting allows the integration to manage the deletion of objects from the database that have not been crawled for a specified period. It is typically used in subsequent runs after the initial crawl. When the value is greater than 0, the integration checks if objects have been seen within the last X days (determined by the expiration period). If the objects are expired, they are deleted from the database. The specific value for `deletedExpiredObjectsDays` depends on your use case and how frequently you crawl data. For example, if you crawl data daily, you can set `deletedExpiredObjectsDays` to 7 days. If you crawl data weekly, you can set `deletedExpiredObjectsDays` to 30 days. ## `performChunking` (type: `boolean`): When set to true, the text will be divided into smaller chunks based on the settings provided below. Proper chunking helps optimize retrieval and ensures accurate and efficient responses. ## `chunkSize` (type: `integer`): Defines the maximum number of characters in each text chunk. Choosing the right size balances between detailed context and system performance. Optimal sizes ensure high relevancy and minimal response time. ## `chunkOverlap` (type: `integer`): Specifies the number of overlapping characters between consecutive text chunks. Adjusting this helps maintain context across chunks, which is crucial for accuracy in retrieval-augmented generation systems. ## `usePineconeIdPrefix` (type: `boolean`): When set to true, this option will use Pinecone ID prefix instead of metadata for handling deltaUpdates. It will create a prefix in the database using the following format: `item_id#chunk_id`. This will results in more efficient updates ## `embeddingBatchSize` (type: `integer`): The number of texts to embed in a single batch. This setting can be used to optimize the performance when embedding texts. If you receive Embedding provider errors, you need to decrease this size. ## Actor input object example ```json { "embeddingsProvider": "OpenAI", "datasetFields": [ "text" ], "dataUpdatesStrategy": "deltaUpdates", "dataUpdatesPrimaryDatasetFields": [ "url" ], "enableDeltaUpdates": true, "deltaUpdatesPrimaryDatasetFields": [ "url" ], "deleteExpiredObjects": true, "expiredObjectDeletionPeriodDays": 30, "performChunking": true, "chunkSize": 2000, "chunkOverlap": 0, "usePineconeIdPrefix": false, "embeddingBatchSize": 1000 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "datasetFields": [ "text" ], "dataUpdatesStrategy": "deltaUpdates", "dataUpdatesPrimaryDatasetFields": [ "url" ], "deltaUpdatesPrimaryDatasetFields": [ "url" ] }; // Run the Actor and wait for it to finish const run = await client.actor("apify/pinecone-integration").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "datasetFields": ["text"], "dataUpdatesStrategy": "deltaUpdates", "dataUpdatesPrimaryDatasetFields": ["url"], "deltaUpdatesPrimaryDatasetFields": ["url"], } # Run the Actor and wait for it to finish run = client.actor("apify/pinecone-integration").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "datasetFields": [ "text" ], "dataUpdatesStrategy": "deltaUpdates", "dataUpdatesPrimaryDatasetFields": [ "url" ], "deltaUpdatesPrimaryDatasetFields": [ "url" ] }' | apify call apify/pinecone-integration --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=apify/pinecone-integration", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Pinecone Integration", "description": "This integration transfers data from Apify Actors to a Pinecone and is a good starting point for a question-answering, search, or RAG use case.", "version": "0.1", "x-build-id": "FeYYVdQFofgp5UYWK" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/apify~pinecone-integration/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-apify-pinecone-integration", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/apify~pinecone-integration/runs": { "post": { "operationId": "runs-sync-apify-pinecone-integration", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/apify~pinecone-integration/run-sync": { "post": { "operationId": "run-sync-apify-pinecone-integration", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "pineconeIndexName", "pineconeApiKey", "embeddingsProvider", "embeddingsApiKey", "datasetFields" ], "properties": { "pineconeApiKey": { "title": "Pinecone API KEY", "type": "string", "description": "Pinecone API KEY" }, "pineconeIndexName": { "title": "Pinecone index name", "type": "string", "description": "Name of the Pinecone index where the data will be stored" }, "pineconeIndexNamespace": { "title": "Pinecone index namespace", "type": "string", "description": "Name of the Pinecone index namespace (partition the records in an index)" }, "embeddingsProvider": { "title": "Embeddings provider (as defined in the langchain API)", "enum": [ "OpenAI", "Cohere" ], "type": "string", "description": "Choose the embeddings provider to use for generating embeddings", "default": "OpenAI" }, "embeddingsConfig": { "title": "Configuration for embeddings provider", "type": "object", "description": "Configure the parameters for the LangChain embedding class. Key points to consider:\n\n1. Typically, you only need to specify the model name. For example, for OpenAI, set the model name as {\"model\": \"text-embedding-3-small\"}.\n\n2. It's required to ensure that the vector size of your embeddings matches the size of embeddings in the database.\n\n3. Here are examples of embedding models:\n - [OpenAI](https://platform.openai.com/docs/guides/embeddings): `text-embedding-3-small`, `text-embedding-3-large`, etc.\n - [Cohere](https://docs.cohere.com/docs/cohere-embed): `embed-english-v3.0`, `embed-multilingual-light-v3.0`, etc.\n\n4. For more details about other parameters, refer to the [LangChain documentation](https://python.langchain.com/docs/integrations/text_embedding/)." }, "embeddingsApiKey": { "title": "Embeddings API KEY (whenever applicable, depends on provider)", "type": "string", "description": "Value of the API KEY for the embeddings provider (if required).\n\n For example for OpenAI it is OPENAI_API_KEY, for Cohere it is COHERE_API_KEY)" }, "datasetFields": { "title": "Dataset fields to select from the dataset results and store in the database", "type": "array", "description": "This array specifies the dataset fields to be selected and stored in the vector store. Only the fields listed here will be included in the vector store.\n\nFor instance, when using the Website Content Crawler, you might choose to include fields such as `text`, `url`, and `metadata.title` in the vector store.", "default": [ "text" ], "items": { "type": "string" } }, "metadataDatasetFields": { "title": "Dataset fields to select from the dataset and store as metadata in the database", "type": "object", "description": "A list of dataset fields which should be selected from the dataset and stored as metadata in the vector stores.\n\nFor example, when using the Website Content Crawler, you might want to store `url` in metadata. In this case, use `metadataDatasetFields parameter as follows {\"url\": \"url\"}`" }, "metadataObject": { "title": "Custom object to be stored as metadata in the vector store database", "type": "object", "description": "This object allows you to store custom metadata for every item in the vector store.\n\nFor example, if you want to store the `domain` as metadata, use the `metadataObject` like this: {\"domain\": \"apify.com\"}." }, "datasetId": { "title": "Dataset ID", "type": "string", "description": "Dataset ID (when running standalone without integration)" }, "dataUpdatesStrategy": { "title": "Update strategy (add, upsert, deltaUpdates (default))", "enum": [ "add", "upsert", "deltaUpdates" ], "type": "string", "description": "Choose the update strategy for the integration. The update strategy determines how the integration updates the data in the database.\n\nThe available options are:\n\n- **Add data** (`add`):\n - Always adds new records to the database.\n - No checks for existing records or updates are performed.\n - Useful when appending data without concern for duplicates.\n\n- **Upsert data** (`upsert`):\n - Updates existing records if they match a key or identifier.\n - Inserts new records into the database if they don't already exist.\n - Ideal for ensuring the database contains the most up-to-date data, avoiding duplicates.\n\n- **Update changed data based on deltas** (`deltaUpdates`):\n - Performs incremental updates by identifying differences (deltas) between the new dataset and the existing records.\n - Only adds new records and updates those that have changed.\n - Unchanged records are left untouched.\n - Maximizes efficiency by reducing unnecessary updates.\n\nSelect the strategy that best fits your use case.", "default": "deltaUpdates" }, "dataUpdatesPrimaryDatasetFields": { "title": "Dataset fields to uniquely identify dataset items (only relevant when dataUpdatesStrategy is `upsert` or `deltaUpdates`)", "type": "array", "description": "This array contains fields that are used to uniquely identify dataset items, which helps to handle content changes across different runs.\n\nFor instance, in a web content crawling scenario, the `url` field could serve as a unique identifier for each item.", "default": [ "url" ], "items": { "type": "string" } }, "enableDeltaUpdates": { "title": "Enable incremental updates for objects based on deltas (deprecated)", "type": "boolean", "description": "When set to true, this setting enables incremental updates for objects in the database by comparing the changes (deltas) between the crawled dataset items and the existing objects, uniquely identified by the `datasetKeysToItemId` field.\n\n The integration will only add new objects and update those that have changed, reducing unnecessary updates. The `datasetFields`, `metadataDatasetFields`, and `metadataObject` fields are used to determine the changes.", "default": true }, "deleteExpiredObjects": { "title": "Delete expired objects from the database", "type": "boolean", "description": "When set to true, delete objects from the database that have not been crawled for a specified period.", "default": true }, "expiredObjectDeletionPeriodDays": { "title": "Delete expired objects from the database after a specified number of days", "minimum": 0, "type": "integer", "description": "This setting allows the integration to manage the deletion of objects from the database that have not been crawled for a specified period. It is typically used in subsequent runs after the initial crawl.\n\nWhen the value is greater than 0, the integration checks if objects have been seen within the last X days (determined by the expiration period). If the objects are expired, they are deleted from the database. The specific value for `deletedExpiredObjectsDays` depends on your use case and how frequently you crawl data.\n\nFor example, if you crawl data daily, you can set `deletedExpiredObjectsDays` to 7 days. If you crawl data weekly, you can set `deletedExpiredObjectsDays` to 30 days.", "default": 30 }, "performChunking": { "title": "Enable text chunking", "type": "boolean", "description": "When set to true, the text will be divided into smaller chunks based on the settings provided below. Proper chunking helps optimize retrieval and ensures accurate and efficient responses.", "default": true }, "chunkSize": { "title": "Maximum chunk size", "minimum": 1, "type": "integer", "description": "Defines the maximum number of characters in each text chunk. Choosing the right size balances between detailed context and system performance. Optimal sizes ensure high relevancy and minimal response time.", "default": 2000 }, "chunkOverlap": { "title": "Chunk overlap", "minimum": 0, "type": "integer", "description": "Specifies the number of overlapping characters between consecutive text chunks. Adjusting this helps maintain context across chunks, which is crucial for accuracy in retrieval-augmented generation systems.", "default": 0 }, "usePineconeIdPrefix": { "title": "Use Pinecone ID prefix", "type": "boolean", "description": "When set to true, this option will use Pinecone ID prefix instead of metadata for handling deltaUpdates. It will create a prefix in the database using the following format: `item_id#chunk_id`. This will results in more efficient updates", "default": false }, "embeddingBatchSize": { "title": "Batch size to use when embedding the texts", "minimum": 1, "type": "integer", "description": "The number of texts to embed in a single batch. This setting can be used to optimize the performance when embedding texts. If you receive Embedding provider errors, you need to decrease this size.", "default": 1000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```