# Commodities Scraper (`nocodeventure/commodities-scraper`) Actor Scrape real-time commodity prices from financial websites. Supports oil, gold, silver, natural gas, and more. Extract price, change, volume, technical indicators, and contract details. Currently supports Investing.com with more sites coming soon. - **URL**: https://apify.com/nocodeventure/commodities-scraper.md - **Developed by:** [No-Code Venture](https://apify.com/nocodeventure) (community) - **Categories:** AI, Automation - **Stats:** 5 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Commodity Price Scraper 📈 **Real-time commodity prices.** Track crude oil, gold, silver, natural gas, and more from leading financial websites. Get current prices, trading data, contract details, and technical indicators — all extracted automatically. **Currently supports:** Investing.com **Coming soon:** More financial data sources ### ✨ Features - **💰 Real-Time Prices** – Current price, change, and change percentage - **📊 Trading Data** – Previous close, open, day's range, 52-week range, volume - **📋 Contract Details** – Settlement date, tick size, contract size, months, rollover dates - **📈 Technical Indicators** – Summary signals (Buy/Sell/Neutral) - **❓ FAQ Extraction** – Optionally extract FAQ questions and answers about the commodity - **📦 Multiple Commodities** – Scrape multiple commodities in a single run - **🔐 Login Support** – Authenticate with email/password for full data access - **🎭 Stealth Mode** – Uses Camoufox (stealthy Firefox fork) to avoid detection - **🔄 Multi-Site Architecture** – Easily extensible to support more financial sites ### 🚀 Getting Started #### Basic Usage Scrape multiple commodity prices from Investing.com: ```json { "site": "investing.com", "commodities": ["crude-oil", "gold", "natural-gas"] } ```` Or scrape a single commodity: ```json { "site": "investing.com", "commodities": ["crude-oil"] } ``` ### 📦 Supported Commodities | Commodity | Slug | Description | |-----------|------|-------------| | Crude Oil WTI | `crude-oil` | West Texas Intermediate crude | | Brent Oil | `brent-oil` | North Sea Brent crude | | Natural Gas | `natural-gas` | Henry Hub natural gas | | Gold | `gold` | Gold futures | | Silver | `silver` | Silver futures | | Copper | `copper` | Copper futures | | Platinum | `platinum` | Platinum futures | | Palladium | `palladium` | Palladium futures | | Heating Oil | `heating-oil` | No. 2 heating oil | | Gasoline RBOB | `gasoline-rbob` | Reformulated gasoline | > 💡 **Tip:** You can use any commodity slug from Investing.com. Just use the URL path (e.g., `https://www.investing.com/commodities/crude-oil` → `crude-oil`). ### 🔑 Authentication Authentication is optional but provides full data access on Investing.com. ```json { "site": "investing.com", "commodities": ["gold", "silver"], "skipAuthentication": false, "investingEmail": "your-email@example.com", "investingPassword": "your-password" } ``` ### ⚙️ Input Parameters | Parameter | Type | Required | Description | Default | |-----------|------|----------|-------------|---------| | `site` | String | ✅ | Website to scrape from | `investing.com` | | `commodities` | String\[] | ✅ | List of commodity slugs (e.g., `["crude-oil", "gold"]`) | `["crude-oil"]` | | `skipAuthentication` | Boolean | ❌ | Skip login (some data may be limited) | `true` | | `investingEmail` | String | ❌ | Investing.com email | — | | `investingPassword` | String | ❌ | Investing.com password | — | | `includeFaq` | Boolean | ❌ | Extract FAQ questions and answers | `false` | | `maxConcurrency` | Integer | ❌ | Max concurrent browsers (1-10) | `1` | | `proxyConfiguration` | Object | ❌ | Proxy settings (not required, works without proxies) | Disabled | | `debugScreenshots` | Boolean | ❌ | Save screenshots for debugging | `true` | ### 📊 Output Example ```json { "name": "Crude Oil WTI", "symbol": "CL", "url": "https://www.investing.com/commodities/crude-oil", "scrapedAt": "2024-01-15T14:30:00.000Z", "source": "investing.com", "price": "72.45", "currency": "USD", "change": "-0.53", "changePercent": "(-0.73%)", "priceType": "Real-time derived", "prevClose": "72.98", "open": "72.98", "dayRange": "71.85-73.12", "weekRange52": "63.64-95.03", "volume": "218,634", "yearChange1": "-12.21%", "settlementDate": "01/22/2024", "settlementType": "Physical", "month": "02, 2024", "baseSymbol": "CL", "pointValue": "1 = $1000", "tickSize": "0.01", "tickValue": "10", "contractSize": "1,000 Barrels", "months": "FGHJKMNQUVXZ", "lastRolloverDay": "01/18/2024", "technicalSummary": "Sell", "faq": [ { "question": "What Is the Current Price of Crude Oil?", "answer": "The current price of Crude Oil WTI is $72.45 per barrel." }, { "question": "What Is the Daily Trading Range for Crude Oil?", "answer": "Today's trading range for Crude Oil is $71.85 - $73.12." } ] } ``` > 💡 **Note:** The `faq` field is only included when `includeFaq` is set to `true`. ### 🎯 Use Cases - **Trading & Investment** – Track real-time commodity prices for trading decisions - **Portfolio Monitoring** – Monitor commodities in your investment portfolio - **Market Research** – Analyze price trends and technical indicators - **Data Analytics** – Build datasets for price analysis and forecasting - **Price Alerts** – Integrate with workflows to trigger alerts on price changes - **Business Intelligence** – Track commodity costs for business planning ### ⚙️ How It Works 1. **Navigate** – Opens the commodity page on Investing.com 2. **Handle Cookies** – Automatically rejects tracking cookies (OneTrust) 3. **Authenticate** – Logs in with your email and password 4. **Extract** – Scrapes all available price and trading data 5. **Output** – Saves structured data to the dataset ### 🛡️ Proxy Support **Proxies are disabled by default and typically not needed.** The scraper uses Camoufox (a stealthy Firefox fork) which works well without proxies in most cases. If you experience blocking issues, you can enable proxies: ```json { "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ``` > 💡 **Tip:** Try running without proxies first. Only enable them if you encounter blocking. ### 💡 Tips - **Check commodity slugs** – Use the exact slug from the site's URL - **Enable debug screenshots** – Helpful for troubleshooting issues - **Multiple commodities** – Add as many commodities as you need to the `commodities` array - **Speed up scraping** – Increase `maxConcurrency` (up to 10) to run multiple browsers in parallel ### 🔮 Coming Soon - **More Sites** – Support for additional financial data sources - **Historical Data** – Extract historical price data - **Price Alerts** – Built-in price threshold notifications - **Forex & Indices** – Support for currency pairs and market indices ### 📜 Legal & Ethical Use This Actor is intended for legitimate data collection purposes. Please: - Respect each site's Terms of Service - Use reasonable request rates - Comply with applicable data protection laws - Don't use for market manipulation or illegal activities ### 🤝 Support Having issues? Found a bug? Want a feature? - Open an issue on the Actor's page - Contact the developer through Apify *** **Built by [nocodeventure](https://apify.com/nocodeventure)** • Made with ❤️ for the Apify community # Actor input Schema ## `commodities` (type: `array`): List of commodities to scrape price data for. Enter commodity slugs (e.g., 'crude-oil', 'gold', 'natural-gas'). ## `site` (type: `string`): The website to scrape commodity prices from. Only one site can be selected per run. ## `skipAuthentication` (type: `boolean`): When enabled, skips the login process. Some data may be limited without authentication. ## `investingEmail` (type: `string`): Your Investing.com email for authentication. Not required if Skip Authentication is enabled. ## `investingPassword` (type: `string`): Your Investing.com password. Not required if Skip Authentication is enabled. ## `maxConcurrency` (type: `integer`): Maximum number of browser instances running in parallel. Higher values speed up scraping but use more memory. ## `useSameTab` (type: `boolean`): When enabled (default), navigates to all URLs in the same browser tab - much faster. When disabled, opens a fresh browser for each URL - slower but more robust and reduces CPU pressure. ## `proxyConfiguration` (type: `object`): Proxy settings. The scraper works without proxies by default. Only enable if you experience blocking issues. ## `includeFaq` (type: `boolean`): When enabled, extracts FAQ data from the commodity page (questions and answers about the commodity). ## `debugScreenshots` (type: `boolean`): When enabled, saves screenshots on failures for debugging. Screenshots are saved as JPEG with low quality to save space. ## Actor input object example ```json { "commodities": [ "crude-oil", "gold" ], "site": "investing.com", "skipAuthentication": true, "maxConcurrency": 1, "useSameTab": true, "proxyConfiguration": { "useApifyProxy": false }, "includeFaq": false, "debugScreenshots": false } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "commodities": [ "crude-oil", "gold" ] }; // Run the Actor and wait for it to finish const run = await client.actor("nocodeventure/commodities-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "commodities": [ "crude-oil", "gold", ] } # Run the Actor and wait for it to finish run = client.actor("nocodeventure/commodities-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "commodities": [ "crude-oil", "gold" ] }' | apify call nocodeventure/commodities-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=nocodeventure/commodities-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Commodities Scraper", "description": "Scrape real-time commodity prices from financial websites. Supports oil, gold, silver, natural gas, and more. Extract price, change, volume, technical indicators, and contract details. Currently supports Investing.com with more sites coming soon.", "version": "0.0", "x-build-id": "e9gFdprjsai8b3tBe" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/nocodeventure~commodities-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-nocodeventure-commodities-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/nocodeventure~commodities-scraper/runs": { "post": { "operationId": "runs-sync-nocodeventure-commodities-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/nocodeventure~commodities-scraper/run-sync": { "post": { "operationId": "run-sync-nocodeventure-commodities-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "site", "commodities" ], "properties": { "commodities": { "title": "Commodities", "type": "array", "description": "List of commodities to scrape price data for. Enter commodity slugs (e.g., 'crude-oil', 'gold', 'natural-gas').", "items": { "type": "string" }, "default": [ "crude-oil" ] }, "site": { "title": "Website", "enum": [ "investing.com" ], "type": "string", "description": "The website to scrape commodity prices from. Only one site can be selected per run.", "default": "investing.com" }, "skipAuthentication": { "title": "Skip Authentication", "type": "boolean", "description": "When enabled, skips the login process. Some data may be limited without authentication.", "default": true }, "investingEmail": { "title": "Investing.com Email", "type": "string", "description": "Your Investing.com email for authentication. Not required if Skip Authentication is enabled." }, "investingPassword": { "title": "Investing.com Password", "type": "string", "description": "Your Investing.com password. Not required if Skip Authentication is enabled." }, "maxConcurrency": { "title": "Max Concurrent Browsers", "minimum": 1, "maximum": 10, "type": "integer", "description": "Maximum number of browser instances running in parallel. Higher values speed up scraping but use more memory.", "default": 1 }, "useSameTab": { "title": "Use Same Browser Tab", "type": "boolean", "description": "When enabled (default), navigates to all URLs in the same browser tab - much faster. When disabled, opens a fresh browser for each URL - slower but more robust and reduces CPU pressure.", "default": true }, "proxyConfiguration": { "title": "Proxy Configuration", "type": "object", "description": "Proxy settings. The scraper works without proxies by default. Only enable if you experience blocking issues.", "default": { "useApifyProxy": false } }, "includeFaq": { "title": "Include FAQ", "type": "boolean", "description": "When enabled, extracts FAQ data from the commodity page (questions and answers about the commodity).", "default": false }, "debugScreenshots": { "title": "Debug Screenshots", "type": "boolean", "description": "When enabled, saves screenshots on failures for debugging. Screenshots are saved as JPEG with low quality to save space.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```