# Walmart Scraper (`oneary/walmart-scraper`) Actor ๐Ÿช Scrape Walmart.com product data โ€” extract titles, prices, ratings, reviews, stock status, seller info, and product specs. Monitor competitor pricing, track inventory changes, and analyze the world's largest retailer. Filter by category, price range, and availability. Built for e-commerce - **URL**: https://apify.com/oneary/walmart-scraper.md - **Developed by:** [Luan M.](https://apify.com/oneary) (community) - **Categories:** E-commerce - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $18.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## ๐Ÿ›’ Walmart Product Scraper [![Apify Actor](https://img.shields.io/badge/Apify-Actor-blue)](https://apify.com/) [![Node.js](https://img.shields.io/badge/Node.js-20-green)](https://nodejs.org/) [![License](https://img.shields.io/badge/License-Apache%202.0-blue)](LICENSE) Extract comprehensive product data from **Walmart.com** at scale. Built with **Crawlee** and **Playwright** for reliable, browser-based scraping that handles JavaScript-rendered content, dynamic pricing, and anti-bot protections. --- ### โœจ Features - **Product Details** โ€” Name, SKU/ID, full description, high-res image URLs, seller information - **Price Comparison** โ€” Current prices, currency tracking, historical price snapshots via re-runs - **Availability Tracking** โ€” Real-time stock status with fulfillment badge capture - **Ratings & Reviews** โ€” Star ratings (1โ€“5) and total review count extraction - **Category Navigation** โ€” Automatic breadcrumb extraction for taxonomy mapping - **Search & Browse** โ€” Start from search queries, category URLs, or a list of product pages - **Price Filtering** โ€” Set `minPrice` and `maxPrice` to target specific price ranges - **Multilingual Support** โ€” Works across Walmart domains (US, Canada, Mexico, and more) - **Review Extraction** โ€” Optional review data alongside product info - **Residential Proxy** โ€” Built-in Apify proxy with residential group for reliable access - **Anti-blocking** โ€” Real browser fingerprints, smart request throttling, and resource blocking for speed --- ### ๐Ÿ“‹ Input Parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `startUrls` | Array | No | Walmart laptops browse | List of product or category URLs to start scraping | | `searchQuery` | String | No | โ€” | Search term (e.g., "iPhone 15", "Nike shoes") | | `categoryUrl` | String | No | โ€” | Full Walmart category browse URL | | `maxProducts` | Integer | No | `100` | Maximum products to scrape (`0` = unlimited) | | `includeReviews` | Boolean | No | `false` | Extract review data alongside product info | | `minPrice` | Number | No | โ€” | Minimum price filter | | `maxPrice` | Number | No | โ€” | Maximum price filter | | `proxyConfiguration` | Object | No | Residential proxy | Apify proxy configuration | > **Note:** If no `startUrls`, `searchQuery`, or `categoryUrl` is provided, the scraper defaults to Walmart's laptops category. --- ### ๐Ÿ“ค Output Fields Each product is saved as a separate dataset item with the following fields: | Field | Type | Description | |-------|------|-------------| | `name` | String | Product title | | `price` | String | Current price (formatted) | | `currency` | String | Currency code (e.g., USD, CAD, MXN) | | `rating` | Number | Star rating (0โ€“5) | | `reviewCount` | Number | Number of customer reviews | | `availability` | String | Stock status / fulfillment badge | | `sku` | String | Walmart product ID / SKU | | `category` | String | Full category breadcrumb path | | `seller` | String | Seller name (defaults to "Walmart") | | `description` | String | Product description (truncated to 2000 chars) | | `imageUrls` | Array | Array of high-res product image URLs | | `url` | String | Original product page URL | | `scrapedAt` | String | ISO 8601 timestamp of extraction | | `error` | String | Error message if scraping failed (optional) | --- ### ๐Ÿš€ Quick Start #### Apify Platform 1. Open the [Walmart Product Scraper](https://apify.com/) on Apify Store 2. Click **Try** or **Use** 3. Configure your input parameters 4. Click **Run** and wait for results 5. Export data as JSON, CSV, XML, or Excel #### Local Development ```bash ## Clone and install git clone https://github.com/your-org/walmart-scraper.git cd walmart-scraper npm install ## Run with default input npx playwright install chromium node src/main.js ```` *** ### ๐Ÿงช Use Cases - **Price Monitoring** โ€” Track price changes over time by running the scraper periodically - **Competitor Analysis** โ€” Compare product assortments, pricing, and availability - **Inventory Tracking** โ€” Monitor stock levels and out-of-stock patterns - **Market Research** โ€” Catalog entire categories with ratings and review counts - **Feed Generation** โ€” Build product feeds for comparison shopping engines - **Brand Monitoring** โ€” Track authorized and third-party sellers for specific brands *** ### ๐ŸŒ Multilingual Domains The scraper can target any Walmart regional domain by providing the appropriate URL in `startUrls` or `categoryUrl`: - `https://www.walmart.com` โ€” United States - `https://www.walmart.ca` โ€” Canada - `https://www.walmart.com.mx` โ€” Mexico Use the appropriate proxy group for best results with regional domains. *** ### โš™๏ธ Technical Details - **Runtime:** Node.js 20 on Apify platform - **Engine:** Crawlee `PlaywrightCrawler` with full browser automation - **Browser:** Chromium (headless) - **Proxy:** Apify proxy with residential group support - **Concurrency:** Auto-scaled based on available resources - **Retry:** Automatic retry with exponential backoff for failed requests - **Storage:** Apify Dataset with structured output *** ### ๐Ÿ”’ Error Handling Failed product pages are still recorded in the dataset with the original URL and an error message, ensuring no data loss. The scraper continues processing remaining items even when individual pages fail. *** ### ๐Ÿ“„ License This project is licensed under the Apache 2.0 License โ€” see the [LICENSE](LICENSE) file for details. # Actor input Schema ## `startUrls` (type: `array`): List of Walmart product or category URLs to start scraping from ## `searchQuery` (type: `string`): Search term to find products on Walmart ## `categoryUrl` (type: `string`): Full Walmart category URL to browse (e.g., https://www.walmart.com/browse/electronics/laptops/3944\_3951\_1089430) ## `maxProducts` (type: `integer`): Maximum number of products to scrape (0 = unlimited) ## `includeReviews` (type: `boolean`): Whether to extract review data alongside product info ## `minPrice` (type: `number`): Filter products with price above this value ## `maxPrice` (type: `number`): Filter products with price below this value ## `proxyConfiguration` (type: `object`): Apify proxy configuration for region-specific scraping and anti-blocking ## Actor input object example ```json { "startUrls": [ { "url": "https://www.walmart.com/browse/electronics/laptops/3944_3951_1089430" } ], "maxProducts": 100, "includeReviews": false, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("oneary/walmart-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], } } # Run the Actor and wait for it to finish run = client.actor("oneary/walmart-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("๐Ÿ’พ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # ๐Ÿ“š Want to learn more ๐Ÿ“–? Go to โ†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call oneary/walmart-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=oneary/walmart-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Walmart Scraper", "description": "๐Ÿช Scrape Walmart.com product data โ€” extract titles, prices, ratings, reviews, stock status, seller info, and product specs. Monitor competitor pricing, track inventory changes, and analyze the world's largest retailer. Filter by category, price range, and availability. Built for e-commerce", "version": "1.0", "x-build-id": "oYVJCga91Cy68LxnM" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/oneary~walmart-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-oneary-walmart-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/oneary~walmart-scraper/runs": { "post": { "operationId": "runs-sync-oneary-walmart-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/oneary~walmart-scraper/run-sync": { "post": { "operationId": "run-sync-oneary-walmart-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "List of Walmart product or category URLs to start scraping from", "default": [ { "url": "https://www.walmart.com/browse/electronics/laptops/3944_3951_1089430" } ], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "searchQuery": { "title": "Search query", "type": "string", "description": "Search term to find products on Walmart" }, "categoryUrl": { "title": "Category URL", "type": "string", "description": "Full Walmart category URL to browse (e.g., https://www.walmart.com/browse/electronics/laptops/3944_3951_1089430)" }, "maxProducts": { "title": "Max products", "minimum": 0, "maximum": 100000, "type": "integer", "description": "Maximum number of products to scrape (0 = unlimited)", "default": 100 }, "includeReviews": { "title": "Include reviews", "type": "boolean", "description": "Whether to extract review data alongside product info", "default": false }, "minPrice": { "title": "Minimum price", "minimum": 0, "type": "number", "description": "Filter products with price above this value" }, "maxPrice": { "title": "Maximum price", "minimum": 0, "type": "number", "description": "Filter products with price below this value" }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Apify proxy configuration for region-specific scraping and anti-blocking", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```