# Universal Store Catalog Scraper (Shopify/Woo/Generic) (`maninder/store-catalog-scraper`) Actor

Extract clean, normalized product-catalog data from any Shopify, WooCommerce, or generic e-commerce storefront. AI-agent / MCP ready: every store comes back in one identical schema.

- **URL**: https://apify.com/maninder/store-catalog-scraper.md
- **Developed by:** [Maninder Pal Singh](https://apify.com/maninder) (community)
- **Categories:** E-commerce
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Universal Store Catalog Scraper 🛍️ → 🤖

Turn **any Shopify, WooCommerce, or generic e-commerce storefront** into a **clean, normalized product feed** — the same schema for every store, every time. Built **MCP-first** so an AI agent can consume it as a structured catalog tool.

This is a **storefront-platform scraper, not a marketplace scraper**. It targets independent merchant stores (each its own site) using **public, non-personal product data only**. The core value is **cross-platform schema consistency**: a Shopify store, a WooCommerce store, and a plain JSON-LD store all come back in *one identical shape*.

***

### ✨ Why this Actor

- **One schema, every platform.** Agents don't have to special-case Shopify vs. Woo vs. generic — the output is always the same record.
- **Tiered, cheap-first extraction.** Uses fast public JSON endpoints where they exist and only spins up a browser as a last resort, so runs stay fast and cheap.
- **Reliable & graceful.** A malformed or blocked store degrades to fewer fields or a skip record — it never fails the whole run.
- **Polite by default.** Respects `robots.txt`, rate-limits per domain, honest User-Agent, public data only.

***

### 🧱 How it extracts (tiered engine)

| Platform | Primary path | Fallback |
|---|---|---|
| **Shopify** | Public `/products.json` (paginated, `?limit=250&page=N`) | Generic JSON-LD |
| **WooCommerce** | Store API `/wp-json/wc/store/v1/products` | Generic JSON-LD / HTML |
| **Generic** | schema.org `Product` JSON-LD | Open Graph → microdata → **Playwright render** |

Detection is a single lightweight request (asset/header/global fingerprints, with a cheap endpoint probe when ambiguous) and is overridable via `forcePlatform`.

***

### 📥 Input

Two modes — use either or combine them.

**Mode 1 — Direct URLs (default, safest)**

```json
{
  "storeUrls": ["https://www.allbirds.com", "https://some-woostore.com/shop"],
  "maxProductsPerStore": 200,
  "includeVariants": true,
  "includeImages": true
}
```

**Mode 2 — Discovery (opt-in, capped)**

```json
{
  "discovery": {
    "enabled": true,
    "keywords": ["handmade ceramic mugs", "merino wool socks"],
    "platformHint": "any",
    "maxStores": 10
  },
  "maxProductsPerStore": 100
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `storeUrls` | string\[] | `[]` | Store home / collection / product URLs |
| `discovery.enabled` | bool | `false` | Opt-in store discovery |
| `discovery.keywords` | string\[] | `[]` | Niche/product terms |
| `discovery.platformHint` | enum | `any` | `shopify` | `woocommerce` | `any` |
| `discovery.maxStores` | int | `10` | **Hard max 50** |
| `maxProductsPerStore` | int | `200` | **Hard max 5000** — cost ceiling |
| `includeVariants` | bool | `true` | Per-variant array |
| `includeImages` | bool | `true` | Image URLs (primary first) |
| `includeDescriptionHtml` | bool | `false` | Raw HTML descriptions (bulky) |
| `forcePlatform` | enum | `auto` | Override detection |
| `respectRobotsTxt` | bool | `true` | Honor robots.txt |
| `maxConcurrency` | int | `5` | Stores in parallel |
| `proxyConfiguration` | object | Apify auto | — |

***

### 📤 Output — one normalized record per product

```json
{
  "storeUrl": "https://www.allbirds.com",
  "storeName": "Allbirds",
  "platform": "shopify",
  "productId": "6592289505353",
  "productUrl": "https://www.allbirds.com/products/mens-wool-runners",
  "title": "Men's Wool Runners",
  "descriptionText": "Our iconic everyday sneaker made from merino wool...",
  "brand": "Allbirds",
  "productType": "Shoes",
  "categories": ["Shoes"],
  "price": 110.0,
  "compareAtPrice": null,
  "currency": "USD",
  "onSale": false,
  "availability": "in_stock",
  "sku": "WR-NAT-GRY-8",
  "variants": [
    { "variantId": "39472394814025", "title": "Natural Grey / 8", "price": 110.0,
      "compareAtPrice": null, "sku": "WR-NAT-GRY-8", "availability": "in_stock",
      "options": { "Color": "Natural Grey", "Size": "8" } }
  ],
  "images": ["https://cdn.shopify.com/.../wool-runner-1.jpg"],
  "tags": ["wool", "runners"],
  "ratingValue": null,
  "ratingCount": null,
  "scrapedAt": "2026-06-30T10:15:42.118Z",
  "extractionMethod": "shopify:products.json"
}
```

**Field guarantees that matter for agents:**

- `price` / `compareAtPrice` are always **numbers** (never `"$19.99"`); `currency` is a separate **ISO-4217** code.
- `availability` is one of `in_stock` | `out_of_stock` | `preorder` | `unknown`.
- `platform` is one of `shopify` | `woocommerce` | `generic`.
- `extractionMethod` tells you exactly which path produced the record.
- Stores that yield nothing produce a `{ storeUrl, skipped, error }` row instead of disappearing.

See [`samples/`](./samples) for a full input + output example per platform.

***

### 🤖 MCP / AI-agent usage

Apify exposes every Actor as an **MCP tool**, so an agent can call this scraper directly. Because the input schema is crisp and the output is deterministic and identical across platforms, the agent experience is clean.

**Connect** (Apify MCP server): `https://mcp.apify.com` — add `YOUR_USERNAME/store-catalog-scraper` as an available tool.

**An agent calls it like this** (conceptual tool call):

```json
{
  "tool": "YOUR_USERNAME/store-catalog-scraper",
  "input": {
    "storeUrls": ["https://www.allbirds.com"],
    "maxProductsPerStore": 50,
    "includeVariants": true
  }
}
```

The agent then reads the run's **dataset** — every item is one normalized product it can reason over (compare prices across stores, check stock, build a catalog) without per-store parsing logic.

> Tip for agent builders: keep `maxProductsPerStore` modest for interactive use, and rely on `platform` + `extractionMethod` to gauge data confidence.

***

### 💸 Pricing (pay-per-event)

| Event | Rate (USD) | When |
|---|---|---|
| **Result (product scraped)** | **$0.004** ($4 / 1,000) | Per normalized product written to the dataset — the main charge |
| **Actor start** | $0.02 | Once per run |

Platform compute + proxy are **included** (absorbed by the Actor), so you pay only the events above — pricing is predictable regardless of a store's size or speed.

**Worked cost example** — scrape **10 stores × 200 products = 2,000 products**:

```
Actor start:        1 × $0.02  = $0.02
Products scraped: 2000 × $0.004 = $8.00
-------------------------------------------
Total                            ≈ $8.02   (~$0.004 per product)
```

You are only charged for **real products**: stores that fail or return nothing are written to a separate, **non-billed** dataset (`skipped-stores`).

***

### 🔒 Responsible use

- **Public, non-personal product data only.** This Actor does **not** log into stores, collect merchant/customer personal data, or access gated content. If a store requires authentication to view products, it is skipped.
- **You own compliance.** You are responsible for complying with each target store's Terms of Service and applicable law in your jurisdiction.
- **Discovery is best-effort and user-directed.** Mode 2 finds *candidate* public storefronts from your keywords and verifies them before crawling; you remain responsible for what you choose to crawl. It is hard-capped at 50 stores.
- **Politeness is built in:** `robots.txt` is respected by default, requests are rate-limited per domain with jitter, and an honest User-Agent is sent.

***

### ⚠️ Limitations

- WooCommerce per-variant **prices** aren't exposed by the public Store API without extra calls, so Woo variants surface attribute combinations with `price: null` (product-level price is always populated).
- Generic JSON-LD stores vary widely; fields present depend on what the store emits. Missing fields are `null`/`[]`, never invented.
- Shopify currency is read from the storefront; if a store hides it, `currency` may be `null`.
- The Playwright fallback only triggers when cheap paths return nothing **and** the page looks JS-rendered; escalations are capped per run for cost.

***

### 🧩 Extending

Platform modules are cleanly separated (`src/shopify.js`, `src/woocommerce.js`, `src/generic.js`) and all funnel through `src/normalize.js`. Adding **BigCommerce** or **Squarespace** is a drop-in: add a detector fingerprint + an extractor that emits into the shared normalizer.

# Actor input Schema

## `storeUrls` (type: `array`):

Mode 1 (default, safest). Store homepages, collection URLs, or product URLs. Each entry is treated as its own storefront.

## `discovery` (type: `object`):

Mode 2. Best-effort discovery of stores from keywords. Disabled by default. You are responsible for what you crawl.

## `maxProductsPerStore` (type: `integer`):

Hard cost ceiling per store. Pagination stops here even if more products exist.

## `includeVariants` (type: `boolean`):

Emit the per-variant array (price, sku, options, availability).

## `includeImages` (type: `boolean`):

Emit the images array (primary image first).

## `includeDescriptionHtml` (type: `boolean`):

Raw HTML descriptions are bulky. Off by default — clean text is always provided in descriptionText.

## `forcePlatform` (type: `string`):

Override auto-detection.

## `respectRobotsTxt` (type: `boolean`):

Honor each store's robots.txt before requesting a URL. Strongly recommended.

## `maxConcurrency` (type: `integer`):

How many stores to process in parallel. Kept low to stay polite per merchant.

## `proxyConfiguration` (type: `object`):

Proxy used for all requests. Apify auto proxy by default.

## Actor input object example

```json
{
  "storeUrls": [
    "https://www.allbirds.com",
    "https://www.example-woostore.com"
  ],
  "discovery": {
    "enabled": false,
    "keywords": [],
    "platformHint": "any",
    "maxStores": 10
  },
  "maxProductsPerStore": 200,
  "includeVariants": true,
  "includeImages": true,
  "includeDescriptionHtml": false,
  "forcePlatform": "auto",
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `products` (type: `string`):

Normalized product records stored in the default dataset. This is the Actor's main output.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "storeUrls": [
        "https://www.allbirds.com",
        "https://www.example-woostore.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("maninder/store-catalog-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "storeUrls": [
        "https://www.allbirds.com",
        "https://www.example-woostore.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("maninder/store-catalog-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "storeUrls": [
    "https://www.allbirds.com",
    "https://www.example-woostore.com"
  ]
}' |
apify call maninder/store-catalog-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=maninder/store-catalog-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ViJvM6FNXTLVpZu2U/builds/K6ozLI00m6uoLCy6R/openapi.json