# E-commerce Product Scraper & JSON-LD Extractor (`egeusta/product-data-extractor`) Actor

Extract product prices, availability, images, identifiers and variants from public product pages. Discover URLs from store sitemaps, filter by data completeness, and export JSON or CSV. Free-plan sample: up to 25 pages per run.

- **URL**: https://apify.com/egeusta/product-data-extractor.md
- **Developed by:** [Scopras Data Labs](https://apify.com/egeusta) (community)
- **Categories:** E-commerce, Developer tools, Lead generation
- **Stats:** 23 total users, 10 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## E-commerce Product Scraper & JSON-LD Extractor

Turns product pages into one clean row each — parsed prices, normalised stock
status, variants, absolute image URLs — from any storefront, not just Shopify.

Give it product URLs, or give it a store's homepage and let it find them.

```json
{
  "startUrls": [{ "url": "https://www.gymshark.com" }],
  "discoverProducts": true,
  "maxProducts": 10,
  "minCompleteness": 50
}
```

Illustrative output shape (not a live product quote). Every field is flat —
no nested objects or arrays — so the Console's CSV/Excel export button
renders a clean table without opening a single item:

```json
{
  "name": "Legacy Fitted T-Shirt",
  "brand": "Gymshark",
  "sku": "GSTS-4471",
  "price": 32,
  "priceCurrency": "USD",
  "availability": "InStock",
  "inStock": true,
  "imageCount": 1,
  "primaryImage": "https://cdn.shopify.com/s/files/…/legacy-tee.jpg",
  "imagesJson": "[\"https://cdn.shopify.com/s/files/…/legacy-tee.jpg\"]",
  "variantCount": 2,
  "variantsJson": "[{\"name\":\"Legacy Tee — S\",\"sku\":\"GSTS-4471-S\",\"price\":32,\"availability\":\"InStock\"},{\"name\":\"Legacy Tee — M\",\"sku\":\"GSTS-4471-M\",\"price\":32,\"availability\":\"OutOfStock\"}]",
  "source": "jsonld",
  "completeness": 1,
  "missingFields": [],
  "ok": true
}
```

- `price` / `priceCurrency` — the price is always a clean number with the
  currency symbol and thousands/decimal formatting resolved, in a column
  separate from its ISO currency code (see "What the extractor handles"
  below for the locale-parsing rule).
- `inStock` — a plain boolean derived from `availability` (`null` when the
  page gave no availability signal at all), for price monitors and feeds
  that just need yes/no rather than nine schema.org spellings.
- `imageCount` / `primaryImage` / `imagesJson` and `variantCount` /
  `variantsJson` — the genuinely nested parts of a product (multiple images,
  per-variant price/sku/stock) are summarised into flat columns
  (`imageCount`, `variantCount`, `primaryImage`) and the full detail is kept
  as a JSON string you can `JSON.parse()` back into an array on re-import,
  instead of a raw array that CSV/Excel would render unreadably.

### Pricing

The Pricing tab shows the current price. Unreachable pages and rows below your completeness filter are never charged.

Every Apify plan, including Free, can process full batches within your configured limit and run budget. The prefilled input now discovers 10 products from a store sitemap with a 50% completeness filter; replace the example store with your target.

To scrape individual product URLs, turn discovery off. To process a whole store, turn it on. Set a maximum total charge before starting a larger job.

### What the extractor handles

**Prices come out as numbers, in the right magnitude.** `"1.299,90"` and
`"1,299.90"` are the same amount written by two halves of the world. Returning
the string hands you a problem you discover in row 40,000; guessing wrong makes
it a thousandfold error. Both are parsed, and anything genuinely unreadable —
"call for pricing" — comes back absent rather than as `0`.

**Variants are not lost.** A `ProductGroup` keeps name and brand on the parent
and SKU, price and stock on each child. Read either node alone and you get half
a product. Both are read, and every variant is returned with its own row of
fields.

**Non-English stores stay legible.** Pages served as windows-1254 or ISO-8859-9
are decoded in the encoding the server declared, and HTML entities are resolved
with case respected — `Ç` is Ç, `ç` is ç. No mojibake, no
`ü` in your spreadsheet.

**It works past the Shopify convention.** Product URLs are found from the
store's own sitemap filing rather than guessed from the path, so flat-URL
stores work too — `hisar.com.tr/hisar-16-parca-granit-ceyiz-seti` has no `/products/`
segment anywhere and is found regardless.

**Three sources, in the right order.** JSON-LD first, then microdata, then
OpenGraph — and a fallback only ever fills a gap, never overwrites. Each row
tells you which sources it needed (`source: "jsonld" | "mixed" | …`).

**Every row grades itself.** `completeness` is the share of the seven fields
that matter — name, brand, identifier, price, currency, availability, image —
and `missingFields` names the ones that were absent. Set
**Minimum completeness** to drop thin rows, and you are never charged for them.

**A name that is really a SKU gets fixed.** Some stores publish
`"name": "PCDP253G13IY"`. Where the page offers something better, you get the
real title.

### Input

| Field | What it does |
|---|---|
| `startUrls` | Product URLs, or store homepages when discovery is on |
| `discoverProducts` | Find product pages from robots.txt and the sitemaps it declares |
| `maxProducts` | Hard cap for the run |
| `minCompleteness` | Drop rows below this percentage. Dropped rows are never charged |
| `includeVariants` | Keep the per-variant array, or flatten to one row per product |
| `crawlerIdentity` | Your own User-Agent. Whoever runs the scrape answers for the traffic |

### Output

Two kinds of row, so a run is always auditable:

- **Products** — `ok: true`, plus everything above.
- **Unreachable pages** — `ok: false` with `url`, `httpStatus` and `error`.
  Recorded so you can see them; **never charged.**

### Politeness

Requests to one host are spaced out, bodies are capped, timeouts are hard, and
a `429` is honoured with a single backed-off retry. These are not configurable.
A scrape that gets your IP banned was not a bargain, and the stores being read
did not ask to be.

### Limits, stated plainly

- **Public pages only.** URLs that point at localhost, private networks or non-http(s) schemes are refused.
- **No JavaScript rendering.** Products whose data is drawn client-side and
  published in no markup at all will come back thin. `completeness` will say so
  rather than pretending otherwise.
- **Compressed sitemaps (`.xml.gz`) are read** like plain ones, with the same size cap after decompression.
- **Discovery depends on the store publishing a sitemap.** Without one, only
  the homepage's own product links are found — the run says so explicitly when
  that happens, because 40 rows against a catalogue of 4,000 is a number you
  need to know the provenance of.
- **At most 15 sitemap documents are read per store**, and no more than 25 are
  requested. A store whose index lists hundreds of dead children will not
  spend your run collecting 404s.
- **A run that reaches your maximum total charge stops there** and says so in
  the log and the run's status. It does not keep fetching pages it cannot
  deliver — raise the budget and run it again for the rest.

***

Built on the engine behind **Scopras**, an audit tool that grades a store's
product markup against what AI shopping agents need. Same extraction rules:
this actor gives you the data, Scopras tells you what is wrong with it.

### Build a recurring ecommerce workflow

Use [XML Sitemap Scraper](https://apify.com/egeusta/sitemap-intelligence) to inventory URLs, extract the product records here, then compare recurring datasets with [Product Catalog Change Monitor](https://apify.com/egeusta/ecommerce-assortment-change-monitor). For public Shopify catalogs and variant snapshots, use [Shopify Price Monitor](https://apify.com/egeusta/shopify-competitor-price-stock-monitor).

You can save inputs as an Apify task and schedule recurring collection. Export the Output dataset as JSON, CSV or Excel, or connect it to your existing workflow through the API and integrations.

For support, open an issue with a run ID and public product URL. Do not post tokens or private customer information.

### FAQ

**Which stores does this product scraper work on?** Any storefront that publishes product markup (JSON-LD, microdata or OpenGraph) in its HTML — Shopify, WooCommerce, Magento, BigCommerce and custom stores. Pages that draw products only with JavaScript come back thin, and the completeness score says so.

**Can it find every product in a store?** With discovery on, it reads the store's robots.txt and sitemaps to list product URLs, then extracts each one. Cap the run with Max products.

**How much does it cost?** See the Pricing tab. Unreachable pages and rows below your completeness filter are never charged.

**How do I track price changes?** Run it on a schedule and compare datasets with [Product Catalog Change Monitor](https://apify.com/egeusta/ecommerce-assortment-change-monitor).

**Is it legal?** It reads publicly available pages only, with no login. You are responsible for respecting each site's terms and the privacy rules that apply to your use of the data.

**Can I run it through the API or on a schedule?** Yes. Save your input as a task, schedule it in Apify Console, or call it from the Apify API. Results export as JSON, CSV or Excel and connect to Google Sheets, Slack, Make, Zapier or a webhook.

### Related Actors

- [Shopify Price Monitor](https://apify.com/egeusta/shopify-competitor-price-stock-monitor)
- [XML Sitemap Scraper](https://apify.com/egeusta/sitemap-intelligence)
- [Product Catalog Change Monitor](https://apify.com/egeusta/ecommerce-assortment-change-monitor)
- [Product Schema Auditor](https://apify.com/egeusta/ai-readiness-auditor)

# Changelog

This Actor's version history is a separate document: https://apify.com/egeusta/product-data-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Product page URLs to extract. To hand over whole stores instead, paste the store homepages and switch on 'Discover product URLs' below.

## `discoverProducts` (type: `boolean`):

Treat the URLs above as stores and find their product pages from robots.txt and the sitemaps it declares. Leave off when you are supplying product URLs directly.

## `maxProducts` (type: `integer`):

Hard cap on how many product pages are fetched in this run.

## `minCompleteness` (type: `integer`):

Drop rows that carry less than this share of the seven scored fields (name, brand, identifier, price, currency, availability, image). Rows dropped here are never charged. 0 keeps everything.

## `includeVariants` (type: `boolean`):

Keep per-variant detail (size and colour rows with their own SKU, price and stock) in the variantCount / variantsJson columns. Turn off to drop variants entirely.

## `crawlerIdentity` (type: `string`):

How the crawler introduces itself to the stores you scrape. Leave blank for the default. Whoever runs the scrape answers for the traffic, so putting a real contact in here is the courteous choice.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.gymshark.com"
    }
  ],
  "discoverProducts": true,
  "maxProducts": 10,
  "minCompleteness": 50,
  "includeVariants": true
}
```

# Actor output Schema

## `products` (type: `string`):

Structured product records stored in the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.gymshark.com"
        }
    ],
    "discoverProducts": true,
    "maxProducts": 10,
    "minCompleteness": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("egeusta/product-data-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.gymshark.com" }],
    "discoverProducts": True,
    "maxProducts": 10,
    "minCompleteness": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("egeusta/product-data-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.gymshark.com"
    }
  ],
  "discoverProducts": true,
  "maxProducts": 10,
  "minCompleteness": 50
}' |
apify call egeusta/product-data-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,egeusta/product-data-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Sky0jZtaV0wEPFmZD/builds/HmaFeCJQKHRXFo61Z/openapi.json