# Shopify Product Feed & Price Scraper (`tpa_d/shopify-beauty-radar`) Actor

Scrape public Shopify products.json feeds for product catalogs, variants, prices, compare-at discounts, availability, tags, images, and new product snapshots.

- **URL**: https://apify.com/tpa\_d/shopify-beauty-radar.md
- **Developed by:** [The Product Analyst Data](https://apify.com/tpa_d) (community)
- **Categories:** E-commerce
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 product / variant rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Shopify Product Feed & Price Scraper 🛍️📈

Scrape public Shopify product feeds from compatible stores and export clean product, variant, price, availability, tag, image, and discount rows.

This Actor reads Shopify’s public `/products.json` catalog endpoint — **no browser, login, proxy, or Shopify admin access required** — and turns it into structured data you can export to JSON, CSV, Excel, Google Sheets, or consume through the Apify API.

It works especially well for **competitor product catalog snapshots**, **Shopify price monitoring**, **discount tracking**, **new product drop discovery**, and **variant-level ecommerce research**. Beauty, skincare, and hair-care stores are the default examples, but the Actor can scan any Shopify store that exposes the public product feed.

![Shopify Product Feed & Price Scraper sample results](assets/product-radar-results-preview.svg)

### What you get

- **Shopify product feed scraping** — product title, vendor, product type, tags, product URL, image URL, and publish date.
- **Variant-level rows** — capture size, shade, bundle, SKU-like variant title, price, compare-at price, and availability at the variant level.
- **Price and discount intelligence** — detect real Shopify `compare_at_price` markdowns and compute discount percentage.
- **New product and catalog snapshots** — sort by `newest` to identify recently published products.
- **Keyword filtering** — scan only matching products such as `serum`, `spf`, `bundle`, `dress`, `supplement`, or any category terms.
- **Opportunity scoring** — rank in-stock markdowns and sale opportunities with a bounded `dropOrSaleScore`.
- **Structured diagnostics** — invalid, blocked, disabled, or no-match feeds return diagnostic rows instead of disappearing.
- **Automation-ready output** — schedule repeated runs and diff datasets for restocks, price changes, new drops, and assortment changes.

### Best use cases

- Scrape a Shopify competitor’s public catalog into a spreadsheet.
- Monitor competitor prices, compare-at prices, and markdowns.
- Track new product drops by sorting products by publish date.
- Compare product assortment, tags, product types, vendors, and price bands across stores.
- Build daily or weekly Shopify catalog snapshots for alerts and BI workflows.
- Research beauty, skincare, hair-care, fashion, supplement, home goods, or other Shopify-powered ecommerce niches.

### Buyer-ready offer

> Send me 3-5 Shopify competitor stores. I’ll return a clean product/variant/price/discount spreadsheet with top markdowns and new-product signals. First sample is free.

Best buyers: Shopify agencies, DTC brands, ecommerce operators, retention/CRO teams, and analysts who need a repeatable competitor catalog feed.

### Input example

```json
{
  "storeUrls": [
    "https://mielleorganics.com",
    "https://baskandlatherco.com",
    "https://kaleidoscopehairproducts.com",
    "https://patternbeauty.com",
    "https://themanemagic.com"
  ],
  "keywords": ["shampoo", "conditioner", "oil", "serum", "leave-in", "curl", "edge", "mask"],
  "maxProductsPerStore": 100,
  "minDiscountPercent": 0,
  "includeVariants": true,
  "sortBy": "opportunity",
  "includeDiagnostics": true
}
```

| Field | What it controls |
|---|---|
| `storeUrls` **required** | Shopify store URLs to scan. Bare domains are accepted. Paths are normalized to the store origin. |
| `keywords` | Filter products by title, vendor, product type, and tags. Empty = keep all products. |
| `maxProductsPerStore` | Product cap per store before variant expansion. |
| `minDiscountPercent` | Minimum markdown percent to return. Use `0` for all matches. |
| `includeVariants` | `true` = one row per variant; `false` = one row per product using its best-deal variant. |
| `sortBy` | `opportunity`, `discount`, `price`, or `newest`. |
| `includeDiagnostics` | Include invalid, blocked, failed, or no-match rows for auditability. |

### Sample output

Real top row from the sample input:

```json
{
  "rowType": "product",
  "storeUrl": "https://mielleorganics.com",
  "productId": 6752188989553,
  "title": "Mango & Tulsi Nourishing Conditioner",
  "vendor": "MIELLE",
  "productType": "Conditioners",
  "productUrl": "https://mielleorganics.com/products/mango-tulsi-nourishing-conditioner",
  "imageUrl": "https://cdn.shopify.com/s/files/1/0763/8199/files/250513-Mielle_SHA-Mielle-MT-Conditioner-3.jpg?v=1747750810",
  "publishedAt": "2022-09-02T17:31:07-05:00",
  "variantId": 39844163518577,
  "variantTitle": "Default Title",
  "price": 5,
  "compareAtPrice": 14.99,
  "discountPercent": 67,
  "available": true,
  "matchedKeywords": ["conditioner"],
  "dropOrSaleScore": 72,
  "scrapedAt": "2026-07-04T02:50:20.087Z"
}
```

Example run result from the sample input:

| Store | Result |
|---|---:|
| `mielleorganics.com` | 53 rows |
| `baskandlatherco.com` | 38 rows |
| `kaleidoscopehairproducts.com` | 41 rows |
| `patternbeauty.com` | 59 rows |
| `themanemagic.com` | 1 diagnostic row |
| **Total** | **192 dataset rows** |

The top finding was a **67% off** in-stock Mielle conditioner with a `dropOrSaleScore` of `72`.

### Output fields

| Field | Meaning |
|---|---|
| `rowType` | `product` or `diagnostic`. |
| `storeUrl` | Normalized Shopify store origin. |
| `productId`, `title`, `vendor`, `productType`, `tags` | Shopify product metadata. |
| `productUrl`, `imageUrl` | Product page and primary image. |
| `publishedAt` | Shopify publish timestamp. |
| `variantId`, `variantTitle` | Specific variant scanned. |
| `price`, `compareAtPrice` | Current and compare-at prices parsed to numbers. |
| `discountPercent` | Rounded markdown percent when `compareAtPrice > price`. |
| `available` | Shopify variant availability boolean. |
| `matchedKeywords` | Which configured keywords matched. |
| `dropOrSaleScore` | 0–100 opportunity score; deeper, in-stock markdowns rank higher. |
| `scrapedAt` | Run timestamp. |

### Diagnostic rows

When `includeDiagnostics` is enabled, stores that fail or return no usable feed produce rows like:

```json
{
  "rowType": "diagnostic",
  "storeUrl": "https://themanemagic.com",
  "statusCode": 200,
  "errorType": "parse_error",
  "message": "Could not parse products.json: Unexpected token '<' ... is not valid JSON",
  "scrapedAt": "2026-07-04T02:50:20.087Z"
}
```

Possible `errorType` values: `invalid_url`, `http_error`, `network_error`, `parse_error`, `invalid_response`, `no_matching_rows`.

### How it works

This Actor queries each store’s public Shopify product feed at:

```text
/products.json
```

It paginates the feed, normalizes products and variants, applies optional keyword and discount filters, sorts rows, and writes product or diagnostic rows to the default dataset.

It does **not** log in, scrape private data, bypass paywalls, or access hidden Shopify admin data.

### Pricing

This Actor is configured for **Pay per event** pricing:

- `apify-actor-start`: **$0.00005 per run start**.
- `product-row`: **$0.001 per real product / variant row**.
- Diagnostic rows are emitted for transparency but are **not** charged as value rows.

Displayed price: **from $1.00 / 1,000 product or variant rows**.

Example: a 1,263-row competitor scan costs about **$1.263** in Actor event charges, plus/minus Apify platform usage depending on the user’s plan/settings.

### Limitations

- Works only with stores exposing Shopify’s public `/products.json` endpoint.
- Some stores disable, block, or alter the public product feed.
- Detects markdowns from `compare_at_price`; coupon/cart-only discounts are not visible.
- Inventory is a boolean (`available`), not exact stock count.
- Region-specific pricing and hidden/draft products are not included.
- A run is a snapshot; schedule repeated runs and compare datasets for true change tracking.
- Not affiliated with Shopify or any listed brand. Use responsibly and respect each site’s terms.

### Ecommerce Data Toolkit

This Actor is part of a focused ecommerce / retail-intelligence toolkit:

| Need | Actor |
|---|---|
| Shopify product feeds, variants, prices, discounts | [Shopify Product Feed & Price Scraper](https://apify.com/fierce_quoll/shopify-beauty-radar) |
| Brand retail footprint and where-to-buy / store-locator rows | [Brand Store Locator & Stockist Scraper](https://apify.com/fierce_quoll/beauty-stockist-radar) |
| Salon, stylist, and certified-pro locator rows | [Beauty Salon & Pro Locator Scraper](https://apify.com/fierce_quoll/beauty-pro-locator-radar) |

Use the three together to map **what brands sell**, **where products are stocked**, and **which salons/pros carry or certify around professional lines**.

### FAQ

#### Does this work on every Shopify store?

No. It works when the store exposes the public `/products.json` endpoint. Some stores disable or alter that endpoint.

#### Can I monitor changes over time?

Yes. Schedule this Actor daily or hourly, export datasets, and compare new runs against previous runs.

#### Does it require proxies?

No proxy is required for the default public-feed approach.

#### Can I export to CSV or Excel?

Yes. Apify datasets can be exported to JSON, CSV, Excel, XML, RSS, or consumed through API integrations.

### Quality / verification

This project now ships with a reusable sample input and standard local checks:

```bash
npm run check
cp examples/sample-input.json storage/key_value_stores/default/INPUT.json
CRAWLEE_STORAGE_DIR=./storage APIFY_LOCAL_STORAGE_DIR=./storage node src/main.js
```

The buyer demo package lives at:

```text
/opt/data/apify-projects/outreach/shopify-product-feed-demo.zip
```

### API usage

Run from the Apify CLI:

```bash
apify call fierce_quoll/shopify-beauty-radar --input-file examples/live-input.json
```

Or use the API tab on the Actor page for language-specific snippets.

### Local development

```bash
npm install
npm test
/opt/data/bin/apify validate-schema .actor/input_schema.json
mkdir -p storage/key_value_stores/default
cp examples/live-input.json storage/key_value_stores/default/INPUT.json
CRAWLEE_STORAGE_DIR=./storage APIFY_LOCAL_STORAGE_DIR=./storage node src/main.js
```

# Actor input Schema

## `storeUrls` (type: `array`):

Public Shopify store URLs to scan. Bare domains like 'examplestore.com' are accepted and normalized. The Actor reads each store's public /products.json feed when available.

## `keywords` (type: `array`):

Case-insensitive keywords matched against product title, vendor, product type, and tags. Leave empty to include every product. Matched keywords are reported per row.

## `maxProductsPerStore` (type: `integer`):

Maximum number of products to fetch from each store before variant expansion. Pagination stops once this many products are collected.

## `minDiscountPercent` (type: `number`):

Only emit rows whose discount (compare-at vs current price) is at least this percent. Use 0 to include full-price products too.

## `includeVariants` (type: `boolean`):

When enabled, emit one row per product variant. When disabled, emit one row per product using its best-deal (highest-discount) variant.

## `sortBy` (type: `string`):

opportunity = drop/sale score; discount = discount percent; price = current price ascending; newest = most recently published first. Sorting is applied within each store.

## `includeDiagnostics` (type: `boolean`):

Emit diagnostic rows (rowType='diagnostic') when a store is unreachable, returns a non-200 status, blocks the /products.json feed, or yields no matching products.

## Actor input object example

```json
{
  "storeUrls": [
    "https://mielleorganics.com",
    "https://baskandlatherco.com",
    "https://kaleidoscopehairproducts.com",
    "https://patternbeauty.com"
  ],
  "keywords": [
    "serum",
    "cleanser",
    "spf",
    "lip",
    "bundle"
  ],
  "maxProductsPerStore": 250,
  "minDiscountPercent": 0,
  "includeVariants": true,
  "sortBy": "opportunity",
  "includeDiagnostics": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "storeUrls": [
        "https://mielleorganics.com",
        "https://baskandlatherco.com",
        "https://kaleidoscopehairproducts.com",
        "https://patternbeauty.com"
    ],
    "keywords": [
        "serum",
        "cleanser",
        "spf",
        "lip",
        "bundle"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tpa_d/shopify-beauty-radar").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "storeUrls": [
        "https://mielleorganics.com",
        "https://baskandlatherco.com",
        "https://kaleidoscopehairproducts.com",
        "https://patternbeauty.com",
    ],
    "keywords": [
        "serum",
        "cleanser",
        "spf",
        "lip",
        "bundle",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("tpa_d/shopify-beauty-radar").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "storeUrls": [
    "https://mielleorganics.com",
    "https://baskandlatherco.com",
    "https://kaleidoscopehairproducts.com",
    "https://patternbeauty.com"
  ],
  "keywords": [
    "serum",
    "cleanser",
    "spf",
    "lip",
    "bundle"
  ]
}' |
apify call tpa_d/shopify-beauty-radar --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=tpa_d/shopify-beauty-radar",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gZiaqKHcUYIkA6Otv/builds/k0LVOQrtYx8FFyO7q/openapi.json