# Lidl Product Scraper (`incognito_mode/lidl-product-scraper`) Actor

Scrape every Lidl (lidl.nl) product from the official API: name, brand, price, Lidl Plus price, discount, size, category and image. Clean JSON/CSV. National pricing, no login. Needs a Dutch (NL) proxy. Failed lookups are never billed.

- **URL**: https://apify.com/incognito\_mode/lidl-product-scraper.md
- **Developed by:** [Elena Vance](https://apify.com/incognito_mode) (community)
- **Categories:** E-commerce, Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 product results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Lidl Product Scraper — Prices, Lidl Plus Member Prices, Discounts & Images from Lidl.nl

**Turn the entire Lidl Netherlands (lidl.nl) assortment into clean, structured
JSON or CSV: product name, brand, regular price, **Lidl Plus member price**,
offer/discount, pack size, category, image, and the product link — one tidy
record per product.**

Browse the whole catalog by category, or search by keyword, and get back data
ready to drop into a spreadsheet, database, price-comparison site, or app. No
login, no account — and **you are never billed for failed requests**.

> **Good to know:** the default **Automatic (datacenter) proxy** works fine and
> keeps costs low — no Dutch proxy required. (If you ever see blocked requests,
> switch to a Dutch residential proxy.) Lidl uses a single **national** price
> list, so one run captures the price every shopper in the Netherlands sees.

***

### Why this Actor

- **Lidl Plus member prices, captured.** When a product has a Lidl Plus loyalty
  price, the Actor surfaces it as the current `price`, flags the record with
  `isLidlPlusPrice: true`, and keeps the regular price in `priceBeforeOffer` — so
  you see both the member price *and* what non-members pay, in one record.
- **National pricing in one run.** Lidl NL publishes one price list for the whole
  country — no per-store fan-out, no store IDs. One run = the price everyone sees.
- **Offers and discounts, detected for you.** Each record carries the current
  `price`, the pre-discount `priceBeforeOffer`, an `isOffer` flag, and a
  `discountLabel` (e.g. `-25%`) — filter the whole assortment down to this week's
  deals in one expression.
- **The full assortment, or a focused slice.** Leave everything default to scrape
  every leaf category, list specific category IDs, or pass a search keyword.
- **Complete product detail.** Brand, description as clean text, pack size,
  category, the product image, and the canonical lidl.nl product URL — one
  normalized record.
- **You never pay for failures.** Timeouts, blocks, and removed products are
  reported in the run summary — **not** written to your dataset and **not billed**.
- **Fast and lightweight.** Runs are quick and compute stays minimal.
- **Clean, consistent EUR output.** Prices as euro floats, whitespace-normalized
  text, JSON-safe values throughout.

***

### Problems this Actor solves

| If you are… | Your problem | How this Actor solves it |
|---|---|---|
| **A price-comparison / deals site** | Keeping a grocery price database current by hand is impossible | Schedule a daily run; ingest every product with regular price, Lidl Plus price, and offer as JSON |
| **A market researcher / analyst** | Grocery-price and inflation tracking needs structured, repeatable data | Dated, normalized records per product — export straight to pandas, Sheets, or BI |
| **A category manager at another retailer** | Benchmarking a competitor's assortment and pricing is slow and partial | Full category-level coverage, including loyalty (Lidl Plus) pricing, in a single run |
| **An app / chatbot / agent developer** | You need Lidl catalog data without building and babysitting a scraper | Pay per record on demand; a stable, normalized schema you can rely on |
| **A deal hunter / content creator** | Finding the best weekly discounts means clicking through the site | Pull the assortment and filter on `isOffer` / `discountLabel` / `isLidlPlusPrice` |

***

### What data you get

Each available product becomes one dataset record:

| Field | Description |
|---|---|
| `id` / `erpNumber` | Lidl's stable article number (the record's unique id) |
| `title` | Product name |
| `brand` | Brand name, when Lidl flags it as shown (else `null`) |
| `description` | Product description as clean text (short + supplemental, when present) |
| `categoryId` / `categoryName` | Category id and title for the product |
| `price` | Current price as a EUR float — the **Lidl Plus member price** when one exists, otherwise the regular price |
| `currency` | Always `EUR` |
| `priceBeforeOffer` | The pre-discount / regular price when the product is on offer or has a member price (else `null`) |
| `isOffer` | `true` when the product is discounted |
| `isLidlPlusPrice` | `true` when `price` is a Lidl Plus loyalty price (not the regular shelf price) |
| `discountLabel` | e.g. `-25%` — from the site, or computed from old/new price |
| `unitSize` | Pack / sales-unit text where Lidl provides it (e.g. `500 g`) |
| `imageUrl` | Product image URL |
| `productUrl` | Canonical lidl.nl product page |
| `scrapedAt` | ISO 8601 timestamp of the run |
| `rawData` | *Optional:* the full raw Lidl product object, when **Include raw product payload** is on |

#### Example output

```jsonc
{
  "id": "10024896",
  "erpNumber": "10024896",
  "title": "Milbona Griekse Yoghurt 500 g",
  "brand": "Milbona",
  "description": "Romige yoghurt naar Griekse stijl, 500 gram.",
  "categoryId": "h10071018",
  "categoryName": "Zuivel & eieren",
  "price": 0.89,             // Lidl Plus member price (see isLidlPlusPrice)
  "currency": "EUR",
  "priceBeforeOffer": 1.19,  // the regular (non-member) price
  "isOffer": true,
  "isLidlPlusPrice": true,
  "discountLabel": "-25%",
  "unitSize": "500 g",
  "imageUrl": "https://www.lidl.nl/.../milbona-griekse-yoghurt.jpg",
  "productUrl": "https://www.lidl.nl/p/milbona-griekse-yoghurt/p10024896",
  "scrapedAt": "2026-06-18T08:30:00+00:00"
}
```

Inputs that could not be fetched are **not** written to the dataset (and never
billed). They are listed in the run's `SUMMARY` record in the key-value store —
`{ "failures": [ { "input": "…", "error": "…" } ] }` — and the run's status
message tells you at a glance how many products succeeded.

***

### How to use it (60 seconds)

1. Click **Try for free / Start**.
2. Choose what to scrape:
   - **Everything (default):** leave the inputs empty to scrape the full Lidl
     assortment (all discovered leaf categories).
   - **Specific categories:** add one category id per line under **Category IDs**,
     e.g. `h10071012` (or just `10071012`).
   - **A keyword:** type a term like `kaas` under **Search keyword**.
3. Leave the **Automatic (datacenter) proxy** as-is — it works out of the box.
   (Only if you hit blocked requests, switch to a Dutch residential proxy.)
4. Click **Save & Start**. Download results as **JSON, CSV, Excel, or via API**
   from the **Dataset** tab; check **Key-value store → SUMMARY** for run totals
   and any failed requests.

***

### Input reference

| Field | Type | Default | Description |
|---|---|---|---|
| **Category IDs** | list | *(empty)* | One Lidl category id per line (e.g. `h10071012`, or just `10071012`). Empty = scrape all leaf categories (full assortment). |
| **Search keyword** | string | – | Scrape products matching a term instead of browsing categories. When set, the category list is ignored. |
| **Include raw product payload** | boolean | `false` | Adds the full raw Lidl product object under `rawData`. Increases item size. |
| **Proxy configuration** | object | Apify Automatic | Automatic (datacenter) proxies work fine and keep costs low. Switch to a Dutch (NL) residential proxy only if you see blocked requests. |
| **Max concurrency** | integer | `4` | Parallel requests (1–20). Kept moderate to be respectful. |
| **Delay between requests** | integer | `0` | Politeness delay in seconds before each request (0–10). ~0.3s is plenty; `0` is fine at moderate concurrency. |
| **Max items** | integer | `0` | Stop after N product records (`0` = unlimited). |

***

### Good to know

- **Category mode** covers the full assortment (or the category IDs you list);
  **search mode** returns products matching your keyword. Products are
  de-duplicated, and when a Lidl Plus member price is present it becomes the
  headline `price` (with the regular price preserved in `priceBeforeOffer`).
- Some categories — notably Lidl's weekly-rotating **"Eten & Drinken"** food
  range — are small or temporarily empty; an empty category is normal, not an
  error.

***

### Pricing — what a run costs

This Actor uses transparent **pay-per-event** pricing with a built-in **volume
discount**: a small Actor-start fee, a fixed price per successful product record
for the first tier of results in a run, and a cheaper rate for every result
beyond the volume threshold. **No subscription, no minimums, and failures are
never charged.** The exact per-result rate is shown on the Actor's **Pricing** tab.

- **Failed requests are free.** Timeouts, blocks, and removed products are
  reported in the summary, never billed.
- **Bigger runs cost less per item.** The discount tier resets per run, so one
  large run is cheaper than the same job split into many small ones.
- **Try it free:** an Apify free account includes **$5 of monthly platform
  credit** — enough to try the Actor before paying anything.
- **Stay in control:** set **Max items** and Apify's **maximum charge per run**;
  the Actor stops gracefully at your cap, keeping everything already scraped.

***

### Compared to the alternatives

| | **This Actor** | Build your own scraper | Manual checking |
|---|---|---|---|
| Full assortment, normalized | Yes | You maintain it | Impractical |
| **Lidl Plus member prices** | Yes | You build it yourself | One product at a time |
| Offers & discount labels | Yes | You maintain it | One product at a time |
| Works out of the box | Yes | You build and babysit it | – |
| Proxy handled for you | Yes (datacenter works) | You source proxies | – |
| Never billed for failures | Yes | – | – |
| Export JSON / CSV / Excel / API | Yes, built-in | DIY | Copy-paste |
| Setup time | ~60 seconds | Days; breaks on site changes | Hours per run |

***

### Integrate the data

- **Exports:** JSON, CSV, Excel, XML from the Dataset tab — or fetch
  programmatically:
  ```
  GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json
  ```
- **Run on a schedule:** use Apify **Schedules** to refresh prices daily or
  weekly, and **webhooks** to push finished runs into your pipeline (Sheets,
  Slack, your backend).
- **From code:** call the Actor with the [Apify API or SDKs](https://docs.apify.com/api)
  (Python / JavaScript) and read the dataset when the run finishes.
- **Run summary:** every run writes a `SUMMARY` record (key-value store) with
  totals, successes, failures, and billing counts — ideal for monitoring
  automated pipelines.
- **Need the category tree itself?** The companion **Lidl Category Scraper**
  emits one record per Lidl category (main categories and leaf subcategories) —
  handy for building your own category navigation or driving targeted runs of
  this Actor.

***

### FAQ

**Do I need a Lidl account or login?**
No. There is no login or account required.

**What is the Lidl Plus price I see in the data?**
Lidl Plus is Lidl's loyalty programme, and some products carry a members-only
price. When one exists, the Actor reports it as the current `price` and sets
`isLidlPlusPrice: true`, while keeping the regular shelf price in
`priceBeforeOffer` — so you always have both.

**Do I need a Dutch proxy?**
No. The default Automatic (datacenter) proxy works fine — Lidl's API returns the
national (NL) assortment and prices regardless of where the request comes from.
Only if you ever see blocked requests, switch to a Dutch (NL) residential proxy.

**Are prices per store?**
No — Lidl Netherlands uses a single national price list, so one run reflects the
price nationwide.

**How do I get only discounted products?**
Scrape, then filter on `isOffer == true` (or a non-null `discountLabel`). To
isolate loyalty deals, filter on `isLidlPlusPrice == true`.

**One of the food categories came back empty — is that a bug?**
No. Lidl's "Eten & Drinken" food range rotates weekly and is often small or
temporarily empty. Empty categories are expected and are not reported as failures.

**How fresh is the data?**
Each run fetches live data, so you get exactly what lidl.nl shows at that moment.
Schedule the Actor to keep your dataset as fresh as you need.

**What formats can I export?**
JSON, CSV, Excel, XML — from the Console or via the Apify API.

***

### Related Actors

**Pair this with the Lidl Category Scraper.** The companion **Lidl Category
Scraper** emits one record per Lidl category — every main category and h-type leaf
subcategory, with ids, names, and product counts. Run it **first** to map the
current Lidl category tree, then feed the leaf category IDs back into this Actor's
**Category IDs** input to scrape exactly the branches you want. Category first to
navigate, product second to harvest.

Part of a family of Dutch-supermarket product scrapers — same clean schema and
pay-per-event billing across every chain, so you can mix and match for full market
coverage:

- **Albert Heijn Product Scraper** — Bonus discounts, national pricing.
- **Plus Product Scraper** — per-store pricing.
- **Dirk van den Broek Product Scraper** — store-specific pricing.
- **DekaMarkt Product Scraper** — store-specific pricing.
- **Hoogvliet Product Scraper** (pairs with the **Hoogvliet Category Scraper**).

Same normalized output and the same "never billed for failures" guarantee across
the whole family — combine them to benchmark prices across the Dutch market.

***

### Disclaimer

This Actor is intended for personal and research use. You are responsible for
ensuring your use complies with Lidl's terms and applicable law. Please scrape
responsibly — keep concurrency moderate and delays reasonable. This project is
not affiliated with, endorsed by, or sponsored by Lidl.

# Actor input Schema

## `categoryIds` (type: `array`):

One Lidl category id per line to scrape only those categories (e.g. h10071012 for Fruit & groenten, or just 10071012). Leave empty to scrape all leaf categories (the full assortment).

## `searchQuery` (type: `string`):

Scrape products matching a search term (e.g. "kaas") instead of browsing categories. When set, the category list above is ignored.

## `includeRawData` (type: `boolean`):

Adds the full raw Lidl product object to each record under 'rawData'. Increases item size; leave off unless you need fields beyond the normalized ones.

## `proxyConfiguration` (type: `object`):

Proxies used for requests. Automatic (datacenter) proxies are fine and keep costs near zero. Switch to Dutch (NL) residential only if you see blocked requests.

## `maxConcurrency` (type: `integer`):

Maximum number of parallel requests. Keep this moderate to stay respectful.

## `requestDelaySecs` (type: `integer`):

Politeness delay applied before each request. ~0.3s is plenty; 0 is fine at moderate concurrency.

## `maxItems` (type: `integer`):

Stop after producing this many product records. Set to 0 for no limit (full assortment).

## Actor input object example

```json
{
  "categoryIds": [
    "h10071012"
  ],
  "includeRawData": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 4,
  "requestDelaySecs": 0,
  "maxItems": 0
}
```

# Actor output Schema

## `products` (type: `string`):

All scraped product records (one item per product).

## `summary` (type: `string`):

JSON summary of the run: totals, successes, failures, and billing counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "categoryIds": [],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("incognito_mode/lidl-product-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "categoryIds": [],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("incognito_mode/lidl-product-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "categoryIds": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call incognito_mode/lidl-product-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=incognito_mode/lidl-product-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1mwvuYee3dTnO4GcE/builds/r3EU7m1PXRrrZZanW/openapi.json