# Shopify New Store Scraper – Daily Leads by Pixels (`diplomatic_skull112/pixelspiece`) Actor

Shopify store scraper that detects newly launched stores in real time. Get a deduplicated list of new Shopify store URLs with country and first-seen date, updated daily. Ideal for ecommerce lead generation, cold outreach, and Shopify app marketing. Export to CSV, JSON, or Excel.

- **URL**: https://apify.com/diplomatic\_skull112/pixelspiece.md
- **Developed by:** [Mohit Shyani](https://apify.com/diplomatic_skull112) (community)
- **Categories:** Lead generation, E-commerce
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 2 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $5.00 / 1,000 store detecteds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Store Radar — Shopify new-store detector (Railway-ready)

Always-on service that watches the Certificate Transparency firehose, verifies which
newly-issued certs belong to Shopify stores, and writes one CSV per day. A built-in
web page lets you watch status and download the leads from anywhere — no SSH.

It needs a **certstream firehose** to read from. The public one
(`certstream.calidog.io`) is offline most of the time, so you run your own. Total
setup is two Railway services and about five minutes.

***

### Architecture

```
  ┌─────────────────────────┐        ┌────────────────────────────┐
  │  certstream (service 1) │  ws    │  store-radar (service 2)   │
  │  reloading01/           │───────▶│  this repo                 │
  │  certstream-server-rust │        │  • DNS-verifies vs Shopify │
  │  (public Docker image)  │        │  • writes daily CSV        │
  └─────────────────────────┘        │  • serves status + CSV     │
                                      └────────────────────────────┘
                                                  │  https
                                                  ▼
                                         you: open the URL,
                                         download today.csv
```

***

### Step 1 — Deploy the certstream firehose (service 1)

1. In your Railway project: **New → Deploy from Docker Image**.
2. Image: `ghcr.io/reloading01/certstream-server-rust:latest`
3. Under **Settings → Networking**, click **Generate Domain** (public URL). The
   server listens on **8080**; set the public port to 8080 if asked.
4. Copy the public domain, e.g. `certstream-production.up.railway.app`.

That's the whole service — no code. (Optional: set a bearer token via its env vars
if you don't want the stream open to the world. See its docs.)

> Why a public domain instead of Railway's private network? Private networking is
> IPv6-only and the rust server binds IPv4 by default, so the public URL is the
> reliable path. The traffic is public CT data anyway.

### Step 2 — Deploy this detector (service 2)

1. Push this folder to a GitHub repo (or use the Railway CLI).

2. Railway: **New → Deploy from GitHub repo** → pick it. It builds from the
   `Dockerfile` automatically (config in `railway.json`).

3. **Variables** → add:

   | Variable | Value |
   |---|---|
   | `CERTSTREAM_URL` | `wss://certstream-production.up.railway.app/` (your service-1 domain, `wss://`) |
   | `OUT_DIR` | `/data` |
   | `SMART_FILTER` | `1` |
   | `HTTP_VERIFY` | `0` (set `1` to also grab `.myshopify.com` handles — slower) |
   | `NICHE` | *(optional)* `apparel,bespoke,jewel` to log only matching domains |
   | `WORKERS` | `200` (concurrent DNS-verification workers) |
   | `DNS_THREADS` | `256` (getaddrinfo thread pool; raise for more DNS throughput) |
   | `DNS_TIMEOUT` | `5` (seconds before a slow DNS lookup is given up on) |
   | `NEW_ONLY` | `0` (set `1` to log only *new* domains, dropping cert renewals of existing stores) |
   | `NEW_WINDOW_DAYS` | `7` (a domain is "new" if its earliest-ever cert is within this many days) |
   | `KEEP_UNKNOWN` | `1` (in `NEW_ONLY` mode, still log rows whose age couldn't be determined; set `0` to drop them) |
   | `CRTSH_CONCURRENCY` | `3` (max simultaneous crt.sh history lookups) |

4. **Settings → Networking → Generate Domain** for this service too.

5. **Settings → Volumes → New Volume**, mount path `/data`. (Railway disks are
   ephemeral; the volume keeps your CSVs across restarts and redeploys.)

### Step 3 — Use it

Open the detector's public URL:

- `/` — live status: certs seen, stores found, finds/min, recent hits (now with a
  country column). Auto-refreshes.
- `/today.csv` — download today's leads.
- `/shopify_new_stores_YYYY-MM-DD.csv` — any past day (listed on the status page).
- `/stores.json` — last 200 finds as JSON (handy to wire into your pipeline).
- `/countries` — **deduped leads split by country.** One downloadable CSV per
  country plus a combined deduped file.

***

### Deduped, country-split leads

The raw daily CSVs log **every** detected host, so the same store often appears
several times: `bellemo.shop`, `www.bellemo.shop`, `account.bellemo.shop`. The
organizer collapses every host to its **registrable domain** (so those three
become the single store `bellemo.shop`) and splits the result by country.

Country comes from the domain's TLD: country-code TLDs (`.co.uk` → UK, `.de` →
Germany, `.com.au` → Australia, `.ca` → Canada …) map to a country; generic TLDs
(`.com`, `.shop`, `.store`, `.online` …) and vanity ccTLDs (`.co`, `.io`, `.me` …)
go to a **GENERIC** bucket. (GeoIP on the resolved IP can't help — every store
points at Shopify's own IP block, so the IP only tells you where Shopify is.)

**From the web UI:** open `/countries`, click **Rebuild from raw CSVs**, then
download per-country files (`/by-country/UK.csv`, `/by-country/AU.csv`, …) or the
whole deduped set (`/deduped.csv`).

**From the command line:**

```bash
python organize_stores.py                 # organize every CSV in ./out
python organize_stores.py --in-dir ./out --out-dir ./out/organized
python organize_stores.py "out/200000+ stores.csv"   # one specific file
python organize_stores.py --colombia-real # treat .co/.io/.me as their real country
```

Output (under `--out-dir`, default `./out/organized`):

| File | What |
|---|---|
| `all_stores_deduped.csv` | every unique store, with `registrable_domain`, `country_code`, `country`, and the `subdomains` that were collapsed |
| `by_country/<CC>.csv` | one file per country (`UK.csv`, `AU.csv`, `DE.csv`, `GENERIC.csv`, …) |
| `by_country/_index.csv` | country, store count, file — a summary |

Columns: `registrable_domain, country_code, country, seen_host, subdomains,
detected_at, myshopify_handle, resolved_ip, type, signal, freshness`.

***

### Run locally (to test before deploying)

```bash
pip install -r requirements.txt

## Point at your own certstream, or try the public one (often down):
export CERTSTREAM_URL="wss://YOUR-certstream-host/"
export OUT_DIR="./out"
python store_radar_service.py
## open http://localhost:8080
```

Run certstream locally too if you want a fully local stack:

```bash
docker run -d -p 8080:8080 ghcr.io/reloading01/certstream-server-rust:latest
export CERTSTREAM_URL="ws://localhost:8080/"
```

***

### What it catches (and what it doesn't)

- **Strong:** stores that connect a custom domain and enable managed SSL — your best
  leads, caught within seconds of going live.
- **Weak:** free `*.myshopify.com` stores often sit behind a wildcard cert and won't
  always appear; Cloudflare-fronted stores won't resolve to Shopify's IPs.
- `SMART_FILTER` narrows to Let's-Encrypt + few-SAN certs (the managed-SSL pattern)
  to keep DNS volume sane. Turn it off to check everything (much heavier).
- This is a high-signal daily stream, not a complete census of every new store.

### New vs renewal

The CT firehose fires on certificate **issuance**, which mixes brand-new stores with
existing stores whose SSL cert just auto-renewed. To tell them apart, each confirmed
custom-domain store is checked against [crt.sh](https://crt.sh): we take the earliest
certificate ever issued for that domain and classify it.

- **`new`** — earliest cert is within `NEW_WINDOW_DAYS` (default 7) → likely a store
  that just came online.
- **`renewal`** — earliest cert is older → an existing store renewing SSL.
- **`unknown`** — crt.sh lookup failed, timed out, or returned nothing. crt.sh is slow
  and rate-limited, so this happens; `*.myshopify.com` (cert-SAN) rows are always
  `unknown` because wildcard certs make crt.sh unreliable for them.

Set `NEW_ONLY=1` to write only `new` (and `unknown`, unless `KEEP_UNKNOWN=0`) rows.
Lookups run behind a small concurrency cap (`CRTSH_CONCURRENCY`) with a 10s timeout,
one retry, and an in-memory cache, so they never starve DNS verification.

> **Caveat:** this detects a *new domain*, not strictly a new Shopify store. A domain
> that existed on another platform (Wix, WooCommerce, a static site…) and later migrated
> to Shopify will have old certs and so look like a `renewal`, even though the *store* is
> brand new on Shopify. Treat `new` as high-confidence and `renewal`/`unknown` as worth
> a second look rather than a hard exclusion.

### Output columns

`detected_at, domain, myshopify_handle, resolved_ip, type, signal, freshness`

`type` is `myshopify` (cert SAN) or `custom`; `signal` is how it was confirmed
(`cert`, `dns`, or `fingerprint`); `freshness` is `new`, `renewal`, or `unknown`
(see [New vs renewal](#new-vs-renewal)).

# Actor input Schema

## `maxItems` (type: `integer`):

Stop the run after this many stores are detected. Keeps your cost predictable. Set 0 for no limit (then the time limit ends the run).

## `maxRuntimeSecs` (type: `integer`):

Hard stop after this many seconds (includes ~20s firehose warm-up). Set 0 for no time limit (then the store limit ends the run). If both limits are 0, the run defaults to 600s so it can never run forever.

## `niche` (type: `array`):

Only keep domains whose name contains one of these words, e.g. apparel, jewel, print. Leave empty to get every detected Shopify store.

## `countries` (type: `array`):

Keep only these country-code TLDs, e.g. UK, AU, DE, CA. Leave empty for all countries. Generic TLDs (.com, .shop, .store) are labelled GENERIC.

## `newOnly` (type: `boolean`):

Cross-check crt.sh and keep only domains whose earliest certificate is recent (a genuinely new store, not an SSL renewal of an existing one). Higher signal, slower.

## `httpVerify` (type: `boolean`):

Also load each store's homepage to confirm Shopify and capture its .myshopify.com handle. Richer data, slower.

## `workers` (type: `integer`):

Concurrent DNS lookups. Higher drains bursts faster; 200 is a good default.

## Actor input object example

```json
{
  "maxItems": 100,
  "maxRuntimeSecs": 600,
  "niche": [
    "apparel",
    "jewel",
    "print"
  ],
  "countries": [],
  "newOnly": false,
  "httpVerify": false,
  "workers": 200
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "niche": [
        "apparel",
        "jewel",
        "print"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("diplomatic_skull112/pixelspiece").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "niche": [
        "apparel",
        "jewel",
        "print",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("diplomatic_skull112/pixelspiece").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "niche": [
    "apparel",
    "jewel",
    "print"
  ]
}' |
apify call diplomatic_skull112/pixelspiece --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=diplomatic_skull112/pixelspiece",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/HyHXaEkdtEkIQR6bA/builds/D1vPQORPl2KyLk0ys/openapi.json