# Common Crawl Scraper (`crawlerbros/common-crawl-scraper`) Actor

Query the Common Crawl URL Index for any domain or URL pattern. Discover a site's archived pages, historical URLs, capture dates, HTTP statuses and MIME types for SEO, domain intelligence and research. Also lists the available monthly crawls.

- **URL**: https://apify.com/crawlerbros/common-crawl-scraper.md
- **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 3 total users, 2 monthly users, 69.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Common Crawl Scraper

Query the **Common Crawl URL Index** for any domain or URL pattern and get back every archived capture — the historical URLs a site exposed, when they were crawled, their HTTP status and MIME type, and the exact location of the raw page in Common Crawl's public archive. Perfect for SEO audits, domain intelligence, historical URL discovery and web-scale research. HTTP-only, no login, no proxy.

### What this actor does

- **Two modes:** `urlCaptures` and `listCrawls`
- **Wildcard lookups:** `example.com`, `example.com/*`, `en.wikipedia.org/wiki/*`, `*.example.com`
- **125+ monthly crawls:** query the latest crawl or any historical one
- **Server-side filters:** date range; **client-side filters:** HTTP status, MIME type
- **WARC location** for every capture so you can fetch the raw archived page
- **Empty fields are omitted** — every field in a record is populated

### Modes

| Mode | What it does | Needs |
|---|---|---|
| `urlCaptures` | Look up all archived captures for a domain / URL pattern | `urlPattern` (+ optional `crawl`, filters) |
| `listCrawls` | List every available Common Crawl monthly crawl | – |

### Output — `urlCaptures` (one row per archived capture)

- `url` — the archived URL
- `urlKey` — Common Crawl's canonical (SURT) key
- `timestamp` — capture time, `YYYYMMDDHHMMSS`
- `captureDate` — the same time as ISO 8601
- `status` — HTTP status at capture time
- `mime` — declared MIME type
- `mimeDetected` — MIME type detected from content
- `digest` — content digest (dedupe identical pages)
- `length` — record byte length
- `offset` — byte offset within the WARC file
- `filename` — WARC file path in the archive
- `languages` — detected language codes
- `encoding` — character encoding
- `redirectUrl` — redirect target (for 3xx captures)
- `truncated` — truncation reason (when present)
- `crawlId` — which crawl this came from
- `warcUrl` — direct link to the WARC file on `data.commoncrawl.org`
- `recordType: "capture"`, `sourceUrl`, `scrapedAt`

### Output — `listCrawls` (one row per crawl)

- `crawlId` — e.g. `CC-MAIN-2024-10`
- `name` — human-readable name (e.g. `February/March 2024 Index`)
- `fromDate`, `toDate` — crawl time window
- `cdxApiUrl` — the crawl's index API endpoint
- `timegateUrl` — the crawl's timegate
- `recordType: "crawl"`, `sourceUrl`, `scrapedAt`

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `urlCaptures` | `urlCaptures` / `listCrawls` |
| `urlPattern` | string | `en.wikipedia.org/wiki/*` | Domain or URL, `*` wildcards allowed |
| `crawl` | string | `latest` | `latest` or a crawl id like `CC-MAIN-2024-10` |
| `matchType` | string | `auto` | `auto` / `exact` / `prefix` / `host` / `domain` |
| `statusFilter` | int | – | Keep only this HTTP status (e.g. `200`) |
| `mimeFilter` | string | – | Keep only MIME types containing this text |
| `fromDate` | string | – | `YYYYMMDD` lower bound |
| `toDate` | string | – | `YYYYMMDD` upper bound |
| `maxItems` | int | `100` | Hard cap (1–5000) |

#### Example: all archived Wikipedia article URLs in the latest crawl

```json
{ "mode": "urlCaptures", "urlPattern": "en.wikipedia.org/wiki/*", "crawl": "latest", "maxItems": 500 }
```

#### Example: only successful HTML pages of a domain

```json
{ "mode": "urlCaptures", "urlPattern": "example.com/*", "statusFilter": 200, "mimeFilter": "text/html" }
```

#### Example: a whole domain including subdomains, in a specific crawl

```json
{ "mode": "urlCaptures", "urlPattern": "wikipedia.org", "matchType": "domain", "crawl": "CC-MAIN-2024-10" }
```

#### Example: list every available crawl

```json
{ "mode": "listCrawls" }
```

### Use cases

- **SEO & site audits** — discover every URL a domain has ever exposed to crawlers
- **Domain intelligence** — profile a competitor's URL structure and content types
- **Historical URL discovery** — recover old / removed pages for migration or research
- **Data engineering** — get WARC offsets to pull raw archived pages at scale
- **Security research** — enumerate a domain's historical footprint

### Data source

Data comes from the public [Common Crawl URL Index](https://index.commoncrawl.org/) (the CDX API) and the crawl archive at `data.commoncrawl.org`, both published openly by the Common Crawl Foundation. No account, API key, or proxy is required. Coverage spans well over a decade of monthly crawls; run `listCrawls` to see every crawl currently available and its date range.

### FAQ

**What is Common Crawl?**  A free, open repository of web crawl data covering billions of pages, updated roughly monthly. Its URL Index lets you look up which URLs were captured and where they live in the archive. See [commoncrawl.org](https://commoncrawl.org).

**What's a "crawl"?**  Each monthly snapshot is a crawl with an id like `CC-MAIN-2024-10`. Use `latest` for the newest, or run `listCrawls` to see all available ids and their date ranges.

**How do wildcards work?**  `example.com` matches that host's pages; `example.com/*` matches everything under it; `*.example.com` matches all subdomains. You can also set `matchType` explicitly to `exact`, `prefix`, `host` or `domain`.

**Why did I get no results for a big-name site?**  Some sites exclude crawlers via robots.txt, so they aren't in Common Crawl. Try a different domain or crawl.

**Can I fetch the actual page content?**  Each capture includes `warcUrl`, `offset` and `length` — enough to download the exact archived response from Common Crawl's public `data.commoncrawl.org` store.

**What does `digest` do?**  It's a content hash — identical `digest` values across captures mean the page content didn't change, which is handy for change detection and deduplication.

**How far back does the data go?**  Common Crawl has crawls stretching back over a decade; `listCrawls` shows every one currently available.

# Actor input Schema

## `mode` (type: `string`):

What to fetch.

## `urlPattern` (type: `string`):

Domain or URL to look up. Use `*` wildcards, e.g. `example.com`, `example.com/*`, `en.wikipedia.org/wiki/*`, or `*.example.com`.

## `crawl` (type: `string`):

Which monthly crawl to query. Use `latest` for the newest, or a crawl id like `CC-MAIN-2024-10`. Run mode=listCrawls to discover valid ids.

## `matchType` (type: `string`):

How to interpret the pattern. `auto` lets the `*` wildcards decide.

## `statusFilter` (type: `integer`):

Only keep captures with this HTTP status (e.g. 200). Leave empty for all.

## `mimeFilter` (type: `string`):

Only keep captures whose MIME type contains this text (e.g. `text/html`, `pdf`, `json`).

## `fromDate` (type: `string`):

Only include captures on or after this date, e.g. `20240101`.

## `toDate` (type: `string`):

Only include captures on or before this date, e.g. `20241231`.

## `maxItems` (type: `integer`):

Hard cap on emitted records.

## Actor input object example

```json
{
  "mode": "urlCaptures",
  "urlPattern": "en.wikipedia.org/wiki/*",
  "crawl": "latest",
  "matchType": "auto",
  "maxItems": 15
}
```

# Actor output Schema

## `captures` (type: `string`):

Dataset containing all scraped Common Crawl records.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "urlCaptures",
    "urlPattern": "en.wikipedia.org/wiki/*",
    "crawl": "latest",
    "matchType": "auto",
    "maxItems": 15
};

// Run the Actor and wait for it to finish
const run = await client.actor("crawlerbros/common-crawl-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "urlCaptures",
    "urlPattern": "en.wikipedia.org/wiki/*",
    "crawl": "latest",
    "matchType": "auto",
    "maxItems": 15,
}

# Run the Actor and wait for it to finish
run = client.actor("crawlerbros/common-crawl-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "urlCaptures",
  "urlPattern": "en.wikipedia.org/wiki/*",
  "crawl": "latest",
  "matchType": "auto",
  "maxItems": 15
}' |
apify call crawlerbros/common-crawl-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=crawlerbros/common-crawl-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/btF0SucZyc8UChaSA/builds/hWF0gztdINx8D1eke/openapi.json