# RAG Web Crawler: Clean Markdown + Token-Sized Chunks (`commonelements/rag-ready-crawler`) Actor

Turn any website into embeddings-ready chunks for RAG and vector databases. Structure-aware token-sized chunking, clean LLM-ready markdown, per-chunk citations and metadata, dedup, and junk filtering. Pay per result, no surprise compute bills.

- **URL**: https://apify.com/commonelements/rag-ready-crawler.md
- **Developed by:** [Harry Schoeller](https://apify.com/commonelements) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 dataset item scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Web Crawler — Clean Markdown + Token-Sized Chunks, Pay-Per-Result

Turn any website into **embeddings-ready chunks** with citations and predictable
per-chunk pricing. No CSS tuning, no runaway compute bills.

Generic crawlers hand you raw pages and make you build the RAG pipeline yourself.
This actor hands you clean, token-sized, deduplicated, citable chunks — at a fixed
price **per chunk you keep**.

### What it does

- **Clean LLM-ready Markdown** — `@mozilla/readability` strips nav, footers, ads,
  and cookie banners; `turndown` + GFM converts the cleaned DOM to Markdown with
  heading hierarchy, fenced code blocks, and tables preserved.
- **Structure-aware, token-budgeted chunking** — splits on the heading tree, then
  recursively sub-splits oversized sections to your token budget (default 512)
  with overlap (default 75). Code blocks and tables are kept intact, never split
  mid-block.
- **Rich per-chunk provenance** — every chunk ships source URL + deep anchor,
  page title, full headings path, content hash, token count, content type, and
  language for metadata-filtered vector search and deep-link citations.
- **Dedup + junk filtering** — exact content-hash dedup plus 64-bit SimHash
  near-duplicate collapsing, and low-information / nav-residue chunk filtering.
- **Four output formats** — `chunks-jsonl` (one record per chunk), `markdown`
  (one record per page), `langchain` (drop-in `{page_content, metadata}` Document
  JSON), and `jsonl-bulk` (flat one-record-per-chunk for DB/`COPY`/pgvector).
- **Incremental / delta sync** — on scheduled re-runs, only NEW or CHANGED pages
  are re-emitted (and billed). Makes daily/weekly crawls cheap.
- **Budget guarantee** — `maxPages` is a hard ceiling; billing is per emitted
  result, so a runaway crawl can never produce a runaway bill.

### Incremental / delta sync — cheap scheduled re-runs

Turn on **Incremental sync** (`incremental: true`) and schedule the actor to run
daily or weekly. The first run does a full crawl and seeds a per-URL content-hash
state in a named key-value store. Every later run crawls the site, but only
re-emits chunks for pages that are **new** or **changed** — unchanged pages cost
nothing. A typical weekly docs re-crawl re-emits a handful of pages instead of
hundreds.

- **State is automatic.** The state store name defaults to a deterministic hash of
  your start URLs, so a scheduled task reuses its own prior state with zero config.
  Set `stateStoreName` explicitly to share state across tasks/schedules.
- **`forceFullCrawl: true`** re-emits everything and rebuilds the baseline — use
  after changing chunking settings or to refresh a stale index.
- **`emitDeletions: true`** writes a tombstone record (`{ deleted: true, url, ... }`)
  to a separate `deletions` dataset for every URL that disappeared since the last
  run, so downstream vector stores can purge stale vectors. Tombstones are not billed.
- When incremental is ON, each emitted record carries a `change_status`
  (`new` | `changed`) in its metadata.

The run summary (`OUTPUT` key-value record) includes a delta block:
`pages_new`, `pages_changed`, `pages_unchanged`, `pages_deleted`,
`chunks_skipped_unchanged` (the spend you saved), `state_store`, `prior_run_id`.

When all incremental options are OFF (the default), behavior and output are
byte-for-byte identical to v1.0.

### Output (chunks-jsonl)

````json
{
  "id": "a1f3c9e29b2c4d10",
  "url": "https://docs.example.com/guide/install",
  "title": "Getting Started — Example Docs",
  "chunkIndex": 3,
  "chunkTotal": 11,
  "headingsPath": ["Getting Started", "Setup", "Installation"],
  "text": "## Installation\n\nInstall via npm:\n\n```bash\nnpm install crawlee\n```",
  "tokenEstimate": 498,
  "fetchedAt": "2026-06-20T14:02:11Z",
  "content_hash": "sha256:...",
  "metadata": {
    "source_url": "https://docs.example.com/guide/install",
    "deep_link": "https://docs.example.com/guide/install#installation",
    "anchor": "installation",
    "canonical_url": "https://docs.example.com/guide/install",
    "page_title": "Getting Started — Example Docs",
    "char_count": 2104,
    "content_type": "mixed",
    "language": "en",
    "last_modified": null,
    "crawl_timestamp": "2026-06-20T14:02:11Z"
  }
}
````

Each record maps 1:1 to a vector-DB upsert: `{ id, values=embed(text), metadata }`.

### Output (langchain)

One record per chunk, drop-in for LangChain — `[Document(**r) for r in dataset]`:

```json
{
  "page_content": "## Installation\n\nInstall via npm...",
  "metadata": {
    "id": "a1f3c9e29b2c4d10",
    "source": "https://docs.example.com/guide/install",
    "title": "Getting Started — Example Docs",
    "deep_link": "https://docs.example.com/guide/install#installation",
    "canonical_url": "https://docs.example.com/guide/install",
    "headings_path": ["Getting Started", "Setup", "Installation"],
    "chunk_index": 3,
    "chunk_total": 11,
    "content_type": "mixed",
    "language": "en",
    "token_estimate": 498,
    "char_count": 2104,
    "content_hash": "sha256:...",
    "last_modified": null,
    "crawl_timestamp": "2026-06-20T14:02:11Z"
  }
}
```

### Output (jsonl-bulk)

Fully flat one-record-per-chunk for generic bulk import (DB `COPY` / pgvector):

```json
{
  "id": "a1f3c9e29b2c4d10",
  "text": "## Installation\n\nInstall via npm...",
  "source_url": "https://docs.example.com/guide/install",
  "deep_link": "https://docs.example.com/guide/install#installation",
  "canonical_url": "https://docs.example.com/guide/install",
  "title": "Getting Started — Example Docs",
  "headings_path": "Getting Started > Setup > Installation",
  "chunk_index": 3,
  "chunk_total": 11,
  "content_type": "mixed",
  "language": "en",
  "token_estimate": 498,
  "char_count": 2104,
  "content_hash": "sha256:...",
  "last_modified": null,
  "crawl_timestamp": "2026-06-20T14:02:11Z"
}
```

### Input

See `.actor/input_schema.json`. Key fields: `startUrls`, `crawlScope`,
`maxCrawlDepth`, `maxPages`, `renderJs`, `outputFormat`, `chunkSize`,
`chunkOverlap`, `dedupNearDuplicates`, `filterJunkChunks`, and the incremental
sync fields `incremental`, `forceFullCrawl`, `stateStoreName`, `emitDeletions`.

### Pricing

Pay-Per-Event. Billable unit = one emitted dataset item. Deduped and
junk-filtered chunks are **not** billed.

| Event | Price |
|---|---|
| Per chunk emitted (chunks-jsonl) | $0.0008 / chunk ($0.80 / 1,000) |
| Per page emitted (markdown) | $0.002 / page ($2.00 / 1,000) |

### Run locally

```bash
npm install
npm run build
apify run    # reads .actor/INPUT.json
```

### Roadmap (v1.2+)

Inline embeddings, direct vector-DB push (Pinecone/Qdrant/Weaviate/pgvector),
`missedGraceRuns` before tombstoning, Standby low-latency mode.

# Actor input Schema

## `startUrls` (type: `array`):

One or more seed URLs to crawl.

## `crawlScope` (type: `string`):

Which links to follow.

## `maxCrawlDepth` (type: `integer`):

Link hops from a start URL. 0 = only the start URLs.

## `includeGlobs` (type: `array`):

Only crawl URLs matching these glob patterns, e.g. https://x.com/docs/\*\*

## `excludeGlobs` (type: `array`):

Skip URLs matching these globs, e.g. **/tag/**, \*\*?print=1

## `maxPages` (type: `integer`):

Hard ceiling on pages crawled. Protects against runaway crawls and runaway bills.

## `renderJs` (type: `boolean`):

Use a headless browser for JS-heavy/SPA sites. Off = fast static HTTP crawl (recommended default).

## `outputFormat` (type: `string`):

RAG chunks = one record per chunk (recommended). Markdown = one record per page. LangChain = {page\_content, metadata} Document JSON per chunk. JSONL bulk = flat one-record-per-chunk for bulk import.

## `enableChunking` (type: `boolean`):

Split each page into token-sized retrieval units. Forced ON when output format is RAG chunks.

## `chunkSize` (type: `integer`):

Target token budget per chunk (sized for your embedding model).

## `chunkOverlap` (type: `integer`):

Token overlap between adjacent chunks (typically 10-20% of chunk size).

## `dedupNearDuplicates` (type: `boolean`):

Drop exact + near-duplicate chunks (shared boilerplate, syndicated content) via content hash + SimHash.

## `filterJunkChunks` (type: `boolean`):

Drop near-empty, nav-residue, and low-information chunks.

## `proxyConfiguration` (type: `object`):

Proxy configuration for the crawl. Apify Proxy recommended for large or rate-limited sites.

## `incremental` (type: `boolean`):

When ON, pages whose content is unchanged since the last run are crawled but NOT re-emitted (not billed). Requires the actor to persist state in a named key-value store (see 'State store name').

## `forceFullCrawl` (type: `boolean`):

Override: emit every page this run even if incremental is ON, then overwrite the saved state. Use after changing chunking settings or to rebuild a stale index.

## `stateStoreName` (type: `string`):

Named key-value store that persists per-URL content hashes between runs. Defaults to a deterministic name derived from the start URLs so scheduled re-runs of the same task share state automatically. Set explicitly to share state across tasks/schedules.

## `emitDeletions` (type: `boolean`):

When incremental is ON, emit a deletion record for each URL seen last run but missing this run, so downstream vector stores can purge stale vectors. Tombstones are NOT billed.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ],
  "crawlScope": "same-subdomain",
  "maxCrawlDepth": 3,
  "includeGlobs": [],
  "excludeGlobs": [],
  "maxPages": 200,
  "renderJs": false,
  "outputFormat": "chunks-jsonl",
  "enableChunking": true,
  "chunkSize": 512,
  "chunkOverlap": 75,
  "dedupNearDuplicates": true,
  "filterJunkChunks": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "incremental": false,
  "forceFullCrawl": false,
  "stateStoreName": "",
  "emitDeletions": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("commonelements/rag-ready-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("commonelements/rag-ready-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com"
    }
  ]
}' |
apify call commonelements/rag-ready-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=commonelements/rag-ready-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/EXhO6t07UyAPd88yK/builds/cqhSZTFeSPfHr8CBl/openapi.json