# API Documentation Crawler for RAG & AI Agents (`ahampton83/api-docs-rag-crawler`) Actor

Crawl API documentation sites and extract structured, LLM-ready content. Use via Apify Console/API or connect as an MCP server for Claude, Cursor, and other AI agents.

- **URL**: https://apify.com/ahampton83/api-docs-rag-crawler.md
- **Developed by:** [Aaron Hampton](https://apify.com/ahampton83) (community)
- **Categories:** AI, Developer tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 page crawleds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## API Documentation Crawler for RAG & AI Agents

Crawl API documentation sites and extract clean, structured content for AI models, RAG pipelines, and vector databases. Use it as a normal Apify Actor or connect it to Claude, Cursor, and other MCP-compatible clients.

### What it does

This Actor crawls API documentation websites and returns structured records for each page:

- **Title & description**
- **Heading hierarchy** (with anchor IDs)
- **Markdown-formatted content**
- **Plain text** (for simple embedding)
- **Detected API endpoints** (method + path, when visible on the page)
- **Code examples**
- **Outbound links**
- **Metadata** (canonical URL, OpenGraph tags)

You can also switch to **chunking mode** to get pre-split text snippets ready for vector DB ingestion.

### Why use this?

- **Feed RAG pipelines** with clean, structured API docs.
- **Power AI agents** that need to answer questions about an API.
- **Avoid brittle copy-paste** from docs into LLM context windows.
- **MCP-native:** call it as a tool from Claude, Cursor, ChatGPT, and other MCP clients.

### How to use

#### Apify Console / API

Provide one or more start URLs and optional limits:

```json
{
  "startUrls": ["https://docs.apify.com/api/v2"],
  "maxPages": 20,
  "maxDepth": 2,
  "outputFormat": "full"
}
```

#### MCP mode

Connect the Actor as an MCP server in Cursor/Claude Desktop:

```json
{
  "mcpServers": {
    "api-docs-crawler": {
      "url": "https://AHampton83--api-docs-rag-crawler.apify.actor/mcp"
    }
  }
}
```

Available tools:

- `crawl_api_docs(startUrl, maxPages?, maxDepth?)` — crawl docs and return structured page summaries.
- `get_page_chunks(startUrl, maxPages?, chunkSize?, chunkOverlap?)` — return pre-chunked text for vector DBs.

### Output schema

#### Full format (`outputFormat: "full"`)

```json
{
  "url": "https://docs.example.com/api/v1/users",
  "title": "Users API",
  "description": "Manage users via the REST API",
  "headings": [{ "level": 1, "text": "Users API", "id": "users-api" }],
  "markdown": "# Users API\n\n## Get all users\n\n`GET /api/v1/users`",
  "text": "Users API Get all users GET /api/v1/users ...",
  "endpoints": [{ "method": "GET", "path": "/api/v1/users", "confidence": "high" }],
  "codeBlocks": [{ "language": "bash", "content": "curl /api/v1/users" }],
  "links": ["https://docs.example.com/api/v1/orders"],
  "metadata": { "canonicalUrl": "...", "ogTitle": "...", "ogDescription": "..." },
  "crawledAt": "2026-07-04T00:00:00.000Z"
}
```

#### Chunks format (`outputFormat: "chunks"`)

```json
{
  "chunkIndex": 0,
  "url": "https://docs.example.com/api/v1/users",
  "title": "Users API",
  "text": "...",
  "headings": [{ "level": 2, "text": "Get all users" }],
  "tokenEstimate": 250
}
```

### Pricing

This Actor uses **Pay Per Event (PPE)** billing:

| Event | Price | Notes |
|---|---|---|
| `apify-actor-start` | $0.00005 | Platform-provided synthetic start event |
| `page-crawled` | $0.005 | First 10 pages per run/tool-call are free |
| `chunk-generated` | $0.001 | First 10 chunks per run/tool-call are free |
| `tool-call` | $0.01 | Per MCP tool call |

Volume discounts apply based on the user's Apify plan tier.

### Publish & monetize

To activate monetization and publish this Actor on Apify Store:

1. Open the Actor in Apify Console: https://console.apify.com/actors/Dtlhc1snKkwxoikZh
2. Go to the **Publication** tab and click **Set up monetization**.
3. In the pricing step, keep `apify-actor-start` and add these custom events with the exact slugs used in the code:
   - `page-crawled` — $0.005
   - `chunk-generated` — $0.001
   - `tool-call` — $0.01
4. **Do not enable `apify-default-dataset-item`**: the Actor already charges manually per page/chunk, so the synthetic dataset-item event would double-charge users.
5. Complete the **Primary event** and **Review** steps, then submit for Store approval.

Payouts require a verified developer account with tax and payment details in Apify Console.

### Limits

- `maxPages`: up to 500 per normal run, 100 per MCP tool call.
- Crawls only same-origin links by default (configurable via `includeUrlGlobs`).
- Static HTML only — Cheerio crawler does not execute JavaScript.

### Use cases

- Build a custom API assistant for your product.
- Keep vector store embeddings in sync with live docs.
- Generate SDK examples from existing documentation.
- Competitive analysis of public APIs.

### Support

Open an issue or reach out via the Apify Store page.

# Actor input Schema

## `startUrls` (type: `array`):

One or more API documentation URLs where crawling should begin.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl per run.

## `maxDepth` (type: `integer`):

Maximum link depth to follow from start URLs.

## `outputFormat` (type: `string`):

Return full page records or pre-chunked text for vector DBs.

## `chunkSize` (type: `integer`):

Approximate words per chunk when outputFormat is 'chunks'.

## `chunkOverlap` (type: `integer`):

Words of overlap between chunks.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs matching these glob patterns (e.g. **/api/**). Leave empty to crawl all same-origin links.

## `excludeUrlGlobs` (type: `array`):

Skip URLs matching these glob patterns (e.g. **/blog/**, **/changelog/**).

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/api/v2"
    }
  ],
  "maxPages": 50,
  "maxDepth": 3,
  "outputFormat": "full",
  "chunkSize": 1000,
  "chunkOverlap": 100,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": []
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/api/v2"
        }
    ],
    "includeUrlGlobs": [],
    "excludeUrlGlobs": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahampton83/api-docs-rag-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/api/v2" }],
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
}

# Run the Actor and wait for it to finish
run = client.actor("ahampton83/api-docs-rag-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/api/v2"
    }
  ],
  "includeUrlGlobs": [],
  "excludeUrlGlobs": []
}' |
apify call ahampton83/api-docs-rag-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=ahampton83/api-docs-rag-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Dtlhc1snKkwxoikZh/builds/1SC5VDUgJOCHcgSJO/openapi.json