# Website to Markdown for RAG & LLMs (`hereditary_model/website-to-markdown`) Actor

Crawls a website and converts every page into clean, LLM-ready Markdown for RAG pipelines, vector databases, and AI agents. Removes nav, ads, and boilerplate. Predictable pricing: $0.004 per page converted.

- **URL**: https://apify.com/hereditary\_model/website-to-markdown.md
- **Developed by:** [Aaron Marxsen](https://apify.com/hereditary_model) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$4.00 / 1,000 page converteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website to Markdown Converter for RAG and LLMs

Convert any website into clean, structured Markdown ready for RAG pipelines, vector databases, LLM context, and AI agents. This actor crawls a site page by page, strips away navigation, footers, cookie banners, and scripts, and returns the real content of every page as Markdown. It is the fastest way to turn a website into LLM ready text without writing a scraper or cleaning HTML by hand.

Built for AI engineers, agent developers, and anyone feeding web content to a large language model.

### Features

- Convert a website to Markdown for RAG and vector database ingestion
- Crawl an entire site or a single section with URL glob filters
- Strip navigation, headers, footers, sidebars, and cookie banners automatically
- Prefer main and article content when a page provides it
- GitHub style Markdown with ATX headings and fenced code blocks
- Skip near empty pages so your dataset stays clean
- Pay per page converted, no subscription

### Use cases

- Build a RAG knowledge base from your product documentation
- Keep an LLM assistant in sync with a changing website on a schedule
- Create clean text corpora for evaluation and fine tuning
- Give an AI agent a tool to read and summarize any website
- Migrate CMS content to Markdown

### Input

| Field | Type | Description |
| --- | --- | --- |
| `startUrls` | array (required) | Where the crawl starts (stays on the same domain) |
| `maxPages` | integer (default 50) | Maximum pages to crawl and convert |
| `includeUrlGlobs` | array (optional) | Only crawl URLs matching these globs |
| `minTextLength` | integer (default 100) | Skip pages with less extracted text than this |

```json
{ "startUrls": [{ "url": "https://docs.apify.com/platform" }], "maxPages": 100 }
```

### Output

One dataset item per page:

```json
{
  "url": "https://docs.apify.com/platform/actors",
  "title": "Actors | Platform | Apify Documentation",
  "description": "Learn how Apify Actors work ...",
  "markdown": "# Actors\n\nAn Actor is a serverless program ...",
  "charCount": 5321,
  "crawledAt": "2026-07-07T12:00:00.000Z"
}
```

### Pricing: worked example

Pay per event. You are charged `page-converted` at $0.004 for each page that is converted and returned. Skipped near empty pages cost nothing.

Converting 100 pages triggers 100 page events at $0.004 each. Total: $0.40.

### How to use

1. Open the actor and click Try for free.
2. Paste one or more start URLs (your docs site, blog, or knowledge base).
3. Set `maxPages` to bound cost, and optionally `includeUrlGlobs` to target a section.
4. Click Start.
5. Watch the run convert each page to Markdown in real time.
6. Download the results as JSON from the Storage tab, or stream them via the API.
7. Feed the Markdown into your RAG loader, vector database, or LLM context.

### FAQ

**How do I convert a website to Markdown for a RAG pipeline?**
Give this actor your start URLs and it crawls the site, cleans each page, and returns Markdown you can load straight into LangChain, LlamaIndex, or a vector database.

**Does it remove navigation and ads?**
Yes. It removes nav, headers, footers, sidebars, cookie banners, and scripts, and prefers main or article content.

**Can it convert a whole documentation site?**
Yes. Raise `maxPages`. Cost scales linearly and predictably with pages converted.

**What format is the output?**
GitHub style Markdown (ATX headings, fenced code blocks) plus the page title, description, and character count.

**Can an AI agent call this automatically?**
Yes. It supports agentic payments, so an agent can discover, run, and pay for it without a human in the loop.

### API and agents

Run it from the [Apify API](https://docs.apify.com/api/v2), or let an AI agent call it. This actor supports agentic payments via x402 and Skyfire (pay per event, limited permissions, no Standby), so agents can buy exactly the pages they need.

```bash
curl -X POST "https://api.apify.com/v2/acts/hereditary_model~website-to-markdown/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "startUrls": [{ "url": "https://example.com" }], "maxPages": 50 }'
```

### Related actors

Part of a connected toolkit. Chain these in your agent workflows:

- [Broken Link Checker](https://apify.com/hereditary_model/broken-link-checker)
- [Local SEO Audit](https://apify.com/hereditary_model/local-seo-audit)
- [Contact Details Extractor](https://apify.com/hereditary_model/contact-details-extractor)
- [Domain Intelligence API](https://apify.com/hereditary_model/domain-intelligence)
- [ATS Job Postings Scraper](https://apify.com/hereditary_model/ats-job-postings-scraper)
- [NPI Healthcare Provider Directory](https://apify.com/hereditary_model/healthcare-provider-directory)
- [Bandcamp Discography Scraper](https://apify.com/hereditary_model/bandcamp-discography-scraper)

# Actor input Schema

## `startUrls` (type: `array`):

Website URL(s) where the crawl starts. The crawler stays on the same domain.

## `maxPages` (type: `integer`):

Maximum number of pages to crawl and convert. Each converted page is one billable event.

## `includeUrlGlobs` (type: `array`):

Only crawl URLs matching these glob patterns (e.g. https://example.com/docs/\*\*). Leave empty to crawl the whole domain.

## `minTextLength` (type: `integer`):

Skip pages whose extracted text is shorter than this many characters (filters out empty or navigation-only pages).

## `proxyConfiguration` (type: `object`):

Proxy settings used for crawling. Defaults to no proxy.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "maxPages": 50,
  "includeUrlGlobs": [],
  "minTextLength": 100
}
```

# Actor output Schema

## `results` (type: `string`):

All crawled pages converted to clean Markdown, one dataset item per page.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hereditary_model/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://docs.apify.com/platform" }] }

# Run the Actor and wait for it to finish
run = client.actor("hereditary_model/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ]
}' |
apify call hereditary_model/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=hereditary_model/website-to-markdown",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/vbJmzelPujqEzrqhV/builds/nSMoz1bJgkvhSqwGS/openapi.json