# Website to Markdown for LLMs & RAG — Content Extractor (`fetchbase/website-to-markdown`) Actor

Turn any URL or whole site into clean, LLM-ready Markdown, text, or JSON. Strips nav/ads/boilerplate; keeps headings, links, tables, code. Sitemap-aware concurrent crawl, URL filters, robots.txt respected, rich metadata. Charged only per page extracted — no startup fee.

- **URL**: https://apify.com/fetchbase/website-to-markdown.md
- **Developed by:** [Fetchbase](https://apify.com/fetchbase) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 page extracteds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website to Markdown for LLMs & RAG — Clean Content Extractor

Turn any URL — or a whole site — into **clean, LLM-ready Markdown**, plain text,
or structured JSON. Navigation, ads, cookie bars and boilerplate are stripped;
headings, links, **tables** and code blocks survive intact. Drop the output
straight into a prompt, a vector store, or a fine-tuning set.

**You pay only per page actually extracted.** No startup fee — most alternatives
charge one on every run. Failed pages cost $0.

### Quick start

```json
{
  "urls": ["https://en.wikipedia.org/wiki/Large_language_model"]
}
```

→ one dataset item with clean Markdown plus every metadata field you'd want:

```json
{
  "url": "https://en.wikipedia.org/wiki/Large_language_model",
  "title": "Large language model",
  "description": "A large language model (LLM) is a neural network trained on…",
  "author": "", "publishedDate": "", "siteName": "Wikipedia", "lang": "en",
  "wordCount": 13299,
  "markdown": "A **large language model** (**LLM**) is a [neural network](…) …",
  "links": [{ "href": "…", "text": "…" }]
}
```

### Crawl an entire site — the fast, polite way

```json
{
  "urls": ["https://docs.example.com"],
  "crawl": true,
  "useSitemap": true,
  "maxPages": 200,
  "maxConcurrency": 8,
  "includeUrlPatterns": ["*/docs/*"],
  "excludeUrlPatterns": ["*/changelog/*", "*.pdf"]
}
```

- **Sitemap discovery** seeds the crawl from the site's own `sitemap.xml`
  (including `Sitemap:` entries in robots.txt) — the fastest, most complete way
  to reach every page, no link-hopping required.
- **Include/exclude glob filters** keep the crawl (and your bill) exactly on the
  content you want.
- **Concurrent fetching** (up to 15 parallel) with per-run page caps and depth
  limits, so cost is always bounded and predictable.
- **robots.txt respected by default.**

### Features

| | |
|---|---|
| **Output formats** | Markdown (GFM: tables, fenced code), plain text, cleaned HTML, or JSON with everything |
| **Main-content extraction** | Readability isolates the article; or convert the full page body — your choice |
| **JS rendering** | Real Chromium for SPAs/React/Vue; switch it off for a much faster static fetch |
| **Metadata** | title, description, author, published date, site name, language, OG image, word count |
| **Crawling** | Same-domain BFS + sitemap seeding, URL glob filters, max pages/depth/concurrency |
| **Clean-up** | Remove any elements by CSS selector before extraction |
| **Reliability** | Optional Apify Proxy (incl. residential) for bot-protected sites |
| **Politeness** | robots.txt respected by default; bounded concurrency |

### Built for AI pipelines

- **RAG ingestion:** crawl your docs/blog → chunk the `markdown` field → embed.
  Re-run on a Schedule to keep the index fresh.
- **Agent tool:** call via the Apify MCP server or API and give your agent clean
  page content instead of raw HTML soup.
- **Dataset building:** sitemap + filters → every article on a site as uniform
  Markdown records with metadata.
- **Content migration:** legacy CMS → Markdown files in one run.

```bash
curl -X POST "https://api.apify.com/v2/acts/fetchbase~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://blog.example.com"], "crawl": true, "useSitemap": true, "maxPages": 50}'
```

### Pricing

A small fixed price **per successfully extracted page**, plus standard Apify
platform usage for the compute your run consumes. **No startup fee** (most
alternatives charge $0.0015–$0.09 per run before extracting anything). Failed
pages are never billed. Tip: turn off `jsRendering` on static sites — runs get
several times faster and cheaper.

### FAQ

**How is this different from a raw scraper?** You get *reading-quality* content:
boilerplate stripped, structure preserved, ready for a model — not a DOM dump.

**Does it respect site owners?** Yes — robots.txt is honored by default, sitemap
seeding uses the URL list the site itself publishes for crawlers, and
concurrency is bounded. Extraction is user-directed: you choose the URLs, and
you're responsible for complying with the target site's terms and applicable law.

**JavaScript-heavy site returns thin content?** Keep `jsRendering: true` (default)
and try `waitUntil: "networkidle"`.

**Blocked by anti-bot?** Enable Apify **residential proxy** in the input.

***

Missing a feature you need? **Open an issue** — this actor ships improvements
fast. If it saves you time, a ⭐ review helps others find it.

### More tools by Fetchbase

Part of a suite of fast, no-nonsense web utilities — all pay-per-result, charged only on success, no startup fee:

- [Website Screenshot API](https://apify.com/fetchbase/web-screenshot-pro) — full-page PNG / JPEG / WebP / PDF
- [Website to Markdown](https://apify.com/fetchbase/website-to-markdown) — clean Markdown for LLMs & RAG
- [PDF & DOCX to Markdown](https://apify.com/fetchbase/document-to-markdown) — documents → Markdown for RAG
- [SEO Audit + Core Web Vitals](https://apify.com/fetchbase/website-seo-audit) — scored on-page audit with fixes
- [Website Performance Audit](https://apify.com/fetchbase/website-performance-audit) — bulk Core Web Vitals & page speed
- [Tech Stack Detector](https://apify.com/fetchbase/tech-stack-detector) — CMS, frameworks, analytics, hosting
- [Domain, DNS & WHOIS Lookup](https://apify.com/fetchbase/domain-dns-intelligence) — records, registration, SSL
- [RSS Feed Reader](https://apify.com/fetchbase/rss-feed-reader) — RSS / Atom / JSON → normalized JSON
- [Job Postings API](https://apify.com/fetchbase/job-postings-scraper) — Greenhouse, Lever, Ashby & more

### Use with AI agents (MCP)

This Actor is callable by AI agents through the [Apify MCP server](https://mcp.apify.com/). Agents in Claude, Cursor, Windsurf, LangGraph, CrewAI and others can discover it via `search-actors` and run it as a tool — its inputs and outputs are fully described in the schema for reliable agent use.

***

#### Was this Actor useful?

If it saved you time, please consider leaving an honest review. Reviews are the main way
independent Actors get discovered on Apify — a single one makes a real difference, and
critical feedback is just as welcome as praise.

Something missing or broken? Open an issue instead and it will get fixed.

# Actor input Schema

## `urls` (type: `array`):

One or more URLs to convert. Each extracted page produces one dataset item and one billable event. You may omit the https:// prefix.

## `outputFormat` (type: `string`):

markdown (best for LLMs/RAG), text (plain), html (cleaned main-content HTML), or json (everything: markdown + text + metadata + links).

## `onlyMainContent` (type: `boolean`):

Use readability to keep only the main article and strip navigation, sidebars, ads, footers and boilerplate. Turn off to convert the whole page body.

## `jsRendering` (type: `boolean`):

Load the page in a real browser so JS-rendered content (SPAs, React/Vue sites) is captured. Turn off for a much faster fetch on static/server-rendered pages.

## `includeLinks` (type: `boolean`):

Include the list of links found on each page in the output (also used as crawl targets when crawling is on).

## `crawl` (type: `boolean`):

Beyond the URLs you provide, follow links to discover and convert more pages, up to the limits below.

## `maxPages` (type: `integer`):

Hard cap on the total number of pages converted (across all seeds). Protects you from runaway crawls and cost.

## `maxDepth` (type: `integer`):

How many link-hops away from your seed URLs to follow. 0 = only the URLs you gave. 1 = them plus pages they link to, etc.

## `sameDomainOnly` (type: `boolean`):

Only follow links that stay on the seed page's domain.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed by the site's robots.txt. Recommended — polite crawling.

## `useSitemap` (type: `boolean`):

When crawling, also seed from the site's own sitemap.xml (and any Sitemap: entries in robots.txt) — the fastest, most complete way to reach every page. Subject to the URL filters below.

## `includeUrlPatterns` (type: `array`):

Only crawl/keep discovered URLs matching these glob patterns (\* = wildcard), e.g. https://site.com/blog/\* or */docs/*. Empty = allow all. Does not filter the URLs you explicitly provide.

## `excludeUrlPatterns` (type: `array`):

Skip discovered URLs matching these glob patterns (\* = wildcard), e.g. */tag/*, */login*, \*.pdf.

## `maxConcurrency` (type: `integer`):

How many pages to fetch in parallel. Higher = faster crawls, but uses more memory — be considerate of the target site.

## `removeSelectors` (type: `array`):

Extra CSS selectors to delete before extraction, e.g. '.related-articles' or '#comments'.

## `waitUntil` (type: `string`):

When to consider a page loaded (only applies when Render JavaScript is on).

## `timeoutMs` (type: `integer`):

Max time to wait for a page before giving up on it.

## `proxyConfiguration` (type: `object`):

Off by default (fast, works on most sites). Enable Apify Proxy — including residential — for bot-protected sites that block datacenter IPs.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://docs.apify.com"
  ],
  "outputFormat": "markdown",
  "onlyMainContent": true,
  "jsRendering": true,
  "includeLinks": true,
  "crawl": false,
  "maxPages": 10,
  "maxDepth": 1,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "useSitemap": false,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "maxConcurrency": 5,
  "removeSelectors": [],
  "waitUntil": "load",
  "timeoutMs": 45000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Large_language_model"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fetchbase/website-to-markdown").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://en.wikipedia.org/wiki/Large_language_model"] }

# Run the Actor and wait for it to finish
run = client.actor("fetchbase/website-to-markdown").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Large_language_model"
  ]
}' |
apify call fetchbase/website-to-markdown --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=fetchbase/website-to-markdown",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/drF5UMUrd2yJn2qVf/builds/7VSZzVqvEEecmSSVu/openapi.json