# URL to Clean Markdown for LLMs & RAG (`eltociear/url-to-markdown-for-llms`) Actor

Turn a list of URLs into clean Markdown for RAG and LLM context. Strips nav/ads/boilerplate; returns title, byline, date and word count per page. Pay only per page successfully extracted.

- **URL**: https://apify.com/eltociear/url-to-markdown-for-llms.md
- **Developed by:** [Ikko Eltociear Ashimine](https://apify.com/eltociear) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$3.00 / 1,000 page extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## URL to Clean Markdown for LLMs & RAG

**Turn a list of URLs into clean Markdown, ready for RAG ingestion and LLM context.** Give it
the pages you already have; it fetches each one, strips the nav, ads and boilerplate, and
returns the main content as Markdown — plus title, byline, date and word count.

This is the "give me the readable text of this page" job, in bulk. Not a crawler, not a search
scraper: you supply the exact URLs, and you pay only per page it actually extracts.

### Why this and not a crawler

| | This actor | Site crawlers / search scrapers |
| --- | --- | --- |
| Input | The exact URLs you already have | A start URL or a search query |
| Work | Fetch + clean-extract each page | Discover, crawl, paginate, render |
| Cost | Per page **extracted** ($0.003) | Per page crawled, browser time, proxies |
| Best for | RAG ingestion of a known URL set | Mapping an unknown site |

If you already know which pages you want in your vector store, crawling them is overkill. This
does exactly the extraction step, cheaply and in parallel.

### Input

| field | meaning |
| --- | --- |
| **URLs** | The pages to convert. Paste a list, or wire this to another actor's output. |
| **Keep hyperlinks** | Keep in-text links in the Markdown. Turn off for the cleanest embedding text. |
| **Only output pages that extracted** | Off also writes a row for each failed URL with its error, so nothing is silently dropped. |
| **Maximum URLs** | Hard cap on how many are processed. |
| **Per-URL timeout / Parallel fetches** | Tune speed vs. politeness on slow sites. |

### Output

One dataset item per URL:

```jsonc
{
  "url": "https://example.com/article",
  "final_url": "https://example.com/article",
  "http_status": 200,
  "ok": true,
  "title": "Article title",
  "byline": "Jane Doe",
  "date": "2026-05-01",
  "markdown": "# Article title\n\nMain content…",
  "word_count": 1234,
  "error": null
}
```

A URL that fails to fetch or has no extractable main content gets `ok: false` with the reason
in `error` — and is **not charged**.

### Pricing

Pay-per-event: **$0.003 per page successfully extracted**. Failed fetches and pages with no
main content cost nothing. Duplicate URLs in your input are de-duplicated, so the same page is
never charged twice in a run.

### Use cases

- **RAG ingestion** — convert a known list of documentation, blog or news URLs into clean
  Markdown chunks for a vector store.
- **LLM context building** — feed an agent the readable text of specific pages instead of raw
  HTML, cutting token cost and prompt noise.
- **Dataset construction** — bulk HTML-to-Markdown conversion for fine-tuning or evaluation
  corpora, with per-page metadata.
- **Content monitoring** — re-extract the same URL set on a schedule and diff the Markdown.

### FAQ

**Does it crawl or follow links?**
No. It converts exactly the URLs you supply. That is the point: no crawl budget, no surprise
pages, no charge for pages you did not ask for.

**How is this different from a local HTML-to-Markdown library?**
A library converts HTML you already fetched. This fetches at scale in parallel, applies
main-content extraction (Readability-style boilerplate removal), and returns structured
metadata — with failures reported instead of silently producing junk text.

**What happens to pages that need JavaScript?**
They return `ok: false` with the reason, and are not charged. No browser is rendered, which is
what keeps the price at a third of a cent per page.

**Can I chain it to another Actor?**
Yes — wire any Actor's dataset of URLs into the **URLs** input and this becomes the extraction
stage of your pipeline.

**Is robots.txt respected?**
Fetches are plain HTTP GETs of URLs you supply; you remain responsible for having the right to
retrieve them.

### How it works

Extraction is [trafilatura](https://trafilatura.readthedocs.io/) — a well-tested content
extractor — the same engine behind our live `clean-read` micro-service. It reads the fetched
HTML and returns the main article; it does not execute page scripts or render a browser, which
is what keeps it fast and cheap. Pages that hard-require JavaScript to render their content, or
that block datacenter traffic, will come back as `ok: false` rather than as partial text.

# Actor input Schema

## `urls` (type: `array`):

The pages to convert to clean Markdown. Paste a list, or wire this to another actor's output.

## `includeLinks` (type: `boolean`):

Keep in-text links in the Markdown. Turn off for the cleanest text for embeddings.

## `onlySuccessful` (type: `boolean`):

Off also writes a row for each failed URL (with the error), so nothing is silently dropped.

## `maxUrls` (type: `integer`):

Hard cap on how many URLs are processed. You pay only per page successfully extracted.

## `timeoutSecs` (type: `integer`):

How long to wait for a single page before recording it as failed.

## `concurrency` (type: `integer`):

Higher finishes sooner but is heavier on slow sites.

## Actor input object example

```json
{
  "urls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://en.wikipedia.org/wiki/Large_language_model"
  ],
  "includeLinks": true,
  "onlySuccessful": false,
  "maxUrls": 1000,
  "timeoutSecs": 20,
  "concurrency": 8
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
        "https://en.wikipedia.org/wiki/Large_language_model"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("eltociear/url-to-markdown-for-llms").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
        "https://en.wikipedia.org/wiki/Large_language_model",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("eltociear/url-to-markdown-for-llms").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "https://en.wikipedia.org/wiki/Large_language_model"
  ]
}' |
apify call eltociear/url-to-markdown-for-llms --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=eltociear/url-to-markdown-for-llms",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/la9h8vTtMG79gpCTP/builds/rTxrEt0efp8qrbSGC/openapi.json