# Website Content Crawler — Text, Markdown & HTML for AI/LLM (`hichemdev/website-content-crawler`) Actor

Crawl any website and extract clean text, Markdown, and HTML from every page — ready for LLM, RAG, and AI ingestion.

- **URL**: https://apify.com/hichemdev/website-content-crawler.md
- **Developed by:** [Hichem Ben Moussa](https://apify.com/hichemdev) (community)
- **Categories:** AI, Developer tools
- **Stats:** 4 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 pages

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🕸️ Website Content Crawler — Text, Markdown & HTML for AI/LLM

**Crawl any website and get clean, structured content from every page — as Markdown, plain text, and HTML. Built to feed LLMs, RAG pipelines, chatbots, and search indexes.**

Point it at one or more URLs, set how deep to crawl, and get back the *readable* content of each page with the navigation, ads, cookie banners, and boilerplate stripped out. No API keys, no glue code.

***

### ✨ What you get for every page

| Field | Description |
|---|---|
| **url / loadedUrl** | The page address (after redirects) |
| **title** | Page title |
| **description** | Meta / OG description |
| **markdown** | Clean Markdown of the main content — perfect for LLM ingestion |
| **text** | Clean plain-text version |
| **html** | Cleaned HTML of the main content block (optional) |
| **canonicalUrl / languageCode** | Page metadata |
| **crawl** | Depth + referrer URL |

***

### 🎯 Who it's for

- **AI / LLM builders** — turn a documentation site or knowledge base into Markdown for **RAG** and fine-tuning.
- **Chatbot & support teams** — index your help center so an assistant can answer from it.
- **SEO & content teams** — audit every page's real text content at scale.
- **Researchers & data teams** — snapshot a whole site into structured records.

***

### 🚀 How to use it

1. Add one or more **Start URLs**.
2. Choose a **crawler type**:
   - **Raw HTTP (Cheerio)** — fast and cheap, ideal for static or server-rendered sites.
   - **Headless browser (Playwright)** — renders JavaScript, for single-page apps and JS-heavy sites.
3. Set **max pages** and **crawl depth**.
4. (Optional) Narrow the crawl with **include / exclude URL patterns**.
5. Click **Start**, then export as **JSON, CSV, Excel, or Markdown**.

***

### 📥 Example input

```json
{
  "startUrls": [{ "url": "https://docs.apify.com" }],
  "crawlerType": "cheerio",
  "maxCrawlPages": 50,
  "maxCrawlDepth": 2,
  "stayWithinDomain": true,
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false
}
```

Narrow to just a blog:

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "includeGlobs": ["https://example.com/blog/**"],
  "excludeGlobs": ["**/tag/**", "**/*.pdf"]
}
```

***

### 📤 Example output

One record per page:

```json
{
  "url": "https://docs.apify.com/platform",
  "title": "Apify platform | Apify Documentation",
  "description": "Learn how the Apify platform works.",
  "canonicalUrl": "https://docs.apify.com/platform",
  "languageCode": "en",
  "markdown": "# Apify platform\n\nThe Apify platform helps you...",
  "text": "Apify platform\nThe Apify platform helps you...",
  "textLength": 1843,
  "crawl": { "depth": 1, "referrerUrl": "https://docs.apify.com", "loadedUrl": "https://docs.apify.com/platform" }
}
```

Export in one click to **JSON, CSV, Excel, or Markdown**, or pull results via the **Apify API**.

***

### 🔌 Integrations & scheduling

- **Schedule** the crawler to keep your knowledge base / index fresh.
- Pipe the Markdown straight into a **vector database** (Pinecone, Weaviate, pgvector) for RAG.
- Combine with **LangChain / LlamaIndex** — the Markdown output drops right into a document loader.

***

### ❓ FAQ

**Cheerio or Playwright — which do I pick?**
Start with **Cheerio** (fast, cheap). If pages come back empty or missing content, the site renders with JavaScript — switch to **Playwright**.

**How does it decide what's "main content"?**
It removes nav, header, footer, sidebars, ads, and cookie banners, then extracts the primary `<main>` / `<article>` region (falling back to the body). You get the article, not the chrome.

**Will it stay on my site?**
By default it only follows links on the **same domain**. Use include/exclude globs for finer control.

**Do I need a proxy?**
Apify Proxy is on by default to reduce blocking. For open public sites you can turn it off to save cost.

**Is crawling legal?**
It fetches publicly available pages. You're responsible for respecting each site's terms and robots policies and for how you use the content.

***

### 🗺️ Roadmap

- `robots.txt` and sitemap.xml support
- Readability-grade main-content scoring
- Per-page screenshots (Playwright mode)
- Automatic language detection & chunking for RAG

***

*Built and maintained by [hichemdev](https://apify.com/hichemdev). Found a bug or want a feature? Open an issue on the Actor's **Issues** tab.*

# Actor input Schema

## `startUrls` (type: `array`):

One or more URLs to start crawling from. The crawler follows links from these pages up to the crawl depth.

## `crawlerType` (type: `string`):

How to fetch pages. "Raw HTTP (Cheerio)" is fast and cheap — best for static or server-rendered sites. "Headless browser (Playwright)" renders JavaScript — use it for single-page apps and JS-heavy sites.

## `maxCrawlPages` (type: `integer`):

Maximum number of pages to crawl and store across all start URLs.

## `maxCrawlDepth` (type: `integer`):

How many links deep to follow from the start URLs. 0 = only the start URLs; 1 = start URLs plus the pages they link to; and so on.

## `stayWithinDomain` (type: `boolean`):

Only follow links that stay on the same domain as the start URL.

## `includeGlobs` (type: `array`):

Optional glob patterns. If set, only URLs matching one of them are crawled, e.g. "https://example.com/blog/\*\*".

## `excludeGlobs` (type: `array`):

Optional glob patterns for URLs to skip, e.g. "**/tag/**" or "\*\*/\*.pdf".

## `saveMarkdown` (type: `boolean`):

Include clean Markdown of the main content — ideal for LLM / RAG ingestion.

## `saveText` (type: `boolean`):

Include a clean plain-text version of the main content.

## `saveHtml` (type: `boolean`):

Include the cleaned HTML of the main content block.

## `proxyConfiguration` (type: `object`):

Proxy settings. Apify Proxy (automatic) helps avoid blocking. You can disable it for public sites to save cost.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "crawlerType": "cheerio",
  "maxCrawlPages": 10,
  "maxCrawlDepth": 1,
  "stayWithinDomain": true,
  "includeGlobs": [],
  "excludeGlobs": [],
  "saveMarkdown": true,
  "saveText": true,
  "saveHtml": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "crawlerType": "cheerio",
    "maxCrawlPages": 10,
    "maxCrawlDepth": 1,
    "includeGlobs": [],
    "excludeGlobs": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("hichemdev/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "crawlerType": "cheerio",
    "maxCrawlPages": 10,
    "maxCrawlDepth": 1,
    "includeGlobs": [],
    "excludeGlobs": [],
}

# Run the Actor and wait for it to finish
run = client.actor("hichemdev/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "crawlerType": "cheerio",
  "maxCrawlPages": 10,
  "maxCrawlDepth": 1,
  "includeGlobs": [],
  "excludeGlobs": []
}' |
apify call hichemdev/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=hichemdev/website-content-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/EFfBybdQVCfqrgTKv/builds/hh1ZFOuFlx8SmMDmc/openapi.json