# 🌐 Website Content Crawler (`citrine_venus/website-content-crawler`) Actor

Crawl any website into clean text, Markdown & rich metadata — AI-ready output for RAG, LLMs & vector databases. Streams to your dataset in real time.

- **URL**: https://apify.com/citrine\_venus/website-content-crawler.md
- **Developed by:** [Data Minds](https://apify.com/citrine_venus) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 row\_results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 🌐 Website Content Crawler — Web Scraping & AI Content Extraction Actor

**Website Content Crawler** is a powerful [Apify Actor](https://docs.apify.com/platform/actors) for **website crawling**, **web scraping**, **content extraction**, and **AI data preparation**. It crawls any website starting from your seed URLs, follows internal links, extracts clean **plain text**, **Markdown**, and rich **metadata** (title, description, canonical URL, Open Graph, JSON-LD, language, headers), and writes structured page data to an Apify **Dataset** in real time — perfect for **RAG pipelines**, **LLM knowledge bases**, **vector database ingestion**, **semantic search**, and **SEO content audits**.

> 💡 **Need a custom version, private integration, or a tailored crawler?** Email **<hello.dataminds@gmail.com>**.

It is a focused alternative to the official [apify/website-content-crawler](https://apify.com/apify/website-content-crawler): a tighter feature surface, predictable cost, and AI-first output (text + Markdown + metadata) ready for **LangChain**, **LlamaIndex**, **OpenAI**, **Pinecone**, **Qdrant**, and **Weaviate** pipelines.

***

### 📑 Table of contents

- [What is Website Content Crawler?](#-what-is-website-content-crawler)
- [Main features](#-main-features)
- [Who is this Actor for?](#-who-is-this-actor-for)
- [Built for generative AI, LLMs, and RAG](#-built-for-generative-ai-llms-and-rag)
- [What the crawler does](#%EF%B8%8F-what-the-crawler-does)
- [Crawling configuration](#-crawling-configuration)
- [HTML cleaning & content extraction](#-html-cleaning--content-extraction)
- [Output format (Dataset)](#-output-format-dataset)
- [Example output (JSON)](#-example-output-json)
- [Quick start](#-quick-start)
- [Input parameters reference](#%EF%B8%8F-input-parameters-reference)
- [Integrations: LangChain, LlamaIndex, OpenAI, Pinecone](#-integrations-langchain-llamaindex-openai-pinecone)
- [Pricing & how to control cost](#-pricing--how-to-control-cost)
- [Frequently asked questions (FAQ)](#-frequently-asked-questions-faq)
- [Troubleshooting](#%EF%B8%8F-troubleshooting)
- [Help, support & custom builds](#-help-support--custom-builds)
- [Is web scraping legal?](#%EF%B8%8F-is-web-scraping-legal)
- [SEO keywords targeted](#-seo-keywords-targeted)

***

### 🌐 What is Website Content Crawler?

**Website Content Crawler** is an Apify-hosted **web scraping API** and **website crawler** that turns any public website into clean, structured, AI-ready data. Unlike generic HTTP scrapers, this Actor:

- **Crawls** an entire site from one or more start URLs, staying on the same registrable domain.
- **Renders JavaScript** automatically when a page needs it, so single-page apps still yield real content.
- **Cleans HTML** by stripping navigation, footers, scripts, cookie banners, modals, and other noise.
- **Extracts content** as readable plain text and optional **Markdown** for token-efficient LLM ingestion.
- **Captures metadata** (title, description, canonical, Open Graph, JSON-LD, headers, language).
- **Saves results** as structured rows in an Apify **Dataset**, exportable to JSON, CSV, Excel, or via the [Apify API](https://docs.apify.com/api/v2).

If you need a **website-to-dataset** pipeline for **AI agents**, **chatbot grounding**, **GPT knowledge files**, **RAG retrieval**, or **content analytics**, this is the Actor.

***

### ✨ Main features

- 🌱 **Multi-URL crawl seeds** — start from one URL or hundreds; the crawler stays on the same domain by default.
- 🎯 **Glob-based URL filtering** — `includeUrlGlobs` / `excludeUrlGlobs` for surgical scope (`/blog/*`, `/docs/**`, skip `/admin/**`). Path-only patterns work as written; `*` stops at `/`, `**` crosses folders.
- 🗺️ **Discovery sources** — seed the queue from `sitemap.xml` (`useSitemaps`, follows sitemap indexes) and `llms.txt` (`useLlmsTxt`) to reach pages that aren't linked, and honour `robots.txt` (`respectRobotsTxtFile`).
- 🛡️ **Apify Proxy support** — datacenter or residential groups, country targeting, or your own proxy URLs, with automatic fallback.
- 🍪 **Initial cookies** — `initialCookies` sent on every request and loaded into the browser for session or light-login access.
- 🖱️ **Rendered-page interaction** — `maxScrollHeightPixels` auto-scroll for lazy/infinite-scroll content, `clickElementsCssSelector` to expand accordions / "load more", and `blockMedia` to skip images/fonts/video for faster, cheaper renders.
- 🚫 **Ad/tracker blocking** — `blockAds` aborts known ad and tracker domains on rendered pages, on top of `blockMedia`.
- 📱 **Mobile emulation** — `mobileView` renders with a real mobile device profile (viewport, UA, touch) instead of desktop.
- 🕵️ **PII redaction (best-effort)** — `redactPii` masks emails, phone numbers, card-like numbers, and SSNs in `text`/`markdown`.
- 📐 **Screenshot mode** — `screenshotFullPage` (default on) toggles full-page vs viewport-only screenshots.
- 📕 **PDF text extraction** — `extractPdfText` opens discovered PDF links and extracts real text into `text`, instead of only listing them as file links.
- 📂 **Depth & page limits** — `maxCrawlDepth`, `maxCrawlPages`, `maxResults` to keep cost and runtime predictable.
- ⚡ **Async concurrency** — tune `maxConcurrency` from 1 to 50 parallel requests.
- 🖥️ **Three rendering modes** — `auto` (render only when a page needs it), `always`, or `never` for the cheapest possible run.
- 🪄 **Smart retry** — pages that come back incomplete are automatically retried for a fuller result.
- 🧼 **HTML noise removal** — strip nav, footer, modals, ads, and cookie banners via CSS selectors before extraction.
- 📝 **Markdown export** — token-efficient Markdown alongside plain text, ideal for **LLM context windows**.
- 🏷️ **Rich metadata extraction** — title, description, canonical URL, Open Graph, Twitter Card, JSON-LD, author, keywords, publish/modified dates, language code.
- 🧲 **Keep-only extraction** — `keepElementsCssSelector` to keep just an article body (`article, main`) and discard everything else before extraction.
- 🧾 **Skipped-URL audit** — `storeSkippedUrls` writes every discovered-but-not-crawled URL (with a reason) to `skipped-urls.json`.
- 📸 **HTML snapshots & full-page screenshots** stored in the run's key-value store, surfaced as **direct clickable URLs** on every row (original HTML, noise-removed HTML, readable HTML, PNG).
- ⏱️ **Politeness controls** — `rateLimitDelaySecs`, retry counts, request timeouts, and dynamic-content waits.
- 🎯 **Wait-for-selector** — block extraction until a CSS selector renders (great for React/Vue/Angular SPAs).
- 🛡️ **Custom HTTP headers** — User-Agent, sec-ch-ua, DNT, plus arbitrary cookies and auth headers via JSON.
- 🌐 **Canonical & fragment handling** — dedupe by canonical link or treat `#anchors` as distinct pages.
- 📦 **Real-time Dataset writes** — every page lands in your Dataset the moment it is ready, not at the end of the run.
- 🔧 **Verbose diagnostics** — link counts, status codes, content lengths, and snapshot links on every row.

***

### 👥 Who is this Actor for?

This Apify Actor is built for:

- 🤖 **AI engineers** building **RAG**, **retrieval-augmented generation**, or **LLM grounding** pipelines.
- 🧠 **Data scientists** assembling **training corpora**, **fine-tuning datasets**, or **embedding indexes**.
- 🛠️ **Backend developers** integrating **website-to-database** ETL flows or **knowledge sync** jobs.
- 📈 **SEO professionals** auditing **titles**, **meta descriptions**, **canonical URLs**, **Open Graph**, **schema.org** markup, and **internal linking** at scale.
- 📚 **Documentation teams** building **searchable docs portals** or **AI-powered help bots**.
- 🛒 **Content strategists & competitive analysts** monitoring competitor blogs, landing pages, and changelogs.
- 🔬 **Researchers** collecting **public web text data** for NLP, classification, or sentiment models.

If you've ever needed to "turn this whole website into a clean JSON file my LLM can read," this Actor is the answer.

***

### 🤖 Built for generative AI, LLMs, and RAG

This Actor is designed from the ground up for **AI workflows**:

- **RAG pipelines** — crawl docs, knowledge bases, FAQs, and blogs into chunkable Markdown.
- **LLM knowledge bases** — feed `text` + `metadata` into vector stores for grounded chatbot answers.
- **Semantic search engines** — index page content with embeddings for relevance-ranked retrieval.
- **Vector database ingestion** — drop directly into **Pinecone**, **Qdrant**, **Weaviate**, **Chroma**, **Milvus**, or **pgvector**.
- **Custom GPT knowledge files** — export Dataset JSON and upload to **ChatGPT custom GPTs**, **Claude Projects**, or **Anthropic Files API**.
- **AI agent training & grounding** — build **autonomous agents** with up-to-date web context.
- **Content operations** — summarization, translation, rewrite-assist, content moderation pipelines.
- **Competitive intelligence** — periodic crawls feeding analytics dashboards.

#### Practical AI use cases

1. **Customer support chatbot grounding** — crawl product docs, help center, and FAQ pages; embed `text + metadata` for retrieval.
2. **Documentation intelligence & semantic search** — turn product docs and changelogs into a searchable, embedding-indexed corpus.
3. **SEO and content audit automation** — analyze titles, meta descriptions, canonical links, and structured data sitewide.
4. **Custom GPT / Claude Project knowledge files** — generate a dataset JSON and upload to your AI assistant of choice.
5. **Internal LLM knowledge base** — sync intranet, wiki, or Notion-published content into a private vector store.
6. **Lead enrichment & company intelligence** — crawl prospect websites for descriptions, services, and contact pages.
7. **Compliance and content monitoring** — track changes to legal pages, terms of service, or pricing pages.

***

### ⚙️ What the crawler does

- **Crawls** from your seed URLs, follows internal links within domain scope, applies your `includeUrlGlobs` / `excludeUrlGlobs` filters, and respects `maxCrawlDepth`, `maxCrawlPages`, and `maxResults`.
- **Handles JavaScript sites** according to your `renderingMode` — `auto` renders only the pages that need it, `always` renders everything, `never` is the fastest and cheapest.
- **Cleans each page** with your CSS selectors (nav, footer, cookie banners, modals) before extracting text and Markdown.
- **Writes every row to the Apify Dataset the moment it is ready** — `url`, flat metadata columns, `crawl` info, `metadata`, `text`, `markdown`, snapshot links, and `screenshotUrl`. Downstream systems can consume rows while the crawl is still running.
- **Keeps going when a page fails** — bad URLs, non-HTML files, blocked pages, and oversized pages are handled per page, never by aborting the run.

***

### 🧭 Crawling configuration

- **Start URLs** are the only required input. Add as many as you like.
- By default, the crawler stays on the **same registrable domain** as your seeds — no accidental crawling of `facebook.com` from your blog's share buttons.
- **`includeUrlGlobs`** — keep only matching paths (e.g., `/blog/*`, `**/blog/**`).
- **`excludeUrlGlobs`** — skip paths (e.g., `/admin/**`, `**/login`, `**/*.pdf`).
- Filters apply to **discovered links only** — your start URLs are always crawled, so a narrow include pattern can't accidentally produce an empty run.
- **`useSitemaps`** — also seed the queue from `robots.txt`-advertised sitemaps and `/sitemap.xml` (following sitemap indexes). Reaches orphan pages that aren't linked from your seeds. Your include/exclude filters still apply.
- **`useLlmsTxt`** — also seed the queue from URLs listed in each start domain's `/llms.txt`.
- **`respectRobotsTxtFile`** — skip discovered URLs disallowed by the site's `robots.txt` (your explicit start URLs are always crawled).
- **`keepUrlFragments`** — treat `#anchors` as distinct URLs (off by default).
- **`ignoreCanonicalUrl`** — disable canonical-based dedup if you want every URL crawled distinctly.
- **`maxCrawlDepth`** — `0` = only seeds; `1` = seeds + their direct links; etc.
- **`maxCrawlPages`** — total unique pages opened (`0` = unlimited).
- **`maxResults`** — total successful records saved.

🧪 Test glob patterns with the [DigitalOcean glob tester](https://www.digitalocean.com/community/tools/glob).

***

### 🧼 HTML cleaning & content extraction

The Actor produces **clean, readable text** suitable for AI consumption:

- **`keepElementsCssSelector`** — keep **only** the elements matching this selector (e.g. `article, main`) and discard everything else. Applied before the remove list below; empty keeps the whole page.
- **`removeElementsCssSelector`** — comma-separated CSS selectors to strip before extraction (defaults remove nav, footer, scripts, cookie banners, modals, alerts).
- **`readableTextCharThreshold`** — how much text a page must have before it counts as complete.
- **`waitForSelector`** — pause until a key CSS selector appears (perfect for SPAs that mount content lazily).
- **`dynamicContentWaitSecs`** — extra settle time for late-arriving dynamic content.
- **`saveMarkdown`** — include a Markdown version of the page alongside plain text.

The result: **token-efficient, noise-free content** ready for embedding, indexing, or LLM context.

#### Rendered-page interaction

These apply only to rendered pages (`renderingMode: auto` when a page is rendered, or `always`):

- **`maxScrollHeightPixels`** — auto-scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content (`0` = off).
- **`clickElementsCssSelector`** — best-effort click on matching elements (accordions, "show more", expanders) before extracting; e.g. `[aria-expanded="false"]`.
- **`blockMedia`** — skip downloading images, fonts, and video while rendering for faster, cheaper runs (text and links unaffected).

***

### 📤 Output format (Dataset)

Results land in the default [Apify Dataset](https://docs.apify.com/platform/storage/dataset). You can:

- Browse rows in the Apify Console with multiple **prebuilt views** (Quick View, Crawl Details, Metadata, Content, Storage & Debug).
- Export to **JSON**, **CSV**, **Excel**, **XML**, **HTML**, or **RSS**.
- Pull data programmatically via the [Apify API](https://docs.apify.com/api/v2).
- Stream into **AI**, **ETL**, or **analytics** pipelines while the run is still active.

Each row includes:

| Field | Description |
|---|---|
| `orderNo` | 1-based order in which the row was saved during the run. |
| `url` | **The page's own URL** — its canonical URL when one exists (and `ignoreCanonicalUrl` is off), otherwise the loaded URL. |
| `loadedUrl` | The URL actually fetched, after redirects. |
| `referrerUrl` | The page that linked to this one (`referrerUrl` input for start URLs). |
| `depth` | Link distance from the nearest start URL (`0` for seeds). |
| `httpStatusCode` | HTTP status of the response. |
| `title`, `description`, `canonicalUrl`, `author`, `keywords`, `datePublished`, `dateModified`, `languageCode` | Flat metadata aliases so Dataset **table views** and CSV exports work without nesting. |
| `crawl` | Nested `loadedUrl`, `loadedTime`, `referrerUrl`, `depth`, `httpStatusCode`. |
| `metadata` | `canonicalUrl`, `title`, `description`, `author`, `keywords`, `datePublished`, `dateModified`, `languageCode`, `openGraph` (Open Graph + article + Twitter Card tags), `jsonLd`, `headers`. |
| `text` | Clean, readable plain text with paragraph structure preserved. |
| `markdown` | Real Markdown — headings, links, bold/italic, nested lists, blockquotes, GFM tables, fenced code (when `saveMarkdown` is on). |
| `html` | Cleaned readable HTML inline (when `saveHtml` is on). |
| `htmlUrl` | **Direct URL** to the readable HTML snapshot in the key-value store. |
| `screenshotUrl` | **Direct URL** to the full-page PNG (when `saveScreenshots` is on). |
| `files` | Non-HTML links found on the page (PDF, DOCX, XLSX, ZIP, …), or `null`. |
| `debug` | Per-row diagnostics — page/file link counts, response content type, snapshot URLs, and content byte lengths. |
| `debugTotalLinks`, `debugStatusCode` | Flat debug aliases for table views. |

> ℹ️ Rows over Apify's 9 MB item limit are trimmed automatically (`html` → `markdown` → `text`) and flagged with `debug.truncated: true`, so a single huge page never fails the run.

> 🧾 With `storeSkippedUrls: true`, discovered-but-not-crawled URLs are written to `skipped-urls.json` in the run's key-value store, each with a `reason` (`off-domain`, `excluded`, `not-included`, `robots-disallow`, `non-http`).

***

### 🧪 Example output (JSON)

Real output from a run against `https://docs.apify.com/` (text and Markdown abbreviated):

```json
{
  "orderNo": 4,
  "url": "https://docs.apify.com/proxy",
  "loadedUrl": "https://docs.apify.com/proxy",
  "loadedTime": "2026-07-23T14:31:52.015Z",
  "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
  "depth": 1,
  "httpStatusCode": 200,
  "title": "Proxy | Platform | Apify Documentation",
  "description": "Learn to anonymously access websites in scraping/automation jobs.",
  "canonicalUrl": "https://docs.apify.com/proxy",
  "author": null,
  "keywords": null,
  "datePublished": null,
  "dateModified": null,
  "languageCode": "en",
  "crawl": {
    "loadedUrl": "https://docs.apify.com/proxy",
    "loadedTime": "2026-07-23T14:31:52.015Z",
    "referrerUrl": "https://docs.apify.com/academy/scraping-basics-javascript",
    "depth": 1,
    "httpStatusCode": 200
  },
  "metadata": {
    "canonicalUrl": "https://docs.apify.com/proxy",
    "title": "Proxy | Platform | Apify Documentation",
    "description": "Learn to anonymously access websites in scraping/automation jobs.",
    "author": null,
    "keywords": null,
    "datePublished": null,
    "dateModified": null,
    "languageCode": "en",
    "openGraph": [
      { "property": "og:url", "content": "https://docs.apify.com/proxy" },
      { "property": "og:locale", "content": "en" }
    ],
    "jsonLd": null,
    "headers": { "content-type": "text/html; charset=utf-8", "…": "…" }
  },
  "text": "Proxy\n\nApify Proxy lets you rotate IP addresses when scraping to avoid geographic blocking. …",
  "markdown": "# Proxy\n\n[Apify Proxy](https://apify.com/proxy) lets you rotate IP addresses when scraping …",
  "htmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html",
  "screenshotUrl": null,
  "files": null,
  "debug": {
    "totalLinks": 59,
    "pageLinks": 59,
    "fileLinks": 0,
    "response": { "statusCode": 200, "contentType": "text/html; charset=utf-8" },
    "snapshots": {
      "originalHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_original.html",
      "removedElementsHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_removed.html",
      "readableTextHtmlUrl": "https://api.apify.com/v2/key-value-stores/<storeId>/records/kv__bfc3652cf15f843d170b_readable.html"
    },
    "lengths": {
      "originalHtml": 75279,
      "readableHtml": 43168,
      "readableText": 10763,
      "markdown": 19203
    }
  }
}
```

***

### 🚀 Quick start

#### Run in Apify Console

1. Open the Actor in [Apify Console](https://console.apify.com/).
2. Paste one or more **start URLs** (e.g., `https://docs.apify.com/`).
3. (Optional) Set **`maxResults`**, **`maxCrawlDepth`**, **`maxCrawlPages`**, and URL filters.
4. Click **Start** and watch rows stream into your Dataset.
5. Export as **JSON / CSV** or pull via the [Apify API](https://docs.apify.com/api/v2).

#### Run via API

```bash
curl -X POST "https://api.apify.com/v2/acts/<your-actor-id>/runs?token=<APIFY_TOKEN>"   -H "Content-Type: application/json"   -d '{
        "startUrls": ["https://docs.apify.com/"],
        "maxResults": 25,
        "maxCrawlDepth": 2,
        "renderingMode": "auto",
        "saveMarkdown": true
      }'
```

Then read the rows from the run's dataset:

```bash
curl "https://api.apify.com/v2/actor-runs/<runId>/dataset/items?format=json&token=<APIFY_TOKEN>"
```

***

### 🛠️ Input parameters reference

| Parameter | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | — | **Required.** Seed URLs. Always crawled — URL filters apply to discovered links only. |
| `referrerUrl` | string | `""` | Value recorded as `referrerUrl` on rows produced from the start URLs. |
| `blockAds` | bool | `false` | Abort known ad/tracker domains on rendered pages. |
| `mobileView` | bool | `false` | Render with a mobile device profile (viewport, UA, touch) instead of desktop. |
| `redactPii` | bool | `false` | Mask emails, phone numbers, card-like numbers, and SSNs in `text`/`markdown` (best-effort). |
| `screenshotFullPage` | bool | `true` | Full-page vs viewport-only screenshots. |
| `extractPdfText` | bool | `false` | Open discovered PDF links and extract real text into `text` instead of only listing them as files. |
| `proxyConfiguration` | object | `{}` | Apify Proxy (datacenter/residential/country) or your own proxy URLs. |
| `includeUrlGlobs` | array | `[]` | Keep only discovered URLs matching these globs (`/blog/*`, `/docs/**`). |
| `excludeUrlGlobs` | array | `[]` | Skip discovered URLs matching these globs (`/admin/**`, `**/*.pdf`). |
| `useSitemaps` | bool | `false` | Seed the queue from `robots.txt`-advertised sitemaps and `/sitemap.xml` (follows sitemap indexes). |
| `useLlmsTxt` | bool | `false` | Seed the queue from URLs listed in each start domain's `/llms.txt`. |
| `respectRobotsTxtFile` | bool | `false` | Skip discovered URLs disallowed by `robots.txt` (seeds always crawled). |
| `keepUrlFragments` | bool | `false` | Treat `#anchors` as distinct URLs. |
| `ignoreCanonicalUrl` | bool | `false` | Use the loaded URL as `url` instead of the canonical one. |
| `maxCrawlDepth` | int | `2` | Link-following depth from seeds (`0` = seeds only). |
| `maxCrawlPages` | int | `0` | Hard cap on unique pages opened (`0` = unlimited). |
| `maxResults` | int | `10` | Cap on successful saved rows. |
| `maxConcurrency` | int | `10` | Parallel page workers (1–50). |
| `rateLimitDelaySecs` | string | `0.1` | Throttle before each fetch; accepts decimals. |
| `maxRequestRetries` | int | `3` | Total attempts per URL (backoff; honours `Retry-After` on 429). |
| `requestTimeoutSecs` | int | `60` | Request / navigation timeout. |
| `dynamicContentWaitSecs` | int | `3` | **Real** settle time in seconds after a rendered page loads. |
| `waitForSelector` | string | — | Wait for this CSS selector; a missing selector triggers a more thorough retry. |
| `renderingMode` | enum | `auto` | `auto` (render only when a page needs it) · `always` · `never` (cheapest). |
| `retryFailedWithRendering` | bool | `true` | Give a failed page one more, more thorough attempt. |
| `blockMedia` | bool | `false` | Skip images/fonts/video on rendered pages for faster, cheaper runs. |
| `maxScrollHeightPixels` | int | `0` | Auto-scroll rendered pages up to N pixels for lazy/infinite-scroll content (`0` = off). |
| `clickElementsCssSelector` | string | — | Best-effort click on matching elements (expanders / "load more") before extracting rendered pages. |
| `ignoreHttpsErrors` | bool | `false` | Accept bad certificates (dev/staging only). |
| `keepElementsCssSelector` | string | — | Keep only elements matching this selector (e.g. `article, main`); empty keeps the whole page. |
| `removeElementsCssSelector` | string | (sensible default) | CSS selectors stripped before extraction. |
| `readableTextCharThreshold` | int | `100` | Below this many characters a page counts as "thin" and is retried. |
| `kvDir` | string | `kv` | Storage prefix for snapshots & screenshots (empty = skip storage writes). |
| `saveScreenshots` | bool | `false` | Full-page PNG per row. **Requires rendering** every page. |
| `saveMarkdown` | bool | `true` | Include Markdown in each row. |
| `saveHtml` | bool | `true` | Include cleaned HTML inline; turn off for much smaller rows. |
| `headerAcceptLanguage` | string | `en-US,en;q=0.9` | `Accept-Language` header. |
| `customHttpHeaders` | object | `{}` | Extra HTTP headers (cookies, auth, lang) — these override everything else. |
| `initialCookies` | array | `[]` | Cookies sent on every request and loaded into the browser: `[{"name":"...","value":"...","domain":"...","path":"/"}]`. |
| `debugLog` | bool | `true` | Drops the run log to DEBUG level. |
| `storeSkippedUrls` | bool | `false` | Write discovered-but-not-crawled URLs (with reasons) to `skipped-urls.json` in the key-value store. |

Plus browser-identity headers (`headerUserAgent`, `headerSecChUa`, `headerSecChUaMobile`, `headerSecChUaPlatform`, `headerDnt`, `headerUpgradeInsecureRequests`). The defaults give the best success rate on most sites; set a custom `headerUserAgent` or `headerSecChUa` and your values are used verbatim instead.

***

### 🔌 Integrations: LangChain, LlamaIndex, OpenAI, Pinecone

This Actor's Dataset output plugs directly into the modern AI stack:

- 🦜 **LangChain** — use `ApifyDatasetLoader` or load JSON into `Document` objects with `text` + `metadata`.
- 🦙 **LlamaIndex** — feed Markdown rows into `MarkdownReader` or build a `VectorStoreIndex`.
- 🧠 **OpenAI Embeddings & Assistants** — chunk `text` / `markdown`, embed with `text-embedding-3`, and upload as **OpenAI File** for Assistants API or Custom GPT knowledge.
- 🤖 **Anthropic Claude** — upload Dataset JSON to **Claude Projects**, the **Files API**, or use as RAG context.
- 📍 **Pinecone**, **Qdrant**, **Weaviate**, **Chroma**, **Milvus**, **pgvector** — embed and upsert with `metadata` filters.
- 📊 **n8n**, **Make.com**, **Zapier** — wire crawl results into automations and notifications.
- 📦 **Snowflake**, **BigQuery**, **PostgreSQL**, **MongoDB**, **S3** — load Dataset JSON/CSV into your warehouse or lake.
- 🧰 **Apify API & SDKs** — orchestrate runs from your own services in any language.

See [Apify Publishing & integration docs](https://docs.apify.com/platform/actors/publishing) for platform-level guidance.

***

### 💸 Pricing & how to control cost

#### Billable event

This Actor charges one **pay-per-event** event:

| Event | When it fires | What you get |
|---|---|---|
| `row_result` | Once per page record written to the Dataset | A complete page row — text, Markdown, metadata, snapshot links |

Pages that fail, are skipped (non-HTML, `404`, filtered by globs), or are dropped after the `maxResults` cap **are not charged**. If the run hits its charging limit (`ACTOR_MAX_TOTAL_CHARGE_USD`), the crawl stops cleanly and everything collected so far stays in the Dataset.

> ⚙️ **Publishing note:** the `row_result` event must exist in *Apify Console → your Actor → Publication → Monetization* with the same name. Until pay-per-event is configured, the Actor still runs and saves data — it just logs `Ignored attempt to charge for an event`.

#### Compute cost

Cost on Apify also scales with **compute units** consumed during a run. The biggest drivers are:

- 📄 **Number of pages crawled** (`maxResults`, `maxCrawlPages`).
- 🖥️ **Rendering mode** — `always` is far heavier than `auto`, and `auto` is heavier than `never`.
- 🧮 **Concurrency** (`maxConcurrency`) and timeout settings.
- 🌐 **Page complexity** and network conditions.

#### To control cost, tune:

- `maxResults` — hard cap on saved rows.
- `maxCrawlPages` — hard cap on URLs opened.
- `maxCrawlDepth` — keep crawls shallow when possible.
- `maxConcurrency` — balance speed vs. politeness.
- `renderingMode: "never"` — the cheapest possible run.
- `blockMedia: true` — skip images/fonts/video on rendered pages to cut render time and bandwidth.
- `blockAds: true` — skip known ad/tracker requests on rendered pages.
- `screenshotFullPage: false` — capture only the viewport instead of the whole scrolled page.
- `retryFailedWithRendering: false` — skip the second attempt on pages that fail once.
- `rateLimitDelaySecs` — throttle to avoid being rate-limited (which wastes retries).
- `saveScreenshots` — leave **off** unless you need images; enabling it renders every page.
- `saveHtml: false` — drops the inline `html` field for much smaller (and cheaper to store) rows.

> 💡 **Tip:** start with `maxResults: 10` for testing, then scale up once your filters and selectors are dialed in.

***

### ❓ Frequently asked questions (FAQ)

#### What is the best web scraper for AI and LLM training data?

A web scraper for **AI / LLM training data** must produce **clean text or Markdown**, preserve **metadata**, handle **JavaScript-rendered pages**, and export to a structured format (JSON, JSONL, Parquet). Website Content Crawler does all four — it strips boilerplate HTML, outputs both plain text and Markdown, captures rich metadata, and writes directly to an Apify Dataset that exports as JSON/CSV.

#### How do I scrape a website for a RAG pipeline?

1. Run Website Content Crawler with your docs/help/blog URLs as `startUrls`.
2. Set `saveMarkdown: true` and tune `removeElementsCssSelector` to strip nav/footer.
3. Export the Dataset as JSON.
4. Chunk each `markdown` field, embed with OpenAI / Cohere / Voyage / local models, and upsert to Pinecone/Qdrant/Weaviate.
5. At query time, retrieve top-K chunks and pass them as LLM context.

#### Can this Actor scrape JavaScript-heavy single-page applications (SPAs)?

Yes. `renderingMode: "auto"` already handles pages that need rendering automatically; set `renderingMode: "always"` when every page must be rendered. Use `waitForSelector` for content that mounts after the initial load, and `dynamicContentWaitSecs` for late-arriving content — it is a genuine wait in seconds.

#### How do I export the crawl results?

Open the run in Apify Console → **Dataset** tab → **Export** → choose JSON, CSV, Excel, XML, HTML, or RSS. You can also pull data programmatically via the [Apify Dataset API](https://docs.apify.com/api/v2#/reference/datasets).

#### Does this Actor respect robots.txt?

Optionally, yes — set `respectRobotsTxtFile: true` and discovered URLs disallowed by the site's `robots.txt` are skipped (your explicit start URLs are always crawled). It is **off by default**. Either way, **you** remain responsible for compliance with `robots.txt`, website terms of service, and applicable laws; you can also add `excludeUrlGlobs` to skip specific paths.

#### How do I avoid being rate-limited or blocked?

- Lower `maxConcurrency` (start at 2–5).
- Increase `rateLimitDelaySecs` (e.g., `1.0`).
- Set realistic `headerUserAgent` and `customHttpHeaders` (cookies, language).
- Enable `proxyConfiguration`, or set `renderingMode: "always"` for sites with aggressive bot detection.

#### Can I crawl multiple websites in one run?

Yes — add multiple URLs to `startUrls`. By default the crawler stays on **each seed's own domain**, so you'll get multi-domain coverage.

#### How is this different from the official `apify/website-content-crawler`?

This Actor keeps the input surface small and the cost predictable: three rendering modes instead of a matrix of engines, Markdown and metadata by default, screenshots and HTML snapshots as direct links, and one billable event you can reason about. The official Apify Actor has a much larger feature surface. Choose this one when you want **simpler input, predictable cost, and AI-first defaults**; choose the official one when you need its full enterprise feature set.

#### What output formats are supported?

JSON, JSONL, CSV, Excel, XML, HTML, RSS — all via standard Apify Dataset exports.

#### Does it support proxies?

Yes — use the **`proxyConfiguration`** input to pick [Apify Proxy](https://docs.apify.com/platform/proxy) datacenter or residential groups, a country, or your own proxy URLs. Requests are spread across several proxy sessions automatically. If the chosen proxy is unavailable, the run continues instead of failing.

#### Can I schedule recurring crawls?

Yes — use [Apify Schedules](https://docs.apify.com/platform/schedules) to run this Actor hourly, daily, or weekly, and pipe new rows into your warehouse, vector DB, or notification channel.

***

### 🛠️ Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| **No rows saved** | Bad `startUrls`, over-restrictive filters, target offline | Verify `startUrls`, relax `includeUrlGlobs` / `excludeUrlGlobs`, check site availability |
| **Crawl too slow** | Rendering every page + high retries | Lower `maxConcurrency`, switch `renderingMode` to `auto` or `never` |
| **Missing content / empty `text`** | JS-rendered SPA, content behind selector | Set `waitForSelector`, increase `dynamicContentWaitSecs`, set `renderingMode: "always"` |
| **Boilerplate in output** | Default `removeElementsCssSelector` doesn't cover site's nav | Add site-specific selectors (e.g., `.sidebar, .related-posts, .ad-slot`) |
| **Duplicate rows** | Canonical handling off, fragments treated as distinct | Enable canonical (default), set `keepUrlFragments: false` |
| **Cert errors on staging** | Self-signed / expired cert | Set `ignoreHttpsErrors: true` (dev only) |
| **Blocked / 403 / 429** | Bot detection, rate limit | Enable `proxyConfiguration` (residential helps most), increase `rateLimitDelaySecs`, lower `maxConcurrency`, try `renderingMode: "always"` |
| **No screenshots in output** | `saveScreenshots` off, `kvDir` empty, or `renderingMode: "never"` | Set `saveScreenshots: true`, keep a `kvDir` prefix, and use `auto` or `always` |
| **Only the start URL is crawled** | `includeUrlGlobs` doesn't match any discovered link, or `maxCrawlDepth: 0` | Widen the globs (`/section/**`) and raise `maxCrawlDepth` |
| **Rows are huge / slow to export** | `html` field included | Set `saveHtml: false` and use `htmlUrl` instead |

***

### 💬 Help, support & custom builds

For **custom solutions**, **private integrations**, **white-label crawlers**, or **implementation help**:

📧 **<hello.dataminds@gmail.com>**

Useful Apify documentation:

- 📘 [Apify documentation](https://docs.apify.com/)
- 🤖 [Actors overview](https://docs.apify.com/platform/actors)
- 🔌 [Apify API reference](https://docs.apify.com/api/v2)
- 📚 [Apify Academy — web scraping](https://docs.apify.com/academy)
- 🧰 [Apify SDKs](https://docs.apify.com/sdk)

***

### ⚖️ Is web scraping legal?

Use this Actor **only on pages you are allowed to access**. Respect:

- ✅ `robots.txt` directives,
- ✅ website terms of service,
- ✅ rate limits and politeness norms,
- ✅ data-protection laws (GDPR, CCPA, etc.) in your jurisdiction,
- ✅ copyright and licensing of extracted content.

You are responsible for **compliant data collection, storage, and use**. When in doubt, consult a qualified attorney.

***

### 🔍 SEO keywords targeted

This README is intentionally keyword-rich for discovery on **Google Search**, **Apify Store search**, and **Bing**. Primary and secondary terms covered include:

**Primary:** website content crawler · web scraping · website crawler · content extraction · web scraping API · Apify Actor · website scraper · site crawler

**Content & format:** Markdown extraction · plain text extraction · metadata extraction · structured data extraction · HTML cleaner · readable text extractor · canonical URL extraction · Open Graph scraper · JSON-LD extraction · schema.org scraper

**AI & LLM:** RAG pipeline · retrieval-augmented generation · LLM training data scraper · vector database ingestion · semantic search crawler · embedding pipeline · custom GPT knowledge file · Claude Projects data · LangChain data loader · LlamaIndex reader · AI agent grounding

**Capabilities:** headless rendering · JavaScript rendering scraper · SPA scraper · React scraper · dynamic content scraper · concurrent crawler · anti-blocking scraper · sitemap crawler · llms.txt crawler · robots.txt compliant crawler · infinite scroll scraper

**Use cases:** documentation scraper · blog crawler · SEO content audit · competitive intelligence crawler · website-to-JSON · website-to-database · website-to-Markdown · knowledge base scraper · help center crawler · changelog scraper

**Integrations:** Pinecone ingestion · Qdrant ingestion · Weaviate · Chroma · pgvector · Snowflake · BigQuery · n8n · Make.com · Zapier · OpenAI Embeddings · Anthropic Files API

***

*Reference patterns for structure and section ordering:*

- 📘 Public Actor: [apify/website-content-crawler](https://apify.com/apify/website-content-crawler)
- 📗 Apify publishing guidance: [Publishing and monetization](https://docs.apify.com/platform/actors/publishing)
- 🔐 Console (access-gated): [Console README](https://console.apify.com/actors/aYG0l9s7dbB7j3gbS/information/version-0/readme)

# Actor input Schema

## `startUrls` (type: `array`):

📝 **Required.** Paste the page(s) where crawling should begin. Example: your docs home, a blog index, or a product category. URL filters below are applied to *discovered* links only — your seeds are always crawled.

## `referrerUrl` (type: `string`):

🏷️ Recorded as `referrerUrl` on rows produced from your start URLs (they have no natural “came from” page). Discovered pages always record the page that linked to them. Leave blank for an empty referrer.

## `blockAds` (type: `boolean`):

🛡️ Abort requests to known ad/tracker hosts (doubleclick, Google Ads, Taboola, Criteo, and more) while rendering — cleaner, faster pages. Only affects rendered pages.

## `mobileView` (type: `boolean`):

📲 Render pages with a mobile viewport and device profile instead of desktop. Useful for sites that serve different content to mobile visitors. A custom User-Agent above still overrides the mobile one.

## `redactPii` (type: `boolean`):

🔒 Mask emails, phone numbers, credit-card-like numbers, and SSNs in `text` and `markdown` before saving. Regex-based heuristics — not a compliance guarantee.

## `screenshotFullPage` (type: `boolean`):

🖼️ When **on** (default), screenshots capture the entire scrolled page. Turn **off** to capture just the visible viewport — faster and smaller. Only applies when `saveScreenshots` is on.

## `extractPdfText` (type: `boolean`):

📄 When **on**, discovered PDF links are opened and their text is extracted into `text` (counted toward your page/result limits like any other page) instead of only being listed as a file link.

## `includeUrlGlobs` (type: `array`):

🧩 **Include patterns.** If you add any, *only* discovered URLs matching at least one pattern are queued. Examples: `/blog/*` · `/docs/**` · `https://site.com/product/**`. Empty = no extra filter.

## `excludeUrlGlobs` (type: `array`):

🛑 URLs matching **any** pattern here are skipped — handy for login, admin, or file folders. Examples: `/admin/**` · `/login` · `**/*.pdf`.

## `useSitemaps` (type: `boolean`):

When **on**, the crawler reads `robots.txt`-advertised sitemaps and `/sitemap.xml` (following sitemap indexes) and queues those URLs — great for reaching pages that aren't linked from your start page. Your include/exclude filters still apply.

## `useLlmsTxt` (type: `boolean`):

When **on**, the crawler reads `/llms.txt` at each start domain and queues the URLs it lists. Your include/exclude filters still apply.

## `respectRobotsTxtFile` (type: `boolean`):

When **on**, discovered URLs disallowed by the site's `robots.txt` are skipped. Your explicit start URLs are always crawled.

## `keepUrlFragments` (type: `boolean`):

📌 Normally `#section` is ignored. Turn **on** if you need each fragment to count as its own URL.

## `ignoreCanonicalUrl` (type: `boolean`):

🔀 When **off**, the page's canonical URL becomes the row's `url` and drives deduplication. When **on**, the loaded URL wins.

## `maxCrawlDepth` (type: `integer`):

0️⃣ = only your start URLs · 1️⃣ = their direct links · 2️⃣ = one more hop… and so on. Great for staying on-topic!

## `maxCrawlPages` (type: `integer`):

🛡️ Hard cap on how many **unique pages** may be opened, including pages that turn out to be empty or blocked. **0** means “no extra cap” (max rows below still applies).

## `maxResults` (type: `integer`):

🎁 Hard stop on how many **successful page records** are saved — perfect for sampling or quick tests.

## `maxConcurrency` (type: `integer`):

🔄 How many pages load **at the same time**. Higher = faster, but be kind to small sites — start moderate.

## `rateLimitDelaySecs` (type: `string`):

🌊 Throttle applied before each fetch, e.g. `0.1` or `1.5`. Use `0` to go as fast as your other limits allow. Non-numeric values fall back to `0`.

## `maxRequestRetries` (type: `integer`):

♻️ Total fetch **attempts** for a URL before giving up (`3` = one try plus two retries, with exponential backoff). `429` responses honour `Retry-After`.

## `requestTimeoutSecs` (type: `integer`):

⏲️ Max time to wait for a page to respond — useful on slow or heavy sites.

## `dynamicContentWaitSecs` (type: `integer`):

📺 Real settle time **after** a rendered page finishes loading, so late-arriving content is captured. Only applies to rendered pages. Higher values cost more compute.

## `waitForSelector` (type: `string`):

✨ Wait until this **CSS selector** appears before extracting — ideal for key content that pops in late. A page missing the selector is treated as incomplete and collected again more thoroughly.

## `renderingMode` (type: `string`):

**Auto** is recommended: most pages are collected quickly, and only the ones that need it are rendered. **Always** renders every page (most reliable for JavaScript-heavy sites, slowest). **Never** is the cheapest and fastest, but cannot capture screenshots or JavaScript-built content.

## `retryFailedWithRendering` (type: `boolean`):

💫 If a page cannot be collected on the first attempt, try it once more with rendering. Great for flaky or protected pages. Ignored when the rendering mode is *Never*.

## `blockMedia` (type: `boolean`):

⚡ Skip downloading images, fonts and video while rendering — faster and cheaper on heavy pages. Only affects rendered pages; text and links are unaffected.

## `maxScrollHeightPixels` (type: `integer`):

🖱️ On rendered pages, scroll down up to this many pixels to trigger lazy-loaded / infinite-scroll content before extracting. `0` disables scrolling. Only applies to rendered pages.

## `clickElementsCssSelector` (type: `string`):

🔽 On rendered pages, click elements matching this selector (accordions, “show more”, expanders) before extracting. Best-effort — unclickable elements are skipped. Example: `[aria-expanded="false"]`.

## `ignoreHttpsErrors` (type: `boolean`):

🔓 **Off** by default (secure). Turn **on** only for local/dev sites with certificate issues — not for production.

## `keepElementsCssSelector` (type: `string`):

🧲 If set, **only** the elements matching this selector are kept — everything else is discarded before extraction. Ideal for grabbing just an article body, e.g. `article, main`. Leave empty to keep the whole page. Applied before the remove list below.

## `removeElementsCssSelector` (type: `string`):

Comma-separated selectors. Commas inside `[attr="a,b"]` or `:is(a,b)` are handled correctly, and an invalid selector is skipped with a warning instead of failing the page.

## `readableTextCharThreshold` (type: `integer`):

📊 A page whose cleaned text is shorter than this counts as **too thin** and, in *Auto* rendering mode, is collected again more thoroughly.

## `kvDir` (type: `string`):

Prefix for saved records (e.g. `kv`). Characters that are illegal in key-value store keys are replaced automatically. Leave **empty** to skip writing files entirely.

## `saveScreenshots` (type: `boolean`):

🖼️ Captures a full-page **PNG** for every saved row. ⚠️ Screenshots require rendering every page, so runs are slower and cost more. Needs a storage prefix and a rendering mode other than *Never*.

## `saveMarkdown` (type: `boolean`):

📒 Adds a real **Markdown** rendering (headings, links, lists, tables, code fences) alongside plain text — turn off if you only need raw text.

## `saveHtml` (type: `boolean`):

🧱 Embeds the cleaned readable HTML in the `html` field. Turn **off** for much smaller dataset rows — the snapshot links in `htmlUrl` still work.

## `proxyConfiguration` (type: `object`):

Pick Apify Proxy (datacenter or residential) or supply custom proxy URLs. If the selected proxy is unavailable, the run continues rather than failing.

## `headerAcceptLanguage` (type: `string`):

Preferred content language, e.g. `en-US,en;q=0.9` or `de-DE,de;q=0.9`.

## `headerDnt` (type: `string`):

Typical value `1`. Leave empty to omit the header.

## `headerUpgradeInsecureRequests` (type: `string`):

Standard browser hint — usually `1`. Leave empty to omit.

## `headerUserAgent` (type: `string`):

🎭 Leave at the default for the best success rate. Change it only if a site needs a specific User-Agent — a custom value is then used verbatim.

## `headerSecChUa` (type: `string`):

Client hint header — keep the default unless you also customise the User-Agent.

## `headerSecChUaMobile` (type: `string`):

Usually `?0` for desktop mode.

## `headerSecChUaPlatform` (type: `string`):

e.g. `"Windows"` for desktop.

## `customHttpHeaders` (type: `object`):

🧩 Merge **custom headers** (cookies, auth, language). These win over every header above. Invalid JSON fails validation.

## `initialCookies` (type: `array`):

🔐 Cookies sent with every request and loaded into the browser — useful for session or light login. Each item: `{ "name": "...", "value": "...", "domain": "...", "path": "/" }` (`domain`/`path` optional). Example: `[{"name":"sessionid","value":"abc123"}]`.

## `debugLog` (type: `boolean`):

📣 When **on**, the run log drops to DEBUG level and prints queue decisions, fallbacks, and parse notes. Noisier, but great for troubleshooting.

## `storeSkippedUrls` (type: `boolean`):

📄 When **on**, URLs that were discovered but not crawled (off-domain, filtered out, robots-disallowed) are written to `skipped-urls.json` in the key-value store, with a reason for each.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "blockAds": false,
  "mobileView": false,
  "redactPii": false,
  "screenshotFullPage": true,
  "extractPdfText": false,
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "useSitemaps": false,
  "useLlmsTxt": false,
  "respectRobotsTxtFile": false,
  "keepUrlFragments": false,
  "ignoreCanonicalUrl": false,
  "maxCrawlDepth": 2,
  "maxCrawlPages": 0,
  "maxResults": 10,
  "maxConcurrency": 10,
  "rateLimitDelaySecs": "0.1",
  "maxRequestRetries": 3,
  "requestTimeoutSecs": 60,
  "dynamicContentWaitSecs": 3,
  "renderingMode": "auto",
  "retryFailedWithRendering": true,
  "blockMedia": false,
  "maxScrollHeightPixels": 0,
  "ignoreHttpsErrors": false,
  "removeElementsCssSelector": "nav, footer, script, style, noscript, svg, img[src^='data:'], [role=\"alert\"], [role=\"banner\"], [role=\"dialog\"], [role=\"alertdialog\"], [role=\"region\"][aria-label*=\"skip\" i], [aria-modal=\"true\"]",
  "readableTextCharThreshold": 100,
  "kvDir": "kv",
  "saveScreenshots": false,
  "saveMarkdown": true,
  "saveHtml": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "headerAcceptLanguage": "en-US,en;q=0.9",
  "headerDnt": "1",
  "headerUpgradeInsecureRequests": "1",
  "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
  "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
  "headerSecChUaMobile": "?0",
  "headerSecChUaPlatform": "\"Windows\"",
  "customHttpHeaders": {},
  "initialCookies": [],
  "debugLog": true,
  "storeSkippedUrls": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/academy/scraping-basics-javascript"
    ],
    "referrerUrl": "",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "rateLimitDelaySecs": "0.1",
    "waitForSelector": "",
    "clickElementsCssSelector": "",
    "keepElementsCssSelector": "",
    "kvDir": "kv",
    "proxyConfiguration": {
        "useApifyProxy": true
    },
    "headerAcceptLanguage": "en-US,en;q=0.9",
    "headerDnt": "1",
    "headerUpgradeInsecureRequests": "1",
    "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
    "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
    "headerSecChUaMobile": "?0",
    "headerSecChUaPlatform": "\"Windows\"",
    "customHttpHeaders": {},
    "initialCookies": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("citrine_venus/website-content-crawler").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/academy/scraping-basics-javascript"],
    "referrerUrl": "",
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [],
    "rateLimitDelaySecs": "0.1",
    "waitForSelector": "",
    "clickElementsCssSelector": "",
    "keepElementsCssSelector": "",
    "kvDir": "kv",
    "proxyConfiguration": { "useApifyProxy": True },
    "headerAcceptLanguage": "en-US,en;q=0.9",
    "headerDnt": "1",
    "headerUpgradeInsecureRequests": "1",
    "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
    "headerSecChUa": "\"Google Chrome\";v=\"143\", \"Chromium\";v=\"143\", \"Not A(Brand\";v=\"24\"",
    "headerSecChUaMobile": "?0",
    "headerSecChUaPlatform": "\"Windows\"",
    "customHttpHeaders": {},
    "initialCookies": [],
}

# Run the Actor and wait for it to finish
run = client.actor("citrine_venus/website-content-crawler").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "referrerUrl": "",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [],
  "rateLimitDelaySecs": "0.1",
  "waitForSelector": "",
  "clickElementsCssSelector": "",
  "keepElementsCssSelector": "",
  "kvDir": "kv",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "headerAcceptLanguage": "en-US,en;q=0.9",
  "headerDnt": "1",
  "headerUpgradeInsecureRequests": "1",
  "headerUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36",
  "headerSecChUa": "\\"Google Chrome\\";v=\\"143\\", \\"Chromium\\";v=\\"143\\", \\"Not A(Brand\\";v=\\"24\\"",
  "headerSecChUaMobile": "?0",
  "headerSecChUaPlatform": "\\"Windows\\"",
  "customHttpHeaders": {},
  "initialCookies": []
}' |
apify call citrine_venus/website-content-crawler --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=citrine_venus/website-content-crawler",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/zwqg9ccBKgqBd5xw9/builds/uPZ6EcAT89iYOKx10/openapi.json