# Contextractor — clean web content extraction for LLMs (`glueo/contextractor`) Actor

Crawl any website and extract clean main-content text as Markdown, plain text, JSON, or HTML — ready for LLMs, RAG pipelines, and vector databases. Built on the rs-trafilatura engine and an adaptive Crawlee + Playwright crawler.

- **URL**: https://apify.com/glueo/contextractor.md
- **Developed by:** [Glueo](https://apify.com/glueo) (community)
- **Categories:** Developer tools, News, AI
- **Stats:** 11 total users, 0 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<table align="right">
  <tbody>
    <tr>
      <td>
        <img width="220" src="https://www.contextractor.com/media/cover-mini.svg" alt="Contextractor" />
        <br />
        <a href="https://www.npmjs.com/package/contextractor"><img src="https://img.shields.io/npm/v/contextractor.svg" alt="npm version" /></a>
        <br />
        <a href="https://www.npmjs.com/package/contextractor"><img src="https://img.shields.io/npm/dm/contextractor.svg" alt="npm downloads" /></a>
        <br />
        <a href="https://github.com/contextractor/contextractor/blob/main/LICENSE"><img src="https://img.shields.io/npm/l/contextractor.svg" alt="license" /></a>
        <h3>Also available as:</h3>
        <ul>
          <li>
            <strong><a href="https://www.contextractor.com/">Online playground</a></strong>
            <br />
            <sub><a href="https://www.contextractor.com/">playground</a>, <a href="https://www.contextractor.com/help/web/">help</a></sub>
          </li>
          <li>
            <strong><a href="https://www.npmjs.com/package/contextractor">NPM package CLI &amp; lib</a></strong>
            <br />
            <sub><a href="https://www.npmjs.com/package/contextractor">package</a>, <a href="https://www.contextractor.com/help/npm/">CLI help</a>, <a href="https://www.contextractor.com/help/npm-lib/">lib help</a></sub>
          </li>
          <li>
            <strong><a href="https://github.com/contextractor/contextractor">Source code on GitHub</a></strong>
          </li>
        </ul>
        <h3>Social:</h3>
        <ul>
          <li>
            <strong><a href="https://x.com/contextractor">X (Twitter)</a></strong>
          </li>
        </ul>
      </td>
    </tr>
  </tbody>
</table>

**Crawl any website and extract clean, boilerplate-free main content** as
**Markdown, plain text, JSON, cleaned HTML, or raw original HTML** — ready to feed
**LLMs, RAG pipelines, and vector databases**. Contextractor uses the
[`rs-trafilatura`](https://github.com/Murrough-Foley/rs-trafilatura) extraction
engine to strip away navigation, ads, and cookie banners, and an adaptive
[Crawlee](https://crawlee.dev/) + Playwright crawler that automatically switches
between a real browser and fast HTTP — with proxy rotation and anti-blocking
handled for you.

Point it at a single page or crawl an entire site: Contextractor returns only the
content that matters, in the exact format your AI workflow needs.

### ✨ What can Contextractor do?

- **Extract clean main content** — the rs-trafilatura engine isolates the article
  body and removes navigation, headers, footers, ads, and cookie banners.
- **Five output formats** — Markdown, plain text (`txt`), JSON, cleaned HTML, and
  the original raw HTML, saved individually or together.
- **Adaptive crawling** — switches between a headless browser (for
  JavaScript-heavy pages) and raw HTTP per page; or force Chromium, Firefox,
  or HTTP-only.
- **Whole-site crawling** — follow links with a CSS selector and scope the crawl
  with include/exclude URL globs, sitemaps, and depth/page limits.
- **Tunable extraction** — choose `precision`, `balanced`, or `recall`, and toggle
  tables, links, images (alt text), and comments.
- **Built-in anti-blocking** — proxy rotation, persistent session pools, and
  automatic IP/fingerprint rotation when a block is detected.
- **Page metadata** — captures title, author, publication date, description, site
  name, and detected language.
- **Handles modern pages** — dismisses cookie modals, waits for selectors or
  network idle, scrolls lazy-loaded content, and accepts custom cookies and HTTP
  headers for logged-in or gated pages.
- **Deduplication** — skip already-seen pages by canonical URL or by
  extracted-content hash.

### 🚀 How to use Contextractor

No code required — run it straight from the Apify Console:

1. **Add your start URLs** — one or more pages or site sections you want to extract.
2. **Choose what to save and where** — the `Save` field takes `format-destination`
   tokens (e.g. `Markdown → Key-value store`, `Original HTML → Dataset`). Pick a
   format for each destination you want; selecting the same format for both the
   dataset and the key-value store saves it to both.
3. *(Optional)* **Set the crawl scope and behavior** — link selector, include/exclude
   URL globs, depth, and page limits to follow links across a site; enable proxy
   rotation, `robots.txt`, or render waits as needed.
4. **Click Start** and watch the run progress live.
5. **Download your data** — from the dataset (JSON, CSV, Excel) or the key-value
   store, or pull it programmatically via the Apify API.

### Input recipes

**Start URLs** are the only required field; everything else has a sensible
default. The complete field-by-field reference lives on the
[Input tab](https://apify.com/glueo/contextractor/input-schema) — the recipes
below cover the common jobs.

Crawl a blog section and save Markdown for RAG ingestion:

```json
{
  "startUrls": [{ "url": "https://blog.example.com/" }],
  "selector": "a[href]",
  "globs": [{ "glob": "https://blog.example.com/**" }],
  "maxCrawlDepth": 2,
  "save": ["markdown-kvs"]
}
```

Extract a single page in several formats at once:

```json
{
  "startUrls": [{ "url": "https://example.com/article" }],
  "maxCrawlDepth": 0,
  "maxRequestsPerCrawl": 1,
  "save": ["markdown-kvs", "json-dataset", "original-kvs"]
}
```

Crawl a site that blocks datacenter traffic, with proxy rotation and a
persistent session pool:

```json
{
  "startUrls": [{ "url": "https://shop.example.com/" }],
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  },
  "proxyRotation": "recommended",
  "sessionPoolName": "shop-example",
  "waitUntil": "networkidle"
}
```

Two shape rules to remember: `startUrls`, `globs`, and `exclude` take arrays of
objects (`{ "url": … }` / `{ "glob": … }`), not bare strings; `save` takes
`format-destination` tokens — format `txt`, `markdown`, `json`, `html`, or
`original`, destination `dataset` or `kvs` (e.g. `markdown-kvs`, `txt-dataset`).

Key settings worth knowing before a big run:

- **Crawler type** — `playwright-adaptive` (default) probes each page and uses a
  real browser only where JavaScript rendering is needed; `cheerio` is raw HTTP
  only — the fastest and cheapest when pages don't need JS.
- **Extraction mode** — `precision` trims aggressively, `recall` keeps more
  borderline content, `balanced` (default) sits between.
- **Deduplication** — `standard` (default) skips pages whose canonical URL was
  already extracted; `aggressive` also drops pages with identical extracted text.
- **Cookie consent** — `closeCookieModals` (default on) blocks trackers and
  dismisses consent walls before extraction.

### What data does Contextractor return?

Every crawled page becomes one dataset record. Successful pages carry
`status: "success"` with the extracted content and metadata; failed and skipped
pages are recorded too, so nothing is silently dropped.

| Field                                 | Description                                                                                                                                                            |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url`                                 | The original request URL.                                                                                                                                              |
| `status`                              | Record outcome: `success`, `failed`, or `skipped`.                                                                                                                     |
| `metadata`                            | Extracted page metadata: `title`, `author`, `publishedAt`, `description`, `siteName`, `languageCode`.                                                                  |
| `crawl`                               | Crawl provenance: `loadedUrl` (final URL after redirects), `loadedTime`, `httpStatusCode`, `depth` (link distance from a start URL), `referrerUrl` (the linking page). |
| `original`                            | The raw page HTML as a content node — `hash` (MD5) and `bytes` always present; `content`, or `key` + `url`, added when `original` is saved.                            |
| `txt`, `markdown`, `json`, `html`     | One content node per saved format — `hash` and `bytes`, plus inline `content` (dataset) or a `key` + `url` reference (key-value store).                                |
| `errors`, `retryCount`, `crawledTime` | On `failed` records only: the error messages, number of retries, and when the request was abandoned.                                                                   |
| `skipReason`                          | On `skipped` records only: `robotsTxt`, `limit`, `enqueueLimit`, `filters`, `redirect`, or `depth`.                                                                    |

Example success record (default settings — Markdown saved to the key-value store):

```json
{
  "url": "https://blog.example.com/why-rag-matters",
  "status": "success",
  "metadata": {
    "title": "Why RAG Matters",
    "author": "Jane Doe",
    "publishedAt": "2026-01-15",
    "description": "A practical look at retrieval-augmented generation.",
    "siteName": "Example Blog",
    "languageCode": "en"
  },
  "crawl": {
    "loadedUrl": "https://blog.example.com/why-rag-matters",
    "loadedTime": "2026-05-31T10:00:00.000Z",
    "httpStatusCode": 200,
    "depth": 1,
    "referrerUrl": "https://blog.example.com/"
  },
  "original": {
    "hash": "f8e6bd335e04d03e1be6798c2c72349c",
    "bytes": 89898
  },
  "markdown": {
    "hash": "43f204bfbee5dbe6862cb38620f257b5",
    "bytes": 5234,
    "key": "markdown-c485356090a92c6a45e8c1155c14d8ee.md",
    "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/<key>"
  }
}
```

#### Where your content is saved

- **Key-value store (default)** — each format is stored as a separate file keyed
  `{format}-{md5(url)}.{ext}` (e.g. `markdown-1a2b3c4d….md`), and the dataset record
  references it by `key` and `url`. Best for large content and bulk download.
- **Dataset** — the extracted content is embedded inline in each record under
  `content`. Best when you want everything in a single JSON, CSV, or Excel export.

Choose one or both with the **Save** option.

### 💰 How much will it cost?

Contextractor is usage-priced: you pay for the Apify compute units (CUs) a run
consumes, plus any proxy traffic, on top of your Apify plan. Cost scales with the
number of pages and the crawler type — the browser crawlers
(`playwright-adaptive`, `playwright-firefox`, `playwright-chromium`) use far more
compute than the HTTP-only `cheerio` crawler, so a large browser-based crawl costs
more than the same number of pages over plain HTTP. Keep runs efficient by using
`cheerio` where JavaScript isn't needed, setting a sensible `maxConcurrency`, and
bounding the crawl with `maxRequestsPerCrawl` and `maxCrawlDepth`. Apify's free
plan includes [$5 of usage monthly](https://apify.com/pricing); see the pricing
page for current compute-unit and proxy rates.

### Designed for LLMs, RAG, and AI pipelines

Contextractor turns messy web pages into clean, structured text that's ready for AI:

- **Build RAG knowledge bases** — crawl docs, blogs, or help centers and ingest
  clean Markdown into a vector database.
- **Feed and contextualize LLMs** — supply boilerplate-free content as context for
  ChatGPT, Claude, or your custom GPTs.
- **Create training and fine-tuning datasets** — gather large volumes of clean
  article text.
- **Bulk content processing** — summarize, translate, classify, or proofread pages
  at scale.
- **Content and SEO research** — archive competitor or reference content as plain
  text or JSON.

Each output format is suited to a different job:

| Format     | Best for                                                                |
| ---------- | ----------------------------------------------------------------------- |
| `markdown` | Chunking and embeddings, chat context, notebooks — the default for RAG. |
| `txt`      | Lightweight NLP, keyword stats, and simple text pipelines.              |
| `json`     | Structured, programmatic downstream processing.                         |
| `html`     | Layout-aware processing or feeding other HTML tools.                    |
| `original` | The full, unmodified page for re-processing, archival, or auditing.     |

### How does it work?

Contextractor runs a three-stage pipeline for every page:

- **Crawl** — an adaptive Crawlee + Playwright crawler fetches each page and follows
  links within the scope you set (selectors, URL globs, depth, sitemaps), respecting
  `robots.txt` when enabled.
- **Extract** — the rs-trafilatura engine isolates the main content and discards
  navigation, ads, and cookie modals, using your chosen precision/balanced/recall
  mode.
- **Output** — each page is emitted in the formats you selected, with an MD5 `hash`
  and byte length, and saved to your dataset or key-value store.

### Integrations and automation

Contextractor outputs standard JSON and Markdown, so its results drop straight into
AI and data pipelines:

- **Apify API & SDKs** — start runs, stream the dataset, and fetch key-value-store
  files programmatically; the [API tab](https://apify.com/glueo/contextractor/api)
  has ready-made JavaScript and Python client examples and an OpenAPI spec.
- **MCP server** — the same tab includes Model Context Protocol setup, so AI agents
  can call the Actor directly.
- **Scheduling & monitoring** — schedule recurring runs and monitor them from the
  Apify Console.
- **No-code connectors** — pipe results into Make, Zapier, n8n, Google Drive, Slack,
  and more via Apify's integrations.
- **LLM frameworks** — feed the extracted Markdown or JSON into LangChain,
  LlamaIndex, or a vector database such as Pinecone, Qdrant, Weaviate, or Chroma for
  retrieval-augmented generation.

### ❓ FAQ

#### Is it legal to scrape website content?

Scraping publicly available, non-personal data is generally legal in most
jurisdictions. Contextractor can honor each site's `robots.txt` (enable **Respect
robots.txt**), and you remain responsible for complying with each site's Terms of
Service and for how you use extracted content — especially copyrighted material you
intend to republish.

#### Why is some content missing or noisy?

Switch the **extraction mode**: `precision` removes more boilerplate (and may drop
borderline content), while `recall` keeps more (and may include some noise). For
pages that load content with JavaScript, add a **Wait for selector**, increase
**Wait for dynamic content**, or raise **Max scroll height** so lazy-loaded sections
appear before extraction.

#### How do I avoid getting blocked?

Enable **Proxy configuration** with proxy rotation, set a **Session pool name** to
reuse working sessions across runs, and allow **session rotations** so the crawler
switches IP and fingerprint when a block is detected.

#### How do I crawl an entire website?

Set a **Link selector** (e.g. `a[href]`) to follow links, then bound the crawl with
**include/exclude URL globs**, **Max crawl depth**, and **Max requests per crawl**. Enable
**Use sitemaps** to also pull URLs from each domain's `sitemap.xml`.

#### How do I remove duplicate pages?

Use **Deduplication**: `standard` (the default) skips pages whose canonical URL was
already extracted; `aggressive` additionally skips pages with identical extracted text;
`minimal` keeps only Crawlee's built-in URL deduplication.

#### Can I start runs from my own code or an AI agent?

Yes — the [API tab](https://apify.com/glueo/contextractor/api) carries ready-made
examples for the JavaScript and Python API clients, the Actor's OpenAPI
specification, and MCP server setup for AI agents.

#### Found a bug or have a feature request?

We respond to issues on the **Issues** tab — please open one and we'll take a
look.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to extract content from

## `crawlerType` (type: `string`):

Browser engine or HTTP client for crawling. playwright-adaptive automatically switches between browser and HTTP client per page. cheerio uses raw HTTP only (fastest, no JS).

## `renderingTypeDetectionRatio` (type: `number`):

(Adaptive only) Ratio (0–1) of pages on which the crawler runs a rendering-type detection probe. Higher values are more accurate but slower.

## `globs` (type: `array`):

Glob patterns matching URLs of pages that will be included in crawling. Setting this option allows you to customize the crawling scope. For example `https://{store,docs}.example.com/**` lets the crawler access all URLs starting with `https://store.example.com/` or `https://docs.example.com/`.

## `exclude` (type: `array`):

Glob patterns matching URLs of pages that will be excluded from crawling. Note that this affects only links found on pages, but not Start URLs, which are always crawled.

## `selector` (type: `string`):

CSS selector for links to enqueue. Leave empty to disable link enqueueing.

## `keepUrlFragment` (type: `boolean`):

URL fragments (the parts of URL after a #) are not considered when the scraper determines whether a URL has already been visited. Turn this on to treat URLs with different fragments as different pages.

## `useSitemaps` (type: `boolean`):

If enabled, the crawler looks for sitemap.xml at the root of each start URL domain and enqueues matching URLs from it in addition to link-following.

## `deduplication` (type: `string`):

Deduplication level applied on top of Crawlee's built-in URL deduplication. standard (default): skip pages whose <link rel="canonical"> was already extracted, across all handler types. aggressive: also skip pages whose extracted text content matches a previously extracted page. minimal: disable additional deduplication — only Crawlee's built-in URL dedup remains active.

## `respectRobotsTxtFile` (type: `boolean`):

If enabled, the crawler will consult the robots.txt file for each domain before crawling pages.

## `initialCookies` (type: `array`):

Cookies that will be pre-set to all pages the scraper opens. This is useful for pages that require login. The value is expected to be a JSON array of objects with `name` and `value` properties. For example:

```json
[
  {
    "name": "cookieName",
    "value": "cookieValue",
    "path": "/",
    "domain": ".example.com"
  }
]
```

You can use the [EditThisCookie](https://docs.apify.com/academy/tools/edit-this-cookie) browser extension to copy browser cookies in this format, and paste it here.

Note that the value is secret and encrypted to protect your login cookies.

## `customHttpHeaders` (type: `object`):

HTTP headers that will be added to all requests made by the crawler. This is useful for setting custom authentication headers or other headers required by the target website. The value is expected to be a JSON object with header names as keys and header values as values. For example: `{ "Authorization": "Bearer token123", "X-Custom-Header": "value" }`.

## `maxRequestsPerCrawl` (type: `integer`):

Maximum number of requests the crawler will handle. Counts handled page outcomes (successes and final failures), including start URLs and pagination pages. The crawler automatically finishes after reaching this number. 0 means unlimited.

## `maxResultsPerCrawl` (type: `integer`):

Maximum number of results that will be saved to dataset. The scraper will terminate after reaching this number. 0 means unlimited.

## `maxCrawlDepth` (type: `integer`):

Maximum link depth from Start URLs. Pages discovered further from start URLs than this limit will not be crawled. 0 means unlimited.

## `initialConcurrency` (type: `integer`):

Initial number of browser pages or HTTP clients running in parallel. Crawlee auto-scales up to maxConcurrency. 0 lets Crawlee pick the default.

## `maxConcurrency` (type: `integer`):

Maximum number of browser pages running in parallel. Kept low by default because the browser crawler cannot abort in-flight pages, so concurrency is the only hard cap on peak memory — large pages can exhaust memory at higher values. Raise it for lightweight pages or the HTTP (cheerio) crawler. This setting also avoids overloading target websites and getting blocked.

## `maxRequestRetries` (type: `integer`):

Maximum number of retries for failed requests on network, proxy, or server errors.

## `mode` (type: `string`):

Extraction mode. precision minimizes noise (may miss some content); recall maximizes content (may include noise); balanced is the default.

## `includeComments` (type: `boolean`):

Include HTML comments in the extracted text.

## `includeTables` (type: `boolean`):

Include table content in the extracted text.

## `includeImages` (type: `boolean`):

Include image alt text and captions in the extracted text.

## `includeLinks` (type: `boolean`):

Include hyperlinks in the extracted text.

## `languageCode` (type: `string`):

Filter extracted content by language code (e.g. "en"). Leave empty to accept any language.

## `save` (type: `array`):

What to save and where, as `format-destination` tokens. Format is one of `txt`, `markdown`, `json`, `html`, `original` (raw page HTML before extraction); destination is `dataset` (inline in the dataset record) or `kvs` (a blob in the key-value store). List a format twice to save it to both, e.g. `markdown-dataset markdown-kvs`. Saving `original` or `html` (large content) to the dataset is not recommended — it risks out-of-memory on large pages; prefer `kvs`.

## `datasetName` (type: `string`):

Name or ID of the dataset for storing results. Leave empty to use the default run dataset.

## `keyValueStoreName` (type: `string`):

Name or ID of the key-value store for content files. Leave empty to use the default store.

## `requestQueueName` (type: `string`):

Name of the request queue for pending URLs. Leave empty to use the default queue.

## `storeSkippedUrls` (type: `boolean`):

If enabled, pushes a dataset record for each URL skipped during crawling (excluded by globs, robots.txt, depth limit, or concurrency cap). Can produce high record volume — enable for auditing only.

## `proxyConfiguration` (type: `object`):

Enables loading websites from IP addresses in specific geographies and to circumvent blocking.

## `proxyRotation` (type: `string`):

Proxy rotation strategy. recommended automatically picks the best proxies. per-request uses a new proxy for each request. until-failure uses one proxy until it fails.

## `sessionPoolName` (type: `string`):

Name for a persistent, shared session pool. Sessions (IP + cookies) are saved under this key and reused across Actor runs. Useful when proxies are frequently blocked — previously working sessions are preferred over random ones.

## `maxSessionRotations` (type: `integer`):

Maximum number of session (IP + browser fingerprint) rotations per request on block detection. Independent of maxRequestRetries. Set to 0 to disable session rotation.

## `navigationTimeoutSecs` (type: `integer`):

Maximum time to wait for page navigation in seconds

## `blockMedia` (type: `boolean`):

Block loading of images, stylesheets, fonts (.woff), PDFs, and ZIPs. On by default: it cuts browser memory and bandwidth substantially, which helps avoid out-of-memory on large pages. Disable it (set to false) if a page needs media to render its content (e.g. image- or CSS-driven lazy loading). Has no effect when using the raw HTTP crawler type or non-Chromium browsers (Chromium only).

## `waitForSelector` (type: `string`):

Wait for this CSS selector to appear before extracting content. The request fails and is retried if the selector does not appear within the timeout. Leave empty to disable.

## `softWaitForSelector` (type: `string`):

Wait for this CSS selector to appear before extracting content. Unlike waitForSelector, the request continues even if the selector does not appear within the timeout. Leave empty to disable.

## `waitForDynamicContentSecs` (type: `integer`):

Maximum seconds to wait for dynamic page content to load after navigation. The crawler continues when the network goes idle or this timeout elapses, whichever comes first. 0 disables this wait. Also used as the timeout for waitForSelector and softWaitForSelector.

## `waitUntil` (type: `string`):

When to consider navigation finished. networkidle waits for 500ms of network silence (best for JS-heavy SPAs, slower); load waits for the load event (default, good for most articles); domcontentloaded is fastest but may fire before client-side rendering completes; commit fires when network response is received and the document has started loading.

## `headless` (type: `boolean`):

Run browser in headless mode

## `ignoreCorsAndCsp` (type: `boolean`):

Ignore Content Security Policy and Cross-Origin Resource Sharing restrictions. Enables free XHR/Fetch requests from pages.

## `closeCookieModals` (type: `boolean`):

Automatically handle cookie consent: Ghostery-based ad/tracker blocking, accepting consent walls that replace the page (e.g. consent-or-pay) via the site’s own consent manager and re-fetching the article, and removing residual consent/CMP containers before extraction.

## `maxScrollHeight` (type: `integer`):

Maximum pixels (px) to scroll down the page until all content is loaded. Setting to 0 disables scrolling.

## `userAgent` (type: `string`):

Custom User-Agent string for the browser. Leave empty to use the default browser User-Agent.

## `ignoreHttpsErrors` (type: `boolean`):

Ignore HTTPS certificate errors. Use at your own risk.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://blog.apify.com/what-is-web-scraping/"
    }
  ],
  "crawlerType": "playwright-adaptive",
  "renderingTypeDetectionRatio": 0.1,
  "globs": [],
  "exclude": [],
  "selector": "",
  "keepUrlFragment": false,
  "useSitemaps": false,
  "deduplication": "standard",
  "respectRobotsTxtFile": false,
  "initialCookies": [],
  "customHttpHeaders": {},
  "maxRequestsPerCrawl": 0,
  "maxResultsPerCrawl": 0,
  "maxCrawlDepth": 0,
  "initialConcurrency": 0,
  "maxConcurrency": 3,
  "maxRequestRetries": 3,
  "mode": "balanced",
  "includeComments": true,
  "includeTables": true,
  "includeImages": false,
  "includeLinks": true,
  "languageCode": "",
  "save": [
    "markdown-kvs"
  ],
  "storeSkippedUrls": false,
  "proxyRotation": "recommended",
  "maxSessionRotations": 10,
  "navigationTimeoutSecs": 60,
  "blockMedia": true,
  "waitForSelector": "",
  "softWaitForSelector": "",
  "waitForDynamicContentSecs": 0,
  "waitUntil": "load",
  "headless": true,
  "ignoreCorsAndCsp": false,
  "closeCookieModals": true,
  "maxScrollHeight": 5000,
  "userAgent": "",
  "ignoreHttpsErrors": false
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://blog.apify.com/what-is-web-scraping/"
        }
    ],
    "initialCookies": [],
    "customHttpHeaders": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("glueo/contextractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://blog.apify.com/what-is-web-scraping/" }],
    "initialCookies": [],
    "customHttpHeaders": {},
}

# Run the Actor and wait for it to finish
run = client.actor("glueo/contextractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://blog.apify.com/what-is-web-scraping/"
    }
  ],
  "initialCookies": [],
  "customHttpHeaders": {}
}' |
apify call glueo/contextractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=glueo/contextractor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/G1As0OV77LoHCPlCW/builds/KuW0xcbR0ZLj80gQz/openapi.json
