# Website & Docs to Markdown + RAG Chunks (`awesome_highboy/aeo-rag-ready-content-structurer`) Actor

Turn websites & docs into clean Markdown plus token-bounded, embeddings-ready RAG chunks (heading lineage + sha256) ready for Pinecone, Weaviate, Qdrant or pgvector. Optional no-hallucination field extraction and AEO mode (FAQ, answer-first, llms.txt, citations). Robots honored; ownership required.

- **URL**: https://apify.com/awesome\_highboy/aeo-rag-ready-content-structurer.md
- **Developed by:** [Adam](https://apify.com/awesome_highboy) (community)
- **Categories:** AI, Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 page processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG-Ready Content Structurer

Turn pages you own into deterministic, embeddings-ready RAG chunks — clean, hashed, and token-bounded.

### What it does

This Actor ingests a list of URLs **that you own or are authorized to crawl**, fetches each page, strips boilerplate, and converts it into clean Markdown. It then produces **RAG chunks**: semantic (heading-aware) or fixed-size chunks with full heading lineage, a deterministic token estimate, and a `sha256` content hash per chunk — ready to upsert into a vector store.

The pipeline is built around a fail-closed gate chain:

1. **Gate A — Ownership attestation.** You must attest you own/are authorized for every URL. If not true, the run is rejected before any fetch, with zero events billed.
2. **Gate B — Paid-plan only.** Default-deny until the Apify paid-plan run flag is confirmed; free/unknown plans run nothing and are not billed.
3. **Gate C — robots.txt.** Always honored and cannot be disabled. Disallowed URLs emit 0 chunks, are charged 0, and are listed in `robots_skipped_urls`.

Cleaning uses Mozilla Readability + Turndown for boilerplate-stripped Markdown. Chunking is deterministic: identical input yields byte-stable output and the same `sha256` content hash, and **no chunk exceeds `maxTokens`**.

> Note: the token count per chunk is a deterministic word-based estimate used to bound chunk size, not a tiktoken count.

### Input

Defined by `INPUT_SCHEMA.json`. Key fields:

| Field | Type | Notes |
|---|---|---|
| `source` | object | The URLs to ingest (URL list) that you own/are authorized to crawl. Includes `maxPages` (default 1000, max 50000). **Required.** |
| `ownership_attestation` | boolean | You attest you own or are authorized for every URL. Must be `true` or the run is rejected (zero billing). **Required.** |
| `render` | enum | `http` (default, cheapest) or `browser` (Playwright/Chromium for JS-heavy pages). |
| `chunking` | object | `strategy` (`semantic`|`fixed`), `maxTokens` (128–2048, default 512), `overlapTokens` (default 64). No chunk exceeds `maxTokens`. |
| `language` | string (nullable) | Optional ISO language-code hint (e.g. `en`). |

### Output

Defined by `dataset_schema.json`. Every record carries a `record_type`. The run path emits:

- **`chunk`** (RAG): `chunk_id` (position-stable key for **idempotent vector-DB upserts** — re-crawling updates the same vectors instead of duplicating them), `source_url`, `page_title`, `section_path` (heading lineage, e.g. `["Guide","Setup","Auth"]`), `heading` (immediate section title), `chunk_index`, `section_chunk_index`/`section_chunk_count` (position within the section), `chunk_text` (clean Markdown), `token_count` (≤ `maxTokens`), `char_count`, `word_count`, `overlap_prev` (chunk carries overlap from the previous one), `content_hash` (`sha256:...`), `language`, `extracted_fields`, `retrieved_at`, `render_mode`.
- **`run_summary`** (exactly one per run): `pages_requested`, `pages_fetched`, `pages_failed` (`[{url, reason}]` — pages that failed extraction are isolated here at **zero charge**, never crashing the run), `chunks_emitted`, `total_tokens`, `robots_skipped_urls`, `output_mode`.

**Chunking quality.** Code fences are kept **atomic** — a chunk never contains a half-open ` ``` ` fence; an oversized code block is split with each piece re-wrapped in its original fence + language, so every chunk is independently valid, embeddable Markdown. Splits prefer natural boundaries (paragraph → sentence → word → char), and a hard character cap (secondary to the token bound) means a whitespace-free blob (minified JS, base64) can't masquerade as a tiny chunk and silently blow an embedding model's real token limit. All output is deterministic and byte-stable across runs.

The dataset schema also defines AEO record types (`faq_pair`, `answer_block`, `llms_txt`, `citation_block`) and a structured-extraction `extracted_fields` shape, with prebuilt dataset **views** (*RAG chunks*, *AEO assets*, *Run summary*). These AEO/extraction outputs are reserved in the schema but are **not produced by the current run path** — the Actor currently emits RAG `chunk` records plus one `run_summary`.

The *RAG chunks* view is ready to upsert into Pinecone / Weaviate / Qdrant / pgvector.

### Pricing

Pay-Per-Event. You are billed only for what actually runs (after the gates), via `Actor.charge()`:

| Event | Price | When charged |
|---|---|---|
| `actor_run_start` | **$0.05** | Once per run, only after the ownership + paid-plan + pilot gates pass. Never on a rejected or free-plan run. |
| `page_processed` | **$0.003** | Per page successfully fetched + converted + chunked. Failed pages and robots-disallowed URLs charge $0. |
| `field_extracted` | **$0.005** | Per `(page × requested field)` pair returning a non-null value. Reserved for structured extraction, which is not active in the current run path, so this event is not charged today. |

The developer keeps 80% and Apify keeps 20% (standard Apify 80/20 split).

**Example run cost** — 100 owned pages, RAG mode:
`$0.05 + (100 × $0.003) = $0.35`.

### Why this Actor

- **Deterministic & idempotent.** Cleaning and chunking are pure and byte-stable: identical input produces identical chunks and identical `sha256` content hashes — safe re-runs, safe vector-store de-duplication.
- **Ownership-gated and robots-compliant by design.** A mandatory ownership/authorization attestation rejects unauthorized runs before any fetch (zero billing), and `robots.txt` is forced on and cannot be disabled.
- **Paid-plan only, fail-closed billing.** Default-deny until a paid plan is confirmed; failed and robots-disallowed pages charge $0.
- **Embeddings-ready output with token bounds.** Chunks carry heading lineage and a token estimate that never exceeds your `maxTokens`, in a schema with a prebuilt view for Pinecone / Weaviate / Qdrant / pgvector.

### About

This Actor is AI-authored and operated under the publisher's LLC. `Actor.charge()` is the only billing path and it **bills the customer only** — the Actor has no payout or money-out capability; revenue settlement is handled entirely by Apify's monetization rail.

# Actor input Schema

## `source` (type: `object`):

The permissioned URLs to ingest: a URL list OR a sitemap you own / are authorized to crawl.

## `ownership_attestation` (type: `boolean`):

You attest that you OWN or are AUTHORIZED to ingest every URL/sitemap above. The run is REJECTED before any fetch with zero events billed if this is not true (M5 Gate A / AC-OwnershipGate). This is the legality keystone (no third-party ToS violation, no anti-bot arms race).

## `render` (type: `string`):

http (default; got-scraping, cheapest) or browser (Playwright/Chromium for JS-heavy pages — higher per-page compute).

## `outputMode` (type: `string`):

rag (default — per-chunk Markdown + embeddings-ready JSON) or aeo (Answer-Engine assets: FAQ pairs / answer-first blocks / llms.txt / citation-structured). Same ownership/robots/paid-plan/QA gates apply to both.

## `chunking` (type: `object`):

strategy: semantic|fixed; maxTokens: int 128-2048 (default 512); overlapTokens: int (default 64). No emitted chunk exceeds maxTokens (AC-ChunkBound).

## `extractSchema` (type: `object`):

Optional per-page extraction. fields: \[{name, type, instruction}]. Values are NEVER hallucinated - a field with no answer returns null. field\_extracted is charged only per (page x field) pair returning a non-null value.

## `aeo` (type: `object`):

assets: which AEO record kinds to emit — any of \['faq\_pair','answer\_block','llms\_txt','citation\_block']. AEO answers obey the no-hallucination contract (an absent answer is OMITTED, never invented).

## `model` (type: `string`):

haiku (default; claude-haiku-4-5, cheapest) or sonnet (claude-sonnet-4-6, higher-fidelity extraction). All Claude calls route via M2 -> M7 LiteLLM gateway (the ONLY Anthropic egress); never raw Anthropic. Opus is VERIFY-only at build time and is NOT a runtime option.

## `respectRobotsTxt` (type: `boolean`):

robots.txt is ALWAYS honored. Disallowed URLs are listed in robots\_skipped\_urls and emit 0 chunks (0 charge). This cannot be set false via the API (M5 Gate C / AC-Robots); the value is forced true regardless of input (EU AI Act machine-readable opt-out compliance).

## `language` (type: `string`):

Optional ISO language-code hint (e.g. 'en').

## Actor input object example

```json
{
  "source": {
    "mode": "url_list",
    "urls": [
      "https://example.com/"
    ],
    "maxPages": 1000
  },
  "ownership_attestation": true,
  "render": "http",
  "outputMode": "rag",
  "chunking": {
    "strategy": "semantic",
    "maxTokens": 512,
    "overlapTokens": 64
  },
  "extractSchema": {
    "fields": [
      {
        "name": "product_name",
        "type": "string",
        "instruction": "The product name stated on the page; null if absent."
      }
    ]
  },
  "aeo": {
    "assets": [
      "faq_pair",
      "answer_block",
      "llms_txt"
    ]
  },
  "model": "haiku",
  "respectRobotsTxt": true
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "source": {
        "mode": "url_list",
        "urls": [
            "https://example.com/"
        ],
        "maxPages": 1000
    },
    "ownership_attestation": true,
    "chunking": {
        "strategy": "semantic",
        "maxTokens": 512,
        "overlapTokens": 64
    },
    "extractSchema": {
        "fields": [
            {
                "name": "product_name",
                "type": "string",
                "instruction": "The product name stated on the page; null if absent."
            }
        ]
    },
    "aeo": {
        "assets": [
            "faq_pair",
            "answer_block",
            "llms_txt"
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("awesome_highboy/aeo-rag-ready-content-structurer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "source": {
        "mode": "url_list",
        "urls": ["https://example.com/"],
        "maxPages": 1000,
    },
    "ownership_attestation": True,
    "chunking": {
        "strategy": "semantic",
        "maxTokens": 512,
        "overlapTokens": 64,
    },
    "extractSchema": { "fields": [{
                "name": "product_name",
                "type": "string",
                "instruction": "The product name stated on the page; null if absent.",
            }] },
    "aeo": { "assets": [
            "faq_pair",
            "answer_block",
            "llms_txt",
        ] },
}

# Run the Actor and wait for it to finish
run = client.actor("awesome_highboy/aeo-rag-ready-content-structurer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "source": {
    "mode": "url_list",
    "urls": [
      "https://example.com/"
    ],
    "maxPages": 1000
  },
  "ownership_attestation": true,
  "chunking": {
    "strategy": "semantic",
    "maxTokens": 512,
    "overlapTokens": 64
  },
  "extractSchema": {
    "fields": [
      {
        "name": "product_name",
        "type": "string",
        "instruction": "The product name stated on the page; null if absent."
      }
    ]
  },
  "aeo": {
    "assets": [
      "faq_pair",
      "answer_block",
      "llms_txt"
    ]
  }
}' |
apify call awesome_highboy/aeo-rag-ready-content-structurer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=awesome_highboy/aeo-rag-ready-content-structurer",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tucjW3FtVROKYpYa4/builds/i9AkwjiLWnWzkO30m/openapi.json