Website Content Crawler avatar

Website Content Crawler

Pricing

from $0.13 / 1,000 page crawleds

Try for free
Go to Apify Store
Website Content Crawler

Website Content Crawler

Try for free

Crawl any website and extract clean, LLM-ready Markdown for RAG pipelines, vector databases and AI agents. Boilerplate, navigation and cookie banners removed. No browser, no login.

Pricing

from $0.13 / 1,000 page crawleds

Rating

0.0

(0)

Developer

MYM

MYM

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

2 days ago

Last modified

Categories

AI

Developer tools

Share

Website Content Crawler — any site to clean Markdown for RAG & LLMs (No API Key)

Apify Actor No API Key required No Login needed No cookies needed Export formats MCP AI agents Pay per result

Turn any website into clean, LLM-ready Markdown. Give it one or more start URLs and it crawls the site breadth-first, strips the navigation, footers, cookie banners and share widgets, and returns just the content — as Markdown, plain text, or cleaned HTML. Built for RAG pipelines, vector databases, AI agents and fine-tuning datasets, where paying tokens for <div> soup is pure waste. No API key, no login, no browser to configure. Every page also carries its title, description, language, heading outline, word count and crawl depth, so you can chunk and filter intelligently. Export to JSON, CSV, Excel, or XML.

Point it at a docs site and press Start — you will have a chunk-ready corpus before your coffee cools.

Table of contents

What can the Website Content Crawler do? {#what-can-it-do}

  • 🌐 Crawl a whole site or just one section — breadth-first from your start URLs.
  • 📝 Clean Markdown output — headings, lists, links, images and code blocks preserved.
  • 🧹 Boilerplate removed — nav, footer, sidebar, cookie banners, share widgets, breadcrumbs, pagination.
  • 🎯 Glob filters — includeUrlGlobs / excludeUrlGlobs to crawl exactly the section you want.
  • 🧭 Scope control — same domain, include subdomains, or follow external links.
  • 🪆 Depth and page caps — maxCrawlDepth and maxPages keep runtime and cost predictable.
  • 🧱 Heading outline per page — chunk on real section boundaries, not arbitrary character counts.
  • 🚫 Duplicate-proof — URL normalisation strips tracking params, fragments and scheme duplicates.
  • 🤖 robots.txt respected by default.
  • ⚡ Parallel fetching with no browser to boot.
  • 📏 minWordCount drops thin pages before they are stored or billed.
  • 📤 Export to JSON, CSV, Excel, or XML, or pull live via API.

Go beyond the limits of the official the web API: no OAuth app, no app review, no quota juggling — paste your input and collect publicly available data.

Who uses the Website Content Crawler? {#who-uses-it}

  • 🤖 AI engineers building RAG pipelines that need a clean corpus, not HTML.
  • 🧠 LLM app developers feeding product docs into a support chatbot.
  • 🗄️ Data teams loading site content into Pinecone, Weaviate, Qdrant or pgvector.
  • 📚 Knowledge-base owners keeping an internal assistant in sync with the public docs.
  • 🔍 SEO and content analysts auditing page copy, headings and word counts at scale.
  • 🏢 Competitive researchers tracking how a rival's messaging changes over time.
  • 🎓 Researchers assembling text corpora from public web sources.

What data can I scrape? {#what-data}

FieldTypeDescription
urlstringFinal URL of the page after redirects.
canonicalUrlstringCanonical URL declared by the page, when present.
titlestringPage title, taken from og:title and falling back to <title>.
descriptionstringMeta description or og:description.
markdownstringMain content as clean Markdown with headings, lists, links and code blocks preserved.
textstringMain content as plain prose (when output format is text).
htmlstringCleaned main-content HTML (when output format is html).
wordCountintegerWord count of the extracted content, after boilerplate removal.
headingsarrayHeading outline as { level, text } pairs - handy for chunking long pages.
langstringLanguage declared on the <html> element.
authorstringAuthor from meta tags, when the page declares one.
publishedAtstringPublish date from article metadata, when present.
imagestringog:image URL, when present.
depthintegerLink hops from the start URL (0 = the start URL itself).
statusCodeintegerHTTP status code of the response.
contentTypestringContent-Type header of the response.
renderedEmptybooleantrue when the page returned almost no text, which usually means it renders content with JavaScript.
linksarrayEvery URL found on the page (only when Include outgoing links is on).
crawledAtstringUTC timestamp of the fetch.

Fields reflect what the web exposes publicly for that item. Values that do not exist for a given item are returned as null rather than dropped, so your schema stays stable.

How to scrape website content {#how-to}

  1. Paste one or more start URLs. A homepage crawls the whole site; a section URL plus a glob crawls just that part.
  2. Set maxPages. This is your budget cap — you are billed per stored page.
  3. Set maxCrawlDepth. 1 is usually enough for a blog index; docs sites often need 3.
  4. Pick a scope. domain stays on the hostname, subdomains includes docs./blog., anywhere follows external links.
  5. (Optional) Add glob filters such as **/docs/** to include, or **/tag/** to exclude.
  6. (Optional) Set minWordCount to drop thin pages before they are stored or billed.
  7. Choose an output format — Markdown for LLM work, text for plain prose, HTML if you post-process yourself.
  8. Press Start, then export as JSON, CSV, Excel, or XML — or pull the dataset from the API.

No login, no cookies, no proxy configuration. Start with one input to see the output shape, then scale the same run — the fields stay identical, so code written against a test run keeps working.

Input example {#input-example}

{
  "startUrls": [
    "https://quotes.toscrape.com/"
  ],
  "maxPages": 50,
  "maxCrawlDepth": 2,
  "crawlScope": "domain",
  "outputFormat": "markdown",
  "minWordCount": 25,
  "respectRobotsTxt": true
}

crawlScope accepts domain, subdomains or anywhere. outputFormat accepts markdown, text or html. Glob patterns use * within a path segment and ** across segments, so **/docs/** matches any depth. maxPages is a hard cap and doubles as your budget limit.

Output example {#output-example}

{
  "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
  "canonicalUrl": null,
  "title": "Quotes to Scrape",
  "description": null,
  "markdown": "# Albert Einstein\n\nBorn: March 14, 1879 in Ulm, Germany\n\n## Description\n\nIn 1879, Albert Einstein was born in Ulm, Germany. He completed his Ph.D. at the University of Zurich by 1909...",
  "wordCount": 619,
  "headings": [
    { "level": 1, "text": "Albert Einstein" },
    { "level": 2, "text": "Description" }
  ],
  "lang": "en",
  "author": null,
  "publishedAt": null,
  "image": null,
  "depth": 1,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "renderedEmpty": false,
  "crawledAt": "2026-07-30T22:41:07.918Z"
}

Every run produces a clean dataset you can page through in the Apify Console or pull via API. Export to JSON, CSV, Excel, or XML in one click.

No browser, by design. Pages are fetched as HTML and converted directly, which is why a page costs a fraction of what a headless-browser crawl costs. Sites that render content purely client-side are reported with renderedEmpty: true instead of being stored as blank pages.

How much will it cost? {#pricing}

Simple, predictable pay-per-result pricing — you pay for the pages you actually receive, not for compute time.

  • 💸 Pay per page — budget is just pages × rate, so you can estimate spend up front.
  • 📉 Volume discounts — Bronze, Silver, Gold, Platinum, and Diamond Apify plans get progressively lower rates.
  • 🆓 Free to try — new Apify accounts include monthly free usage, so your first run costs nothing.
  • 🚫 No hidden fees — no proxy add-on, no per-request API charge, no minimum spend.

See the live rate on the Apify store page.

Tips {#tips}

  • 🎯 Start with maxPages: 10 to see the shape of the output, then scale the same run.
  • 🧱 Chunk on headings, not character counts — retrieval quality improves noticeably.
  • 📏 Set minWordCount: 25 to drop login, tag and pagination stubs for free.
  • 🎛️ Use includeUrlGlobs instead of a big maxPages when you only need one section — it is cheaper and faster.
  • 🕵️ Watch renderedEmpty — a cluster of true values means the site renders client-side and needs a different approach.
  • 🧭 Prefer domain scope unless you deliberately want to leave the site; anywhere can grow without bound.
  • ⏰ Schedule a weekly re-crawl to keep a knowledge base current, and diff the Markdown to see what changed.
  • 🐢 Lower maxConcurrency if the target site starts returning errors — some hosts rate-limit aggressively.
  • 🔗 Turn on includeLinks when you need a site map or link graph rather than just content.

Want other the web data? {#want-other-data}

Building a multi-platform dataset? The same no-login, pay-per-result approach covers the other networks:

Browse everything on the myagizm store profile.

Use the data with AI agents (MCP) {#mcp}

The Website Content Crawler is available through the Apify MCP server, so AI assistants can call it as a tool and pull website content on demand.

  • 🧩 Claude — add the Apify MCP server and let Claude fetch and analyse the data.
  • 🤖 ChatGPT — use the actor as a tool inside research and reporting workflows.
  • 🖊️ Cursor — pull structured results straight into your coding sessions.

Common AI-agent recipes:

  • 🧠 "Crawl our docs site and summarise the getting-started flow" — the agent crawls, reads the Markdown, and answers.
  • 📚 "Build me a knowledge base from these three URLs" — one run per site, straight into your vector store.
  • 🔎 "What changed on this competitor's pricing page since last week?" — schedule a crawl and diff the Markdown.
  • 🧾 "Extract every heading from this handbook" — the headings array gives the outline without extra parsing.

Integrations & API {#integrations}

Connect the actor to the tools you already use: Make, Zapier, n8n, Slack, Airbyte, Gumloop, CrewAI, IFTTT, Lindy, GitHub, Google Sheets, and Google Drive — plus webhooks, Apify Schedules, and the MCP server.

Run synchronously and get the dataset (Apify API)

curl -X POST "https://api.apify.com/v2/acts/myagizm~website-content-crawler/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://quotes.toscrape.com/"], "maxPages": 50, "maxCrawlDepth": 2, "crawlScope": "domain", "outputFormat": "markdown", "minWordCount": 25, "respectRobotsTxt": true}'

Start a run asynchronously

curl -X POST "https://api.apify.com/v2/acts/myagizm~website-content-crawler/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://quotes.toscrape.com/"], "maxPages": 50, "maxCrawlDepth": 2, "crawlScope": "domain", "outputFormat": "markdown", "minWordCount": 25, "respectRobotsTxt": true}'

Fetch dataset items later

$curl "https://api.apify.com/v2/acts/myagizm~website-content-crawler/runs/last/dataset/items?token=YOUR_APIFY_TOKEN&format=csv"

Change format to json, csv, xlsx, or xml to export in your preferred format. The Apify Console also auto-generates ready-to-paste JavaScript, Python, and CLI snippets plus an OpenAPI spec for this actor.

Support {#support}

Questions, feature requests, or something not looking right? Open an issue on the actor's Issues tab. Please include the input you used so we can reproduce and fix it quickly.

FAQ {#faq}

Do I need an API key, login, or cookies? No. Point it at a public URL and press Start. There is nothing to authorise and no account to connect.

What exactly is "clean" about the Markdown? Scripts, styles, <nav>, <footer>, <aside>, forms, cookie banners, share widgets, breadcrumbs, tag clouds and pagination blocks are removed first. Then the densest of <main> / <article> / [role=main] / #content wins. What is left is the part of the page a reader would call the content - which is exactly what you want in a vector database.

Why Markdown instead of raw HTML? Because embeddings and LLM context windows are paid for by the token. Raw HTML wastes most of them on markup. Markdown keeps the structure that actually carries meaning - headings, lists, links, code - at a fraction of the size.

Can it crawl JavaScript-rendered sites? It fetches HTML without running a browser, so content that only appears after client-side rendering will not be captured. When that happens the item is returned with renderedEmpty: true and a low wordCount, so you can detect and route those URLs instead of silently storing blank pages. Many modern frameworks server-render by default and work fine.

How do I keep the crawl from wandering off the site? crawlScope defaults to domain, which never leaves the exact hostname. Use subdomains to include docs. and blog. style hosts, and anywhere only when you deliberately want external links followed.

How do I crawl only one section of a site? Set includeUrlGlobs to something like **/docs/**. Only URLs matching one of your patterns are crawled. excludeUrlGlobs is applied afterwards, so you can carve out **/tag/** or **/*.pdf.

What is the difference between maxPages and maxCrawlDepth? maxPages is a hard budget cap on stored pages. maxCrawlDepth limits how far from the start URL the crawler may travel. A deep site with maxCrawlDepth: 1 yields few pages even with a high maxPages; a shallow site can hit maxPages at depth 1.

Does it respect robots.txt? Yes, by default. Disallow rules are read per origin and cached for the run. You can turn this off for sites you own or are authorised to crawl.

Will it crawl the same page twice? No. URLs are normalised before queueing: the fragment is dropped, tracking parameters (utm_*, gclid, fbclid, …) are stripped, query parameters are sorted, trailing slashes are unified, and http:// links on a host you seeded over https:// are upgraded. That stops the most common duplicate-billing traps.

How do I skip thin pages like login and tag stubs? Set minWordCount. Pages under the threshold are dropped before they are stored, so you are not billed for them.

Can I get a link graph or site map? Turn on Include outgoing links and every item gains a links array with all URLs found on that page.

How fast is it? Fetching is parallel and there is no browser to boot. A small documentation site typically finishes in seconds. Lower maxConcurrency if the target rate-limits you.

How do I chunk the output for a vector database? Use headings to split on section boundaries rather than a fixed character count - retrieval quality is noticeably better when chunks follow the document's own structure.

Is the output stable enough to build on? Yes. Fields that do not exist for a page are returned as null rather than omitted, so your schema stays the same across runs and sites.

Can I schedule recurring crawls? Yes. Use Apify Schedules to re-crawl daily or weekly and keep a knowledge base fresh.

What does it cost? You pay per stored page. maxPages is therefore also your budget cap, and pages filtered out by minWordCount are not billed.

Can AI agents call it directly? Yes, through the Apify MCP server - see the MCP section above.

website content crawler · website crawler for RAG · crawl website to markdown · html to markdown api · website scraper for LLM · web crawler for AI training data · scrape website text content · extract article text from url · website to markdown converter · crawl docs site for chatbot · knowledge base crawler · vector database ingestion crawler · RAG pipeline web crawler · site crawler markdown export · scrape entire website content · web page text extractor · boilerplate removal scraper · readability api alternative · crawl blog posts to markdown · llm context web scraper · documentation scraper · sitemap content crawler · bulk url content extractor · web content api no key · scrape website for embeddings · crawl subdomain content · markdown scraper for chatgpt · web crawler for agents · clean text extraction from html · website corpus builder

Disclaimer {#disclaimer}

This is an independent tool and is not affiliated with, endorsed by, or sponsored by any website you crawl or its operators. All product names, logos, and brands are property of their respective owners.

The actor collects only publicly available data — the same content any visitor can see without logging in. It never logs in, never bypasses access controls, and never touches private or protected content. You are responsible for how you use the data: if it contains personal data, follow GDPR, CCPA, and other applicable regulations, and respect the platform's terms.

Resources {#resources}

中文说明 —— 网站内容抓取工具：将任意网站转为适合 RAG 与大模型的干净 Markdown

把任意网站转换成适合大模型使用的干净 Markdown。 只需给出一个或多个起始地址，它会广度优先地抓取整站，移除导航、页脚、Cookie 提示与分享控件，只返回正文 —— 可选 Markdown、纯文本或清洗后的 HTML。专为 RAG 流水线、向量数据库、AI 智能体与微调数据集打造：把 token 花在 <div> 标签上纯属浪费。无需 API Key、无需登录、无需配置浏览器。每个页面还附带标题、描述、语言、标题大纲、词数与抓取深度，方便你智能切块与过滤。支持导出 JSON、CSV、Excel、XML。

两分钟内免费拿到第一批数据 —— 填入内容、点击 Start 即可。无需配置，试用不需要信用卡。

能做什么

  • 🌐 抓取整站或指定板块 —— 从起始地址广度优先爬取。
  • 📝 干净的 Markdown 输出 —— 保留标题、列表、链接、图片与代码块。
  • 🧹 自动去除模板内容 —— 导航、页脚、侧栏、Cookie 提示、分享控件、面包屑与分页。
  • 🎯 Glob 过滤 —— 用 includeUrlGlobs / excludeUrlGlobs 精确圈定要抓的板块。
  • 🧭 范围控制 —— 仅同域名、包含子域名，或跟随外链。
  • 🪆 深度与页数上限 —— maxCrawlDepth 与 maxPages 让耗时与成本可预期。
  • 🧱 逐页标题大纲 —— 按真实章节边界切块，而不是固定字符数。
  • 🚫 天然去重 —— 地址规范化会剥离跟踪参数、锚点与协议重复。
  • 🤖 默认遵守 robots.txt。
  • ⚡ 并行抓取，无需启动浏览器。
  • 📏 minWordCount 在存储与计费前丢弃内容过少的页面。
  • 📤 导出 JSON、CSV、Excel、XML，或通过 API 实时获取。

无需 OAuth 应用、无需应用审核、无需配额管理 —— 填入内容即可采集公开数据。

谁在用

  • 🤖 AI 工程师 —— 为 RAG 流水线准备干净语料，而不是一堆 HTML。
  • 🧠 大模型应用开发者 —— 把产品文档喂给客服机器人。
  • 🗄️ 数据团队 —— 将站点内容导入 Pinecone、Weaviate、Qdrant 或 pgvector。
  • 📚 知识库维护者 —— 让内部助手与公开文档保持同步。
  • 🔍 SEO 与内容分析师 —— 批量审计页面文案、标题结构与词数。
  • 🏢 竞品研究人员 —— 追踪对手的信息表达随时间的变化。
  • 🎓 研究人员 —— 从公开网页构建文本语料库。

可以抓取哪些数据

字段类型说明
urlstring重定向后的最终页面地址。
canonicalUrlstring页面声明的规范地址（如果有）。
titlestring页面标题，优先取 og:title，否则取 <title>。
descriptionstringmeta description 或 og:description。
markdownstring正文内容，转换为保留标题、列表、链接与代码块的干净 Markdown。
textstring纯文本正文（输出格式选 text 时）。
htmlstring清洗后的正文 HTML（输出格式选 html 时）。
wordCountinteger去除模板内容后的正文词数。
headingsarray标题大纲，{ level, text } 结构，便于长页面切块。
langstring<html> 元素上声明的语言。
authorstringmeta 标签中的作者（如果有）。
publishedAtstring文章元数据中的发布时间（如果有）。
imagestringog:image 地址（如果有）。
depthinteger距离起始地址的跳数（0 表示起始地址本身）。
statusCodeintegerHTTP 状态码。
contentTypestring响应的 Content-Type。
renderedEmptyboolean为 true 表示页面几乎没有文本，通常说明内容由 JavaScript 渲染。
linksarray页面上发现的全部链接（仅在开启 Include outgoing links 时返回）。
crawledAtstring抓取时间（UTC）。

字段取决于平台对外公开了什么。某个条目没有的值会返回 null 而不是直接省略，因此你的数据结构始终稳定。

使用步骤

  1. 填入一个或多个起始地址。 填首页可抓全站；填板块地址并配合 glob 可只抓该板块。
  2. 设置 maxPages。 这就是预算上限 —— 按存储页数计费。
  3. 设置 maxCrawlDepth。 博客列表通常 1 就够，文档站常需要 3。
  4. 选择抓取范围。 domain 仅同主机名，subdomains 包含 docs./blog.，anywhere 会跟随外链。
  5. (可选) 添加 glob 过滤，如包含 **/docs/**，或排除 **/tag/**。
  6. (可选) 设置 minWordCount，在存储与计费前丢弃内容过少的页面。
  7. 选择输出格式 —— 大模型场景选 Markdown，纯文本选 text，需自行后处理选 HTML。
  8. 点击 Start，然后导出 JSON、CSV、Excel、XML，或通过 API 拉取数据集。

输入示例

{
  "startUrls": [
    "https://quotes.toscrape.com/"
  ],
  "maxPages": 50,
  "maxCrawlDepth": 2,
  "crawlScope": "domain",
  "outputFormat": "markdown",
  "minWordCount": 25,
  "respectRobotsTxt": true
}

输出示例

{
  "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
  "canonicalUrl": null,
  "title": "Quotes to Scrape",
  "description": null,
  "markdown": "# Albert Einstein\n\nBorn: March 14, 1879 in Ulm, Germany\n\n## Description\n\nIn 1879, Albert Einstein was born in Ulm, Germany. He completed his Ph.D. at the University of Zurich by 1909...",
  "wordCount": 619,
  "headings": [
    { "level": 1, "text": "Albert Einstein" },
    { "level": 2, "text": "Description" }
  ],
  "lang": "en",
  "author": null,
  "publishedAt": null,
  "image": null,
  "depth": 1,
  "statusCode": 200,
  "contentType": "text/html; charset=utf-8",
  "renderedEmpty": false,
  "crawledAt": "2026-07-30T22:41:07.918Z"
}

刻意不使用浏览器。 页面以 HTML 抓取后直接转换，因此单页成本远低于无头浏览器方案。纯客户端渲染的站点会被标记为 renderedEmpty: true，而不是把空白页存下来。

价格

采用按结果计费：只为真正拿到的数据付费，不按运行时间计费。

  • 💸 成本 = 结果数量 × 单价，预算可提前算清。
  • 📉 Apify 的 Bronze / Silver / Gold / Platinum / Diamond 套餐享阶梯折扣。
  • 🆓 新账号自带每月免费额度，第一次运行不花钱。
  • 🚫 无代理附加费、无单次请求费、无最低消费。

实时价格见 Apify 商店页面。

使用建议

  • 🎯 先用 maxPages: 10 试跑，确认输出结构后再放大同一个任务。
  • 🧱 按 headings 切块，而不是按字符数 —— 检索质量提升明显。
  • 📏 设置 minWordCount: 25，免费过滤掉登录页、标签页与分页占位页。
  • 🎛️ 只需要某个板块时用 includeUrlGlobs，比调大 maxPages 更省更快。
  • 🕵️ 关注 renderedEmpty —— 大量为 true 说明该站为客户端渲染，需要换方案。
  • 🧭 优先使用 domain 范围，除非确实要离开本站；anywhere 可能无限扩散。
  • ⏰ 设置每周重爬保持知识库最新，并对比 Markdown 查看变化。
  • 🐢 目标站点报错时调低 maxConcurrency，部分主机限速较严。
  • 🔗 需要站点地图或链接图谱时开启 includeLinks。

其他网页数据

需要多平台数据？以下工具采用同样的免登录、按结果计费方式：

全部工具见 myagizm 商店主页。

配合 AI 智能体使用（MCP）

本 Actor 已接入 Apify MCP 服务器，Claude、ChatGPT、Cursor 等助手可以把它当作工具直接调用，用自然语言提问即可拿到结构化结果，无需写胶水代码。

集成与 API

支持 Make、Zapier、n8n、Slack、Airbyte、Gumloop、CrewAI、IFTTT、Lindy、GitHub、Google Sheets、Google Drive，以及 Webhook、定时任务与 MCP 服务器。

curl -X POST "https://api.apify.com/v2/acts/myagizm~website-content-crawler/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://quotes.toscrape.com/"], "maxPages": 50, "maxCrawlDepth": 2, "crawlScope": "domain", "outputFormat": "markdown", "minWordCount": 25, "respectRobotsTxt": true}'

把 format 改成 json、csv、xlsx 或 xml 即可导出不同格式。

常见问题

需要 API Key、登录或 Cookie 吗？ 不需要。填入公开网址点击 Start 即可，无需授权，也不用绑定任何账号。

Markdown 的"干净"具体指什么？ 先移除脚本、样式、<nav>、<footer>、<aside>、表单、Cookie 提示、分享控件、面包屑、标签云与分页区块，再从 <main> / <article> / [role=main] / #content 中选出文字最密集的一块。剩下的就是读者眼中的正文 —— 也正是向量库需要的内容。

为什么用 Markdown 而不是原始 HTML？ 因为向量化和大模型上下文都是按 token 计费的，原始 HTML 会把额度浪费在标签上。Markdown 用很小的体积保留了真正承载语义的结构：标题、列表、链接与代码。

能抓取 JavaScript 渲染的网站吗？ 本工具直接抓取 HTML，不启动浏览器，因此纯客户端渲染的内容无法获取。遇到这种页面时会返回 renderedEmpty: true 且 wordCount 很低，方便你识别并单独处理，而不是把空白页悄悄存进库里。多数现代框架默认服务端渲染，可正常抓取。

如何避免爬到站外？ crawlScope 默认为 domain，只留在完全相同的主机名内；subdomains 会包含 docs.、blog. 等子域；只有确实需要跟随外链时才用 anywhere。

只想抓网站的某个板块怎么办？ 把 includeUrlGlobs 设为 **/docs/** 之类的模式，只有匹配的地址才会被抓取；excludeUrlGlobs 随后生效，可排除 **/tag/** 或 **/*.pdf。

maxPages 和 maxCrawlDepth 有什么区别？ maxPages 是存储页数的硬性预算上限；maxCrawlDepth 限制离起始地址的跳数。层级很深的站点即使 maxPages 很大，maxCrawlDepth: 1 也只能拿到很少页面。

遵守 robots.txt 吗？ 默认遵守，按来源读取并在本次运行内缓存。对自有或已获授权的站点可以关闭。

同一个页面会被抓两次吗？ 不会。入队前会先规范化地址：去掉锚点、剥离 utm_*、gclid、fbclid 等跟踪参数、排序查询参数、统一结尾斜杠，并把已用 https:// 作为种子的主机上的 http:// 链接升级为 https，避免重复计费。

如何跳过登录页、标签页这类内容很少的页面？ 设置 minWordCount，低于阈值的页面在存储前就被丢弃，不计费。

能拿到链接图谱或站点地图吗？ 开启 Include outgoing links，每条结果都会带上包含该页全部链接的 links 数组。

速度如何？ 并行抓取且无需启动浏览器，小型文档站通常几秒完成。若目标站点限速，可调低 maxConcurrency。

如何为向量库切块？ 建议用 headings 按章节边界切分，而不是固定字符数 —— 顺着文档自身结构切块，检索质量明显更好。

输出结构稳定吗？ 稳定。页面没有的字段返回 null 而不是省略，因此不同站点、不同批次的结构保持一致。

可以定时重复抓取吗？ 可以，使用 Apify Schedules 按天或按周重新抓取，保持知识库最新。

费用怎么算？ 按存储的页数计费，因此 maxPages 同时也是预算上限；被 minWordCount 过滤掉的页面不计费。

AI 智能体可以直接调用吗？ 可以，通过 Apify MCP 服务器调用，详见上文 MCP 章节。

免责声明

本工具为独立第三方工具，与 你所抓取的网站及其运营方 无任何隶属、认可或赞助关系，所有商标归各自所有者。

本 Actor 只采集公开可见的数据 —— 即任何未登录访客都能看到的内容。它不登录、不绕过访问控制、不获取私密内容。数据使用的合规责任由使用者承担：如涉及个人数据，请遵守 GDPR、CCPA 等法规与平台条款。

相关搜索

网站内容抓取 · RAG 网页爬虫 · 网站转 Markdown · HTML 转 Markdown 接口 · 大模型语料抓取 · 网页正文提取 · 知识库爬虫 · 向量数据库入库爬虫 · 文档站抓取 · 网站全站内容抓取 · 网页文本提取工具 · 去除模板内容抓取 · 博客文章转 Markdown · 大模型上下文抓取 · 站点地图内容抓取 · 批量网址正文提取 · 免密钥网页内容接口 · 嵌入向量网页抓取 · 子域名内容抓取 · AI 智能体网页爬虫 · 网页语料库构建

You might also like

Website Content Crawler avatar

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

User avatar

Apify

144K

4.5

(213)

FW

Fast Website Content Crawler

6sigmag/fast-website-content-crawler

A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

User avatar

David

4.3K

4.9

(7)

🔥 FireScrape AI Website Content Markdown Scraper avatar

🔥 FireScrape AI Website Content Markdown Scraper

mohamedgb00714/fireScraper-AI-Website-Content-Markdown-Scraper

Advanced web scraper powered by Crawlee and Puppeteer — extracts website content, converts it to Markdown, and structures it for LLM training datasets.

User avatar

mohamed el hadi msaid

303

1.9

(2)

Website Content Crawler avatar

Website Content Crawler

parseforge/website-content-crawler

Crawl any website and pull clean Markdown content ready for AI! Follow links across a whole domain and extract page text, titles, headings, images, and metadata. Perfect for building RAG pipelines, training datasets, knowledge bases, and vector databases. Start crawling content in minutes!

User avatar

ParseForge

72

DW

Deep Website Content Crawler

6sigmag/deep-website-content-crawler

Scrape Failed Killer! A high-performance web scraper that rapidly extracts and analyzes content from multiple websites simultaneously. Perfect for competitive research, content aggregation, and website structure analysis.

User avatar

David

1.1K

3.0

(1)

Website Content Crawler avatar

Website Content Crawler

mikolabs/website-content-crawler

Deep-crawl websites to extract clean text, Markdown, or HTML for AI/LLM apps, RAG pipelines, and vector databases. Supports adaptive crawling, HTML cleaning, file downloads, and structured dataset output. Easily integrates with LangChain, LlamaIndex, and other LLM tools.

User avatar

Mikolabs

30

5.0

(1)

🧪High-Volume Website Content & Media Scraper avatar

🧪High-Volume Website Content & Media Scraper

caring_dizi/blog-content-scraper-fixed

🧪Crawling Done Right! Let me now what you think, what or where or how i can improve my actor, and i am all for constructive criticism. So please message if you have any questions. Enjoy and have a good day.

User avatar

Jeff Halverson

150

5.0

(2)

Website Content Crawler avatar

Website Content Crawler

alizarin_refrigerator-owner/website-crawler

Crawl websites for SEO audits. Extracts HTML, title, meta tags, headings, links, & text content from pages. Automatic sitemap detection & parsing Extracts metadata (title, description, OG tags) Heading structure (H1, H2, H3) Internal & external link analysis Image extraction w/alt text Word count

User avatar

The Howlers

124

Website Content Text Extractor avatar

Website Content Text Extractor

smart-digital/website-content-text-extractor

Extract visible text content from websites as structured JSON blocks. Supports multi-URL batch processing, header/footer/cookie exclusion, and optional form extraction. Perfect for content analysis and translation workflows.

SD

My Smart Digital

83

5.0

(1)

WC

Website Content Crawler

jasondev/website-content-crawler

A powerful web crawler that extracts text content from websites, optimized for AI models, Large Language Models (LLMs), vector databases, and Retrieval-Augmented Generation (RAG) pipelines.

User avatar

Jason Giang

68

WC

Website Content Crawler for AI — Clean Markdown, 4x Cheaper

joyouscam35875/website-content-crawler

Crawl any website and extract clean text/markdown for LLMs, RAG pipelines, vector databases. BFS crawl with depth control, robots.txt support, boilerplate removal. Perfect for feeding AI models. $0.001/page — 4x cheaper than the official Apify crawler.

User avatar

Ken Digital

67