Article to Clean Markdown & JSONL for AI

Extract Japanese and English articles into clean Markdown, JSON, JSONL, text, and RAG-ready chunks. Static-first extraction with optional browser fallback.

What this Actor does

This Actor turns public article URLs into structured, reusable content. It extracts the main article, removes common navigation and advertising elements, normalizes metadata, and produces clean Markdown, JSON, JSONL, plain text, optional sanitized HTML, and RAG-ready chunks.

Static HTTP extraction is tried first in auto mode. Browser rendering is used only when the static response cannot produce a usable article. The Actor does not use a paid AI API and does not require an external model.

Who it is for

• Developers preparing permitted public content for search, analysis, or retrieval
• AI and RAG teams that need consistent Markdown and chunk metadata
• Japanese and English content workflows
• Researchers and publishers processing pages they are allowed to access and reuse
• Automation users who want structured error results instead of silent failures

Main features

• Static-first extraction with optional browser fallback
• Markdown, JSON, JSONL, clean text, and optional sanitized HTML
• Canonical URL, author, dates, language, JSON-LD, links, and image metadata
• Japanese-aware content detection and chunking
• Configurable chunk size and overlap
• Duplicate removal by canonical URL and content hash
• Structured results for redirects, 404 responses, robots denial, CAPTCHA, and paywalls
• Conservative concurrency and response-size limits
• Dynamic 2048/4096 MB selection based on browser pressure and HTML size, not URL count

Japanese article support

Japanese article extraction does not depend on space-separated word counts. The Actor uses Japanese-aware content-length checks, preserves UTF-8 output, detects supported legacy encodings such as Shift_JIS where possible, and produces chunks without assuming English sentence boundaries. Extraction quality still depends on the page structure and encoding declarations.

Input example

{
  "startUrls": [{ "url": "https://example.org/guides/tiny-garden" }],
  "maxPages": 1,
  "renderMode": "auto",
  "outputFormats": ["json", "jsonl", "markdown"],
  "generateChunks": true,
  "chunkSize": 800,
  "sameDomainOnly": true,
  "respectRobotsTxt": true
}

Use only public HTTP(S) pages that you are permitted to process. respectRobotsTxt defaults to true.

Output example

The following is a short, fictional example written for this README. It does not contain third-party article text.

{
  "title": "A Small Balcony Garden",
  "canonicalUrl": "https://example.org/guides/tiny-garden",
  "language": "en",
  "markdown": "# A Small Balcony Garden\n\nThree pots can make a useful herb corner.",
  "cleanText": "A Small Balcony Garden\nThree pots can make a useful herb corner.",
  "wordCount": 12,
  "extractionMethod": "readability",
  "extractionConfidence": 0.91,
  "chunks": [
    {
      "id": "tiny-garden-0",
      "index": 0,
      "text": "A Small Balcony Garden\nThree pots can make a useful herb corner."
    }
  ],
  "status": "ok"
}

Each dataset item can also include source/final URLs, description, author, dates, headings, outbound links, images, JSON-LD, content hash, render method, structured error information, and capture time.

RAG and AI use cases

• Build permitted document collections for retrieval-augmented generation
• Convert mixed Japanese and English pages into consistent Markdown
• Create JSONL records for batch import
• Split long articles into traceable chunks with hashes and offsets
• Preserve canonical URLs and metadata for source attribution

This Actor prepares data; it does not determine whether the source content may be used for a particular model, publication, or commercial purpose.

Pricing

The confirmed Apify Console Pay per event prices are:

Event	When it occurs	Price
apify-actor-start	Synthetic event emitted automatically by Apify per allocated GB	$0.00005 / allocated GB
article-success	Once for each successfully extracted unique article	$0.0035 / article
browser-success	Additional event for a successfully browser-rendered article	$0.0055 / article

A 2048 MB run has a synthetic start charge of $0.00010. Application code never emits or manually charges a start event. The only custom events emitted by the Actor are article-success and browser-success.

A successfully extracted static article costs $0.0035. When browser rendering is required, an additional $0.0055 is charged, for a total of $0.0090 for that article. The browser total is not a second $0.0090 charge.

Platform usage is included in the event prices because Pay per event + usage is OFF. The primary event is article-success. The Actor stops before starting another URL when the user-defined maximum charge limit no longer permits the required success events; results completed before that limit remain available.

The prices are configured in Apify Console. The Actor remains private and Store unpublished. Only a human may perform the final Store publication step.

Browser fallback

• auto: Static extraction first. Browser rendering is used only when necessary and can incur the browser surcharge.
• static: Lowest-cost option. JavaScript-only content may not be extracted.
• browser: Uses browser rendering and incurs the browser surcharge for each successful article.

Normal low-concurrency work uses 2048 MB. Three or more concurrent browsers, or a per-page HTML limit above 6 MB, selects 4096 MB. A large URL list alone does not select 4096 MB. The unsupported 1024 MB configuration is not used.

Failed-page charging policy

Failed, skipped, duplicate, robots-blocked, 404, CAPTCHA, and paywall results do not trigger article-success or browser-success. In a partially successful run, only successfully extracted unique articles trigger custom success events. Apify independently emits the synthetic apify-actor-start event; Actor code does not control or duplicate it.

robots.txt policy

respectRobotsTxt is enabled by default. A robots-denied page is returned as a structured skipped result and is not charged as a successful extraction. Disabling the option does not waive website terms, copyright, contractual restrictions, or applicable law. Use the Actor only where you have permission.

Paywall and CAPTCHA limitations

The Actor does not log in, accept terms, solve CAPTCHA, bypass paywalls, evade access controls, rotate evasive proxies, or use credentials to retrieve protected content. Detected restricted pages are skipped and do not trigger success events.

Copyright and permitted-use notice

The Actor extracts content; it does not grant rights to copy, train on, republish, or commercialize that content. You are responsible for the source website's terms, robots.txt policy, copyright, privacy rules, database rights, and any required permission. Avoid collecting personal or sensitive data.

Known limitations

• Highly interactive pages may require browser mode and cost more.
• Extraction depends on page markup and may include or omit content on unusual layouts.
• Metadata such as author and publication date is null when it cannot be verified.
• CAPTCHA, paywalls, login-only content, and access-control bypass are unsupported.
• Scanned PDFs, video/audio transcription, and arbitrary file conversion are outside scope.
• Browser fallback does not guarantee successful extraction.

FAQ

Is the browser fee charged in addition to $0.0090?

No. A browser success emits article-success ($0.0035) plus browser-success ($0.0055), totaling $0.0090 for that article.

Are failed pages charged?

They do not receive either custom successful-extraction event. Apify's synthetic apify-actor-start charge is platform-managed and is separate from extraction success.

Can the Actor bypass a paywall or CAPTCHA?

No. It detects and skips restricted pages where possible.

Does the Actor support Japanese?

Yes. It includes Japanese-aware extraction and chunking, subject to the source page's markup and encoding quality.

Does it call an AI model?

No. The extraction and chunking pipeline runs without an external AI API.

What happens at my maximum charge limit?

The Actor may stop when the user-defined maximum charge limit is reached. Already saved results remain available, so a run can finish with partial results.

Support

Use the Actor's support or Issues section in Apify Console. Include the run ID, input settings, public URL, and error code, but do not send passwords, tokens, private page content, or copyrighted article bodies. Extraction cannot be guaranteed for every site.

Local development

Windows PowerShell:

npm install
npm run lint
npm run typecheck
npm test
npm run build
apify validate-schema

The local simulation uses an injected billing client and does not create actual charges:

npm run simulate:ppe

See PRICING_FINAL.md and PUBLISH_CHECKLIST.md before publication. Do not place APIFY_TOKEN, proxy passwords, or other secrets in this repository.

日本語の簡易説明

公開され、利用許可を確認できる日本語・英語の記事ページを、Markdown、JSON、 JSONL、プレーンテキスト、RAG向けチャンクへ整理するActorです。autoは静的取得を 先に試し、必要な場合だけbrowserへ切り替えます。robots.txtを既定で尊重し、 404・robots拒否・CAPTCHA・ペイウォール・重複は成功課金の対象にしません。

利用者は対象サイトの規約、著作権、個人情報、再利用条件を確認してください。 ログイン、CAPTCHA回避、ペイウォール突破、アクセス制御の回避には対応しません。 現在はprivate・Store未公開です。ConsoleのPay per event設定は確定済みですが、 一般公開とStore申請の最終操作は人間がApify Consoleで行います。