# Article to Clean Markdown & JSONL for AI (`fxosie-create/article-to-clean-markdown-jsonl`) Actor

Extract Japanese and English articles into clean Markdown, JSONL, structured metadata, and RAG-ready chunks. Static-first with optional browser fallback.

- **URL**: https://apify.com/fxosie-create/article-to-clean-markdown-jsonl.md
- **Developed by:** [YMRLab](https://apify.com/fxosie-create) (community)
- **Categories:** AI, Developer tools, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 successful articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Article to Clean Markdown & JSONL for AI

Extract Japanese and English articles into clean Markdown, JSON, JSONL, text, and
RAG-ready chunks. Static-first extraction with optional browser fallback.

### What this Actor does

This Actor turns public article URLs into structured, reusable content. It extracts the
main article, removes common navigation and advertising elements, normalizes metadata,
and produces clean Markdown, JSON, JSONL, plain text, optional sanitized HTML, and
RAG-ready chunks.

Static HTTP extraction is tried first in `auto` mode. Browser rendering is used only
when the static response cannot produce a usable article. The Actor does not use a paid
AI API and does not require an external model.

### Who it is for

- Developers preparing permitted public content for search, analysis, or retrieval
- AI and RAG teams that need consistent Markdown and chunk metadata
- Japanese and English content workflows
- Researchers and publishers processing pages they are allowed to access and reuse
- Automation users who want structured error results instead of silent failures

### Main features

- Static-first extraction with optional browser fallback
- Markdown, JSON, JSONL, clean text, and optional sanitized HTML
- Canonical URL, author, dates, language, JSON-LD, links, and image metadata
- Japanese-aware content detection and chunking
- Configurable chunk size and overlap
- Duplicate removal by canonical URL and content hash
- Structured results for redirects, 404 responses, robots denial, CAPTCHA, and paywalls
- Conservative concurrency and response-size limits
- Dynamic 2048/4096 MB selection based on browser pressure and HTML size, not URL count

### Japanese article support

Japanese article extraction does not depend on space-separated word counts. The Actor
uses Japanese-aware content-length checks, preserves UTF-8 output, detects supported
legacy encodings such as Shift\_JIS where possible, and produces chunks without assuming
English sentence boundaries. Extraction quality still depends on the page structure and
encoding declarations.

### Input example

```json
{
  "startUrls": [{ "url": "https://example.org/guides/tiny-garden" }],
  "maxPages": 1,
  "renderMode": "auto",
  "outputFormats": ["json", "jsonl", "markdown"],
  "generateChunks": true,
  "chunkSize": 800,
  "sameDomainOnly": true,
  "respectRobotsTxt": true
}
```

Use only public HTTP(S) pages that you are permitted to process. `respectRobotsTxt`
defaults to `true`.

### Output example

The following is a short, fictional example written for this README. It does not contain
third-party article text.

```json
{
  "title": "A Small Balcony Garden",
  "canonicalUrl": "https://example.org/guides/tiny-garden",
  "language": "en",
  "markdown": "# A Small Balcony Garden\n\nThree pots can make a useful herb corner.",
  "cleanText": "A Small Balcony Garden\nThree pots can make a useful herb corner.",
  "wordCount": 12,
  "extractionMethod": "readability",
  "extractionConfidence": 0.91,
  "chunks": [
    {
      "id": "tiny-garden-0",
      "index": 0,
      "text": "A Small Balcony Garden\nThree pots can make a useful herb corner."
    }
  ],
  "status": "ok"
}
```

Each dataset item can also include source/final URLs, description, author, dates,
headings, outbound links, images, JSON-LD, content hash, render method, structured error
information, and capture time.

### RAG and AI use cases

- Build permitted document collections for retrieval-augmented generation
- Convert mixed Japanese and English pages into consistent Markdown
- Create JSONL records for batch import
- Split long articles into traceable chunks with hashes and offsets
- Preserve canonical URLs and metadata for source attribution

This Actor prepares data; it does not determine whether the source content may be used
for a particular model, publication, or commercial purpose.

### Pricing

The confirmed Apify Console Pay per event prices are:

| Event               | When it occurs                                                  |                       Price |
| ------------------- | --------------------------------------------------------------- | --------------------------: |
| `apify-actor-start` | Synthetic event emitted automatically by Apify per allocated GB | **$0.00005 / allocated GB** |
| `article-success`   | Once for each successfully extracted unique article             |       **$0.0035 / article** |
| `browser-success`   | Additional event for a successfully browser-rendered article    |       **$0.0055 / article** |

A 2048 MB run has a synthetic start charge of **$0.00010**. Application code never
emits or manually charges a start event. The only custom events emitted by the Actor
are `article-success` and `browser-success`.

A successfully extracted static article costs **$0.0035**. When browser rendering is
required, an additional **$0.0055** is charged, for a total of **$0.0090** for that
article. The browser total is not a second $0.0090 charge.

Platform usage is included in the event prices because **Pay per event + usage is OFF**.
The primary event is `article-success`. The Actor stops before starting another URL when
the user-defined maximum charge limit no longer permits the required success events;
results completed before that limit remain available.

The prices are configured in Apify Console. The Actor remains private and Store
unpublished. Only a human may perform the final Store publication step.

### Browser fallback

- `auto`: Static extraction first. Browser rendering is used only when necessary and can
  incur the browser surcharge.
- `static`: Lowest-cost option. JavaScript-only content may not be extracted.
- `browser`: Uses browser rendering and incurs the browser surcharge for each successful
  article.

Normal low-concurrency work uses 2048 MB. Three or more concurrent browsers, or a
per-page HTML limit above 6 MB, selects 4096 MB. A large URL list alone does not select
4096 MB. The unsupported 1024 MB configuration is not used.

### Failed-page charging policy

Failed, skipped, duplicate, robots-blocked, 404, CAPTCHA, and paywall results do not
trigger `article-success` or `browser-success`. In a partially successful run, only
successfully extracted unique articles trigger custom success events. Apify independently
emits the synthetic `apify-actor-start` event; Actor code does not control or duplicate it.

### robots.txt policy

`respectRobotsTxt` is enabled by default. A robots-denied page is returned as a
structured skipped result and is not charged as a successful extraction. Disabling the
option does not waive website terms, copyright, contractual restrictions, or applicable
law. Use the Actor only where you have permission.

### Paywall and CAPTCHA limitations

The Actor does not log in, accept terms, solve CAPTCHA, bypass paywalls, evade access
controls, rotate evasive proxies, or use credentials to retrieve protected content.
Detected restricted pages are skipped and do not trigger success events.

### Copyright and permitted-use notice

The Actor extracts content; it does not grant rights to copy, train on, republish, or
commercialize that content. You are responsible for the source website's terms,
robots.txt policy, copyright, privacy rules, database rights, and any required
permission. Avoid collecting personal or sensitive data.

### Known limitations

- Highly interactive pages may require browser mode and cost more.
- Extraction depends on page markup and may include or omit content on unusual layouts.
- Metadata such as author and publication date is `null` when it cannot be verified.
- CAPTCHA, paywalls, login-only content, and access-control bypass are unsupported.
- Scanned PDFs, video/audio transcription, and arbitrary file conversion are outside
  scope.
- Browser fallback does not guarantee successful extraction.

### FAQ

#### Is the browser fee charged in addition to $0.0090?

No. A browser success emits `article-success` ($0.0035) plus `browser-success`
($0.0055), totaling $0.0090 for that article.

#### Are failed pages charged?

They do not receive either custom successful-extraction event. Apify's synthetic
`apify-actor-start` charge is platform-managed and is separate from extraction success.

#### Can the Actor bypass a paywall or CAPTCHA?

No. It detects and skips restricted pages where possible.

#### Does the Actor support Japanese?

Yes. It includes Japanese-aware extraction and chunking, subject to the source page's
markup and encoding quality.

#### Does it call an AI model?

No. The extraction and chunking pipeline runs without an external AI API.

#### What happens at my maximum charge limit?

The Actor may stop when the user-defined maximum charge limit is reached. Already saved
results remain available, so a run can finish with partial results.

### Support

Use the Actor's support or Issues section in Apify Console. Include the run ID, input
settings, public URL, and error code, but do not send passwords, tokens, private page
content, or copyrighted article bodies. Extraction cannot be guaranteed for every site.

### Local development

Windows PowerShell:

```powershell
npm install
npm run lint
npm run typecheck
npm test
npm run build
apify validate-schema
```

The local simulation uses an injected billing client and does not create actual charges:

```powershell
npm run simulate:ppe
```

See [PRICING\_FINAL.md](PRICING_FINAL.md) and
[PUBLISH\_CHECKLIST.md](PUBLISH_CHECKLIST.md) before publication. Do not place
`APIFY_TOKEN`, proxy passwords, or other secrets in this repository.

### 日本語の簡易説明

公開され、利用許可を確認できる日本語・英語の記事ページを、Markdown、JSON、
JSONL、プレーンテキスト、RAG向けチャンクへ整理するActorです。`auto`は静的取得を
先に試し、必要な場合だけbrowserへ切り替えます。robots.txtを既定で尊重し、
404・robots拒否・CAPTCHA・ペイウォール・重複は成功課金の対象にしません。

利用者は対象サイトの規約、著作権、個人情報、再利用条件を確認してください。
ログイン、CAPTCHA回避、ペイウォール突破、アクセス制御の回避には対応しません。
現在はprivate・Store未公開です。ConsoleのPay per event設定は確定済みですが、
一般公開とStore申請の最終操作は人間がApify Consoleで行います。

# Actor input Schema

## `startUrls` (type: `array`):

Public HTTP(S) article or listing URLs.

## `maxPages` (type: `integer`):

Maximum number of pages processed in one run.

## `renderMode` (type: `string`):

auto: Static extraction first. Browser rendering is used only when necessary and can incur the browser surcharge. static: Lowest-cost option. JavaScript-only content may not be extracted. browser: Uses browser rendering and incurs the browser surcharge for each successful article.

## `outputFormats` (type: `array`):

JSON is always stored in the dataset.

## `generateChunks` (type: `boolean`):

Generate Japanese-aware text and Markdown chunks for downstream RAG use.

## `chunkSize` (type: `integer`):

Approximate maximum number of characters in each generated chunk.

## `sameDomainOnly` (type: `boolean`):

Only enqueue links on the start URL's registered domain.

## `respectRobotsTxt` (type: `boolean`):

Enabled by default. Disabling this does not waive site terms, copyright, or legal obligations.

## `maxDepth` (type: `integer`):

Maximum link depth from each start URL.

## `includeSubdomains` (type: `boolean`):

Allow subdomains when same-domain crawling is enabled.

## `minArticleWords` (type: `integer`):

Minimum extracted word or Japanese character-unit count for a usable article.

## `removeNavigation` (type: `boolean`):

Remove common navigation and breadcrumb elements before extraction.

## `removeAdvertisements` (type: `boolean`):

Remove common advertisement elements before extraction.

## `removeRelatedArticles` (type: `boolean`):

Remove common related and recommended article blocks before extraction.

## `includeImages` (type: `boolean`):

Include article image metadata in each result.

## `includeLinks` (type: `boolean`):

Include normalized outbound links found in the extracted article.

## `includeJsonLd` (type: `boolean`):

Include parsed JSON-LD objects found in the page.

## `chunkOverlap` (type: `integer`):

Approximate number of characters repeated between adjacent chunks.

## `languageHint` (type: `string`):

Optional BCP-47-style hint such as ja or en.

## `requestTimeoutSeconds` (type: `integer`):

Per-request timeout for static and browser retrieval.

## `maxConcurrency` (type: `integer`):

Maximum simultaneous page tasks. URL count alone does not increase memory. Keep this low and use the request interval to avoid target-site load.

## `maxBrowserConcurrency` (type: `integer`):

Maximum simultaneous browser renderers for auto/browser mode. Effective browser concurrency is the lowest of this value, maxConcurrency, and expected browser work. One or two uses 2048 MB; three or four selects 4096 MB. Ignored by static extraction at runtime.

## `expectedBrowserFallbacks` (type: `integer`):

Planning estimate used only for memory selection. Use 0 for static, estimate auto fallbacks, and use the planned page count for browser mode. A large URL list stays at 2048 MB when browser concurrency is 1-2.

## `maxHtmlBytes` (type: `integer`):

Maximum accepted HTML response size. Up to 6,000,000 bytes uses 2048 MB; a higher limit selects 4096 MB.

## `minRequestIntervalMillis` (type: `integer`):

Minimum delay between page requests to the same origin. Use a larger value when a site's policy requires it.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "maxPages": 20,
  "renderMode": "auto",
  "outputFormats": [
    "json",
    "jsonl",
    "markdown"
  ],
  "generateChunks": true,
  "chunkSize": 1400,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "maxDepth": 1,
  "includeSubdomains": false,
  "minArticleWords": 80,
  "removeNavigation": true,
  "removeAdvertisements": true,
  "removeRelatedArticles": true,
  "includeImages": true,
  "includeLinks": true,
  "includeJsonLd": true,
  "chunkOverlap": 150,
  "languageHint": "",
  "requestTimeoutSeconds": 30,
  "maxConcurrency": 3,
  "maxBrowserConcurrency": 2,
  "expectedBrowserFallbacks": 1,
  "maxHtmlBytes": 6000000,
  "minRequestIntervalMillis": 500
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `jsonl` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fxosie-create/article-to-clean-markdown-jsonl").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("fxosie-create/article-to-clean-markdown-jsonl").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call fxosie-create/article-to-clean-markdown-jsonl --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=fxosie-create/article-to-clean-markdown-jsonl",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Crg30xNjPR8xuIPWb/builds/FJaOWmKcVKhWNB0w2/openapi.json