# Wikipedia Data Scraper Pro (`moving_beacon-owner1/my-actor-39`) Actor

An automated crawler that extracts textual content and metadata from Wikipedia pages for building knowledge bases.

- **URL**: https://apify.com/moving\_beacon-owner1/my-actor-39.md
- **Developed by:** [Jamshaid Arif](https://apify.com/moving_beacon-owner1) (community)
- **Categories:** Automation, SEO tools, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00/month + usage

To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Wikipedia Scraper

Extract structured data from Wikipedia at any scale — articles, categories, sections, links, categories, and multilingual translations — without managing infrastructure.

***

### What Does This Actor Do?

Wikipedia Scraper fetches public Wikipedia data through the official MediaWiki API. It supports three modes:

| Mode | Use Case |
|------|----------|
| **📄 Article Pages** | Scrape one or many articles by title |
| **📂 Category Crawl** | Collect every article under a category (and its subcategories) |
| **🌐 Translation Comparison** | Fetch the same article across multiple language editions |

Every result is pushed to the Apify Dataset as a structured record you can download as JSON, CSV, Excel, or XML.

***

### Output Fields

Each dataset item contains:

| Field | Type | Description |
|-------|------|-------------|
| `title` | string | Wikipedia article title |
| `language` | string | Language code (`en`, `de`, `fr`, …) |
| `pageId` | integer | Wikipedia internal page ID |
| `url` | string | Full URL to the article |
| `scrapedAt` | ISO date | Timestamp of extraction |
| `summary` | string | Full lead section text |
| `summaryPreview` | string | First 200 characters of summary |
| `sections` | array | Nested section tree (title + text + subsections) |
| `links` | object | Outbound wiki links `{ title → url }` |
| `categories` | object | Article categories `{ name → url }` |
| `translations` | object | Other language editions `{ lang → { title, url } }` |
| `numSections` | integer | Count of top-level sections |
| `numLinks` | integer | Count of outbound links returned |
| `numCategories` | integer | Count of categories returned |
| `numTranslations` | integer | Count of available language editions |
| `status` | string | `ok`, `not_found`, `network_error`, or `error` |

When **Translation Comparison** mode is used, items also include a `comparisonBaseTitle` field identifying the base English article.

***

### Input Configuration

#### Mode: Article Pages

```json
{
  "mode": "page",
  "topics": ["Python (programming language)", "Alan Turing", "Machine learning"],
  "language": "en",
  "includeSections": true,
  "includeLinks": true,
  "includeCategories": true,
  "includeTranslations": false,
  "includeFullText": false
}
```

#### Mode: Category Crawl

```json
{
  "mode": "category",
  "categoryTitle": "Category:Machine learning",
  "language": "en",
  "categoryMaxDepth": 1,
  "maxPages": 50,
  "includeSections": true,
  "includeLinks": false
}
```

#### Mode: Translation Comparison

```json
{
  "mode": "translations",
  "comparisonTitle": "Artificial intelligence",
  "translationLanguages": ["en", "de", "fr", "ja", "ar", "es", "zh", "ru"]
}
```

***

### Usage Examples

#### Using the Apify API (Python)

```python
import apify_client

client = apify_client.ApifyClient("YOUR_API_TOKEN")

run = client.actor("YOUR_ACTOR_ID").call(run_input={
    "mode": "page",
    "topics": ["Deep learning", "Neural network", "Transformer (machine learning model)"],
    "language": "en",
    "includeSections": True,
    "includeLinks": True
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], "→", item["url"])
    print("  Summary:", item["summaryPreview"])
    print("  Sections:", item["numSections"])
```

#### Using the Apify API (JavaScript/Node.js)

```javascript
const { ApifyClient } = require('apify-client');

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('YOUR_ACTOR_ID').call({
    mode: 'category',
    categoryTitle: 'Category:Physics',
    categoryMaxDepth: 1,
    maxPages: 30,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach(item => console.log(item.title, item.url));
```

#### Using the Apify CLI

```bash
## Install CLI
npm install -g apify-cli

## Run locally (requires .actor/ directory)
apify run --input='{"mode":"page","topics":["Quantum computing"]}'

## Deploy to Apify platform
apify push
```

***

### Sections Structure Example

When `includeSections` is `true`, each article item contains a `sections` array:

```json
{
  "sections": [
    {
      "level": 1,
      "title": "History",
      "text": "Python was conceived in the late 1980s by Guido van Rossum...",
      "subsections": [
        {
          "level": 2,
          "title": "Early development",
          "text": "Python 0.9.0 was published to alt.sources in February 1991...",
          "subsections": []
        }
      ]
    },
    {
      "level": 1,
      "title": "Design philosophy",
      "text": "Python is a multi-paradigm programming language...",
      "subsections": []
    }
  ]
}
```

***

### Rate Limiting & Politeness

This actor follows the [Wikimedia User-Agent policy](https://meta.wikimedia.org/wiki/User-Agent_policy):

- Uses a descriptive `User-Agent` header identifying itself as `Wikipedia Scraper / Apify Actor`
- Introduces a configurable delay (default **0.5 s**) between every API call
- Respects Wikipedia's public API — no login or authentication required
- Does not scrape HTML; uses the official MediaWiki REST API exclusively

If you encounter rate-limiting errors, increase the **Request Delay** setting to `1.0`–`2.0` seconds.

***

### Performance & Memory

| Input size | Recommended memory |
|---|---|
| 1–20 articles | 256 MB |
| 20–100 articles | 512 MB |
| Category crawl (100+ pages) | 1024 MB |

***

### Limitations

- Wikipedia's API caps some response sizes (links, categories). This actor returns up to **50 links** and **50 categories** per page.
- Some Wikipedia editions have incomplete `langlinks` metadata.
- Full-text extraction (`includeFullText: true`) significantly increases dataset size. Enable only when needed.
- Wikipedia may throttle aggressive requests. Keep `requestDelay` ≥ 0.5 seconds.

***

### Legal & Attribution

This actor accesses only **publicly available** Wikipedia content through the official MediaWiki API, in compliance with Wikipedia's [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) and [Creative Commons Attribution-ShareAlike 4.0 License](https://creativecommons.org/licenses/by-sa/4.0/).

All extracted content remains subject to Wikipedia's licensing. When republishing Wikipedia content, you must attribute Wikipedia and link to the original article.

# Actor input Schema

## `mode` (type: `string`):

Choose what to scrape:<br><b>page</b> – one or more article pages<br><b>category</b> – all pages inside a category tree<br><b>translations</b> – compare one article across multiple languages

## `topics` (type: `array`):

Exact Wikipedia article titles to scrape (used in <b>page</b> mode). E.g. <i>Python (programming language)</i>, <i>Alan Turing</i>.

## `categoryTitle` (type: `string`):

Full category name to crawl (used in <b>category</b> mode). Must include the <i>Category:</i> prefix. E.g. <i>Category:Physics</i>.

## `comparisonTitle` (type: `string`):

English Wikipedia title to compare across languages (used in <b>translations</b> mode).

## `language` (type: `string`):

Language code for the Wikipedia edition to scrape (ISO 639-1). Ignored in <b>translations</b> mode.

## `translationLanguages` (type: `array`):

Language codes to fetch in <b>translations</b> mode. Leave empty to fetch all available translations.

## `categoryMaxDepth` (type: `integer`):

How many subcategory levels to follow in <b>category</b> mode. 0 = top-level articles only.

## `maxPages` (type: `integer`):

Maximum number of article pages to scrape in <b>category</b> mode.

## `includeSections` (type: `boolean`):

Store the full nested section structure (title + text + subsections). Note: flat section text is always included in <b>fullTextBySections</b> regardless of this setting.

## `includeLinks` (type: `boolean`):

Collect links to other Wikipedia articles found on each page (capped at 50).

## `includeCategories` (type: `boolean`):

Collect the Wikipedia categories each article belongs to.

## `includeTranslations` (type: `boolean`):

Collect URLs of the same article in other languages (langlinks). Adds one extra API call per page.

## `requestDelay` (type: `number`):

Seconds to wait between Wikipedia API calls. Increase if you encounter rate-limiting.

## `proxyConfiguration` (type: `object`):

Optional proxy settings for Wikipedia API requests.

## Actor input object example

```json
{
  "mode": "page",
  "topics": [
    "Machine learning",
    "Alan Turing",
    "Ada Lovelace"
  ],
  "categoryTitle": "Category:Machine learning",
  "comparisonTitle": "Artificial intelligence",
  "language": "en",
  "translationLanguages": [
    "en",
    "de",
    "fr",
    "ja"
  ],
  "categoryMaxDepth": 1,
  "maxPages": 20,
  "includeSections": true,
  "includeLinks": true,
  "includeCategories": true,
  "includeTranslations": false,
  "requestDelay": 0.5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "topics": [
        "Machine learning"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("moving_beacon-owner1/my-actor-39").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "topics": ["Machine learning"] }

# Run the Actor and wait for it to finish
run = client.actor("moving_beacon-owner1/my-actor-39").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "topics": [
    "Machine learning"
  ]
}' |
apify call moving_beacon-owner1/my-actor-39 --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=moving_beacon-owner1/my-actor-39",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/b7Rge0bmElecXGlNV/builds/El3QCPERcfBnkKdBA/openapi.json