# AI Web Scraper — Any Site to JSON with GPT or Claude (`flash_scraper/ai-universal-scraper`) Actor

AI web scraper that turns any URL into clean, structured JSON. List the fields you want or describe them in plain English, bring your own OpenAI (GPT) or Anthropic (Claude) key, and the model reads the page like a human — no CSS selectors, no per-site code. Export JSON, CSV, or Excel.

- **URL**: https://apify.com/flash\_scraper/ai-universal-scraper.md
- **Developed by:** [Flash Scrape](https://apify.com/flash_scraper) (community)
- **Categories:** Automation, AI, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Universal Scraper — scrape any website with GPT or Claude, no selectors needed

**An AI web scraper that turns any page into clean, structured JSON.** Paste a URL, list the fields you want (`title`, `price`, `author`…), plug in your own OpenAI or Anthropic API key, and a language model reads the page like a human and returns exactly those fields. No CSS selectors to maintain, no per-site configuration, no broken parser every time a site changes its HTML — which is exactly where traditional scrapers fall over.

### What it does

- **Extracts structured data from any URL** — you provide a list of pages and either a list of field names, a plain-English instruction, or both.
- **Uses your own LLM key** — supports **OpenAI** (default model `gpt-4o-mini`) and **Anthropic** (default model `claude-haiku-4-5-20251001`), with an optional model override so you can trade cost for quality.
- **Cleans pages before sending them to the model** — scripts, styles and markup are stripped so you only pay tokens for actual content, capped at a character limit you control (`maxChars`, default 12,000).
- **Returns strict JSON** — the model is instructed to answer with pure JSON, and the reply is parsed defensively (code fences and stray prose are handled).
- **One row per item** — a single-object answer becomes one dataset row per URL; when the page clearly lists multiple items (a news feed, a product listing), the model returns an array and you get one row per item, each tagged with the source `url`.
- **Fails soft** — an unreachable URL or a failed extraction is pushed as a row with an `error` field and the run continues with the remaining URLs.

### Use cases

- **Competitor product monitoring** — pull `name`, `price`, `availability` from competitor product pages without writing a scraper per site.
- **Content and news pipelines** — turn any article or feed page into `{title, summary, author}` rows, ready for a newsletter digest or a research database.
- **AI data extraction pipelines** — use it as the "scrape any website with GPT" step in a larger workflow: feed the dataset into Zapier, Make, or your own code via the Apify API.
- **Lead and directory extraction** — extract names, roles and links from listing pages that have no API and no purpose-built scraper.
- **One-off datasets** — when a client asks for structured data from an obscure site, describe the fields instead of reverse-engineering the HTML.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | — (required) | One or more page URLs to extract data from. |
| `fields` | array | — | The data points you want, e.g. `"title"`, `"price"`, `"author"`, `"rating"`. Leave empty if you use Instructions instead. |
| `instructions` | string | — | Natural-language guidance, e.g. "Extract every news item as a separate object with title and points." |
| `llmProvider` | string | `openai` | Which model provider to use: `openai` or `anthropic`. |
| `apiKey` | string | — (optional) | Your OpenAI or Anthropic API key. Stored encrypted; used only to call the model. Leave empty to run in no-LLM preview mode. |
| `model` | string | — | Override the model. Defaults: OpenAI → `gpt-4o-mini`, Anthropic → `claude-haiku-4-5`. |
| `maxChars` | integer | `12000` | How much page text to send to the model (controls cost). Minimum 1,000. |

You must set `startUrls`; for full AI extraction also provide `apiKey` plus at least one of `fields` or `instructions`. The `apiKey` is optional — without it the Actor runs in a **no-LLM preview mode** that uses built-in heuristics to return each page's title, meta description, and a short text preview. That's handy for a free test drive, but the structured field extraction described below needs your OpenAI or Anthropic key.

```json
{
  "startUrls": ["https://news.ycombinator.com"],
  "fields": ["title", "summary", "author"],
  "llmProvider": "openai",
  "apiKey": "sk-...",
  "maxChars": 12000
}
```

#### Where to get an API key

- **OpenAI** — create a key at [platform.openai.com/api-keys](https://platform.openai.com/api-keys).
- **Anthropic** — create a key at [console.anthropic.com](https://console.anthropic.com/) under API Keys.

The key is marked secret in the input schema, so Apify stores it encrypted and it never appears in logs or shared run links.

### Output

Each dataset row is the JSON object the model extracted, plus the source `url`. The field names are the ones **you** asked for — the schema of the output is effectively yours to define. With the example input above, a row looks like:

```json
{
  "url": "https://news.ycombinator.com",
  "title": "Show HN: I built a self-hosted analytics tool in a weekend",
  "summary": "A lightweight, privacy-first alternative to hosted analytics, written in Go.",
  "author": "jdoe42"
}
```

If a URL can't be fetched or the model's reply can't be parsed, you get a row like `{"url": "https://…", "error": "extraction failed: …"}` instead, so nothing disappears silently. Export everything as JSON, CSV, or Excel from the run's Dataset tab, or pull it via the Apify API.

### Pricing

This actor uses **Apify pay-per-event pricing charged per `result`** — you pay only for the rows actually delivered to your dataset, never for setup or empty runs. See the actor's Pricing tab for the current per-result rate. A **free Apify plan is enough to try it**.

Separately, **you pay your LLM provider directly for tokens** on your own key. With the default `gpt-4o-mini` or `claude-haiku-4-5` and the 12,000-character cap, a page typically costs a fraction of a cent; lower `maxChars` for cheaper runs or raise it for long pages.

### Tips / FAQ

**Does it work on JavaScript-heavy sites?**
It fetches the raw HTML over HTTP (no headless browser). Server-rendered pages — most articles, listings, product pages, docs — work well. A single-page app that renders everything client-side may return little usable text; check the row's content before scaling up.

**Which provider and model should I pick?**
Extraction quality depends on the model you choose. The defaults (`gpt-4o-mini`, `claude-haiku-4-5`) are fast and cheap and handle most pages. For messy or ambiguous pages, set `model` to a stronger one (e.g. `gpt-4o` or a Claude Sonnet model) — same input, better judgment, higher token cost.

**How do I get multiple rows from one page?**
Use `instructions`, e.g. "Extract every news item as a separate object with title and points." When the model returns an array, each object becomes its own dataset row.

**How do I control cost?**
Only the first `maxChars` characters of cleaned page text are sent to the model (default 12,000). That cap, plus your choice of model, is your cost lever.

**Does it follow links or crawl a whole site?**
No — it processes exactly the URLs you provide, one LLM call per page. For crawling, generate the URL list first (e.g. from a sitemap) and pass it in.

**What happens on errors?**
Failed fetches and failed extractions are logged and pushed as `{url, error}` rows; the run continues with the next URL rather than aborting.

### Related actors

- [Smart Scraper by Example](https://apify.com/flash_scraper/smart-scraper-by-example) — show it one example of the data you want and it scrapes the rest of the site to match.
- [AI Web Scraper](https://apify.com/flash_scraper/ai-web-scraper) — a sibling AI extraction actor for structured web data.
- [Company & Domain Enricher](https://apify.com/flash_scraper/company-domain-enricher) — enrich the domains you scrape into full company records.

***

Found a bug or need a feature? Open an Issue on this actor's **Issues** tab — typical response within 1 business day.

# Actor input Schema

## `startUrls` (type: `array`):

One or more page URLs to extract data from.

## `fields` (type: `array`):

The data points you want, e.g. "title", "price", "author", "rating". Leave empty if you use Instructions instead.

## `instructions` (type: `string`):

Natural-language guidance, e.g. "Extract every news item as a separate object with title and points."

## `llmProvider` (type: `string`):

AI provider used for extraction — OpenAI (GPT) or Anthropic (Claude). Must match the API key you provide below.

## `apiKey` (type: `string`):

Optional — without a key the Actor runs in no-LLM preview mode using built-in heuristic extraction. Your OpenAI or Anthropic API key. Stored encrypted; used only to call the model.

## `model` (type: `string`):

Override the model. Defaults: OpenAI → gpt-4o-mini, Anthropic → claude-haiku-4-5.

## `maxChars` (type: `integer`):

How many characters of page text to send to the model per page — higher captures more of the page but costs more tokens, e.g. 12000.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com/product/123"
  ],
  "fields": [
    "title",
    "summary",
    "author"
  ],
  "llmProvider": "openai",
  "maxChars": 12000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://news.ycombinator.com"
    ],
    "fields": [
        "title",
        "summary",
        "author"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("flash_scraper/ai-universal-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://news.ycombinator.com"],
    "fields": [
        "title",
        "summary",
        "author",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("flash_scraper/ai-universal-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://news.ycombinator.com"
  ],
  "fields": [
    "title",
    "summary",
    "author"
  ]
}' |
apify call flash_scraper/ai-universal-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=flash_scraper/ai-universal-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qjC9cn2XYGbcO2lcc/builds/mj7ZVLaMTN5ITKblt/openapi.json