Python Web Scraper — Scrape or Crawl Any Website

🔗 Part of the Apify actors collection — actors that chain: scrape → clean → use.

Scrape any website — or crawl a whole section of one — with your own Python page function. This is a custom Python scraper running a real headless Chromium browser through Playwright, so JavaScript-rendered pages work the same as static ones. You decide what gets extracted; the Actor handles the browser, the crawl queue, the retries and the limits.

You pay $0.003 per record delivered. Nothing else — not pages visited, not retries, not time.

What you get back, and how fast

Your records, not ours. Whatever your page function returns is the dataset record — this Actor adds no fields and removes none. The default page function returns url, title and text (first 5 000 characters of the body), so a run works before you have written a line of your own.

Measured on our last front-door check: 5 pages of a static site in 28 seconds, one browser, no proxy. JavaScript-heavy pages are slower — that is Chromium rendering, not queue overhead — and a wider crawl scales with maxConcurrency rather than with time per page.

$3.00 per 1 000 records ($0.003 each). A run that delivers nothing costs nothing: pages visited, retries and browser time are not billed.

---

What can this Python web scraper do?

• Scrape a single page — give it a URL and a page function, get a record back.
• Crawl a whole section of a site — follow links by CSS selector, restrict them with glob patterns, stop at a hard request limit.
• Extract exactly the shape you want — your function returns a Python dict, and that dict is the output record. No fixed template to fight, no fields you have to accept.
• Run JavaScript-heavy pages — real Chromium, real rendering, configurable wait condition.
• Chain into other Actors — the output dataset feeds straight into Dataset Deduplicator & Cleaner or any Apify integration.

What websites can I scrape with it?

Any site that a browser can open without credentials, and that allows crawling in its robots.txt.

That includes: documentation sites, product catalogues, listing pages, blogs, news archives, government registers, price pages, job boards that aren't behind a login, and the long tail of sites nobody has written a dedicated scraper for.

It does not include sites behind industrial anti-bot protection or a login — see When is this the wrong tool? below. That section is there because a wasted run costs you money, and we would rather you not spend it.

How do I use it?

Two fields are all you need. The prefilled example works as-is — press Start and you get a record back.

{
  "startUrls": [{ "url": "https://apify.com" }],
  "pageFunction": "async def page_function(page, context, request):\n    return {\n        \"url\": page.url,\n        \"title\": await page.title()\n    }"
}

Result:

{ "url": "https://apify.com", "title": "Apify: Full-stack web scraping and data extraction platform" }

How do I write the page function?

It is an ordinary Python async function with a fixed signature:

async def page_function(page, context, request):
    # page      – Playwright Page instance (page.locator, page.title, page.content, …)
    # context   – Actor context (key-value store, dataset, logging)
    # request   – Current Request object (request.url, request.user_data, …)
    # Return a dict, or a list of dicts. Each dict becomes one result item.
    return {
        "url": page.url,
        "title": await page.title(),
        "h1": await page.locator("h1").first.inner_text(),
    }

Rules worth knowing:

• Return a dict → one record. Return a list of dicts → many records from one page.
• Return None → no record from this page, and no charge. That is the supported way of saying "this page had nothing for me".
• The function is compiled before the crawl starts. A syntax error fails the run in the first second, not after an hour of crawling.
• An exception thrown for one page does not kill the run. That page is counted as an error, the crawl continues, and the count is reported in the final status message.

How do I crawl and follow links?

{
  "startUrls": [{ "url": "https://example.com/products" }],
  "linkSelector": "a.product-link",
  "includeGlobs": ["https://example.com/products/*"],
  "maxRequestsPerCrawl": 200,
  "pageFunction": "async def page_function(page, context, request):\n    return {\n        \"url\": page.url,\n        \"name\": await page.locator(\"h1\").inner_text(),\n        \"price\": await page.locator(\".price\").inner_text()\n    }"
}

• linkSelector finds links on each visited page.
• includeGlobs decides which of those links are worth enqueueing. Without it, a crawl will happily wander off into the rest of the site — and every page it visits is a page you waited for.
• maxRequestsPerCrawl is a hard stop on total requests.

⬇️ Input

Field	Type	Default	Description
startUrls	array of objects	—	Seed URLs, as { "url": "…" }. At least one.
pageFunction	string	—	Python source of the async page function.
linkSelector	string	—	CSS selector for links to follow. Empty = no crawling.
includeGlobs	array of strings	—	Glob patterns a URL must match to be enqueued.
maxRequestsPerCrawl	integer	100	Hard limit on requests for the whole run.
maxConcurrency	integer	5	Parallel browser contexts.
requestTimeoutSecs	integer	45	Per-page timeout. Bounds a page that never settles.
waitUntil	string	load	Playwright wait condition: load, domcontentloaded, networkidle.
respectRobotsTxt	boolean	true	Obey robots.txt. Disallowed URLs are skipped before being visited.
proxyConfiguration	object	Apify proxy	Proxy settings.
maxItems	integer	0 (no limit)	Stop after this many records. Hard cap 50 000 per run.

⬆️ Output — what you get back

The shape of each record is yours. Whatever dictionary your page function returns is written to the dataset unchanged. That is the whole point of this Actor, and it is why it ships no fixed output schema: any schema would be a promise about code you wrote, not code we wrote.

One record per page

{ "url": "https://example.com/products/wrench", "name": "Torque wrench 1/2\"", "price": "€89.00" }

Many records from one page

Return a list, and each element becomes its own record — useful for a listing page:

async def page_function(page, context, request):
    rows = []
    for card in await page.locator(".product-card").all():
        rows.append({
            "name": await card.locator(".name").inner_text(),
            "price": await card.locator(".price").inner_text(),
        })
    return rows

What the Actor guarantees, and what your page function decides

The Actor guarantees:

• every record you return is written exactly once, and charged exactly once;
• a page that throws is isolated — the crawl continues and you get the records that worked;
• maxRequestsPerCrawl and maxItems are hard stops, so a crawl cannot run away with your budget;
• requestTimeoutSecs bounds a page that never settles;
• a run that delivers nothing because something broke fails loudly instead of reporting success.

Your page function decides everything else: which elements to read, how to name the fields, what to skip.

What happens when something fails?

Errors are not written into your dataset as fake records — you are never charged for a failure. They are counted and reported, so a run always tells you what actually happened.

Situation	What the Actor does
pageFunction has a syntax error	Fails before the crawl starts, with the syntax error in the status message.
startUrls is empty	Fails immediately with a clear message.
One page throws inside your function	Counted as a page error, crawl continues. Reported as (N page(s) errored in the page function).
A URL is unreachable after retries	Counted as a failed request. Reported as (N request(s) failed at network/navigation).
Zero records, and pages errored or requests failed	The run fails, with the reason: every page errored in the page function, or all requests failed (site unreachable/blocked?). A broken run never reports success.
Zero records, and nothing errored	Succeeds with Scraped 0 item(s) — your selectors matched nothing. Costs you nothing.
maxItems reached	Stops cleanly, status says (reached maxItems=N).
Billing call fails repeatedly	The run stops rather than doing unpaid work, and says how many items it delivered first.

That fifth row is the one that matters most. A scraper that returns an empty dataset and calls it success is the single most expensive failure mode there is, because your pipeline keeps running on nothing. This Actor refuses to do it.

How much does it cost?

• $0.003 per record delivered (result-item), flat, regardless of how many pages were walked to find it.
• An empty run costs nothing. No record, no charge.
• Nothing is charged for pages visited, retries, browser time, or bandwidth.
• To try it cheaply, set a small maxItems — you pay only for what you actually receive.

For 1 000 records that is $3.00. A crawl of 200 pages that yields 200 records costs $0.60, whether those pages took two minutes or twenty.

When is this the wrong tool?

Be honest with yourself about the target before you spend a run:

• Sites behind industrial anti-bot protection — major marketplaces, search engines, large social networks. They rate-limit or block datacenter IP addresses regardless of how good the browser automation is. Those targets need a residential proxy, and a generic crawler is not the reason they work or fail.
• Content behind a login. This Actor does not carry your credentials, by design.
• A site somebody already solved well. If a dedicated Actor exists for your target, it will handle that site's quirks better than a page function you write today.

Use this when you need your own extraction logic on a site nobody has built a dedicated tool for.

FAQ

Can I use it with the Apify API?

Yes. Start it like any Actor — POST /v2/acts/eszetael_lab~reliable-playwright-scraper/runs with your input as the JSON body, then read the run's dataset. Everything this Actor does is available through the standard API, CLI and client libraries.

Can I use it through an MCP server?

Yes. It is exposed through Apify's Actors MCP server like any other public Actor, so an AI agent can call it as a tool. It is also enabled for agentic payments, meaning an agent can run it and be charged directly without a human in the loop.

Can I schedule it to run every day?

Yes — use Apify Schedules. A common pattern is a daily crawl with a maxItems cap, chained into the Dataset Deduplicator & Cleaner so you only ever look at what is new.

Does it respect robots.txt?

By default, yes. Disallowed URLs are skipped before they are visited, so a blocked page never costs you a request. Turn respectRobotsTxt off only for sites you own or are otherwise authorised to crawl.

Can it log in to a website?

No, and that is deliberate. This Actor does not carry credentials. If a site requires a session, this is the wrong tool.

Where does my page function run?

Inside your own run, on your own account, like any other code you run on the platform. A per-page timeout bounds runaway code, and maxRequestsPerCrawl bounds runaway crawls.

Is it legal to scrape with this Actor?

Scraping publicly available data is broadly lawful in the EU and the US, but "broadly" is not "always". You are responsible for ensuring your use complies with applicable law and the target site's terms of service, and for not collecting personal data you have no basis to collect. robots.txt is respected by default because it is the site's own machine-readable statement of what it permits.

Your feedback

Found a bug, or a site where this behaves badly? Open an issue on the Actor's Issues tab. Real failure reports are worth more to us than feature requests.

Related actors

Three tools built to chain into each other — scrape, then clean, then use.

• Dataset Deduplicator & Cleaner — pass this Actor's dataset ID straight in to remove duplicates across runs and clean the fields before analysis. Six times cheaper than a scraper, because it processes data you already paid for.
• Bluesky Scraper — when your target is Bluesky rather than a website, use the protocol directly instead of a browser: no login, no proxy, and an incremental mode that returns only new posts.

All three are on pay-per-result pricing, and an empty run costs nothing in every one of them.