# Python Web Scraper — Playwright, Any Website (`eszetael_lab/reliable-playwright-scraper`) Actor Scrape or crawl any website with your own Python page function — a custom Python scraper on Playwright and a headless Chromium browser. Full control over what is extracted, no template to fight. Respects robots.txt, retries failures. Empty runs cost nothing. - **URL**: https://apify.com/eszetael\_lab/reliable-playwright-scraper.md - **Developed by:** [Radosław Szal](https://apify.com/eszetael_lab) (community) - **Categories:** AI, Automation, Developer tools - **Stats:** 3 total users, 3 monthly users, 64.3% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $3.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Python Web Scraper — Scrape or Crawl Any Website > 🔗 Part of the **[Apify actors collection](https://github.com/Eszetael/apify-actors)** — actors that chain: scrape → clean → use. Scrape any website — or crawl a whole section of one — with your own Python page function. This is a **custom Python scraper** running a real headless Chromium browser through Playwright, so JavaScript-rendered pages work the same as static ones. You decide what gets extracted; the Actor handles the browser, the crawl queue, the retries and the limits. You pay **$0.003 per record delivered**. Nothing else — not pages visited, not retries, not time. ### What you get back, and how fast **Your records, not ours.** Whatever your page function returns *is* the dataset record — this Actor adds no fields and removes none. The default page function returns `url`, `title` and `text` (first 5 000 characters of the body), so a run works before you have written a line of your own. Measured on our last front-door check: **5 pages of a static site in 28 seconds**, one browser, no proxy. JavaScript-heavy pages are slower — that is Chromium rendering, not queue overhead — and a wider crawl scales with `maxConcurrency` rather than with time per page. **$3.00 per 1 000 records** ($0.003 each). A run that delivers nothing costs nothing: pages visited, retries and browser time are not billed. *** ### What can this Python web scraper do? - **Scrape a single page** — give it a URL and a page function, get a record back. - **Crawl a whole section of a site** — follow links by CSS selector, restrict them with glob patterns, stop at a hard request limit. - **Extract exactly the shape you want** — your function returns a Python dict, and that dict *is* the output record. No fixed template to fight, no fields you have to accept. - **Run JavaScript-heavy pages** — real Chromium, real rendering, configurable wait condition. - **Chain into other Actors** — the output dataset feeds straight into [Dataset Deduplicator & Cleaner](https://apify.com/eszetael_lab/dataset-deduplicator-cleaner) or any Apify integration. ### What websites can I scrape with it? Any site that a browser can open without credentials, and that allows crawling in its `robots.txt`. That includes: documentation sites, product catalogues, listing pages, blogs, news archives, government registers, price pages, job boards that aren't behind a login, and the long tail of sites nobody has written a dedicated scraper for. It does **not** include sites behind industrial anti-bot protection or a login — see [When is this the wrong tool?](#when-is-this-the-wrong-tool) below. That section is there because a wasted run costs you money, and we would rather you not spend it. ### How do I use it? Two fields are all you need. The prefilled example works as-is — press **Start** and you get a record back. ```json { "startUrls": [{ "url": "https://apify.com" }], "pageFunction": "async def page_function(page, context, request):\n return {\n \"url\": page.url,\n \"title\": await page.title()\n }" } ``` Result: ```json { "url": "https://apify.com", "title": "Apify: Full-stack web scraping and data extraction platform" } ``` ### How do I write the page function? It is an ordinary Python `async` function with a fixed signature: ```python async def page_function(page, context, request): # page – Playwright Page instance (page.locator, page.title, page.content, …) # context – Actor context (key-value store, dataset, logging) # request – Current Request object (request.url, request.user_data, …) # Return a dict, or a list of dicts. Each dict becomes one result item. return { "url": page.url, "title": await page.title(), "h1": await page.locator("h1").first.inner_text(), } ``` Rules worth knowing: - **Return a dict** → one record. **Return a list of dicts** → many records from one page. - **Return `None`** → no record from this page, and no charge. That is the supported way of saying "this page had nothing for me". - The function is compiled **before the crawl starts**. A syntax error fails the run in the first second, not after an hour of crawling. - An exception thrown for one page does not kill the run. That page is counted as an error, the crawl continues, and the count is reported in the final status message. ### How do I crawl and follow links? ```json { "startUrls": [{ "url": "https://example.com/products" }], "linkSelector": "a.product-link", "includeGlobs": ["https://example.com/products/*"], "maxRequestsPerCrawl": 200, "pageFunction": "async def page_function(page, context, request):\n return {\n \"url\": page.url,\n \"name\": await page.locator(\"h1\").inner_text(),\n \"price\": await page.locator(\".price\").inner_text()\n }" } ``` - **`linkSelector`** finds links on each visited page. - **`includeGlobs`** decides which of those links are worth enqueueing. Without it, a crawl will happily wander off into the rest of the site — and every page it visits is a page you waited for. - **`maxRequestsPerCrawl`** is a hard stop on total requests. ### ⬇️ Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `startUrls` | array of objects | — | Seed URLs, as `{ "url": "…" }`. At least one. | | `pageFunction` | string | — | Python source of the async page function. | | `linkSelector` | string | — | CSS selector for links to follow. Empty = no crawling. | | `includeGlobs` | array of strings | — | Glob patterns a URL must match to be enqueued. | | `maxRequestsPerCrawl` | integer | 100 | Hard limit on requests for the whole run. | | `maxConcurrency` | integer | 5 | Parallel browser contexts. | | `requestTimeoutSecs` | integer | 45 | Per-page timeout. Bounds a page that never settles. | | `waitUntil` | string | `load` | Playwright wait condition: `load`, `domcontentloaded`, `networkidle`. | | `respectRobotsTxt` | boolean | `true` | Obey `robots.txt`. Disallowed URLs are skipped *before* being visited. | | `proxyConfiguration` | object | Apify proxy | Proxy settings. | | `maxItems` | integer | 0 (no limit) | Stop after this many records. Hard cap 50 000 per run. | ### ⬆️ Output — what you get back **The shape of each record is yours.** Whatever dictionary your page function returns is written to the dataset unchanged. That is the whole point of this Actor, and it is why it ships no fixed output schema: any schema would be a promise about code *you* wrote, not code *we* wrote. #### One record per page ```json { "url": "https://example.com/products/wrench", "name": "Torque wrench 1/2\"", "price": "€89.00" } ``` #### Many records from one page Return a list, and each element becomes its own record — useful for a listing page: ```python async def page_function(page, context, request): rows = [] for card in await page.locator(".product-card").all(): rows.append({ "name": await card.locator(".name").inner_text(), "price": await card.locator(".price").inner_text(), }) return rows ``` #### What the Actor guarantees, and what your page function decides The Actor guarantees: - every record you return is written **exactly once**, and charged **exactly once**; - a page that throws is isolated — the crawl continues and you get the records that worked; - `maxRequestsPerCrawl` and `maxItems` are hard stops, so a crawl cannot run away with your budget; - `requestTimeoutSecs` bounds a page that never settles; - a run that delivers nothing **because something broke** fails loudly instead of reporting success. Your page function decides everything else: which elements to read, how to name the fields, what to skip. ### What happens when something fails? Errors are **not** written into your dataset as fake records — you are never charged for a failure. They are counted and reported, so a run always tells you what actually happened. | Situation | What the Actor does | |---|---| | `pageFunction` has a syntax error | Fails **before the crawl starts**, with the syntax error in the status message. | | `startUrls` is empty | Fails immediately with a clear message. | | One page throws inside your function | Counted as a page error, crawl continues. Reported as `(N page(s) errored in the page function)`. | | A URL is unreachable after retries | Counted as a failed request. Reported as `(N request(s) failed at network/navigation)`. | | **Zero records, and pages errored or requests failed** | The run **fails**, with the reason: *every page errored in the page function*, or *all requests failed (site unreachable/blocked?)*. A broken run never reports success. | | Zero records, and nothing errored | Succeeds with `Scraped 0 item(s)` — your selectors matched nothing. Costs you nothing. | | `maxItems` reached | Stops cleanly, status says `(reached maxItems=N)`. | | Billing call fails repeatedly | The run **stops** rather than doing unpaid work, and says how many items it delivered first. | That fifth row is the one that matters most. A scraper that returns an empty dataset and calls it success is the single most expensive failure mode there is, because your pipeline keeps running on nothing. This Actor refuses to do it. ### How much does it cost? - **$0.003 per record** delivered (`result-item`), flat, regardless of how many pages were walked to find it. - **An empty run costs nothing.** No record, no charge. - Nothing is charged for pages visited, retries, browser time, or bandwidth. - To try it cheaply, set a small `maxItems` — you pay only for what you actually receive. For 1 000 records that is **$3.00**. A crawl of 200 pages that yields 200 records costs $0.60, whether those pages took two minutes or twenty. ### When is this the wrong tool? Be honest with yourself about the target before you spend a run: - **Sites behind industrial anti-bot protection** — major marketplaces, search engines, large social networks. They rate-limit or block datacenter IP addresses regardless of how good the browser automation is. Those targets need a residential proxy, and a generic crawler is not the reason they work or fail. - **Content behind a login.** This Actor does not carry your credentials, by design. - **A site somebody already solved well.** If a dedicated Actor exists for your target, it will handle that site's quirks better than a page function you write today. Use this when you need *your own* extraction logic on a site nobody has built a dedicated tool for. ### FAQ #### Can I use it with the Apify API? Yes. Start it like any Actor — `POST /v2/acts/eszetael_lab~reliable-playwright-scraper/runs` with your input as the JSON body, then read the run's dataset. Everything this Actor does is available through the standard API, CLI and client libraries. #### Can I use it through an MCP server? Yes. It is exposed through Apify's Actors MCP server like any other public Actor, so an AI agent can call it as a tool. It is also enabled for **agentic payments**, meaning an agent can run it and be charged directly without a human in the loop. #### Can I schedule it to run every day? Yes — use Apify **Schedules**. A common pattern is a daily crawl with a `maxItems` cap, chained into the Dataset Deduplicator & Cleaner so you only ever look at what is new. #### Does it respect robots.txt? By default, yes. Disallowed URLs are skipped *before* they are visited, so a blocked page never costs you a request. Turn `respectRobotsTxt` off only for sites you own or are otherwise authorised to crawl. #### Can it log in to a website? No, and that is deliberate. This Actor does not carry credentials. If a site requires a session, this is the wrong tool. #### Where does my page function run? Inside your own run, on your own account, like any other code you run on the platform. A per-page timeout bounds runaway code, and `maxRequestsPerCrawl` bounds runaway crawls. #### Is it legal to scrape with this Actor? Scraping publicly available data is broadly lawful in the EU and the US, but "broadly" is not "always". You are responsible for ensuring your use complies with applicable law and the target site's terms of service, and for not collecting personal data you have no basis to collect. `robots.txt` is respected by default because it is the site's own machine-readable statement of what it permits. #### Your feedback Found a bug, or a site where this behaves badly? Open an issue on the Actor's **Issues** tab. Real failure reports are worth more to us than feature requests. ### Related actors Three tools built to chain into each other — scrape, then clean, then use. - **[Dataset Deduplicator & Cleaner](https://apify.com/eszetael_lab/dataset-deduplicator-cleaner)** — pass this Actor's dataset ID straight in to remove duplicates across runs and clean the fields before analysis. Six times cheaper than a scraper, because it processes data you already paid for. - **[Bluesky Scraper](https://apify.com/eszetael_lab/bluesky-scraper)** — when your target is Bluesky rather than a website, use the protocol directly instead of a browser: no login, no proxy, and an incremental mode that returns only new posts. All three are on pay-per-result pricing, and an empty run costs nothing in every one of them. # Actor input Schema ## `startUrls` (type: `array`): URLs to start crawling from. ## `pageFunction` (type: `string`): Async Python run on each page. Signature: async def page\_function(page, context, request). `page` is a Playwright Page, `request.url` is the current URL. Return a dict or list of dicts to save to the dataset (or None to skip). If omitted, a default function returns the page URL, title and the first 5000 characters of visible text — so a minimal call works without writing any code. ## `linkSelector` (type: `string`): CSS selector for links to enqueue and follow (e.g. a\[href]). Empty = scrape only the start URLs. ## `includeGlobs` (type: `array`): Only enqueue URLs matching these glob patterns, e.g. https://example.com/product/\*. Both shapes are accepted: a plain string, or the {"glob": "..."} object the Console editor writes. ## `maxRequestsPerCrawl` (type: `integer`): Hard cap on the number of pages crawled. ## `maxConcurrency` (type: `integer`): Maximum pages loaded in parallel. ## `requestTimeoutSecs` (type: `integer`): How long to wait for a single page (load + page function) before giving up. ## `waitUntil` (type: `string`): When to consider a page loaded before running the page function. ## `respectRobotsTxt` (type: `boolean`): Skip URLs disallowed by the site's robots.txt or TDM reservation. Keep on unless you own the target site. ## `proxyConfiguration` (type: `object`): Proxy for the requests. Datacenter is fine for most sites; use RESIDENTIAL for heavily protected ones. ## `blockResources` (type: `boolean`): Stop the browser downloading images, fonts, media and stylesheets. The HTML, the DOM and every URL in it are unchanged — you still get image src attributes, you just do not pay to fetch the bytes. On residential proxy (billed per gigabyte) this is usually the single largest saving available, and pages load faster. Turn it off if your page function takes screenshots or needs rendered layout. ## `sessionMode` (type: `string`): How browser sessions (cookies, storage, proxy exit node) are reused across pages. 'auto' rotates them, which is right for crawling public pages at scale. 'sticky' keeps ONE session for the whole run — use it when your page function logs in, because a rotating session drops the cookie the site just issued and you end up in a login loop. 'off' disables session handling entirely. ## `maxItems` (type: `integer`): Hard cap on delivered records. You are billed per delivered record ($0.003), so this is also your cost ceiling: the default of 100 caps a run at $0.30. Set 0 for no cap (up to the safety limit of 50000, i.e. $150). ## `skipEmptyRecords` (type: `boolean`): A page that did not open — a bot wall, a timeout mid-render — still produces a record, usually {"url": ..., "title": "", "text": ""}. You are charged per delivered record, so those cost you full price for nothing. With this on they are neither delivered nor charged, and the run log says how many there were. Turn it off if you want a row for every page attempted, including the ones that came back empty. Responses that are not a document (PDF, video, archive) or that declare a size above 25 MB are skipped the same way. Note the limit of that check: a server that does not send Content-Length cannot be judged before its body is downloaded, so an undeclared huge response still reaches the browser. ## Actor input object example ```json { "startUrls": [ { "url": "https://quotes.toscrape.com" } ], "pageFunction": "async def page_function(page, context, request):\n rows = []\n for q in await page.locator('.quote').all():\n rows.append({\n 'text': await q.locator('.text').inner_text(),\n 'author': await q.locator('.author').inner_text(),\n 'tags': await q.locator('.tags a.tag').all_inner_texts(),\n })\n return rows", "maxRequestsPerCrawl": 1, "maxConcurrency": 5, "requestTimeoutSecs": 45, "waitUntil": "load", "respectRobotsTxt": true, "proxyConfiguration": { "useApifyProxy": true }, "blockResources": true, "sessionMode": "auto", "maxItems": 5, "skipEmptyRecords": true } ``` # Actor output Schema ## `records` (type: `string`): One record per page, exactly as your pageFunction returned it. With the default pageFunction: url, title, text (first 5000 characters of the body). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://quotes.toscrape.com" } ], "pageFunction": `async def page_function(page, context, request): rows = [] for q in await page.locator('.quote').all(): rows.append({ 'text': await q.locator('.text').inner_text(), 'author': await q.locator('.author').inner_text(), 'tags': await q.locator('.tags a.tag').all_inner_texts(), }) return rows`, "maxRequestsPerCrawl": 1, "waitUntil": "load", "respectRobotsTxt": true, "maxItems": 5 }; // Run the Actor and wait for it to finish const run = await client.actor("eszetael_lab/reliable-playwright-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://quotes.toscrape.com" }], "pageFunction": """async def page_function(page, context, request): rows = [] for q in await page.locator('.quote').all(): rows.append({ 'text': await q.locator('.text').inner_text(), 'author': await q.locator('.author').inner_text(), 'tags': await q.locator('.tags a.tag').all_inner_texts(), }) return rows""", "maxRequestsPerCrawl": 1, "waitUntil": "load", "respectRobotsTxt": True, "maxItems": 5, } # Run the Actor and wait for it to finish run = client.actor("eszetael_lab/reliable-playwright-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://quotes.toscrape.com" } ], "pageFunction": "async def page_function(page, context, request):\\n rows = []\\n for q in await page.locator('\''.quote'\'').all():\\n rows.append({\\n '\''text'\'': await q.locator('\''.text'\'').inner_text(),\\n '\''author'\'': await q.locator('\''.author'\'').inner_text(),\\n '\''tags'\'': await q.locator('\''.tags a.tag'\'').all_inner_texts(),\\n })\\n return rows", "maxRequestsPerCrawl": 1, "waitUntil": "load", "respectRobotsTxt": true, "maxItems": 5 }' | apify call eszetael_lab/reliable-playwright-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=eszetael_lab/reliable-playwright-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/acts/9JNwFHbVdWLt1wRfy/builds/noucsIhgQ7q2nfWVg/openapi.json