# Camoufox Scraper (`josef.prochazka/camoufox-scraper`) Actor Simple actor that uses Playwright with Camoufox to test if a specific website blocking mechanisms can be bypassed by using Camoufox. - **URL**: https://apify.com/josef.prochazka/camoufox-scraper.md - **Developed by:** [Josef Procházka](https://apify.com/josef.prochazka) (community) - **Categories:** Developer tools, Open source - **Stats:** 24 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Camoufox Scraper Camoufox Scraper is a ready-made solution for crawling websites using [Playwright](https://playwright.dev/python/) with [Camoufox](https://camoufox.com/) browser. It provides [page](https://playwright.dev/python/docs/api/class-page) to your defined function, which you can use to extract any data from the page. ### Usage To get started with Camoufox Scraper, you only need two things. First, tell the scraper which web pages it should load. Second, tell it how to extract data from each page. The scraper starts by loading the pages specified in the [**Start URLs**](#start-urls) field. You can make the scraper follow page links on the fly by setting a [**Link selector**](#link-selector), and [**Link patterns**](#link-patterns) to tell the scraper which links it should add to the crawling queue. This is useful for the recursive crawling of entire websites, e.g. to find all products in an online store. To tell the scraper how to extract data from web pages, you need to provide a [**Page function**](#page-function). This is Python code that is executed for every web page loaded. In summary, Camoufox Scraper works as follows: 1. Adds each [Start URL](#start-urls) to the crawling queue. 2. Fetches the first URL from the queue and constructs a DOM from the fetched HTML string. 3. Executes the [**Page function**](#page-function) on the loaded page and saves its results. 4. Optionally, finds all links from the page using the [**Link selector**](#link-selector). If a link matches any of the [**Link selector**](#link-selector) and has not yet been visited, add it to the queue. 5. If there are more items in the queue, repeats step 2, otherwise finish. ### Input configuration As input, the Beautifulsoup Scraper Actor accepts a number of configurations. These can be entered either manually in the user interface in [Apify Console](https://console.apify.com), or programmatically in a JSON object using the [Apify API](https://apify.com/docs/api/v2#/reference/actors/run-collection/run-actor). For a complete list of input fields and their types, please visit the [Input](https://apify.com/apify/beautifulsoup-scraper/input-schema) tab. #### Page function The **Page function** (`pageFunction`) field contains a Python script with a single function that enables the user to extract data from the web page, access its DOM, add new URLs to the request queue, and otherwise control Beautifulsoup Scraper's operation. Example: ```python from typing import Any from crawlee.crawlers import PlaywrightCrawlingContext async def page_function(context: PlaywrightCrawlingContext) -> Any: url = context.request["url"] title = await context.page.locator("title").first.inner_text() return {"url": url, "title": title} ``` #### Context The code runs in Python 3.12 and the `page_function` accepts a single argument `context` of type [PlaywrightCrawlingContext](https://crawlee.dev/python/api/class/PlaywrightCrawlingContext). See documentation link for further details ### Proxy configuration The **Proxy configuration** (`proxyConfiguration`) option enables you to set proxies that will be used by the scraper in order to prevent its detection by target web pages. You can use both the [Apify Proxy](https://apify.com/proxy) and custom HTTP or SOCKS5 proxy servers. Proxy is required to run the scraper. The following table lists the available options for the proxy configuration setting:
Apify Proxy (automatic) The scraper will load all web pages using the Apify Proxy in automatic mode. In this mode, the proxy uses all proxy groups that are available to the user. For each new web page, it automatically selects the proxy that hasn't been used in the longest time for the specific hostname in order to reduce the chance of detection by the web page. You can view the list of available proxy groups on the Proxy page in Apify Console.
Apify Proxy (selected groups) The scraper will load all web pages using the Apify Proxy with specific groups of target proxy servers.
Custom proxies

The scraper will use a custom list of proxy servers. The proxies must be specified in the scheme://user:password@host:port format. Multiple proxies should be separated by a space or new line. The URL scheme can be either http or socks5. The user and password might be omitted, but the port must always be present.

Example: 

http://bob:password@proxy1.example.com:8000
http://bob:password@proxy2.example.com:8000
The proxy configuration can be set programmatically when calling the Actor using the API by setting the `proxyConfiguration` field. It accepts a JSON object with the following structure: ```javascript { // Indicates whether to use the Apify Proxy or not. "useApifyProxy": Boolean, // Array of Apify Proxy groups, only used if "useApifyProxy" is true. // If missing or null, the Apify Proxy will use automatic mode. "apifyProxyGroups": String[], // Array of custom proxy URLs, in "scheme://user:password@host:port" format. // If missing or null, custom proxies are not used. "proxyUrls": String[], } ``` ### Results The scraping results returned by [**Page function**](#page-function) are stored in the default dataset associated with the Actor run, from where you can export them to formats such as JSON, XML, CSV, or Excel. To download the results, call the [Get dataset items](https://docs.apify.com/api/v2#/reference/datasets/item-collection) API endpoint: ``` https://api.apify.com/v2/datasets/[DATASET_ID]/items?format=json ``` where `[DATASET_ID]` is the ID of the Actor's run dataset, in which you can find the Run object returned when starting the Actor. Alternatively, you'll find the download links for the results in Apify Console. To skip the `#error` and `#debug` metadata fields from the results and not include empty result records, simply add the `clean=true` query parameter to the API URL, or select the **Clean items** option when downloading the dataset in Apify Console. To get the results in other formats, set the `format` query parameter to `xml`, `xlsx`, `csv`, `html`, etc. For more information, see [Datasets](https://docs.apify.com/storage#dataset) in documentation or the [Get dataset items](https://docs.apify.com/api/v2#/reference/datasets/item-collection) endpoint in Apify API reference. # Actor input Schema ## `startUrls` (type: `array`): A static list of URLs to scrape. ## `maxCrawlingDepth` (type: `integer`): Specifies how many links away from the Start URLs the scraper will descend. Note that pages added using context.request\_queue in Page function are not subject to the maximum depth constraint. ## `maxRequestsPerCrawl` (type: `integer`): Crawler will stop after processing this amount of requests. ## `requestTimeout` (type: `integer`): The maximum duration (in seconds) for the request to complete before timing out. The timeout value is passed to the httpx.AsyncClient object. ## `linkSelector` (type: `string`): A CSS selector stating which links on the page (\ elements with href attribute) shall be followed and added to the request queue. To filter the links added to the queue, use the Link patterns field.

If the Link selector is empty, the page links are ignored. Of course, you can work with the page links and the request queue in the Page function as well. ## `linkPatterns` (type: `array`): Link patterns (regular expressions) to match links in the page that you want to enqueue. Combine with Link selector to tell the scraper where to find links. Omitting the link patterns will cause the scraper to enqueue all links matched by the Link selector. ## `pageFunction` (type: `string`): A Python function, that is executed for every page. Use it to scrape data from the page, perform actions or add new URLs to the request queue. The page function has its own naming scope and you can import any installed modules. Typically you would want to obtain the data from the context.soup object and return them. Identifier page\_function can't be changed. For more information about the context object you get into the page\_function check the github.com/apify/actor-beautifulsoup-scraper#context. Asynchronous functions are supported. ## `proxyConfiguration` (type: `object`): Specifies proxy servers that will be used by the scraper in order to hide its origin. ## Actor input object example ```json { "startUrls": [ { "url": "https://crawlee.dev" } ], "maxCrawlingDepth": 1, "maxRequestsPerCrawl": 1, "requestTimeout": 30, "linkSelector": "a[href]", "linkPatterns": [ ".*crawlee\\.dev.*" ], "pageFunction": "from typing import Any\nfrom crawlee.crawlers import PlaywrightCrawlingContext\n \nasync def page_function(context: PlaywrightCrawlingContext) -> Any:\n url = context.request.url\n title = await context.page.locator(\"title\").first.inner_text()\n return {'url': url, 'title': title}\n", "proxyConfiguration": { "useApifyProxy": true } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://crawlee.dev" } ], "maxCrawlingDepth": 1, "maxRequestsPerCrawl": 1, "requestTimeout": 30, "linkSelector": "a[href]", "linkPatterns": [ ".*crawlee\\.dev.*" ], "pageFunction": `from typing import Any from crawlee.crawlers import PlaywrightCrawlingContext async def page_function(context: PlaywrightCrawlingContext) -> Any: url = context.request.url title = await context.page.locator("title").first.inner_text() return {'url': url, 'title': title}`, "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("josef.prochazka/camoufox-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://crawlee.dev" }], "maxCrawlingDepth": 1, "maxRequestsPerCrawl": 1, "requestTimeout": 30, "linkSelector": "a[href]", "linkPatterns": [".*crawlee\\.dev.*"], "pageFunction": """from typing import Any from crawlee.crawlers import PlaywrightCrawlingContext async def page_function(context: PlaywrightCrawlingContext) -> Any: url = context.request.url title = await context.page.locator(\"title\").first.inner_text() return {'url': url, 'title': title} """, "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("josef.prochazka/camoufox-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://crawlee.dev" } ], "maxCrawlingDepth": 1, "maxRequestsPerCrawl": 1, "requestTimeout": 30, "linkSelector": "a[href]", "linkPatterns": [ ".*crawlee\\\\.dev.*" ], "pageFunction": "from typing import Any\\nfrom crawlee.crawlers import PlaywrightCrawlingContext\\n \\nasync def page_function(context: PlaywrightCrawlingContext) -> Any:\\n url = context.request.url\\n title = await context.page.locator(\\"title\\").first.inner_text()\\n return {'\''url'\'': url, '\''title'\'': title}\\n", "proxyConfiguration": { "useApifyProxy": true } }' | apify call josef.prochazka/camoufox-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=josef.prochazka/camoufox-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/818x19to5pH3FQjgK/builds/Mmp40Lbfq9JHVYoF4/openapi.json