# Scrappey Web Scraper – Managed Browser & Proxy API (`dormic/apify-scrappey`) Actor

Scrape data from modern, JavaScript-heavy web pages using the Scrappey.com API integrated with an Apify Actor. A robust solution for complex scraping scenarios — managed browser sessions, proxy rotation, and full browser automation.

- **URL**: https://apify.com/dormic/apify-scrappey.md
- **Developed by:** [Pim](https://apify.com/dormic) (community)
- **Categories:** Automation, Developer tools, Open source
- **Stats:** 171 total users, 2 monthly users, 100.0% runs succeeded, 10 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Scrappey Web Scraper for Apify

A web scraping actor for reliable access to modern, JavaScript-heavy websites. Handles automated verification flows, managed browser sessions, proxy rotation, and full browser automation. Powered by the [Scrappey.com](https://scrappey.com) API.

### Features

- **Managed Web Access** - Reliably retrieve content from dynamic sites that require a real browser environment
- **Verification Flow Handling** - Automatically work through common challenge and verification steps where access is permitted
- **Browser Automation** - Full browser control with 20+ action types including click, type, scroll, and JavaScript execution
- **Session Management** - Maintain cookies and state across multiple requests
- **Proxy Support** - Built-in proxy rotation with country selection, premium, and mobile proxy options
- **All HTTP Methods** - Support for GET, POST, PUT, DELETE, and PATCH requests
- **Data Extraction** - CSS selectors, regex patterns, screenshots, and video recording

### Quick Start

#### Input Configuration

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "cmd": "request.get"
}
```

Get your API key at <https://scrappey.com>

#### Basic GET Request

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://httpbin.rs/get",
    "cmd": "request.get"
}
```

#### POST Request with Data

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://httpbin.rs/post",
    "cmd": "request.post",
    "postData": {
        "username": "user",
        "password": "pass"
    }
}
```

#### Browser Automation

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com/login",
    "browserActions": [
        { "type": "wait_for_selector", "cssSelector": "#login-form" },
        { "type": "type", "cssSelector": "#username", "text": "myuser" },
        { "type": "type", "cssSelector": "#password", "text": "mypassword" },
        { "type": "click", "cssSelector": "#submit", "waitForSelector": ".dashboard" }
    ]
}
```

#### Automatic Verification Handling

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "automaticallySolveCaptchas": true
}
```

#### Geo-Targeted Requests

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "proxyCountry": "UnitedStates",
    "premiumProxy": true
}
```

#### Screenshot Capture

```json
{
    "scrappeyApiKey": "your-api-key",
    "url": "https://example.com",
    "screenshot": true,
    "screenshotUpload": true
}
```

### Input Options

#### Required

| Option | Type | Description |
|--------|------|-------------|
| `scrappeyApiKey` | string | Your Scrappey API key |
| `url` | string | Target URL |

#### HTTP Method

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `cmd` | string | `request.get` | HTTP method: `request.get`, `request.post`, `request.put`, `request.delete`, `request.patch` |
| `postData` | object | - | Request body for POST, PUT, PATCH |
| `requestType` | string | `browser` | `browser` for full rendering, `request` for HTTP-only (faster) |

#### Proxy Configuration

| Option | Type | Description |
|--------|------|-------------|
| `proxy` | string | Custom proxy URL (http://user:pass@ip:port) |
| `proxyCountry` | string | Request a proxy from a specific country (e.g., UnitedStates, Germany) |
| `premiumProxy` | boolean | Use premium residential proxies |
| `mobileProxy` | boolean | Use mobile carrier proxies |

#### Verification Handling

| Option | Type | Description |
|--------|------|-------------|
| `automaticallySolveCaptchas` | boolean | Automatically work through detected verification steps |

#### Response Options

| Option | Type | Description |
|--------|------|-------------|
| `screenshot` | boolean | Capture page screenshot |
| `screenshotUpload` | boolean | Upload screenshot to storage |
| `video` | boolean | Record browser session |
| `cssSelector` | string | Extract content by CSS selector |
| `innerText` | boolean | Include page text content |
| `includeImages` | boolean | Include image URLs |
| `includeLinks` | boolean | Include link URLs |

#### Session and Cookies

| Option | Type | Description |
|--------|------|-------------|
| `session` | string | Session ID for state persistence |
| `cookiejar` | array | Cookies to set before request |
| `localStorage` | object | Local storage data to set |

### Output Data

```json
{
    "url": "https://example.com",
    "cmd": "request.get",
    "verified": true,
    "statusCode": 200,
    "currentUrl": "https://example.com",
    "userAgent": "Mozilla/5.0...",
    "cookies": [],
    "cookieString": "",
    "responseHeaders": {},
    "requestHeaders": {},
    "html": "<html>...</html>",
    "innerText": "Page content...",
    "ipInfo": {
        "country": "United States",
        "city": "New York"
    },
    "session": "session-id",
    "timeElapsed": 1234,
    "screenshot": "base64...",
    "screenshotUrl": "https://...",
    "timestamp": "2025-01-09T12:00:00.000Z"
}
```

### Browser Actions

Execute automated browser interactions:

| Action | Description |
|--------|-------------|
| `click` | Click an element |
| `type` | Type text into input |
| `wait` | Wait for milliseconds |
| `wait_for_selector` | Wait for element to appear |
| `scroll` | Scroll to element or bottom |
| `hover` | Hover over element |
| `keyboard` | Press keyboard keys |
| `dropdown` | Select dropdown option |
| `execute_js` | Run JavaScript code |
| `if` | Conditional execution |
| `while` | Loop execution |
| `goto` | Navigate to URL |

#### Example: Login Flow

```json
{
    "browserActions": [
        { "type": "wait_for_selector", "cssSelector": "#login-form" },
        { "type": "type", "cssSelector": "#email", "text": "user@example.com" },
        { "type": "type", "cssSelector": "#password", "text": "password123" },
        { "type": "click", "cssSelector": "#submit", "waitForSelector": ".dashboard" },
        { "type": "execute_js", "code": "document.querySelector('.user-name').innerText" }
    ]
}
```

### Error Codes

| Code | Description | Solution |
|------|-------------|----------|
| CODE-0001 | Server overload | Retry with backoff |
| CODE-0002 | Request not completed | Try a different proxy |
| CODE-0007 | Proxy error | Check proxy credentials |
| CODE-0010 | Access not completed | Use a premium proxy |
| CODE-0029 | Too many sessions | Wait for cleanup |
| CODE-0032 | Verification not completed | Retry request |

### Local Development

#### Setup

```bash
git clone https://github.com/pim97/apify-scrappey
cd apify-scrappey
npm install
```

#### Run Locally

```bash
## Set input in storage/key_value_stores/default/INPUT.json
npm run start:dev
```

#### Run Tests

```bash
npm test
```

#### Build

```bash
npm run build
```

#### Deploy to Apify

```bash
apify login
apify push
```

### CI/CD

The repository includes GitHub Actions for automatic deployment:

1. Tests run on every push and pull request
2. Deployment to Apify triggers on:
   - Push to main/master branch
   - New release published

Set `APIFY_TOKEN` secret in your repository settings.

### Resources

- [Scrappey Documentation](https://docs.scrappey.com)
- [Scrappey API Reference](https://docs.scrappey.com/api-reference)
- [Apify SDK Documentation](https://docs.apify.com/sdk/js)
- [Request Builder](https://app.scrappey.com/#/builder)

### Support

- [Scrappey Support](https://scrappey.com/support)
- [GitHub Issues](https://github.com/pim97/apify-scrappey/issues)
- [Apify Discord](https://discord.com/invite/jyEM2PRvMU)

### License

MIT License

# Actor input Schema

## `scrappeyApiKey` (type: `string`):

Your Scrappey API key. Get one at https://scrappey.com

## `url` (type: `string`):

The URL of the website to scrape

## `cmd` (type: `string`):

The HTTP method to use for the request

## `postData` (type: `object`):

Data to send with POST, PUT, or PATCH requests. Can be form data or JSON.

## `requestType` (type: `string`):

Use 'browser' for full browser rendering (slower, handles JS) or 'request' for HTTP-only mode (faster, cheaper)

## `proxyCountry` (type: `string`):

Request a proxy from a specific country (e.g., UnitedStates, Germany, UnitedKingdom)

## `proxy` (type: `string`):

Custom proxy URL in format: http://username:password@ip:port

## `premiumProxy` (type: `boolean`):

Use premium residential proxies for better success rates

## `mobileProxy` (type: `boolean`):

Use mobile carrier proxies

## `session` (type: `string`):

Session ID for maintaining state across requests. Leave empty for automatic session.

## `browserActions` (type: `array`):

Array of browser automation actions (click, type, scroll, wait, etc.)

## `automaticallySolveCaptchas` (type: `boolean`):

Automatically work through detected verification widgets where access is permitted

## `alwaysLoad` (type: `array`):

Verification widget types to always load handlers for

## `cloudflareBypass` (type: `boolean`):

Enable additional optimizations for sites with interstitial challenge pages

## `datadomeBypass` (type: `boolean`):

Enable additional optimizations for sites with advanced verification flows

## `kasadaBypass` (type: `boolean`):

Enable additional optimizations for sites with client-side verification

## `screenshot` (type: `boolean`):

Capture a screenshot of the page

## `screenshotUpload` (type: `boolean`):

Upload screenshot to storage and return URL

## `video` (type: `boolean`):

Record browser session as video

## `cssSelector` (type: `string`):

Extract content matching this CSS selector

## `innerText` (type: `boolean`):

Include the inner text content of the page

## `includeImages` (type: `boolean`):

Include all image URLs found on the page

## `includeLinks` (type: `boolean`):

Include all link URLs found on the page

## `customHeaders` (type: `object`):

Custom HTTP headers to send with the request

## `cookiejar` (type: `array`):

Array of cookies to set before making the request

## `localStorage` (type: `object`):

Local storage data to set in the browser

## `interceptFetchRequest` (type: `string`):

URL pattern(s) to intercept and return (comma-separated for multiple)

## `fullPageLoad` (type: `boolean`):

Wait for full page load including all resources

## `timeout` (type: `integer`):

Request timeout in milliseconds

## Actor input object example

```json
{
  "url": "https://httpbin.rs/get",
  "cmd": "request.get",
  "requestType": "browser",
  "premiumProxy": false,
  "mobileProxy": false,
  "browserActions": [],
  "automaticallySolveCaptchas": false,
  "cloudflareBypass": false,
  "datadomeBypass": false,
  "kasadaBypass": false,
  "screenshot": false,
  "screenshotUpload": false,
  "video": false,
  "innerText": false,
  "includeImages": false,
  "includeLinks": false,
  "fullPageLoad": false,
  "timeout": 300000
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "url": "https://httpbin.rs/get",
    "browserActions": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("dormic/apify-scrappey").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "url": "https://httpbin.rs/get",
    "browserActions": [],
}

# Run the Actor and wait for it to finish
run = client.actor("dormic/apify-scrappey").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "url": "https://httpbin.rs/get",
  "browserActions": []
}' |
apify call dormic/apify-scrappey --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=dormic/apify-scrappey",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cLo1vfjaaMlAzaQao/builds/mI5zqpUKTsi4lGKxU/openapi.json