# RegExp Scraper (`ib4ngz/regexp-scraper`) Actor

This actor scrapes data from a list of provided URLs using regular expressions for precise and customizable pattern matching. It can handle both static and dynamic web pages and supports depth-based crawling to explore links and extract data from multiple levels of the web.

- **URL**: https://apify.com/ib4ngz/regexp-scraper.md
- **Developed by:** [Iqbal R](https://apify.com/ib4ngz) (community)
- **Categories:** Automation, Developer tools
- **Stats:** 10 total users, 0 monthly users, 60.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RegExp Scraper

This actor scrapes data from a list of provided URLs using regular expressions for precise and customizable pattern matching. It can handle both static and dynamic web pages and supports depth-based crawling to explore links and extract data from multiple levels of the web.

### Features

- **RegEx-Based Scraping**: Allows precise and customizable data extraction using regular expressions to match specific patterns on web pages.
- **Static and Dynamic Page Support**: Capable of handling both static content and dynamically loaded content (e.g., JavaScript-rendered pages).
- **Depth-Based Crawling**: Supports crawling with configurable depth, allowing you to scrape data from multiple levels of linked pages.
- **Flexible Input Configuration**: Accepts a list of starting URLs and provides advanced configuration for customizing the crawl behavior.
- **Proxy Configuration**: Advanced proxy support for anonymous scraping and bypassing IP restrictions.
- **Unique Dataset**: Ensures only unique matches are saved by preventing duplicates during the crawling process.

### Input Schema

- **startUrls**: A list of URLs to start the crawling process from. These URLs will be used as entry points for the scraper.
- **maxDepth**: The maximum depth for crawling. It defines how many levels of linked pages will be crawled starting from the start URLs.
- **patterns**: Regular expressions (RegEx) used to extract data from HTML content. Each pattern should be written on a new line.
- **crawlerType**: The type of crawler to use.
  - **Crawlee + Cheerio**: A fast crawler that uses Cheerio for parsing HTML content. It does not execute JavaScript and is suitable for static pages.
  - **Crawlee + Puppeteer + Chrome**: A slower crawler that uses Puppeteer and headless Chrome to render JavaScript and load dynamic content, making it suitable for JavaScript-heavy websites.
- **proxyConfiguration**: Configuration for using proxies to anonymize requests and avoid IP blocking.

### Dataset Schema

- **match**: The match found based on the provided regular expression. It is displayed as text.

### How to Use

1. **Configure Input**:

   - Provide a list of `startUrls` where the scraper should begin its operation. These URLs should be valid web addresses that the scraper will visit.
   - Set the `maxDepth` to control how deep the crawler will follow links on each page. A depth of 1 means only the start page will be scraped, while higher values will scrape linked pages.

2. **Set Regex Patterns**:

   - Define the regex patterns in the `patterns` field. These patterns will be used to search through the HTML content of the scraped pages. Each pattern should be on a new line.

3. **Choose Crawler Type**:

   - Select a `crawlerType` based on your needs:
     - `Crawlee + Cheerio`: Suitable for fast scraping of static HTML pages.
     - `Crawlee + Puppeteer + Chrome`: Required for scraping JavaScript-heavy websites that rely on dynamic content.

4. **Advanced Configuration (Optional)**:
   - Optionally configure proxy settings under `proxyConfiguration` to route your requests through a proxy.

5. **Run the Actor**:

   - Run the actor, and it will start crawling the provided URLs, applying regex patterns to find matches in the page content.
   - The results will be saved in the dataset for later analysis.

6. **Review Results**:
   - After execution, you can access the results through the Output tab. The matches found by the regex patterns will be displayed here.

### Conclusion

The **RegExp Scraper** is a powerful tool designed to scrape both static and dynamic content from websites using flexible regex patterns. Whether you're scraping simple static HTML pages or more complex sites that rely on JavaScript for rendering content, this actor provides the necessary capabilities to handle different types of web scraping tasks.

With customizable options such as crawling depth, regex pattern matching, and the ability to select between different types of crawlers, users can tailor the scraping process to their specific needs. The integration of proxy configuration and the ability to store results in a structured dataset also makes this actor ideal for large-scale scraping operations.

By following the provided input and configuration guidelines, users can easily deploy the RegExp Scraper to gather valuable data from a wide variety of websites. Whether you're a developer, data scientist, or anyone looking to extract structured information from the web, this actor offers a robust and flexible solution for your web scraping needs.

# Actor input Schema

## `startUrls` (type: `array`):

URLs to start with.

## `maxDepth` (type: `integer`):

The maximum depth for crawling.

## `patterns` (type: `string`):

Patterns to search in the HTML content. Each pattern should be on a new line.

## `crawlerType` (type: `string`):

Select the type of crawler to use.

## `proxyConfiguration` (type: `object`):

Select proxies to be used by your crawler.

## `minConcurrency` (type: `integer`):

The minimum number of concurrent requests or pages being processed.

## `maxConcurrency` (type: `integer`):

The maximum number of concurrent requests or pages being processed.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxDepth": 1,
  "patterns": "(?<=href=[\"'])([^\"']+)",
  "crawlerType": "Crawlee + Cheerio",
  "minConcurrency": 1,
  "maxConcurrency": 10
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "patterns": "(?<=href=[\"'])([^\"']+)",
    "crawlerType": "Crawlee + Cheerio"
};

// Run the Actor and wait for it to finish
const run = await client.actor("ib4ngz/regexp-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "patterns": "(?<=href=[\"'])([^\"']+)",
    "crawlerType": "Crawlee + Cheerio",
}

# Run the Actor and wait for it to finish
run = client.actor("ib4ngz/regexp-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "patterns": "(?<=href=[\\"'\''])([^\\"'\'']+)",
  "crawlerType": "Crawlee + Cheerio"
}' |
apify call ib4ngz/regexp-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=ib4ngz/regexp-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WmKUqdLCVcAxwVpQK/builds/E1cjNZGdpdt0U8YAI/openapi.json