# Job Board Aggregator (`mighty_monk/job-board-aggregator`) Actor

Aggregate remote and on-site job listings from Remotive, Arbeitnow, and custom job board URLs. Extract title, company, location, salary, remote flag, description, and apply URL.

- **URL**: https://apify.com/mighty\_monk/job-board-aggregator.md
- **Developed by:** [Harsh](https://apify.com/mighty_monk) (community)
- **Categories:** Automation, News
- **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 job listings

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Job Board Aggregator

Aggregate job listings from multiple sources into a single structured dataset. Built with **TypeScript**, **Crawlee**, and **Cheerio**.

### Features

- **Remotive API** — fetches remote jobs from `https://remotive.com/api/remote-jobs`
- **Arbeitnow API** — fetches jobs from `https://www.arbeitnow.com/api/job-board-api` with automatic pagination
- **Custom job boards** — optional Cheerio scraping for any HTML job listing page
- **Structured output** — title, company, location, salary, remote flag, description, apply URL
- **Rate limiting** — configurable concurrency and requests-per-minute caps
- **Retries** — failed requests are retried up to 3 times with logging

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `includeRemotive` | `boolean` | `true` | Enable Remotive API source |
| `includeArbeitnow` | `boolean` | `true` | Enable Arbeitnow API source |
| `customJobBoardUrls` | `string[]` | `[]` | Optional HTML job board URLs |
| `searchQuery` | `string` | — | Keyword filter for Remotive |
| `category` | `string` | — | Remotive category slug |
| `maxJobs` | `integer` | `500` | Max jobs across all sources |
| `maxRequestsPerCrawl` | `integer` | `200` | Max HTTP requests |
| `maxConcurrency` | `integer` | `2` | Parallel request limit |
| `maxRequestsPerMinute` | `integer` | `30` | Global rate limit |

#### Example input

```json
{
  "includeRemotive": true,
  "includeArbeitnow": true,
  "customJobBoardUrls": [],
  "searchQuery": "typescript",
  "maxJobs": 100,
  "maxConcurrency": 2,
  "maxRequestsPerMinute": 30
}
```

### Output

Each dataset item contains:

| Field | Description |
|-------|-------------|
| `title` | Job title |
| `company` | Company name |
| `location` | Location or region |
| `salary` | Salary text when available |
| `remote` | Whether the job is remote |
| `description` | Plain-text job description |
| `url` | Apply or detail page URL |
| `source` | `remotive`, `arbeitnow`, or `custom` |
| `scrapedAt` | ISO timestamp |

### How it works

1. The Actor builds start requests for each enabled source.
2. **Remotive** and **Arbeitnow** responses are parsed as JSON.
3. **Arbeitnow** pagination follows `links.next` until `maxJobs` is reached.
4. **Custom boards** use Cheerio heuristics for job cards and enqueue next-page links.
5. Results are pushed to the default dataset with per-job logging.

### Local development

```bash
npm install
apify run
```

Edit `storage/key_value_stores/default/INPUT.json` for local test input.

### Deploy

```bash
apify login
apify push
```

Publish to the Apify Store with PPE pricing:

```bash
node ../../scripts/publish-factory.js job-board-aggregator '{"resultPriceUsd":0.002,"resultTitle":"Job listing","resultDescription":"Charged per job listing returned"}'
```

### Notes

- Remotive requests should be kept infrequent (their API advises max ~4 calls/day for production sync).
- Arbeitnow does not expose salary in its public API.
- Custom HTML scraping uses common CSS patterns; highly custom sites may need dedicated selectors.
- Respect each source's terms of service and attribution requirements.

# Actor input Schema

## `includeRemotive` (type: `boolean`):

Fetch jobs from the Remotive public API (remote jobs).

## `includeArbeitnow` (type: `boolean`):

Fetch jobs from the Arbeitnow public API with automatic pagination.

## `customJobBoardUrls` (type: `array`):

Optional HTML job board listing pages to scrape with Cheerio (supports pagination).

## `searchQuery` (type: `string`):

Optional keyword filter passed to the Remotive API.

## `category` (type: `string`):

Optional Remotive category slug (e.g. software-dev, customer-support).

## `maxJobs` (type: `integer`):

Maximum total jobs to save across all sources.

## `maxRequestsPerCrawl` (type: `integer`):

Maximum HTTP requests the crawler may make (pagination + custom boards).

## `maxConcurrency` (type: `integer`):

Maximum parallel requests for rate limiting.

## `maxRequestsPerMinute` (type: `integer`):

Global rate limit to avoid overloading job board APIs.

## `useApifyProxy` (type: `boolean`):

Route requests through Apify Proxy. Disabled by default for public APIs; enable for custom boards behind geo-blocks.

## Actor input object example

```json
{
  "includeRemotive": true,
  "includeArbeitnow": true,
  "customJobBoardUrls": [],
  "maxJobs": 500,
  "maxRequestsPerCrawl": 200,
  "maxConcurrency": 2,
  "maxRequestsPerMinute": 30,
  "useApifyProxy": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "customJobBoardUrls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("mighty_monk/job-board-aggregator").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "customJobBoardUrls": [] }

# Run the Actor and wait for it to finish
run = client.actor("mighty_monk/job-board-aggregator").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "customJobBoardUrls": []
}' |
apify call mighty_monk/job-board-aggregator --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=mighty_monk/job-board-aggregator",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/SO1muXSKfdmVBKaTQ/builds/3RFb6HSZwadY9UhJ2/openapi.json