# Deduped Job Intelligence for AI Agents (`deepapi/job-intelligence-deduper`) Actor

Extract, normalize, and deduplicate public job postings into clean hiring-signal records with source evidence, role classification, and confidence scores.

- **URL**: https://apify.com/deepapi/job-intelligence-deduper.md
- **Developed by:** [DeepAPI](https://apify.com/deepapi) (community)
- **Categories:** Lead generation, AI, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 valid jobs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Deduped Job Intelligence for AI Agents

Extract, normalize, and deduplicate public job postings into clean hiring-signal records with source evidence, role classification, duplicate groups, and confidence scores.

Use this Actor when you need hiring intelligence rather than another raw job feed. Provide public careers, job-board, or direct posting URLs, and the Actor returns validated dataset rows that are ready for sales workflows, recruiting research, market analysis, and AI agents.

### What this Actor does

- Crawls user-supplied public careers, job-board, and direct job posting URLs.
- Extracts job candidates from supported ATS/job-board pages and fallback job links.
- Visits extracted job detail pages to enrich records with location, remote type, seniority, and evidence when available.
- Extracts public salary or compensation ranges from structured job data and visible job detail text when available.
- Normalizes job titles for cleaner grouping and analysis.
- Detects common role functions, seniority signals, locations, and remote types when present.
- Matches optional role keywords such as `account executive`, `customer success`, or `sales engineer`.
- Groups duplicate postings with a stable `duplicateGroupId`.
- Preserves source URLs and evidence for auditability.
- Filters output with `maxJobs` and `minConfidence`.
- Returns validated dataset rows suitable for automation and AI-agent workflows.

### Use cases

- B2B sales teams using hiring as an account signal.
- Lead generation agencies enriching company lists.
- Recruiters monitoring active roles across target companies.
- Investors and analysts tracking hiring momentum.
- AI agents that need normalized hiring data rather than raw HTML or duplicate job feeds.

### Supported source types

Use public source URLs such as company careers pages, public job-board pages, direct public job posting URLs, and public ATS pages.

The output schema currently detects these providers when available: Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Teamtailor, Breezy, and Workday.

Provider coverage depends on the public page structure available at crawl time.

### Input

Use public HTTP(S) `sourceUrls` as the primary input. Optional filters let you cap results, match role keywords, filter by locations, include duplicate source URLs, set clean-mode requirements, and set a minimum confidence threshold.

```json
{
  "sourceUrls": [
    "https://jobs.ashbyhq.com/notion",
    "https://jobs.lever.co/posthog"
  ],
  "maxJobs": 25,
  "roleKeywords": [
    "account executive",
    "customer success",
    "sales engineer",
    "solutions engineer",
    "engineer"
  ],
  "locations": ["United States", "Remote"],
  "includeSourceDuplicates": true,
  "requireLocation": false,
  "requireSalary": false,
  "requireRoleKeywordMatch": false,
  "minConfidence": 0.65
}
```

Clean mode toggles can return only jobs that are ready for downstream routing:

| Toggle | Effect |
|---|---|
| `requireLocation` | Return only jobs with a detected location. |
| `requireSalary` | Return only jobs with a detected public salary range. |
| `requireRoleKeywordMatch` | Return only jobs that matched at least one configured role keyword. |

### Output

Each pushed dataset item is a validated, normalized job intelligence record.

```json
{
  "companyName": "Notion",
  "jobTitle": "Account Executive, Commercial",
  "normalizedTitle": "Account Executive Commercial",
  "function": "sales",
  "location": "New York, United States",
  "salary": {
    "currency": "USD",
    "min": 150000,
    "max": 180000,
    "period": "year"
  },
  "sourceUrl": "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149",
  "sourceUrls": [
    "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149",
    "https://jobs.ashbyhq.com/notion/fdc2a6c0-396a-45db-b465-683bacf4201e"
  ],
  "sourceUrlsText": "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149 | https://jobs.ashbyhq.com/notion/fdc2a6c0-396a-45db-b465-683bacf4201e",
  "jobBoardProvider": "ashby",
  "duplicateGroupId": "notion:account-executive-commercial",
  "duplicateCount": 2,
  "roleKeywords": ["account executive"],
  "roleKeywordsText": "account executive",
  "hiringSignal": "Hiring Account Executive, Commercial",
  "evidence": [
    {
      "type": "job_title",
      "value": "Account Executive, Commercial",
      "sourceUrl": "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149"
    },
    {
      "type": "role_keyword",
      "value": "account executive",
      "sourceUrl": "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149"
    },
    {
      "type": "salary",
      "value": "USD 150000-180000 per year",
      "sourceUrl": "https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149"
    }
  ],
  "evidenceSummary": "job_title: Account Executive, Commercial (https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149) | salary: USD 150000-180000 per year (https://jobs.ashbyhq.com/notion/9526496b-5c39-456b-a454-ebec889e7149)",
  "salaryText": "USD 150000-180000 per year",
  "confidence": 0.95,
  "scrapedAt": "2026-06-29T15:52:57.933Z"
}
```

Spreadsheet preview:

| companyName | normalizedTitle | function | location | salaryText | roleKeywordsText | duplicateCount | sourceUrlsText |
|---|---|---|---|---|---|---:|---|
| Notion | Account Executive Commercial | sales | New York, United States | USD 150000-180000 per year | account executive | 2 | https://jobs.ashbyhq.com/notion/9526496b... | https://jobs.ashbyhq.com/notion/fdc2a6c0... |

### Example results

A local sample run against two public sources produced:

```text
Source URLs: 2
Requests processed: 2
Raw jobs found: 74
Valid jobs returned: 25
Pushed jobs: 25
```

The sample included merged duplicate sources for equivalent postings, role keyword matches, function detection, and source-backed evidence.

### Output Fields For CSV Buyers

The dataset keeps rich arrays and objects for agents, but also includes flat text fields for spreadsheet exports:

| Field | Meaning |
|---|---|
| `sourceUrlsText` | Pipe-separated source URLs represented by the deduplicated job. |
| `roleKeywordsText` | Pipe-separated matched role keywords. |
| `evidenceSummary` | Pipe-separated source evidence summary for review in CSV, Excel, or Sheets. |
| `salaryText` | Human-readable salary range when detected. |

### Troubleshooting

| Symptom | Most likely reason | What to do |
|---|---|---|
| No `validJob` rows | The public page did not expose job links/data matching your filters and `minConfidence`. | Lower `minConfidence`, remove restrictive `locations`, or pass the direct ATS/job-board URL. |
| Jobs are missing locations or salary | The public job detail page does not expose those fields in visible text or JSON-LD. | Keep the row if the title/source evidence is enough, or filter with `locations` only when location is required. |
| Fewer rows than expected | Duplicates were merged, charge limits stopped writes, or `maxJobs` capped output. | Check `OUTPUT.summary`, `duplicateCount`, and `sourceUrlsText`. |

### Limitations

- Works with public pages only.
- Does not log in to private job boards or social networks.
- Does not collect applicant data.
- Does not guarantee complete coverage for every ATS or custom careers site.
- Salary, department, seniority, location, and remote type are returned only when detected from public source content.
- Provider support can vary when public page markup changes.

### Local Development

From the portfolio root:

```bash
pnpm jobs:test
pnpm jobs:typecheck
pnpm jobs:sample
```

The sample script uses a separate `storage-live-sample` local storage directory and writes inspection files to `data/live-sample`.

### Pricing Unit

Recommended pay-per-event unit:

```text
validJob
```

The Actor will charge only when a normalized job passes validation and is pushed to the dataset.

# Actor input Schema

## `sourceUrls` (type: `array`):

Public careers, job-board, or direct job posting URLs to normalize and deduplicate.

## `maxJobs` (type: `integer`):

Maximum number of valid normalized jobs to return.

## `roleKeywords` (type: `array`):

Optional keywords used to classify and filter role relevance.

## `locations` (type: `array`):

Optional location strings used to filter returned jobs and classify matching locations.

## `includeSourceDuplicates` (type: `boolean`):

Include duplicate source URLs in each output record.

## `requireLocation` (type: `boolean`):

Clean mode: return only jobs with a detected location.

## `requireSalary` (type: `boolean`):

Clean mode: return only jobs with a detected public salary range.

## `requireRoleKeywordMatch` (type: `boolean`):

Clean mode: return only jobs that matched at least one configured role keyword.

## `minConfidence` (type: `number`):

Minimum confidence score required before a job is returned.

## Actor input object example

```json
{
  "sourceUrls": [
    "https://jobs.ashbyhq.com/notion",
    "https://jobs.lever.co/posthog"
  ],
  "maxJobs": 1000,
  "roleKeywords": [],
  "locations": [],
  "includeSourceDuplicates": true,
  "requireLocation": false,
  "requireSalary": false,
  "requireRoleKeywordMatch": false,
  "minConfidence": 0.65
}
```

# Actor output Schema

## `normalizedJobsUrl` (type: `string`):

Dataset items returned by this Actor. Each item is a validated deduplicated job record.

## `runOutput` (type: `string`):

OUTPUT key-value store record with run-level counts, filter diagnostics, source-level result counts, and charge-limit status.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sourceUrls": [
        "https://jobs.ashbyhq.com/notion",
        "https://jobs.lever.co/posthog"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("deepapi/job-intelligence-deduper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sourceUrls": [
        "https://jobs.ashbyhq.com/notion",
        "https://jobs.lever.co/posthog",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("deepapi/job-intelligence-deduper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sourceUrls": [
    "https://jobs.ashbyhq.com/notion",
    "https://jobs.lever.co/posthog"
  ]
}' |
apify call deepapi/job-intelligence-deduper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=deepapi/job-intelligence-deduper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2Q304cgUfvevIydXn/builds/K2Syk98S7ODZbY3Nk/openapi.json