# Hacker News Scraper — Stories, Comments, Ask HN, Show HN (`foxlabs/hackernews-intelligence`) Actor

Scrape Hacker News stories and full comment threads via the official Firebase API. Top stories, Ask HN, Show HN, jobs, new. Built for founder/VC sentiment monitoring, topic clustering, trend tracking. Clean JSON, comment tree depth-controlled.

- **URL**: https://apify.com/foxlabs/hackernews-intelligence.md
- **Developed by:** [Berkan Kaplan](https://apify.com/foxlabs) (community)
- **Categories:** Developer tools, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Search, Who-is-Hiring Jobs & Feeds

Search **the entire Hacker News archive** — stories *and* comments, back to 2007 — with **no 1,000-result wall**, turn the monthly **"Ask HN: Who is hiring?"** thread into **structured job posts with contact e-mails**, and pull the live front page. Built on Hacker News' **official APIs**: no key, no login, no proxies, no HTML scraping to break.

- 🔍 **Unlimited-depth search** — most HN scrapers stop at the API's 1,000-result cap; this one pages by time window instead (verified: **3,000 unique, correctly ordered rows in a single run**)
- 💼 **Who-is-hiring, parsed** — the July 2026 thread returned **276 job posts**: company **88.8%**, e-mails on **26.8%** of posts, URLs on **84.1%**, remote flag on every row
- 💬 **Comments are searchable too** — where brand opinions actually live, not just titles
- ⚡ **Fast & free to run** — pure HTTP against official endpoints, no proxy or third-party cost

### Quick start (API)

Every Hacker News mention of "supabase" in the last 90 days, stories **and** comments:

```bash
curl -X POST "https://api.apify.com/v2/acts/foxlabs~hackernews-intelligence/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "mode": "search", "query": "supabase", "searchType": "both", "datePreset": "last_90_days", "maxResults": 500 }'
```

Prefer no code? Open the **Input** tab — the default is the latest Who-is-hiring thread — and click **Start**.

### What you get

Search & feed rows:

| Field | Type | Description |
|---|---|---|
| `type` | string | `story`, `comment` or `job_post` |
| `title` | string | Story title (comments carry their story's title) |
| `url` / `hnUrl` | string | The linked article + the Hacker News discussion page |
| `author` | string | HN username |
| `points` / `numComments` | number | Score and comment count (stories) |
| `text` | string | Story or comment body, HTML decoded to clean text |
| `storyId` / `parentId` | number | Thread position for comments |
| `createdAt` / `fetchedAt` | string | When it was posted / collected (ISO) |

Who-is-hiring rows add:

| Field | Type | Description |
|---|---|---|
| `company` | string | Employer, read from the thread's `Company \| Role \| Location` convention — `null` when a post doesn't follow it, never guessed |
| `headline` / `roleLine` | string | The post's first line, and the part after the company |
| `remote` | boolean | Whether the headline says remote |
| `emails` | array | Every e-mail address in the post (lowercased, de-duplicated) |
| `urls` | array | Application / company links in the post |
| `text` | string | The full post, clean text |
| `threadId` / `threadTitle` | number / string | Which monthly thread it came from |

#### Sample output

Real rows from live runs (July 2026):

```json
{
  "type": "job_post",
  "threadTitle": "Ask HN: Who is hiring? (July 2026)",
  "company": "Founding Technologist + Full Stack Developer",
  "headline": "Founding Technologist + Full Stack Developer | REMOTE | Part-or-full-time | $0 + equity",
  "roleLine": "REMOTE | Part-or-full-time | $0 + equity",
  "remote": true,
  "emails": ["hataginow@gmail.com"],
  "urls": [],
  "author": "baaaadegg",
  "hnUrl": "https://news.ycombinator.com/item?id=48919859",
  "createdAt": "2026-07-15T12:32:31Z"
}
```

```json
{
  "type": "story",
  "id": 49063754,
  "title": "PGSimCity - How PostgreSQL Works",
  "url": "https://nikolays.github.io/PGSimCity/",
  "hnUrl": "https://news.ycombinator.com/item?id=49063754",
  "author": "jonbaer",
  "points": 920,
  "numComments": 91,
  "createdAt": "2026-07-27T00:19:04Z"
}
```

Download as JSON, CSV, Excel or HTML, or pull it straight from the API.

### Modes & filters

- **Mode** — `search` (whole archive), `who_is_hiring` (monthly job thread), `feed` (live lists).
- **Search** — free-text `query`; search in `stories`, `comments` or **both**; `minPoints` / `minComments` to keep only stories that landed; `author` to follow one person; date presets from 24 hours to all time (plus custom).
- **Sort** — **date** (default) streams newest-first with unlimited depth; **relevance** uses the API's ranked order but is capped at 1,000 results by the API, and the run tells you so.
- **Who is hiring** — pick the thread type (**who is hiring** / **who wants to be hired** / **freelancer**) and any `month` (`2026-07`), or leave it empty for the latest.
- **Feed** — `top` / `new` / `best` / `ask` / `show` / `jobs`, optionally walking each comment tree (`commentDepth`, `maxCommentsPerStory`).
- **Max results** — hard cap; `0` really is unlimited.

### Example inputs (copy & paste)

```jsonc
// 1) Brand monitoring — every mention, comments included
{ "mode": "search", "query": "clickhouse", "searchType": "both", "datePreset": "last_30_days" }

// 2) What actually landed — front-page-grade stories on a topic
{ "mode": "search", "query": "postgres", "minPoints": 100, "datePreset": "last_30_days" }

// 3) This month's job market, structured
{ "mode": "who_is_hiring", "threadType": "hiring", "maxResults": 0 }

// 4) Candidates instead of employers
{ "mode": "who_is_hiring", "threadType": "wants_to_be_hired" }

// 5) A specific month's hiring thread
{ "mode": "who_is_hiring", "month": "2026-04" }

// 6) Follow one writer
{ "mode": "search", "author": "patio11", "searchType": "both", "datePreset": "last_year" }

// 7) Today's front page with discussion
{ "mode": "feed", "feed": "top", "includeComments": true, "commentDepth": 2, "maxResults": 30 }
```

### Use cases

- **Developer-tool marketing.** Track every mention of your product *and* your competitors in comments — the place where developers say what they really think — then schedule the run weekly and diff the results.
- **Tech recruiting & job hunting.** Turn the monthly Who-is-hiring thread into a spreadsheet: company, role line, remote flag, application e-mails and links. What used to be an hour of scrolling is one run.
- **Launch & trend tracking.** Watch Show HN and high-score stories in your category to see what is getting traction, with scores and comment counts attached.
- **Research datasets.** Pull a topic's full history — the archive goes back to 2007 and the sliding-window pagination really does walk all of it — for NLP, sentiment or trend analysis.
- **Community & founder intelligence.** Follow specific authors or threads, or sweep Ask HN for recurring pain points to mine for product ideas.

### Performance & cost

Pure HTTP against Hacker News' official endpoints — no browser, no proxy, no third-party API costs. Measured on the platform (2026-07-30): a 3,000-row archive sweep took **73 seconds**; the 276-post July hiring thread took **12 seconds**; a filtered topic search returned in **4 seconds**. Runs use a 512 MB container.

### Integrations

**JavaScript** (`apify-client`):

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('foxlabs/hackernews-intelligence').call({
  mode: 'who_is_hiring', threadType: 'hiring', maxResults: 0,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

**Python** (`apify-client`):

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("foxlabs/hackernews-intelligence").call(run_input={
    "mode": "search", "query": "rust", "searchType": "both", "datePreset": "last_30_days",
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item.get("title"), item.get("points"))
```

Also works with **Make / n8n / Zapier**, scheduled runs, webhooks, and the **Apify MCP server** for AI agents.

### Data quality (measured 2026-07-30)

| What | Result |
|---|---|
| Who-is-hiring parse (n=276, July 2026 thread) | company **88.8%** · headline **100%** · role line **89.1%** · URLs **84.1%** · e-mails **26.8%** of posts · remote flag on every row |
| Verification against live HN pages | **5 of 5** randomly picked job posts matched their HN page exactly (author + company) |
| Unlimited-depth proof | **3,000 / 3,000 unique** rows, all inside the requested window, strictly newest-first |
| Filter accuracy | `minPoints=100` + 30-day window: **13/13** rows satisfied both |

Nothing is fabricated. A field that a post doesn't contain comes back `null` or empty, never guessed.

### Pricing

**Pay per result** — you're billed per row returned, and there's an Apify **free tier** to try it. The underlying APIs are free and public, so there are no proxy or third-party costs baked in.

### FAQ

**Where does the data come from?** Two official Hacker News APIs: the [Algolia HN Search API](https://hn.algolia.com/api) for the archive and the [Firebase HN API](https://github.com/HackerNews/API) for live feeds. Both are public and keyless.

**How do you get past the 1,000-result limit?** The search API caps classic pagination at 1,000 results. In date order this Actor doesn't use page numbers at all — it narrows a timestamp window on each round, so it can keep walking as deep as you ask. (Relevance sort still uses the ranked endpoint and its 1,000 cap; the run logs a warning when you pick it.)

**Can I get the whole history of a keyword?** Yes — set `datePreset: "all_time"` and `maxResults: 0`. HN's archive starts in 2007.

**Why is `company` empty on some job posts?** The hiring thread has a convention (`Company | Role | Location`) that most but not all posters follow — measured 88.8% on the July 2026 thread. When a post doesn't follow it we leave `company` null rather than guess; `headline` and the full `text` are always there.

**Do I get e-mails for every job post?** No — only where the poster included one, measured 26.8% of posts. The rest link to an application page (`urls`, 84.1%).

**Is this legal?** It reads public, official Hacker News APIs — no login, no bypassing anything. Contact details in hiring posts were published by the poster specifically to be contacted about that role; use them for that.

**Can I search comments only?** Yes, `searchType: "comments"` — useful for brand and competitor monitoring, since opinions live in the discussion rather than the title.

**How fresh is it?** Live. Both APIs reflect Hacker News in near real time; schedule the Actor to track a keyword or catch each new hiring thread.

### Troubleshooting

- **0 results** → widen the date window, drop `minPoints`, or simplify the query — the archive is searched exactly as you type it.
- **Off-topic rows in search** → the search API also matches a story's URL, so a keyword can hit a domain name. Add `searchType: "stories"` plus a `minPoints` filter to tighten it.
- **"Capped at 1,000" warning** → you're on relevance sort; switch **Sort by** to date for unlimited depth.
- **No thread found in who-is-hiring mode** → check the `month` format (`YYYY-MM`), or leave it empty to take the latest thread.

### Notes, limits & legal

- Hacker News content belongs to its authors; this Actor redistributes what the official APIs publish. Not affiliated with Hacker News or Y Combinator.
- `company` parsing depends on a community convention (measured 88.8%) — the raw text is always included so you can re-parse it your own way.
- Relevance sort is limited to 1,000 results by the API; date sort is not.

### Support

Questions, a field you'd like added, or a custom build? Open the **Issues** tab, or e-mail **info@foxlabs.com.tr**. We reply fast.

*If this Actor saves you time, a ⭐ review really helps.*

### Changelog

#### 0.2 — 2026-07-30

- **Full-archive search** (stories, comments or both) with points/comment/author filters and date presets back to 2007 — **no 1,000-result wall** in date order, thanks to sliding time-window pagination (verified: 3,000 unique ordered rows in one run).
- **Who-is-hiring parser** — the monthly thread (or *wants to be hired* / *freelancer*, any month) becomes structured job posts: company, headline, role line, remote flag, e-mails and URLs, plus full text. Measured on the July 2026 thread: 276 posts, company 88.8%, e-mails 26.8%, 5/5 verified against live HN pages.
- Feeds kept (top/new/best/ask/show/jobs, optional comment trees); HTML decoded to clean text; failures stay visible instead of ending as an empty "success".

#### 0.1 — 2026-05-20

- Initial release: Firebase feeds with depth-limited comment-thread walking.

***

Part of the **[foXLabs data platform](https://data.foxlabs.com.tr/)** — official public-data company, contact, ownership, jobs, procurement & AI-search intelligence scrapers. Browse the full suite at **[data.foxlabs.com.tr](https://data.foxlabs.com.tr/)**.

# Actor input Schema

## `mode` (type: `string`):

Search = query the full HN archive (stories & comments) with points/date filters — unlimited depth, no 1,000-result wall. Who is hiring = parse the monthly 'Ask HN: Who is hiring?' thread into structured job posts (company, role, remote, e-mails). Feed = live front page / new / best / ask / show / jobs.

## `query` (type: `string`):

Search mode. What to search for — a brand, product, topic (e.g. "postgres", "Supabase", "rust async"). Leave empty to browse everything in the date window.

## `searchType` (type: `string`):

Search mode. Stories = titles/links (launches, articles). Comments = the discussion text (where brand mentions and opinions actually live). Both = the two merged.

## `sortBy` (type: `string`):

Date (default) streams newest-first with UNLIMITED depth. Relevance uses the API's ranked order but is hard-capped at 1,000 results by the API.

## `minPoints` (type: `integer`):

Search mode, stories only. Keep only stories with at least this many upvotes (e.g. 100 = front-page material). Leave empty for all.

## `minComments` (type: `integer`):

Search mode, stories only. Keep only stories with at least this many comments. Leave empty for all.

## `author` (type: `string`):

Search mode. Only items by this HN username (e.g. "patio11"). Leave empty for all authors.

## `threadType` (type: `string`):

Which monthly thread to parse: companies hiring (default), people looking for work, or the freelancer thread.

## `month` (type: `string`):

Which month's thread, e.g. 2026-07. Leave empty for the latest thread.

## `datePreset` (type: `string`):

Search mode: how far back to search. The archive goes back to 2007 — 'All time' really is all time.

## `dateFrom` (type: `string`):

YYYY-MM-DD. Only used when Date range = Custom.

## `dateTo` (type: `string`):

YYYY-MM-DD. Only used when Date range = Custom. Leave empty for 'today'.

## `feed` (type: `string`):

Feed mode: which live list to pull.

## `includeComments` (type: `boolean`):

Feed mode: also walk each story's comment tree into the row (slower).

## `commentDepth` (type: `integer`):

Feed mode, with comments on: how deep to walk each thread (1-5).

## `maxCommentsPerStory` (type: `integer`):

Feed mode, with comments on: cap per story (1-500).

## `maxResults` (type: `integer`):

Hard cap on dataset rows. Set 0 for unlimited (the sliding-window search really can walk the whole archive).

## Actor input object example

```json
{
  "mode": "who_is_hiring",
  "searchType": "stories",
  "sortBy": "date",
  "threadType": "hiring",
  "datePreset": "all_time",
  "feed": "top",
  "includeComments": false,
  "commentDepth": 2,
  "maxCommentsPerStory": 50,
  "maxResults": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "who_is_hiring",
    "query": "",
    "threadType": "hiring",
    "maxResults": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("foxlabs/hackernews-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "who_is_hiring",
    "query": "",
    "threadType": "hiring",
    "maxResults": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("foxlabs/hackernews-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "who_is_hiring",
  "query": "",
  "threadType": "hiring",
  "maxResults": 1000
}' |
apify call foxlabs/hackernews-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=foxlabs/hackernews-intelligence",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IzFuDRcqNz0rwi51I/builds/E2bmLVRg4ts6a7d9f/openapi.json