Hacker News Scraper — Stories, Comments, Ask HN, Show HN avatar

Hacker News Scraper — Stories, Comments, Ask HN, Show HN

Pricing

from $2.00 / 1,000 results

Go to Apify Store
Hacker News Scraper — Stories, Comments, Ask HN, Show HN

Hacker News Scraper — Stories, Comments, Ask HN, Show HN

Scrape Hacker News stories and full comment threads via the official Firebase API. Top stories, Ask HN, Show HN, jobs, new. Built for founder/VC sentiment monitoring, topic clustering, trend tracking. Clean JSON, comment tree depth-controlled.

Pricing

from $2.00 / 1,000 results

Rating

0.0

(0)

Developer

Berkan Kaplan

Berkan Kaplan

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

3 days ago

Last modified

Share

Hacker News Search, Who-is-Hiring Jobs & Feeds

Search the entire Hacker News archive — stories and comments, back to 2007 — with no 1,000-result wall, turn the monthly "Ask HN: Who is hiring?" thread into structured job posts with contact e-mails, and pull the live front page. Built on Hacker News' official APIs: no key, no login, no proxies, no HTML scraping to break.

  • 🔍 Unlimited-depth search — most HN scrapers stop at the API's 1,000-result cap; this one pages by time window instead (verified: 3,000 unique, correctly ordered rows in a single run)
  • 💼 Who-is-hiring, parsed — the July 2026 thread returned 276 job posts: company 88.8%, e-mails on 26.8% of posts, URLs on 84.1%, remote flag on every row
  • 💬 Comments are searchable too — where brand opinions actually live, not just titles
  • Fast & free to run — pure HTTP against official endpoints, no proxy or third-party cost

Quick start (API)

Every Hacker News mention of "supabase" in the last 90 days, stories and comments:

curl -X POST "https://api.apify.com/v2/acts/foxlabs~hackernews-intelligence/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{ "mode": "search", "query": "supabase", "searchType": "both", "datePreset": "last_90_days", "maxResults": 500 }'

Prefer no code? Open the Input tab — the default is the latest Who-is-hiring thread — and click Start.

What you get

Search & feed rows:

FieldTypeDescription
typestringstory, comment or job_post
titlestringStory title (comments carry their story's title)
url / hnUrlstringThe linked article + the Hacker News discussion page
authorstringHN username
points / numCommentsnumberScore and comment count (stories)
textstringStory or comment body, HTML decoded to clean text
storyId / parentIdnumberThread position for comments
createdAt / fetchedAtstringWhen it was posted / collected (ISO)

Who-is-hiring rows add:

FieldTypeDescription
companystringEmployer, read from the thread's Company | Role | Location convention — null when a post doesn't follow it, never guessed
headline / roleLinestringThe post's first line, and the part after the company
remotebooleanWhether the headline says remote
emailsarrayEvery e-mail address in the post (lowercased, de-duplicated)
urlsarrayApplication / company links in the post
textstringThe full post, clean text
threadId / threadTitlenumber / stringWhich monthly thread it came from

Sample output

Real rows from live runs (July 2026):

{
"type": "job_post",
"threadTitle": "Ask HN: Who is hiring? (July 2026)",
"company": "Founding Technologist + Full Stack Developer",
"headline": "Founding Technologist + Full Stack Developer | REMOTE | Part-or-full-time | $0 + equity",
"roleLine": "REMOTE | Part-or-full-time | $0 + equity",
"remote": true,
"emails": ["hataginow@gmail.com"],
"urls": [],
"author": "baaaadegg",
"hnUrl": "https://news.ycombinator.com/item?id=48919859",
"createdAt": "2026-07-15T12:32:31Z"
}
{
"type": "story",
"id": 49063754,
"title": "PGSimCity - How PostgreSQL Works",
"url": "https://nikolays.github.io/PGSimCity/",
"hnUrl": "https://news.ycombinator.com/item?id=49063754",
"author": "jonbaer",
"points": 920,
"numComments": 91,
"createdAt": "2026-07-27T00:19:04Z"
}

Download as JSON, CSV, Excel or HTML, or pull it straight from the API.

Modes & filters

  • Modesearch (whole archive), who_is_hiring (monthly job thread), feed (live lists).
  • Search — free-text query; search in stories, comments or both; minPoints / minComments to keep only stories that landed; author to follow one person; date presets from 24 hours to all time (plus custom).
  • Sortdate (default) streams newest-first with unlimited depth; relevance uses the API's ranked order but is capped at 1,000 results by the API, and the run tells you so.
  • Who is hiring — pick the thread type (who is hiring / who wants to be hired / freelancer) and any month (2026-07), or leave it empty for the latest.
  • Feedtop / new / best / ask / show / jobs, optionally walking each comment tree (commentDepth, maxCommentsPerStory).
  • Max results — hard cap; 0 really is unlimited.

Example inputs (copy & paste)

// 1) Brand monitoring — every mention, comments included
{ "mode": "search", "query": "clickhouse", "searchType": "both", "datePreset": "last_30_days" }
// 2) What actually landed — front-page-grade stories on a topic
{ "mode": "search", "query": "postgres", "minPoints": 100, "datePreset": "last_30_days" }
// 3) This month's job market, structured
{ "mode": "who_is_hiring", "threadType": "hiring", "maxResults": 0 }
// 4) Candidates instead of employers
{ "mode": "who_is_hiring", "threadType": "wants_to_be_hired" }
// 5) A specific month's hiring thread
{ "mode": "who_is_hiring", "month": "2026-04" }
// 6) Follow one writer
{ "mode": "search", "author": "patio11", "searchType": "both", "datePreset": "last_year" }
// 7) Today's front page with discussion
{ "mode": "feed", "feed": "top", "includeComments": true, "commentDepth": 2, "maxResults": 30 }

Use cases

  • Developer-tool marketing. Track every mention of your product and your competitors in comments — the place where developers say what they really think — then schedule the run weekly and diff the results.
  • Tech recruiting & job hunting. Turn the monthly Who-is-hiring thread into a spreadsheet: company, role line, remote flag, application e-mails and links. What used to be an hour of scrolling is one run.
  • Launch & trend tracking. Watch Show HN and high-score stories in your category to see what is getting traction, with scores and comment counts attached.
  • Research datasets. Pull a topic's full history — the archive goes back to 2007 and the sliding-window pagination really does walk all of it — for NLP, sentiment or trend analysis.
  • Community & founder intelligence. Follow specific authors or threads, or sweep Ask HN for recurring pain points to mine for product ideas.

Performance & cost

Pure HTTP against Hacker News' official endpoints — no browser, no proxy, no third-party API costs. Measured on the platform (2026-07-30): a 3,000-row archive sweep took 73 seconds; the 276-post July hiring thread took 12 seconds; a filtered topic search returned in 4 seconds. Runs use a 512 MB container.

Integrations

JavaScript (apify-client):

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('foxlabs/hackernews-intelligence').call({
mode: 'who_is_hiring', threadType: 'hiring', maxResults: 0,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python (apify-client):

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("foxlabs/hackernews-intelligence").call(run_input={
"mode": "search", "query": "rust", "searchType": "both", "datePreset": "last_30_days",
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item.get("title"), item.get("points"))

Also works with Make / n8n / Zapier, scheduled runs, webhooks, and the Apify MCP server for AI agents.

Data quality (measured 2026-07-30)

WhatResult
Who-is-hiring parse (n=276, July 2026 thread)company 88.8% · headline 100% · role line 89.1% · URLs 84.1% · e-mails 26.8% of posts · remote flag on every row
Verification against live HN pages5 of 5 randomly picked job posts matched their HN page exactly (author + company)
Unlimited-depth proof3,000 / 3,000 unique rows, all inside the requested window, strictly newest-first
Filter accuracyminPoints=100 + 30-day window: 13/13 rows satisfied both

Nothing is fabricated. A field that a post doesn't contain comes back null or empty, never guessed.

Pricing

Pay per result — you're billed per row returned, and there's an Apify free tier to try it. The underlying APIs are free and public, so there are no proxy or third-party costs baked in.

FAQ

Where does the data come from? Two official Hacker News APIs: the Algolia HN Search API for the archive and the Firebase HN API for live feeds. Both are public and keyless.

How do you get past the 1,000-result limit? The search API caps classic pagination at 1,000 results. In date order this Actor doesn't use page numbers at all — it narrows a timestamp window on each round, so it can keep walking as deep as you ask. (Relevance sort still uses the ranked endpoint and its 1,000 cap; the run logs a warning when you pick it.)

Can I get the whole history of a keyword? Yes — set datePreset: "all_time" and maxResults: 0. HN's archive starts in 2007.

Why is company empty on some job posts? The hiring thread has a convention (Company | Role | Location) that most but not all posters follow — measured 88.8% on the July 2026 thread. When a post doesn't follow it we leave company null rather than guess; headline and the full text are always there.

Do I get e-mails for every job post? No — only where the poster included one, measured 26.8% of posts. The rest link to an application page (urls, 84.1%).

Is this legal? It reads public, official Hacker News APIs — no login, no bypassing anything. Contact details in hiring posts were published by the poster specifically to be contacted about that role; use them for that.

Can I search comments only? Yes, searchType: "comments" — useful for brand and competitor monitoring, since opinions live in the discussion rather than the title.

How fresh is it? Live. Both APIs reflect Hacker News in near real time; schedule the Actor to track a keyword or catch each new hiring thread.

Troubleshooting

  • 0 results → widen the date window, drop minPoints, or simplify the query — the archive is searched exactly as you type it.
  • Off-topic rows in search → the search API also matches a story's URL, so a keyword can hit a domain name. Add searchType: "stories" plus a minPoints filter to tighten it.
  • "Capped at 1,000" warning → you're on relevance sort; switch Sort by to date for unlimited depth.
  • No thread found in who-is-hiring mode → check the month format (YYYY-MM), or leave it empty to take the latest thread.
  • Hacker News content belongs to its authors; this Actor redistributes what the official APIs publish. Not affiliated with Hacker News or Y Combinator.
  • company parsing depends on a community convention (measured 88.8%) — the raw text is always included so you can re-parse it your own way.
  • Relevance sort is limited to 1,000 results by the API; date sort is not.

Support

Questions, a field you'd like added, or a custom build? Open the Issues tab, or e-mail info@foxlabs.com.tr. We reply fast.

If this Actor saves you time, a ⭐ review really helps.

Changelog

0.2 — 2026-07-30

  • Full-archive search (stories, comments or both) with points/comment/author filters and date presets back to 2007 — no 1,000-result wall in date order, thanks to sliding time-window pagination (verified: 3,000 unique ordered rows in one run).
  • Who-is-hiring parser — the monthly thread (or wants to be hired / freelancer, any month) becomes structured job posts: company, headline, role line, remote flag, e-mails and URLs, plus full text. Measured on the July 2026 thread: 276 posts, company 88.8%, e-mails 26.8%, 5/5 verified against live HN pages.
  • Feeds kept (top/new/best/ask/show/jobs, optional comment trees); HTML decoded to clean text; failures stay visible instead of ending as an empty "success".

0.1 — 2026-05-20

  • Initial release: Firebase feeds with depth-limited comment-thread walking.

Part of the foXLabs data platform — official public-data company, contact, ownership, jobs, procurement & AI-search intelligence scrapers. Browse the full suite at data.foxlabs.com.tr.