Hacker News Scraper — Stories, Comments, Ask HN, Show HN
Pricing
from $2.00 / 1,000 results
Hacker News Scraper — Stories, Comments, Ask HN, Show HN
Scrape Hacker News stories and full comment threads via the official Firebase API. Top stories, Ask HN, Show HN, jobs, new. Built for founder/VC sentiment monitoring, topic clustering, trend tracking. Clean JSON, comment tree depth-controlled.
Pricing
from $2.00 / 1,000 results
Rating
0.0
(0)
Developer
Berkan Kaplan
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
3 days ago
Last modified
Categories
Share
Hacker News Search, Who-is-Hiring Jobs & Feeds
Search the entire Hacker News archive — stories and comments, back to 2007 — with no 1,000-result wall, turn the monthly "Ask HN: Who is hiring?" thread into structured job posts with contact e-mails, and pull the live front page. Built on Hacker News' official APIs: no key, no login, no proxies, no HTML scraping to break.
- 🔍 Unlimited-depth search — most HN scrapers stop at the API's 1,000-result cap; this one pages by time window instead (verified: 3,000 unique, correctly ordered rows in a single run)
- 💼 Who-is-hiring, parsed — the July 2026 thread returned 276 job posts: company 88.8%, e-mails on 26.8% of posts, URLs on 84.1%, remote flag on every row
- 💬 Comments are searchable too — where brand opinions actually live, not just titles
- ⚡ Fast & free to run — pure HTTP against official endpoints, no proxy or third-party cost
Quick start (API)
Every Hacker News mention of "supabase" in the last 90 days, stories and comments:
curl -X POST "https://api.apify.com/v2/acts/foxlabs~hackernews-intelligence/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \-H "Content-Type: application/json" \-d '{ "mode": "search", "query": "supabase", "searchType": "both", "datePreset": "last_90_days", "maxResults": 500 }'
Prefer no code? Open the Input tab — the default is the latest Who-is-hiring thread — and click Start.
What you get
Search & feed rows:
| Field | Type | Description |
|---|---|---|
type | string | story, comment or job_post |
title | string | Story title (comments carry their story's title) |
url / hnUrl | string | The linked article + the Hacker News discussion page |
author | string | HN username |
points / numComments | number | Score and comment count (stories) |
text | string | Story or comment body, HTML decoded to clean text |
storyId / parentId | number | Thread position for comments |
createdAt / fetchedAt | string | When it was posted / collected (ISO) |
Who-is-hiring rows add:
| Field | Type | Description |
|---|---|---|
company | string | Employer, read from the thread's Company | Role | Location convention — null when a post doesn't follow it, never guessed |
headline / roleLine | string | The post's first line, and the part after the company |
remote | boolean | Whether the headline says remote |
emails | array | Every e-mail address in the post (lowercased, de-duplicated) |
urls | array | Application / company links in the post |
text | string | The full post, clean text |
threadId / threadTitle | number / string | Which monthly thread it came from |
Sample output
Real rows from live runs (July 2026):
{"type": "job_post","threadTitle": "Ask HN: Who is hiring? (July 2026)","company": "Founding Technologist + Full Stack Developer","headline": "Founding Technologist + Full Stack Developer | REMOTE | Part-or-full-time | $0 + equity","roleLine": "REMOTE | Part-or-full-time | $0 + equity","remote": true,"emails": ["hataginow@gmail.com"],"urls": [],"author": "baaaadegg","hnUrl": "https://news.ycombinator.com/item?id=48919859","createdAt": "2026-07-15T12:32:31Z"}
{"type": "story","id": 49063754,"title": "PGSimCity - How PostgreSQL Works","url": "https://nikolays.github.io/PGSimCity/","hnUrl": "https://news.ycombinator.com/item?id=49063754","author": "jonbaer","points": 920,"numComments": 91,"createdAt": "2026-07-27T00:19:04Z"}
Download as JSON, CSV, Excel or HTML, or pull it straight from the API.
Modes & filters
- Mode —
search(whole archive),who_is_hiring(monthly job thread),feed(live lists). - Search — free-text
query; search instories,commentsor both;minPoints/minCommentsto keep only stories that landed;authorto follow one person; date presets from 24 hours to all time (plus custom). - Sort — date (default) streams newest-first with unlimited depth; relevance uses the API's ranked order but is capped at 1,000 results by the API, and the run tells you so.
- Who is hiring — pick the thread type (who is hiring / who wants to be hired / freelancer) and any
month(2026-07), or leave it empty for the latest. - Feed —
top/new/best/ask/show/jobs, optionally walking each comment tree (commentDepth,maxCommentsPerStory). - Max results — hard cap;
0really is unlimited.
Example inputs (copy & paste)
// 1) Brand monitoring — every mention, comments included{ "mode": "search", "query": "clickhouse", "searchType": "both", "datePreset": "last_30_days" }// 2) What actually landed — front-page-grade stories on a topic{ "mode": "search", "query": "postgres", "minPoints": 100, "datePreset": "last_30_days" }// 3) This month's job market, structured{ "mode": "who_is_hiring", "threadType": "hiring", "maxResults": 0 }// 4) Candidates instead of employers{ "mode": "who_is_hiring", "threadType": "wants_to_be_hired" }// 5) A specific month's hiring thread{ "mode": "who_is_hiring", "month": "2026-04" }// 6) Follow one writer{ "mode": "search", "author": "patio11", "searchType": "both", "datePreset": "last_year" }// 7) Today's front page with discussion{ "mode": "feed", "feed": "top", "includeComments": true, "commentDepth": 2, "maxResults": 30 }
Use cases
- Developer-tool marketing. Track every mention of your product and your competitors in comments — the place where developers say what they really think — then schedule the run weekly and diff the results.
- Tech recruiting & job hunting. Turn the monthly Who-is-hiring thread into a spreadsheet: company, role line, remote flag, application e-mails and links. What used to be an hour of scrolling is one run.
- Launch & trend tracking. Watch Show HN and high-score stories in your category to see what is getting traction, with scores and comment counts attached.
- Research datasets. Pull a topic's full history — the archive goes back to 2007 and the sliding-window pagination really does walk all of it — for NLP, sentiment or trend analysis.
- Community & founder intelligence. Follow specific authors or threads, or sweep Ask HN for recurring pain points to mine for product ideas.
Performance & cost
Pure HTTP against Hacker News' official endpoints — no browser, no proxy, no third-party API costs. Measured on the platform (2026-07-30): a 3,000-row archive sweep took 73 seconds; the 276-post July hiring thread took 12 seconds; a filtered topic search returned in 4 seconds. Runs use a 512 MB container.
Integrations
JavaScript (apify-client):
import { ApifyClient } from 'apify-client';const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });const run = await client.actor('foxlabs/hackernews-intelligence').call({mode: 'who_is_hiring', threadType: 'hiring', maxResults: 0,});const { items } = await client.dataset(run.defaultDatasetId).listItems();
Python (apify-client):
from apify_client import ApifyClientclient = ApifyClient("YOUR_APIFY_TOKEN")run = client.actor("foxlabs/hackernews-intelligence").call(run_input={"mode": "search", "query": "rust", "searchType": "both", "datePreset": "last_30_days",})for item in client.dataset(run["defaultDatasetId"]).iterate_items():print(item.get("title"), item.get("points"))
Also works with Make / n8n / Zapier, scheduled runs, webhooks, and the Apify MCP server for AI agents.
Data quality (measured 2026-07-30)
| What | Result |
|---|---|
| Who-is-hiring parse (n=276, July 2026 thread) | company 88.8% · headline 100% · role line 89.1% · URLs 84.1% · e-mails 26.8% of posts · remote flag on every row |
| Verification against live HN pages | 5 of 5 randomly picked job posts matched their HN page exactly (author + company) |
| Unlimited-depth proof | 3,000 / 3,000 unique rows, all inside the requested window, strictly newest-first |
| Filter accuracy | minPoints=100 + 30-day window: 13/13 rows satisfied both |
Nothing is fabricated. A field that a post doesn't contain comes back null or empty, never guessed.
Pricing
Pay per result — you're billed per row returned, and there's an Apify free tier to try it. The underlying APIs are free and public, so there are no proxy or third-party costs baked in.
FAQ
Where does the data come from? Two official Hacker News APIs: the Algolia HN Search API for the archive and the Firebase HN API for live feeds. Both are public and keyless.
How do you get past the 1,000-result limit? The search API caps classic pagination at 1,000 results. In date order this Actor doesn't use page numbers at all — it narrows a timestamp window on each round, so it can keep walking as deep as you ask. (Relevance sort still uses the ranked endpoint and its 1,000 cap; the run logs a warning when you pick it.)
Can I get the whole history of a keyword? Yes — set datePreset: "all_time" and maxResults: 0. HN's archive starts in 2007.
Why is company empty on some job posts? The hiring thread has a convention (Company | Role | Location) that most but not all posters follow — measured 88.8% on the July 2026 thread. When a post doesn't follow it we leave company null rather than guess; headline and the full text are always there.
Do I get e-mails for every job post? No — only where the poster included one, measured 26.8% of posts. The rest link to an application page (urls, 84.1%).
Is this legal? It reads public, official Hacker News APIs — no login, no bypassing anything. Contact details in hiring posts were published by the poster specifically to be contacted about that role; use them for that.
Can I search comments only? Yes, searchType: "comments" — useful for brand and competitor monitoring, since opinions live in the discussion rather than the title.
How fresh is it? Live. Both APIs reflect Hacker News in near real time; schedule the Actor to track a keyword or catch each new hiring thread.
Troubleshooting
- 0 results → widen the date window, drop
minPoints, or simplify the query — the archive is searched exactly as you type it. - Off-topic rows in search → the search API also matches a story's URL, so a keyword can hit a domain name. Add
searchType: "stories"plus aminPointsfilter to tighten it. - "Capped at 1,000" warning → you're on relevance sort; switch Sort by to date for unlimited depth.
- No thread found in who-is-hiring mode → check the
monthformat (YYYY-MM), or leave it empty to take the latest thread.
Notes, limits & legal
- Hacker News content belongs to its authors; this Actor redistributes what the official APIs publish. Not affiliated with Hacker News or Y Combinator.
companyparsing depends on a community convention (measured 88.8%) — the raw text is always included so you can re-parse it your own way.- Relevance sort is limited to 1,000 results by the API; date sort is not.
Support
Questions, a field you'd like added, or a custom build? Open the Issues tab, or e-mail info@foxlabs.com.tr. We reply fast.
If this Actor saves you time, a ⭐ review really helps.
Changelog
0.2 — 2026-07-30
- Full-archive search (stories, comments or both) with points/comment/author filters and date presets back to 2007 — no 1,000-result wall in date order, thanks to sliding time-window pagination (verified: 3,000 unique ordered rows in one run).
- Who-is-hiring parser — the monthly thread (or wants to be hired / freelancer, any month) becomes structured job posts: company, headline, role line, remote flag, e-mails and URLs, plus full text. Measured on the July 2026 thread: 276 posts, company 88.8%, e-mails 26.8%, 5/5 verified against live HN pages.
- Feeds kept (top/new/best/ask/show/jobs, optional comment trees); HTML decoded to clean text; failures stay visible instead of ending as an empty "success".
0.1 — 2026-05-20
- Initial release: Firebase feeds with depth-limited comment-thread walking.
Part of the foXLabs data platform — official public-data company, contact, ownership, jobs, procurement & AI-search intelligence scrapers. Browse the full suite at data.foxlabs.com.tr.