# Hacker News Data Scraper (`shahidirfan/hacker-news-data-scraper`) Actor

Unlock the pulse of the tech world by scraping Hacker News effortlessly. Extract top stories, comments, and jobs from Y Combinator's platform. Perfect for market research, sentiment analysis, and staying ahead of startup trends with fast, structured data.

- **URL**: https://apify.com/shahidirfan/hacker-news-data-scraper.md
- **Developed by:** [Shahid Irfan](https://apify.com/shahidirfan) (community)
- **Categories:** News, Developer tools, Agents
- **Stats:** 54 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Data Scraper

Extract comprehensive data from Hacker News using the official API. Collect stories, comments, and job postings from different categories including top stories, new stories, best stories, Ask HN, Show HN, and job listings. Perfect for monitoring trends, analyzing community engagement, and building datasets for research.

### Features

- **Complete Story Data** — Extract titles, scores, comments, and metadata
- **Multiple Categories** — Collect from top, new, best, ask, show, and job stories
- **Fast API Extraction** — Direct access to official Hacker News data
- **Structured JSON Output** — Consistent format for all data types
- **Rate Limit Respect** — Built-in delays for responsible data collection

### Use Cases

#### Community Research

Analyze trending topics and user engagement patterns on Hacker News. Understand what content resonates with the tech community and track discussion trends over time.

#### Job Market Intelligence

Monitor startup job postings and career opportunities. Track hiring trends across different tech companies and identify emerging roles in the industry.

#### Content Analysis

Build comprehensive datasets for machine learning and natural language processing. Study user behavior, content patterns, and community dynamics.

#### News Monitoring

Stay updated on the latest tech news and discussions. Automatically collect and analyze stories that matter to your research or business.

***

### Input Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `storyType` | String | No | `topstories` | Type of stories to collect: `topstories`, `newstories`, `beststories`, `askstories`, `showstories`, `jobstories` |
| `results_wanted` | Integer | No | `20` | Maximum number of stories to collect (1-500) |
| `proxyConfiguration` | Object | No | `{"useApifyProxy": false}` | Proxy settings (optional for HN API) |

***

### Output Data

Each item in the dataset contains:

| Field | Type | Description |
|-------|------|-------------|
| `id` | Integer | Unique story ID |
| `type` | String | Item type (`story`, `comment`, `job`, etc.) |
| `title` | String | Story title |
| `by` | String | Author username |
| `score` | Integer | Story score/upvotes |
| `descendants` | Integer | Number of comments |
| `time` | Integer | Unix timestamp |
| `timestamp` | String | ISO 8601 timestamp |
| `url` | String | Original story URL (if external) |
| `text` | String | Story text content (HTML format) |
| `text_clean` | String | Story text content (clean text format) |
| `hn_url` | String | Hacker News discussion URL |
| `kids` | Array | Comment IDs |
| `deleted` | Boolean | Whether the item is deleted |
| `dead` | Boolean | Whether the item is dead |
| `parent` | Integer | Parent item ID (for comments) |
| `poll` | Integer | Associated poll ID (for poll options) |
| `parts` | Array | Related poll option IDs (for polls) |

***

### Usage Examples

#### Collect Top Stories

Extract the most popular stories from Hacker News:

```json
{
  "storyType": "topstories",
  "results_wanted": 50
}
```

#### Get New Stories

Collect the latest submissions to Hacker News:

```json
{
  "storyType": "newstories",
  "results_wanted": 30
}
```

#### Collect Job Postings

Gather startup job listings from the community:

```json
{
  "storyType": "jobstories",
  "results_wanted": 100
}
```

***

### Sample Output

```json
{
  "id": 45006801,
  "type": "story",
  "title": "Show HN: I built a tool to help developers write better commit messages",
  "by": "developer123",
  "score": 245,
  "descendants": 67,
  "time": 1735689600,
  "timestamp": "2025-01-01T00:00:00.000Z",
  "url": "https://github.com/developer123/commit-helper",
  "text": "<p>A simple tool that analyzes your commit messages and suggests improvements based on conventional commit standards.</p>",
  "text_clean": "A simple tool that analyzes your commit messages and suggests improvements based on conventional commit standards.",
  "hn_url": "https://news.ycombinator.com/item?id=45006801",
  "kids": [45006802, 45006803, 45006804],
  "deleted": false,
  "dead": false,
  "parent": null,
  "poll": null,
  "parts": null
}
```

***

### Tips for Best Results

#### Choose Story Types Wisely

- Use `topstories` for trending content and popular discussions
- Select `newstories` for the latest submissions and fresh content
- Pick `jobstories` for career opportunities and hiring trends

#### Optimize Collection Size

- Start with small numbers (20-50) for testing and exploration
- Increase to 100-200 for comprehensive data collection
- Balance between data volume and processing time

#### Handle Large Datasets

- Export results to JSON or CSV for analysis
- Use filtering and sorting in your analysis tools
- Consider pagination for very large collections

***

### Integrations

Connect your Hacker News data with:

- **Google Sheets** — Export for collaborative analysis
- **Airtable** — Build searchable story databases
- **Slack** — Get notifications for trending stories
- **Make** — Create automated content workflows
- **Zapier** — Trigger actions based on story data

#### Export Formats

Download data in multiple formats:

- **JSON** — For developers and API integrations
- **CSV** — For spreadsheet analysis and reporting
- **Excel** — For business intelligence dashboards

***

### Frequently Asked Questions

#### What's the difference between story types?

`topstories` are ranked by score and popularity, `newstories` by recency, `beststories` by a special algorithm, while `askstories`, `showstories`, and `jobstories` are specific post types.

#### Can I collect comments along with stories?

The current version collects story metadata. Comments can be fetched separately using the `kids` array with additional API calls to the Hacker News API.

#### Is this using the official API?

Yes, this scraper uses the official Hacker News API provided by Y Combinator, ensuring reliable and compliant data collection.

#### How many stories can I collect?

You can collect up to 500 stories per run. The API provides access to the most recent and popular content.

#### What if some fields are empty?

Some fields may be empty depending on the story type. For example, job postings may not have external URLs, and some stories may not have text content.

***

### Support

For issues or feature requests, contact support through the Apify Console.

#### Resources

- [Apify Documentation](https://docs.apify.com/)
- [API Reference](https://docs.apify.com/api/v2)
- [Hacker News API](https://github.com/HackerNews/API)

***

### Legal Notice

This scraper uses the official Hacker News API and complies with their terms of service. The API is provided by Y Combinator for public use. Users are responsible for ensuring compliance with applicable laws and using data responsibly.

# Actor input Schema

## `storyType` (type: `string`):

Type of stories to collect from Hacker News.

## `results_wanted` (type: `integer`):

The maximum number of stories to collect.

## `proxyConfiguration` (type: `object`):

Use Apify Proxy for reliable scraping (optional for HN API).

## Actor input object example

```json
{
  "storyType": "topstories",
  "results_wanted": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "storyType": "topstories",
    "results_wanted": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("shahidirfan/hacker-news-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "storyType": "topstories",
    "results_wanted": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("shahidirfan/hacker-news-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "storyType": "topstories",
  "results_wanted": 20
}' |
apify call shahidirfan/hacker-news-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=shahidirfan/hacker-news-data-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/fhQfeWHDe30sPz4EF/builds/rGeUrjuRIe86B5mB4/openapi.json