# Reddit Posts Scraper (`scrapeengine/reddit-posts-scraper`) Actor 🧰 Reddit Posts Scraper pulls structured Reddit data—titles, bodies, media, flair, author, subreddit, upvotes, comments, awards, dates & links—from subreddits, users, and searches. ⚙️ Exports JSON/CSV. 🚀 Ideal for market research, trend analysis, sentiment & content curation. - **URL**: https://apify.com/scrapeengine/reddit-posts-scraper.md - **Developed by:** [ScrapeEngine](https://apify.com/scrapeengine) (community) - **Categories:** Automation, Lead generation, Social media - **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $19.99/month + usage To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### Reddit Posts Scraper Reddit Posts Scraper is a production-ready Apify actor that extracts public Reddit posts—and optionally comments—from subreddits, full Reddit URLs, or keyword searches, delivering clean, structured JSON you can export to CSV. It solves the pain of manual collection and unreliable tools by providing a reliable Reddit post scraper with parallel comment fetching, automatic proxy fallback, and robust retry logic. Built for marketers, developers, data analysts, and researchers, this subreddit scraper scales from quick one-offs to automated monitoring and bulk runs, enabling large-scale trend tracking, sentiment analysis, and content curation. ### What data / output can you get? Below are the exact fields this Reddit data scraper saves to the dataset for each post: | Data field | Description | Example value | | --- | --- | --- | | subreddit | Community name | "news" | | title | Post title | "Example post title" | | author | Reddit username of the poster | "username123" | | score | Upvotes / score | 156 | | num_comments | Number of comments | 42 | | created_utc | Unix timestamp (UTC) | 1703123456 | | permalink | Permalink to the Reddit thread | "https://www.reddit.com/r/news/comments/abc123/example_post/" | | body | Selftext/body of the post | "Post content..." | | thumbnail_url | Thumbnail image URL | "https://preview.redd.it/..." | | image_url | Main image or media URL (if any) | "https://i.redd.it/..." | | comments | Array of comments with nested replies | [{"author":"userA","body":"Nice post!","score":12,"created_utc":1703123457,"replies":[]}] | | post_id | Reddit post ID | "abc123" | | success | Whether the post was processed successfully | true | | error_message | Error message if processing failed | null | Notes: - comments contains nested replies with the same structure (author, body, score, created_utc, replies). - Some fields (e.g., image_url, thumbnail_url, body) may be empty if not present in the original post. You can export results as JSON or CSV from the Apify dataset or fetch them programmatically via the Apify API—perfect when you need to export Reddit posts to CSV for analysis or dashboards. ### Key features - 🚀 Parallel comment processing Efficiently handle larger batches with parallel fetching of comments for each post—ideal for a bulk Reddit thread scraper. - 🧩 Flexible source targeting Feed subreddits (e.g., news or r/technology), full Reddit URLs, or keyword queries in a single list to scrape Reddit posts at scale. - 🔀 Sorting + time filtering Choose sortOrder (hot, new, top, rising) and apply a timeFilter (hour, day, week, month, year, all) when applicable for focused results. - 🛡️ Automatic proxy fallback Built-in fallback from direct connection to datacenter to residential proxies on blocks (403/429) to maximize reliability for continuous runs. - 🔁 Robust retry logic Retries handle blocks, timeouts, and upstream/proxy errors with backoff strategies to keep your Reddit scraping tool resilient. - 📤 Structured, integration-ready data Clean JSON output with post metadata and nested comments that you can export to CSV or consume via API for downstream workflows. - 🧑‍💻 Developer friendly Call runs and fetch datasets via the Apify API from Python or Node.js—great for building an automated Reddit API scraper pipeline. - 💾 Live dataset streaming Results are pushed to the dataset as they’re processed, so you keep partial data even if long runs stop early. ### How to use Reddit Posts Scraper - step by step 1. 🔑 Sign in at console.apify.com with your Apify account. 2. 🔍 Find “Reddit Posts Scraper” in the Apify Store and open the actor. 3. 🧾 Add your input in the “Reddit URLs / Subreddits / Keywords” field (one per line). You can mix: - Full URLs like https://www.reddit.com/r/news/ - Subreddit names like news or r/news - Keyword searches like artificial intelligence 4. ⚙️ Configure settings as needed: - sortOrder (hot, new, top, rising) - timeFilter (hour, day, week, month, year, all) — only applies to top or rising - maxPosts and maxComments limits - proxyConfiguration for reliable, large-scale runs 5. ▶️ Start the run and watch logs as it collects posts and processes comments in parallel. 6. 💾 Open the Output (Dataset) tab to preview results and export them as JSON or CSV, or pull them via the Apify API. Pro tip: Use the Apify API from your Python Reddit scraper or Node.js scripts to schedule runs and build a Reddit post monitoring workflow that keeps your datasets fresh over time. ### Use cases | Use case | Description | | --- | --- | | Market & trend research | Track trending posts by keyword or subreddit to quantify interest and surface emerging topics. | | Content & SEO planning | Identify high-engagement threads to inspire content calendars and optimize topical coverage. | | Brand monitoring | Monitor mentions and sentiment across communities to detect issues and opportunities early. | | Academic & social research | Collect public discourse (titles, bodies, comments) for reproducible studies and NLP projects. | | Data pipeline / API workflows | Automate runs via Apify API and export Reddit data to CSV/JSON for BI dashboards or databases. | | Competitive intelligence | Analyze discussions around competitors’ products or niches to inform strategy and positioning. | ### Why choose Reddit Posts Scraper? This Reddit web scraper is engineered for reliability and scale with structured outputs, automatic proxy fallback, and parallel processing. - ✅ Accurate structured extraction: Captures subreddit, title, author, score, comment counts, timestamps, permalinks, media, and nested comments. - ⚡ Scales to bulk: Handles multiple sources with parallel comment fetching—true automated subreddit scraping. - 🧑‍💻 Developer access: Control runs and fetch results over the Apify API from Python or Node.js for CI/CD and data pipelines. - 🛡️ Reliable by design: Retries and proxy fallback mitigate 403/429 and upstream errors for consistent results. - 🔒 No login, no browser: Works with public Reddit endpoints for streamlined, safe operations. - 🔗 Integration-ready: Export JSON/CSV to feed analytics stacks, warehouses, or custom apps. Unlike browser extensions or ad‑hoc scripts, this production-ready Reddit posts extractor delivers consistent, scalable outcomes with observability and structured exports. ### Is it legal / ethical to use Reddit Posts Scraper? Yes—when used responsibly. This actor targets publicly available Reddit content and does not require login or access to private subreddits. Guidelines for compliant use: - Scrape only public data and respect Reddit’s terms. - Avoid collecting or misusing personal data from users. - Follow applicable data protection laws (e.g., GDPR, CCPA). - Do not attempt to access private or authenticated content. - Consult your legal team for edge cases or regulated use. ### Input parameters & output format Example JSON input ```json { "startUrls": [ "https://www.reddit.com/r/news/", "news", "artificial intelligence" ], "sortOrder": "top", "timeFilter": "week", "maxPosts": 50, "maxComments": 100, "proxyConfiguration": { "useApifyProxy": false } } ```` Input parameter reference - startUrls (array, required)\ Description: Enter one item per line. Mix full Reddit URLs (e.g., https://www.reddit.com/r/news/), subreddit names (e.g., news or r/news), or search keywords (e.g., artificial intelligence). Duplicate subreddits are merged.\ Default: none - maxPosts (integer)\ Description: Max number of posts to scrape per subreddit or keyword (1–1000). If you have 3 sources and set 50, you can get up to 150 posts total.\ Default: 50 - maxComments (integer)\ Description: Max comments to fetch for each post (0–1000). Set to 0 to skip comments and only get post metadata (faster).\ Default: 100 - sortOrder (string)\ Description: How Reddit should sort the posts. Hot = trending now, New = latest first, Top = most upvoted, Rising = gaining traction.\ Default: "top" - timeFilter (string)\ Description: Time range for results. Only applies when sortOrder is Top or Rising (ignored for Hot and New).\ Default: "week" - proxyConfiguration (object)\ Description: Choose which proxies to use. If Reddit blocks a request, the actor automatically falls back: no proxy → datacenter → residential. Recommended for large runs or when you hit blocks.\ Default: { "useApifyProxy": false } Example JSON output item (one post) ```json { "subreddit": "news", "title": "Example post title", "author": "username", "score": 156, "num_comments": 42, "created_utc": 1703123456, "permalink": "https://www.reddit.com/r/news/comments/abc123/example_post/", "body": "Post content...", "thumbnail_url": "https://preview.redd.it/...", "image_url": "https://i.redd.it/...", "comments": [ { "author": "commenter1", "body": "Comment text...", "score": 23, "created_utc": 1703123457, "replies": [] } ], "post_id": "abc123", "success": true, "error_message": null } ``` Notes: - comments includes nested replies with the same structure (author, body, score, created\_utc, replies). - Some fields (e.g., image\_url, thumbnail\_url, body) may be empty if not present in the original post. ### FAQ #### Does the scraper include comments? Yes. Set maxComments > 0 to fetch comment threads. If you only need post metadata, set maxComments to 0 for faster runs. #### Can I target subreddits, URLs, and keywords together? Yes. Add them all to startUrls (one per line). The actor accepts full Reddit URLs, subreddit names, and search keywords in one list. #### How many posts can I scrape per source? Up to maxPosts per source, between 1 and 1000. For example, with 3 sources and maxPosts set to 50, you can collect up to 150 posts in total. #### Do I need login or API keys? No. The actor works with publicly accessible Reddit endpoints and does not require login or API keys. #### What happens if Reddit blocks requests? The actor attempts automatic proxy fallback (direct → datacenter → residential) and retries on blocks and transient errors to improve success rates. #### Can I export results to CSV? Yes. Open the dataset in the Output tab and export as JSON or CSV, or fetch data programmatically via the Apify API. #### Can developers use this from Python or Node.js? Yes. Trigger runs and download datasets via the Apify API from Python or Node.js to build automated pipelines or integrate with your systems. #### Does keyword search work across Reddit? Yes. If a startUrls entry looks like a keyword query, the actor uses Reddit’s search endpoint to collect matching posts. ### Final thoughts Reddit Posts Scraper is built to extract structured Reddit posts (and comments) from subreddits, URLs, and keyword searches—fast, reliable, and export‑ready. With parallel comment fetching, automatic proxy fallback, and clean JSON/CSV output, it’s ideal for marketers, developers, data analysts, and researchers. Use the Apify API to integrate with Python or Node.js and automate your subreddit monitoring pipeline. Start extracting smarter Reddit insights at scale. # Actor input Schema ## `startUrls` (type: `array`): 📝 Enter one item per line. You can mix: • 🌐 Full URLs — e.g. https://www.reddit.com/r/news/ • 📌 Subreddit names — e.g. news or r/news • 🔍 Search keywords — e.g. artificial intelligence (searches Reddit) Duplicate subreddits are merged. At least one entry is required. ## `maxPosts` (type: `integer`): Max number of posts to scrape **per** subreddit or keyword (1–1000). If you have 3 sources and set 50, you can get up to 150 posts total. ## `maxComments` (type: `integer`): Max comments to fetch for each post (0–1000). Set to **0** to skip comments and only get post metadata (faster). ## `sortOrder` (type: `string`): How Reddit should sort the posts. Hot = trending now, New = latest first, Top = most upvoted, Rising = gaining traction. ## `timeFilter` (type: `string`): Time range for results. ⚠️ Only applies when Sort order is **Top** or **Rising**. Ignored for Hot and New. ## `proxyConfiguration` (type: `object`): Choose which proxies to use. If Reddit blocks a request, the actor automatically falls back: no proxy → datacenter → residential. ✅ Recommended for large runs or when you hit blocks. ## Actor input object example ```json { "startUrls": [ "news", "r/technology", "https://www.reddit.com/r/news/", "artificial intelligence" ], "maxPosts": 50, "maxComments": 100, "sortOrder": "top", "timeFilter": "week", "proxyConfiguration": { "useApifyProxy": false } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.reddit.com/r/news/", "news", "artificial intelligence" ], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapeengine/reddit-posts-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [ "https://www.reddit.com/r/news/", "news", "artificial intelligence", ], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("scrapeengine/reddit-posts-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.reddit.com/r/news/", "news", "artificial intelligence" ], "proxyConfiguration": { "useApifyProxy": false } }' | apify call scrapeengine/reddit-posts-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapeengine/reddit-posts-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Reddit Posts Scraper", "description": "🧰 Reddit Posts Scraper pulls structured Reddit data—titles, bodies, media, flair, author, subreddit, upvotes, comments, awards, dates & links—from subreddits, users, and searches. ⚙️ Exports JSON/CSV. 🚀 Ideal for market research, trend analysis, sentiment & content curation.", "version": "0.1", "x-build-id": "QLcAwW849Zgkhqr7j" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapeengine~reddit-posts-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapeengine-reddit-posts-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapeengine~reddit-posts-scraper/runs": { "post": { "operationId": "runs-sync-scrapeengine-reddit-posts-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapeengine~reddit-posts-scraper/run-sync": { "post": { "operationId": "run-sync-scrapeengine-reddit-posts-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "🔗 Reddit URLs / Subreddits / Keywords", "type": "array", "description": "📝 Enter one item per line. You can mix:\n• 🌐 Full URLs — e.g. https://www.reddit.com/r/news/\n• 📌 Subreddit names — e.g. news or r/news\n• 🔍 Search keywords — e.g. artificial intelligence (searches Reddit)\n\nDuplicate subreddits are merged. At least one entry is required.", "items": { "type": "string" } }, "maxPosts": { "title": "📄 Maximum posts per source", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Max number of posts to scrape **per** subreddit or keyword (1–1000). If you have 3 sources and set 50, you can get up to 150 posts total.", "default": 50 }, "maxComments": { "title": "💬 Maximum comments per post", "minimum": 0, "maximum": 1000, "type": "integer", "description": "Max comments to fetch for each post (0–1000). Set to **0** to skip comments and only get post metadata (faster).", "default": 100 }, "sortOrder": { "title": "📋 Sort order", "enum": [ "hot", "new", "top", "rising" ], "type": "string", "description": "How Reddit should sort the posts. Hot = trending now, New = latest first, Top = most upvoted, Rising = gaining traction.", "default": "top" }, "timeFilter": { "title": "⏱️ Time filter", "enum": [ "hour", "day", "week", "month", "year", "all" ], "type": "string", "description": "Time range for results. ⚠️ Only applies when Sort order is **Top** or **Rising**. Ignored for Hot and New.", "default": "week" }, "proxyConfiguration": { "title": "🔐 Proxy configuration", "type": "object", "description": "Choose which proxies to use. If Reddit blocks a request, the actor automatically falls back: no proxy → datacenter → residential. ✅ Recommended for large runs or when you hit blocks." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```