# Discourse Community Scraper (`rl1987/discourse-community-scraper`) Actor Generalised scraper for any Discourse-based community forum (topics, posts, categories, search) via Discourse's JSON API. - **URL**: https://apify.com/rl1987/discourse-community-scraper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** Social media - **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 dataset items This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does Discourse Community Scraper do? **Discourse Community Scraper** extracts topics and posts from any **Discourse-powered community forum** — no per-site configuration needed. It works out of the box on [community.n8n.io](https://community.n8n.io), [community.smartthings.com](https://community.smartthings.com), [forums.developer.nvidia.com](https://forums.developer.nvidia.com), and thousands of other Discourse instances (Discourse powers most modern developer/product community forums). Instead of parsing HTML, it talks directly to Discourse's built-in JSON API (`/latest.json`, `/c/{category}.json`, `/search.json`, `/t/{id}.json`, ...), which every Discourse site exposes — making the scraper fast, resilient to theme/layout changes, and generic across sites. Run it on the [Apify platform](https://apify.com) for scheduling, API access, webhooks, and proxy rotation without managing any infrastructure yourself. ### Why use Discourse Community Scraper? - **Community research & sentiment analysis** — pull recurring questions, complaints, and feature requests from a product's support forum. - **Competitive intelligence** — monitor what users say about a competitor's product in its official community. - **Support/knowledge-base mining** — build a searchable dataset of solved threads and accepted answers. - **Content/dataset building** — collect real-world Q\&A pairs or discussion threads for training or analysis. ### How to use Discourse Community Scraper 1. Set **Forum URL** to the base URL of any Discourse forum (e.g. `https://community.n8n.io`). 2. Pick a **Mode**: - `categories` — list every category/section on the forum, so you can find the right `categorySlug` without guessing (run this first if you don't already know it). - `latest` / `top` — the site-wide topic list. - `category` — one category (set **Category slug/path** to a category item's `category_slug_for_scraping` from a `categories` run, e.g. `questions/12`). - `search` — Discourse's search endpoint (set **Search query**; supports Discourse search filters like `order:latest #category`). - `topicUrls` — scrape only the specific topic URLs you provide. 3. Toggle **Include full posts** to also fetch every reply in each topic (off = topic metadata only, much faster). 4. Set **Max topics** / **Max posts per topic** to bound the run, and hit **Start**. ### Input See the **Input** tab for the full schema. Key fields: | Field | Description | |---|---| | `forumUrl` | Base URL of the Discourse forum | | `mode` | `categories`, `latest`, `top`, `category`, `search`, or `topicUrls` | | `categorySlug` | Category slug/path (mode=`category`) | | `searchQuery` | Search query (mode=`search`) | | `topicUrls` | List of direct topic URLs (mode=`topicUrls`) | | `includePosts` | Fetch full post stream per topic | | `maxTopics` / `maxPostsPerTopic` | Limits (0 = unlimited) | | `requestDelaySecs` / `maxConcurrency` | Politeness/rate-limit controls | | `proxyConfiguration` | Apify Proxy settings | ### Output Up to three item shapes land in the same dataset, distinguished by `type`: ```json { "type": "category", "forum": "https://community.n8n.io", "category_id": 13, "title": "Jobs", "slug": "jobs", "url": "https://community.n8n.io/c/jobs/13", "parent_category_id": null, "topic_count": 615, "category_slug_for_scraping": "jobs/13" } ``` ```json { "type": "topic", "forum": "https://community.n8n.io", "topic_id": 302663, "title": "AI Assistant on self-hosted n8n: early setup instructions", "slug": "ai-assistant-on-self-hosted-n8n-early-setup-instructions", "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663", "posts_count": 4, "views": 512, "like_count": 3, "created_at": "2026-01-05T10:00:00.000Z" } ``` ```json { "type": "post", "topic_id": 302663, "post_id": 565721, "post_number": 1, "username": "Ophir_Prusak", "cooked_html": "

Here's how to set it up:

  1. Install the self-hosted package
  2. Restart n8n
", "content_markdown": "Here's how to set it up:\n\n1. Install the **self-hosted** package\n2. Restart n8n", "content_text": "Here's how to set it up: Install the self-hosted package Restart n8n", "url": "https://community.n8n.io/t/ai-assistant-on-self-hosted-n8n-early-setup-instructions/302663/1" } ``` `cooked_html` is Discourse's raw rendered post body. `content_markdown` and `content_text` are cleaned conversions of it (via [markdownify](https://pypi.org/project/markdownify/) and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)) — use these for LLM input, search indexing, or anywhere raw HTML is unwanted. Download the dataset as JSON, CSV, Excel, or HTML from the **Storage** tab. #### Data table | Field | Item type | Description | |---|---|---| | `title` / `topic_title` | topic / post | Topic title | | `url` | both | Direct link to the topic or post | | `username` | post | Author's forum username | | `cooked_html` | post | Rendered HTML of the post body | | `content_markdown` | post | Post body as clean Markdown | | `content_text` | post | Post body as clean plain text | | `posts_count` | topic | Number of replies in the topic | | `like_count` | both | Like count | | `created_at` | both | ISO timestamp | ### Example: market research from a community's Jobs category You can run this actor end-to-end through the [Apify MCP server](https://mcp.apify.com/) to answer research questions without writing scraping code. Example: *"What kind of freelance work do n8n users want to buy in 2026?"* 1. **Discover the category** — call the actor with `mode: "categories"` on `https://community.n8n.io`. This lists every category with a ready-to-use `category_slug_for_scraping`, e.g. `"jobs/13"` for the `Jobs` category. 2. **Scrape it with posts** — call the actor again with `mode: "category"`, `categorySlug: "jobs/13"`, `includePosts: true`, `maxTopics: 80`. This returns each job-post topic plus its replies, with `content_text` already cleaned of HTML. 3. **Analyze** — pull `content_text` across all `post` items and count keyword mentions (e.g. "AI agent", "CRM", "voice AI", "RAG", "self-hosted") to see what buyers are actually asking for, and compare "hiring"/"looking for" vs. "for hire"/"available" topic titles to gauge demand vs. supply. A real run against `community.n8n.io/c/jobs` (61 topics, 204 rows) found: **AI agent building** was the top request (72 mentions), usually combined with **CRM integration** (92), **document/RAG/OCR processing** (60/38/22), and a growing **voice AI** niche (Vapi + Twilio + n8n stacks, 31+23 mentions). Buy-side posts outnumbered for-hire posts roughly 2:1, and **self-hosted** n8n (21 mentions) was a common hard requirement. Rates ranged $15–50/hr or $250–1,500 flat, with a shift toward long-term retainers over one-off builds. This same discover → scrape → analyze flow works for any Discourse forum's category (support backlogs, feature requests, showcase/built-with sections, etc.), not just job boards. ### Cost estimation This actor only makes lightweight JSON API requests (no browser rendering), so it's cheap to run — typically well under 1 compute unit for a few hundred topics without full posts. Enabling `includePosts` roughly doubles request volume per topic (one extra request per ~20 posts). ### Tips - Leave `includePosts` off for a quick topic-index crawl, then re-run with `topicUrls` on just the topics you care about. - Lower `maxConcurrency` / raise `requestDelaySecs` on smaller or self-hosted forums to stay polite. - Enable Apify Proxy for large crawls to avoid rate limiting. ### FAQ & disclaimers Only scrape publicly accessible forum content, and respect each forum's Terms of Service and `robots.txt`. This actor is not affiliated with Discourse, n8n, SmartThings, or NVIDIA. Found a Discourse site this doesn't work on? Open an issue in the Issues tab. # Actor input Schema ## `forumUrl` (type: `string`): Base URL of any Discourse-powered forum, e.g. https://community.n8n.io ## `mode` (type: `string`): What to crawl. 'categories' lists the forum's categories/sections (use this to discover a categorySlug for mode=category). 'latest'/'top' pull the site-wide topic list, 'category' pulls one category, 'search' pulls search results, 'topicUrls' scrapes only the topics you list. ## `categorySlug` (type: `string`): Required when mode=category. E.g. 'questions' or 'questions/12' (slug or slug/id) as it appears in the category URL. ## `searchQuery` (type: `string`): Required when mode=search. Uses Discourse's /search.json endpoint (supports its query filters, e.g. 'order:latest #category'). ## `topicUrls` (type: `array`): Required when mode=topicUrls. Direct URLs to Discourse topics. ## `includePosts` (type: `boolean`): If true, fetch every topic's full post stream (original post + all replies) in addition to the topic metadata. If false, only topic-level metadata is scraped (much faster, fewer requests). ## `maxTopics` (type: `integer`): Maximum number of topics to scrape. 0 = unlimited. ## `maxPostsPerTopic` (type: `integer`): Maximum number of posts to fetch per topic (only used when includePosts is true). 0 = unlimited. ## `requestDelaySecs` (type: `integer`): Polite delay between outgoing API requests to avoid overloading the forum. ## `maxConcurrency` (type: `integer`): Maximum number of concurrent HTTP requests. ## `proxyConfiguration` (type: `object`): Proxy settings. Recommended if scraping many pages to avoid rate limiting. ## Actor input object example ```json { "forumUrl": "https://community.n8n.io", "mode": "latest", "topicUrls": [], "includePosts": true, "maxTopics": 100, "maxPostsPerTopic": 0, "requestDelaySecs": 1, "maxConcurrency": 3, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "forumUrl": "https://community.n8n.io" }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/discourse-community-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "forumUrl": "https://community.n8n.io" } # Run the Actor and wait for it to finish run = client.actor("rl1987/discourse-community-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "forumUrl": "https://community.n8n.io" }' | apify call rl1987/discourse-community-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/discourse-community-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/T7Yaojs7XA1pIPx5Y/builds/4n9vQjrj4pbdhTJH7/openapi.json