# Substack Posts & Creator Scraper (`sleek_waveform/substack-creator-scraper`) Actor Scrape posts, engagement metrics, and author data from any Substack publication. Get title, author, publish date, likes, comments, paywall status, and full body in Markdown or HTML. Paginates the full archive automatically. - **URL**: https://apify.com/sleek\_waveform/substack-creator-scraper.md - **Developed by:** [Daniel Dimitrov](https://apify.com/sleek_waveform) (community) - **Categories:** Social media, Lead generation - **Stats:** 14 total users, 4 monthly users, 97.8% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $2.00 / 1,000 posts This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Substack Posts & Creator Scraper ### What does Substack Scraper do? Substack Scraper extracts post content, engagement metrics, and author data from any Substack publication or individual post URL. It accesses Substack's internal JSON structure directly — no headless browser needed — giving you clean, structured data in seconds rather than minutes. With a single publication URL, the scraper automatically paginates through the entire archive and returns every post with its title, author, publish date, total reactions, comment count, paywall status, and full body content in your choice of Markdown or HTML. ### Why scrape Substack? Substack has become the home for thousands of high-quality newsletters and independent journalists. The data available on public posts is invaluable for: - **Competitor and trend analysis** — track what content performs best in your niche, monitor publishing frequency and engagement patterns across publications - **Creator and influencer research** — build lists of authors with engagement benchmarks for outreach and partnership decisions - **Newsletter research** — study the structure, cadence, and topics of top-performing newsletters before launching your own - **Content backup** — archive your own Substack posts with engagement history before platform changes - **AI and NLP training data** — extract clean, structured long-form text with rich metadata at scale - **PR and media monitoring** — track journalist activity and media coverage across Substack publications If you would like more inspiration on how scraping Substack could help your business, check out our [industry pages](https://apify.com/industries). ### Before you start scraping Substack You need a free [Apify account](https://console.apify.com/sign-up) to run this Actor. The free plan includes $5 in monthly credits — enough to scrape several thousand posts. No credit card required. For large-scale jobs (100,000+ posts), the Actor automatically uses Apify datacenter proxies to rotate IPs and avoid Substack rate limiting. ### How to scrape Substack 1. Open the Actor in [Apify Console](https://console.apify.com/) and click **Try for free** 2. Enter one or more Substack publication URLs (e.g., `https://www.astralcodexten.com/`) or direct post URLs 3. Set **Max Posts Per Publication** (default: 100) and choose your preferred **Output Format** 4. Click **Start** and wait for the run to complete 5. Download your data from the **Dataset** tab — available in JSON, CSV, Excel, and HTML ### Substack Scraper input parameters | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `startUrls` | Array | ✅ | — | Substack publication homepages or individual post URLs | | `maxItems` | Number | ❌ | 100 | Max posts to scrape per publication | | `scrapeFormat` | String | ❌ | `"markdown"` | Post body format: `"markdown"`, `"html"`, or `"none"` (metadata only) | | `maxRequestRetries` | Number | ❌ | 3 | Retry attempts before a request is abandoned | | `maxSessionRotations` | Number | ❌ | 10 | Session rotations per request before giving up | | `webhookUrl` | String | ❌ | — | URL to notify when the run finishes (success or failure). Useful for Zapier, Make, and n8n integrations | #### Input examples **Scrape 50 recent posts from a publication** ```json { "startUrls": [{ "url": "https://www.astralcodexten.com/" }], "maxItems": 50, "scrapeFormat": "markdown" } ```` **Scrape specific posts** ```json { "startUrls": [ { "url": "https://www.astralcodexten.com/p/seiu-delenda-est" } ], "scrapeFormat": "html" } ``` **Metadata only — fastest option, no body content** ```json { "startUrls": [ { "url": "https://www.astralcodexten.com/" }, { "url": "https://platformer.news/" } ], "maxItems": 500, "scrapeFormat": "none" } ``` ### Substack Scraper output Each scraped post is stored as a single JSON record in the Actor's dataset: ```json { "url": "https://www.astralcodexten.com/p/seiu-delenda-est", "publicationName": "Astral Codex Ten", "authorName": "Scott Alexander", "title": "SEIU Delenda Est", "subtitle": "", "postDate": "2024-01-15T10:00:00.000Z", "likes": 551, "comments": 655, "isPaywalled": false, "body": "# SEIU Delenda Est\n\nPost content in markdown..." } ``` | Field | Type | Description | |-------|------|-------------| | `url` | String | Canonical post URL | | `publicationName` | String | Name of the Substack publication | | `authorName` | String | Author's display name | | `title` | String | Post title | | `subtitle` | String | Post subtitle (if present) | | `postDate` | String | ISO 8601 publish timestamp | | `likes` | Number | Total reactions across all 8 reaction types (❤ 👍 🎉 🔥 😂 😮 😢 😡) | | `comments` | Number | Number of comments | | `isPaywalled` | Boolean | `true` if the post requires a paid subscription to read in full | | `body` | String|null | Post content in the requested format; `null` when `scrapeFormat` is `"none"` | ### How much will it cost to scrape Substack? This Actor uses **Pay Per Result** pricing — you are charged per post scraped, not per compute time. Apify gives you $5 free usage credits every month on the [Apify Free plan](https://apify.com/pricing). You can scrape around **2,500 Substack posts per month** for that, so those 2,500 results will be completely free! But if you need to get more data regularly from Substack, you should grab an Apify subscription. We recommend our [$49/month Personal plan](https://apify.com/pricing) — you can get up to **25,000 posts every month** with the $49 monthly plan! Or get **250,000+ posts** for $499 with the [Team plan](https://apify.com/pricing) — wow! ### What are the limitations of Substack Scraper? - **Paywalled content** — only free-preview text is available for paid-only posts; full body requires a subscriber session, which is not supported - **Rate limiting** — Substack may throttle aggressive scraping; the Actor uses automatic IP rotation via datacenter proxies to mitigate this - **Frontend changes** — if Substack modifies their internal page structure, the Actor may need an update - **Custom domains** — most custom-domain Substacks work correctly; a small number with non-standard configurations may not - **Comment content** — only the comment count is extracted; individual comment text is not supported ### Is it legal to scrape Substack? This Actor only accesses **publicly available** posts and metadata. Paywalled content is never extracted. Web scraping of publicly accessible data is generally considered lawful in most jurisdictions for research, journalism, and personal use. Note that personal data is protected by GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. You are responsible for complying with [Substack's Terms of Service](https://substack.com/tos) and applicable laws in your jurisdiction. We also recommend that you read our blog post: [is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/) ### Scrape Substack with the Apify API You can trigger this Actor and download results programmatically using the [Apify API](https://docs.apify.com/api/v2). See the **API** tab on this Actor's page for ready-to-use code examples in JavaScript and Python, or check out the [Apify API reference](https://docs.apify.com/api/v2) for full details. ### Substack Scraper integrations This Actor works with any platform that supports webhooks or the Apify API: - **Zapier / Make / n8n** — use the `webhookUrl` input field to receive a POST notification when the run finishes, then pass the `actorRunId` to the Apify API to fetch your results - **Apify Integrations tab** — configure webhooks, scheduled runs, and connections to Google Sheets, Slack, Airtable, and more directly in the [Apify Console](https://console.apify.com/) without writing code - **REST API** — start a run, poll for completion, and download the dataset via the [Apify API v2](https://docs.apify.com/api/v2) #### API example — Python ```python from apify_client import ApifyClient client = ApifyClient("YOUR_APIFY_TOKEN") run = client.actor("sleek_waveform/substack-creator-scraper").call(run_input={ "startUrls": [{"url": "https://www.astralcodexten.com/"}], "maxItems": 100, "scrapeFormat": "markdown" }) posts = client.dataset(run["defaultDatasetId"]).list_items().items for post in posts: print(f"{post['postDate'][:10]} | {post['likes']} likes | {post['title']}") ``` ### FAQ about Substack Scraper **Does this Actor require a Substack account or login?** No. It only extracts publicly available posts and metadata — no login credentials, session cookies, or Substack API key are required. **Can I scrape paid/paywalled posts?** Only the free-preview portion of paywalled posts is accessible. Full body content behind a paid subscription wall is not extracted. The `isPaywalled` field tells you whether a post is behind a paywall. **How do I scrape the full archive of a newsletter?** Set `maxItems` to a high number (e.g., 1000) and point `startUrls` to the publication homepage (e.g., `https://platformer.news/`). The scraper auto-paginates through the entire archive until it hits `maxItems` or exhausts all posts. **Can I scrape multiple publications at once?** Yes. Add multiple URLs to `startUrls`. Each publication is scraped independently, and all results land in the same dataset with `publicationName` as a filter column. **What format does the `body` field use?** Your choice: `"markdown"` (clean prose, good for LLMs and vector databases), `"html"` (preserves formatting for display), or `"none"` (metadata only — fastest option for engagement analysis without needing body text). **How many posts can I scrape on the free plan?** With Apify's $5 monthly free credit, approximately **2,500 posts** per month at no cost. **Does it scrape reader comments?** Comment count is extracted (`comments` field), but individual comment text is not — Substack serves comments via a separate authenticated endpoint. **How do I monitor new posts from a publication weekly?** Set up a scheduled run on Apify: Actor page → Schedule → weekly. Filter for posts newer than a specific date by combining `maxItems: 20` (which always returns the most recent) with the `postDate` field in your downstream processing. **Can I use this for LLM training data?** Yes. The `"markdown"` output format produces clean, boilerplate-free prose ideal for LLM fine-tuning and RAG pipelines. Pair with the [Website to Markdown Scraper](https://apify.com/sleek_waveform/llm-markdown-crawler) to build multi-source AI training datasets. ### High-value Substack publications to scrape | Category | Example publications | |----------|---------------------| | AI / Tech | Stratechery, Import AI, The Batch, AI Supremacy | | Finance | The Diff, Money Stuff (Bloomberg), Odd Lots | | Media / Politics | Semafor, Platformer, The Atlantic | | Growth / Startups | Lenny's Newsletter, The Generalist, Not Boring | | Newsletter operators | The Rebooting, Inbox Collective | ### Other sleek\_waveform Actors you might like - **[Website to Markdown Scraper](https://apify.com/sleek_waveform/llm-markdown-crawler)** — crawl any website and extract clean Markdown for RAG pipelines. Pairs with Substack Scraper to build multi-source LLM datasets. - **[Threads Profile & Post Scraper](https://apify.com/sleek_waveform/threads-scraper)** — scrape Threads posts, hashtags, and engagement metrics. Many Substack writers cross-post to Threads — combine both scrapers for a full picture of a creator's reach. - **[YouTube Trend Scraper](https://apify.com/sleek_waveform/youtube-trend-spotter)** — track trending YouTube videos by keyword. Compare Substack newsletter topics against what's gaining traction on YouTube for cross-platform content strategy. *** *Found this Actor useful? Leave a review on the Apify Store — it takes 30 seconds and helps other developers discover it.* # Actor input Schema ## `startUrls` (type: `array`): Substack publication URLs (e.g., https://www.lennysnewsletter.com/) or specific post URLs (e.g., https://www.lennysnewsletter.com/p/post-slug). You can provide multiple URLs to scrape multiple publications. ## `maxItems` (type: `integer`): Maximum number of posts to scrape from each publication. The actor will automatically paginate through the publication's archive until it reaches this limit or exhausts all available posts. ## `scrapeFormat` (type: `string`): Format for extracting the post body content. Choose 'Markdown' for clean text analysis, 'HTML' to preserve formatting and images, or 'None' for metadata-only extraction (fastest). ## `maxRequestRetries` (type: `integer`): Number of times to retry a failed request before giving up. Increase this value if you encounter frequent timeouts or blocking issues. ## `maxSessionRotations` (type: `integer`): Maximum number of session rotations before giving up on a request. Higher values improve reliability but may increase execution time. ## `webhookUrl` (type: `string`): Optional URL to notify when the Actor run finishes (success or failure). A POST request with run details will be sent to this URL. Useful for integrations with Zapier, Make, n8n, or any custom pipeline. ## Actor input object example ```json { "startUrls": [ { "url": "https://simonw.substack.com/" } ], "maxItems": 3, "scrapeFormat": "none", "maxRequestRetries": 2, "maxSessionRotations": 5 } ``` # Actor output Schema ## `results` (type: `string`): Dataset containing all scraped Substack posts. Each item includes url, publicationName, authorName, title, subtitle, postDate, likes, comments, isPaywalled, and body. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://simonw.substack.com/" } ], "maxItems": 3, "scrapeFormat": "none" }; // Run the Actor and wait for it to finish const run = await client.actor("sleek_waveform/substack-creator-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://simonw.substack.com/" }], "maxItems": 3, "scrapeFormat": "none", } # Run the Actor and wait for it to finish run = client.actor("sleek_waveform/substack-creator-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://simonw.substack.com/" } ], "maxItems": 3, "scrapeFormat": "none" }' | apify call sleek_waveform/substack-creator-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=sleek_waveform/substack-creator-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Substack Posts & Creator Scraper", "description": "Scrape posts, engagement metrics, and author data from any Substack publication. Get title, author, publish date, likes, comments, paywall status, and full body in Markdown or HTML. Paginates the full archive automatically.", "version": "1.0", "x-build-id": "hwCQ9LqTSxT3kgFRO" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/sleek_waveform~substack-creator-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-sleek_waveform-substack-creator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/sleek_waveform~substack-creator-scraper/runs": { "post": { "operationId": "runs-sync-sleek_waveform-substack-creator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/sleek_waveform~substack-creator-scraper/run-sync": { "post": { "operationId": "run-sync-sleek_waveform-substack-creator-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Substack publication URLs (e.g., https://www.lennysnewsletter.com/) or specific post URLs (e.g., https://www.lennysnewsletter.com/p/post-slug). You can provide multiple URLs to scrape multiple publications.", "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "maxItems": { "title": "Max Posts Per Publication", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of posts to scrape from each publication. The actor will automatically paginate through the publication's archive until it reaches this limit or exhausts all available posts.", "default": 100 }, "scrapeFormat": { "title": "Post Body Format", "enum": [ "markdown", "html", "none" ], "type": "string", "description": "Format for extracting the post body content. Choose 'Markdown' for clean text analysis, 'HTML' to preserve formatting and images, or 'None' for metadata-only extraction (fastest).", "default": "markdown" }, "maxRequestRetries": { "title": "Maximum Request Retries", "minimum": 0, "maximum": 10, "type": "integer", "description": "Number of times to retry a failed request before giving up. Increase this value if you encounter frequent timeouts or blocking issues.", "default": 2 }, "maxSessionRotations": { "title": "Maximum Session Rotations", "minimum": 1, "maximum": 50, "type": "integer", "description": "Maximum number of session rotations before giving up on a request. Higher values improve reliability but may increase execution time.", "default": 5 }, "webhookUrl": { "title": "Webhook URL", "pattern": "^https?://.+", "type": "string", "description": "Optional URL to notify when the Actor run finishes (success or failure). A POST request with run details will be sent to this URL. Useful for integrations with Zapier, Make, n8n, or any custom pipeline." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```