# Substack Newsletter Scraper - Articles, Metadata & Full Content (`hata1234/substack-scraper`) Actor Extract articles, metadata, and content from any Substack newsletter via public API. No proxy needed. Supports multiple newsletters, full article body extraction, audience filtering (free/paid), date range, keyword search, and pagination. Works with both substack.com subdomains and custom domains. - **URL**: https://apify.com/hata1234/substack-scraper.md - **Developed by:** [Moris Chao](https://apify.com/hata1234) (community) - **Categories:** News, Social media - **Stats:** 24 total users, 5 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Substack Newsletter Scraper Scrape articles from any Substack newsletter using the public Substack API. Supports both `subdomain.substack.com` URLs and custom domains (e.g., `www.lennysnewsletter.com`). ### Features - Scrape article metadata (title, author, date, reactions, comments, etc.) - Optionally fetch full article HTML body - Filter by audience (free/paid), content type, date range, or keyword - Scrape multiple newsletters in a single run (comma-separated URLs) - Concurrent batch fetching for full article bodies - Polite rate limiting (500ms delay between requests) ### Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `newsletterUrl` | string | *required* | Newsletter URL(s). Comma-separated for multiple. | | `maxItems` | integer | `50` | Max articles per newsletter. `0` = unlimited. | | `sortBy` | string | `"new"` | `"new"` (newest first) or `"top"` (most popular). | | `includeBody` | boolean | `false` | Fetch full HTML body for each article. | | `audienceFilter` | string | `"all"` | `"all"`, `"free"`, or `"paid"`. | | `typeFilter` | string | `"all"` | `"all"`, `"newsletter"`, `"podcast"`, or `"thread"`. | | `dateFrom` | string | — | Only articles on/after this date (YYYY-MM-DD). | | `dateTo` | string | — | Only articles on/before this date (YYYY-MM-DD). | | `searchKeyword` | string | — | Filter by keyword in title or description. | #### Example Input ```json { "newsletterUrl": "https://www.lennysnewsletter.com", "maxItems": 10, "sortBy": "new", "includeBody": false } ```` #### Multiple Newsletters ```json { "newsletterUrl": "https://www.lennysnewsletter.com, https://stratechery.com", "maxItems": 20 } ``` ### Output Each article is saved to the default dataset with the following fields: | Field | Type | Description | |-------|------|-------------| | `id` | number | Substack post ID | | `title` | string | Article title | | `subtitle` | string | Article subtitle | | `slug` | string | URL slug | | `url` | string | Full canonical URL | | `postDate` | string | ISO 8601 publish date | | `audience` | string | `"everyone"` or `"only_paid"` | | `type` | string | `"newsletter"`, `"podcast"`, or `"thread"` | | `wordcount` | number | Word count | | `reactions` | object | Reaction counts (e.g., `{"❤": 532}`) | | `commentCount` | number | Number of comments | | `coverImage` | string | Cover image URL | | `author` | string | Author name(s) | | `description` | string | Article description/excerpt | | `body` | string | Full HTML body (only when `includeBody: true`) | #### Example Output ```json { "id": 123456, "title": "How to build a great product", "subtitle": "Lessons from top PMs", "slug": "how-to-build-a-great-product", "url": "https://www.lennysnewsletter.com/p/how-to-build-a-great-product", "postDate": "2026-03-03T13:45:17.054Z", "audience": "everyone", "type": "newsletter", "wordcount": 2642, "reactions": { "❤": 532 }, "commentCount": 9, "coverImage": "https://substackcdn.com/image/...", "author": "Lenny Rachitsky", "description": "A deep dive into product excellence..." } ``` ### Notes - **Paid articles**: Articles marked as `"only_paid"` may only return a preview of the body content. - **Rate limiting**: The Actor adds a 500ms delay between API requests to avoid overloading Substack servers. - **Custom domains**: Both `newsletter.substack.com` and custom domains like `platformer.news` are supported. - **No authentication required**: Uses Substack's public API endpoints. ### Cost This Actor uses only HTTP API calls (no browser), so it's very lightweight: - \~256MB memory is sufficient - No proxy required - A typical run of 50 articles completes in under a minute # Actor input Schema ## `newsletterUrl` (type: `string`): URL of the Substack newsletter to scrape. Supports both subdomain.substack.com and custom domains (e.g. www.lennysnewsletter.com). Use comma-separated URLs to scrape multiple newsletters. ## `maxItems` (type: `integer`): Maximum number of articles to scrape per newsletter. Set to 0 for unlimited. ## `sortBy` (type: `string`): Sort order for articles. ## `includeBody` (type: `boolean`): Whether to fetch the full HTML body for each article. This requires an additional API call per article and will slow down the scrape. ## `audienceFilter` (type: `string`): Filter articles by audience type. ## `typeFilter` (type: `string`): Filter articles by content type. ## `dateFrom` (type: `string`): Only include articles published on or after this date (YYYY-MM-DD format). ## `dateTo` (type: `string`): Only include articles published on or before this date (YYYY-MM-DD format). ## `searchKeyword` (type: `string`): Filter articles whose title or description contains this keyword (case-insensitive). ## Actor input object example ```json { "newsletterUrl": "https://www.lennysnewsletter.com", "maxItems": 50, "sortBy": "new", "includeBody": false, "audienceFilter": "all", "typeFilter": "all" } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "newsletterUrl": "https://www.lennysnewsletter.com" }; // Run the Actor and wait for it to finish const run = await client.actor("hata1234/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "newsletterUrl": "https://www.lennysnewsletter.com" } # Run the Actor and wait for it to finish run = client.actor("hata1234/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "newsletterUrl": "https://www.lennysnewsletter.com" }' | apify call hata1234/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=hata1234/substack-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Substack Newsletter Scraper - Articles, Metadata & Full Content", "description": "Extract articles, metadata, and content from any Substack newsletter via public API. No proxy needed. Supports multiple newsletters, full article body extraction, audience filtering (free/paid), date range, keyword search, and pagination. Works with both substack.com subdomains and custom domains.", "version": "1.0", "x-build-id": "OnnoefeFzh0RluOAm" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/hata1234~substack-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-hata1234-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/hata1234~substack-scraper/runs": { "post": { "operationId": "runs-sync-hata1234-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/hata1234~substack-scraper/run-sync": { "post": { "operationId": "run-sync-hata1234-substack-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "newsletterUrl" ], "properties": { "newsletterUrl": { "title": "Newsletter URL(s)", "type": "string", "description": "URL of the Substack newsletter to scrape. Supports both subdomain.substack.com and custom domains (e.g. www.lennysnewsletter.com). Use comma-separated URLs to scrape multiple newsletters." }, "maxItems": { "title": "Max Items", "minimum": 0, "type": "integer", "description": "Maximum number of articles to scrape per newsletter. Set to 0 for unlimited.", "default": 50 }, "sortBy": { "title": "Sort By", "enum": [ "new", "top" ], "type": "string", "description": "Sort order for articles.", "default": "new" }, "includeBody": { "title": "Include Full Article Body", "type": "boolean", "description": "Whether to fetch the full HTML body for each article. This requires an additional API call per article and will slow down the scrape.", "default": false }, "audienceFilter": { "title": "Audience Filter", "enum": [ "all", "free", "paid" ], "type": "string", "description": "Filter articles by audience type.", "default": "all" }, "typeFilter": { "title": "Type Filter", "enum": [ "all", "newsletter", "podcast", "thread" ], "type": "string", "description": "Filter articles by content type.", "default": "all" }, "dateFrom": { "title": "Date From", "type": "string", "description": "Only include articles published on or after this date (YYYY-MM-DD format)." }, "dateTo": { "title": "Date To", "type": "string", "description": "Only include articles published on or before this date (YYYY-MM-DD format)." }, "searchKeyword": { "title": "Search Keyword", "type": "string", "description": "Filter articles whose title or description contains this keyword (case-insensitive)." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```