# Substack Newsletter Scraper & Author Extractor (`pure_matai/substack-newsletter-scraper`) Actor Paste Substack URLs to extract structured newsletter data. Export posts, author profiles, and publication details to CSV, Excel, JSON, or XML. Process 1,000 posts for $0.50. No coding or browser needed. Ideal for newsletter research, PR monitoring, sponsorship research, competitive intelligence. - **URL**: https://apify.com/pure\_matai/substack-newsletter-scraper.md - **Developed by:** [AzulGanzes Technologies](https://apify.com/pure_matai) (community) - **Categories:** Developer tools, AI, News - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 per exported posts This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Substack Newsletter Scraper & Author Extractor AZ StackPulse hero banner — every newsletter that matters in 60 seconds **Turn 3 hours of manual newsletter monitoring into 60 seconds. 1,000 posts for $0.50.** *** AZ StackPulse — watch it run: paste Substack URLs, get a CRM-ready dataset in 60 seconds for $0.50 *** ### Why You Need This A PR agency monitoring 50 newsletters manually spends **3 hours a week** reading, copying, and pasting. At a $50/hr billing rate, that's **$125/week** in lost billable time. This tool does the same job in **60 seconds** for **$0.50**. **No browser. No manual reading. No coding.** Paste your Substack URLs, click Start, and download a structured dataset ready for your CRM, newsletter research, or competitive analysis. **Public content only. No login required. No paywall bypass. Robots.txt respected.** *** ### What You Get AZ StackPulse export — 12 posts from 3 newsletters, CRM-ready spreadsheet AZ StackPulse start — one-click input, pre-filled URLs, ready to run | Export Formats | Output Fields | |---|---| | JSON, CSV, Excel (XLSX), XML | 21 fields per post: newsletter metadata, post details, engagement stats, full content, author profile, media links | Sample output (CSV): ```csv newsletterName,newsletterUrl,title,publishedAt,authorName,wordCount,likeCount Lenny's Newsletter,https://lenny.substack.com,"How to run effective brainstorming sessions",2026-07-10,Lenny Rachitsky,2400,847 ``` *** ### How to Run It 1. **Click Start** — Lenny's Newsletter is pre-filled. First 10 posts are free. No typing required. 2. **Configure filters** — date range, keyword match, post limit (optional). 3. **Results appear** — in under 60 seconds. 4. **Download your dataset** — CSV, Excel, JSON, or XML. Ready for your CRM. The tool checks `robots.txt` before every crawl, respects rate limits, and never accesses subscriber-only content. *** ### Pricing | What You Pay | Price | Example Run | |---|---|---| | Per post exported | $0.00050 | **1,000 posts = $0.50** | | Full content fetch | +$0.00030/post | **1,000 posts with article text = $0.80** | | Free preview | First 10 posts | **$0.00** | *** ### For Everyone Else | You're a... | Here's what this tool does for you | |---|---| | **Newsletter Sponsor** | Export engagement stats (likes, comments, restacks) across 20 candidates. Sort by audience activity. Know exactly which newsletter to sponsor. | | **VC or Investor** | Feed 30 thought-leader newsletters through keyword filters. Get a structured deal-flow intelligence feed — not a flooded inbox. | | **Recruiter** | Identify domain experts by newsletter. Export author bios and public profiles for direct outreach. | | **Audience Research Team** | Enrich B2B contact databases with newsletter authorship signals. Every writer is a potential buyer, partner, or influencer. | *** ### Example Use Cases **Weekly PR Coverage Report** ``` newsletterUrls: ["client1.substack.com", "client2.substack.com"] keywordFilter: "ClientBrandName" includeContent: false maxPosts: 0 ``` Exports every mention of your client. Run weekly. ~$0.30/run. **Newsletter Sponsorship Audit** ``` newsletterUrls: ["candidate1.substack.com", "candidate2.substack.com"] includeContent: false maxPosts: 100 ``` Compare engagement across candidates. ~$0.10 per newsletter. **VC Deal Flow Monitor** ``` newsletterUrls: ["analyst1.substack.com", "analyst2.substack.com"] keywordFilter: "fundraising|funding|series A" dateFrom: "2026-01-01" ``` Track funding announcements. ~$0.50/run. *** ### Screenshots AZ StackPulse PR persona — find every client mention instantly with keyword filter AZ StackPulse export — CSV, JSON, XLSX, XML formats, CRM-ready AZ StackPulse strengths — 6 reasons customers choose us AZ StackPulse trust — public content only, no paywall bypass, robots.txt respected *** ### Technical Details The tool uses Substack's public RSS feeds and JSON API — no browser automation, no login, no reverse engineering. | Component | Responsibility | |---|---| | `src/fetcher.py` | RSS parsing, JSON API, robots.txt enforcement, rate-limit handling | | `src/parser.py` | Extracts 21 fields from raw data | | `src/exporter.py` | Outputs JSON, CSV, XLSX, XML | | `src/utils.py` | URL validation, domain extraction | **Depends on:** Python 3.11+, `apify`, `httpx`, `beautifulsoup4`, `lxml`, `openpyxl` # Actor input Schema ## `newsletterUrls` (type: `array`): One or more Substack newsletter URLs (e.g. https://lenny.substack.com). Custom domains pointing to Substack are supported. ## `maxPosts` (type: `integer`): Maximum number of posts to export per newsletter. Default is 10 for the free preview. Set to 0 to fetch all available. ## `includeContent` (type: `boolean`): Fetch full article HTML and plain text. Disable for faster, cheaper metadata-only runs. ## `dateFrom` (type: `string`): Only include posts published on or after this date (YYYY-MM-DD). Leave empty for no filter. ## `dateTo` (type: `string`): Only include posts published on or before this date (YYYY-MM-DD). Leave empty for no filter. ## `keywordFilter` (type: `string`): Case-insensitive substring match on post title. Only matching posts are returned. ## `includeAuthors` (type: `boolean`): Fetch author profile data (name, bio, avatar) for each newsletter. ## `proxy` (type: `object`): Proxy settings for large runs. Apify proxy recommended for runs over 1,000 posts. ## Actor input object example ```json { "newsletterUrls": [ "https://lenny.substack.com" ], "maxPosts": 10, "includeContent": true, "includeAuthors": true, "proxy": { "useApifyProxy": true } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "newsletterUrls": [ "https://lenny.substack.com" ], "proxy": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("pure_matai/substack-newsletter-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "newsletterUrls": ["https://lenny.substack.com"], "proxy": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("pure_matai/substack-newsletter-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "newsletterUrls": [ "https://lenny.substack.com" ], "proxy": { "useApifyProxy": true } }' | apify call pure_matai/substack-newsletter-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=pure_matai/substack-newsletter-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/SAS4G3bDUeSL6YbqS/builds/RqWRBwgBVN9b87Ys6/openapi.json