# Smart Article Extractor (`datapilot/smart-article-extractor`) Actor News Article Extractor Actor fetches article URLs and extracts structured content using Requests, , and Newspaper3k. It collects title, author, publish date, text, summary, keywords, images, and word count. Supports proxy use and outputs clean JSON results. - **URL**: https://apify.com/datapilot/smart-article-extractor.md - **Developed by:** [Data Pilot](https://apify.com/datapilot) (community) - **Categories:** News, Other, Social media - **Stats:** 7 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $10.00/month + usage To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Smart Article Extractor πŸš€ **Smart Article Extractor** is a powerful Apify Actor designed to extract article content from web pages using advanced NLP and parsing techniques. This tool provides comprehensive **Smart Article information**, including titles, authors, full text, summaries, and metadata for any URL. Whether you're conducting **Smart Article research**, content aggregation, or NLP tasks, the Smart Article Extractor delivers accurate **Smart Article data** efficiently. With dual extraction methods using Newspaper3k and , the Smart Article Extractor ensures reliable extraction of article content that may not be available through simple HTML parsing. It focuses on key **Smart Article metrics** like word count, keywords, and publication dates, making it an essential tool for **Smart Article analysis** and content processing. ### πŸ”₯ Features - **Comprehensive Smart Article Extraction** – Extracts detailed **Smart Article data**, including titles, authors, full text, and summaries for any URL. - **Dual Extraction Engine** – Uses both Newspaper3k and for robust **Smart Article content extraction**. - **Homepage Detection** – Automatically skips potential homepages to ensure only **Smart Article content** is processed. - **Proxy Support** – Utilizes Apify's residential proxies to bypass restrictions and ensure high success rates for **Smart Article scraping**. - **Metadata Enrichment** – Provides **Smart Article metadata** like keywords, publication dates, and image URLs. - **Text Cleaning** – Automatically cleans and formats **Smart Article text** for better readability and NLP processing. - **Error Handling** – Robust logging and fallback mechanisms for failed **Smart Article extractions**. - **Dataset Integration** – Automatically uploads **Smart Article data** to your Apify dataset for easy export and analysis. --- ### βš™οΈ How It Works The Smart Article Extractor takes a list of URLs as input and uses requests to fetch the HTML content. It then employs Newspaper3k for initial extraction, falling back to for more complex pages. The extractor returns structured **Smart Article data** on success or error details on failure, providing a reliable way to gather **Smart Article information** for research and analysis. **Key Processing Steps:** 1. **URL Validation** – Parse and validate article URLs 2. **HTML Fetching** – Fetch page content using requests 3. **Newspaper3k Extraction** – Extract using NLP-based method 4. ** Fallback** – Use HTML parsing if primary fails 5. **Homepage Detection** – Skip non-article pages 6. **Metadata Extraction** – Get titles, authors, dates 7. **Text Cleaning** – Format and clean article text 8. **Export** – Push results to dataset in JSON format **Key benefits for Smart Article analysis:** - Access full **Smart Article text** and metadata. - Analyze **Smart Article content** for NLP and sentiment analysis. - Build **Smart Article databases** for content research. - Extract structured data from unstructured web content. - Enable content aggregation and news monitoring. --- ### πŸ“₯ Input The extractor accepts the following input parameters: | Field | Type | Default | Description | |---------|----------------|----------|-------------| | `urls` | string / array | required | List of URLs to extract Smart Article data from, comma-separated or array (e.g., `"https://example.com/article1, https://example.com/article2"`). | **Example input JSON:** ```json { "urls": "https://www.bbc.com/news/live/cz0g2yg3579t, https://example.com/article2" } ```` **Alternative array format:** ```json { "urls": [ "https://example.com/article1", "https://example.com/article2", "https://news.example.com/article3" ] } ``` *** ### πŸ“€ Output The extractor outputs detailed **Smart Article data** in JSON format for each URL. Each record includes: | Field | Type | Description | |---------------|---------|-------------| | `url` | string | Original URL of the Smart Article. | | `domain` | string | Domain of the Smart Article source. | | `title` | string | Title of the Smart Article. | | `author` | array | Author(s) of the Smart Article. | | `publisher` | string | Publisher domain of the Smart Article. | | `published` | string | Publication date of the Smart Article. | | `text` | string | Full text content of the Smart Article. | | `summary` | string | Summary of the Smart Article. | | `keywords` | array | Keywords associated with the Smart Article. | | `word_count` | integer | Word count of the Smart Article. | | `image` | string | Top image URL of the Smart Article. | | `scrapedAt` | string | ISO timestamp of the scrape. | **Example output for Smart Article data:** ```json { "url": "https://www.bbc.com/news", "domain": "example.com", "title": "Example Smart Article Title", "author": ["John Doe"], "publisher": "example.com", "published": "2025-02-14T12:00:00Z", "text": "This is the full text of the smart article...", "summary": "This is a summary of the smart article.", "keywords": ["technology", "news"], "word_count": 500, "image": "https://example.com/image.jpg", "scrapedAt": "2025-02-14T12:00:00Z" } ``` **Example error response:** ```json { "url": "https://www.bbc.com/news", "status": "failed", "error": "Article content not found or page is not an article", "scrapedAt": "2025-02-14T12:00:00Z" } ``` **Example summary record:** ```json { "summary": true, "total_urls": 10, "successful_extractions": 9, "failed_extractions": 1, "average_word_count": 650, "total_keywords": 45, "completed_at": "2025-02-14T12:35:00Z" } ``` *** ### 🧰 Technical Stack - **Article Extraction:** Newspaper3k – Advanced NLP-based extraction - **HTML Parsing:** – Robust fallback parsing - **HTTP Client:** requests – Web page fetching - **NLP:** NLTK – Natural language processing for summaries - **Text Processing:** Natural language tokenization and analysis - **Platform:** Apify Actor – serverless, scalable, integrated with Dataset - **Deployment:** One‑click run on Apify Console or via REST API *** ### 🎯 Use Cases - **Content Aggregation** – Aggregate article content from multiple sources. - **News Monitoring** – Extract and monitor news articles on specific topics. - **Research & Analysis** – Extract content for academic and market research. - **Sentiment Analysis** – Analyze article sentiment and tone. - **Keyword Extraction** – Extract and analyze keywords from articles. - **Content Database Building** – Build databases of article content. - **Archive Creation** – Create archives of important articles. - **Competitor Intelligence** – Monitor competitor-related news and articles. - **SEO Analysis** – Analyze article structure for SEO insights. - **Content Recommendation** – Extract data for content recommendation systems. - **Machine Learning Datasets** – Create datasets for ML and NLP models. - **Summarization Research** – Analyze article summaries and extraction quality. - **Author Analysis** – Track articles by specific authors. - **Publication Analysis** – Analyze content patterns by publication. *** ### πŸš€ Quick Start 1. **Open in Apify Console** – visit the Actor page and click Try for free. 2. **Enter article URLs** – provide one or more article URLs (comma-separated or as array). 3. **Click Start** – the Actor will extract article content using dual extraction engines. 4. **View Results** – check the dataset for extracted article data. 5. **Analyze Content** – examine titles, authors, full text, summaries, and keywords. 6. **Review Metadata** – check publication dates, authors, and image URLs. 7. **Monitor Quality** – review extraction success and any failures. 8. **Export** – download the results as JSON, CSV, or Excel. You can also call this Actor programmatically via Apify SDK or REST API – ideal for automated content extraction and news aggregation pipelines. *** ### πŸ’Ž Why This Extractor? | Feature | Benefit | |---------|---------| | βœ… Full text | Get complete article content, not just snippets. | | βœ… NLP-powered | Smart extraction using natural language processing. | | βœ… Metadata rich | Get titles, authors, dates, keywords, images. | | βœ… Homepage detection | Skip non-article pages automatically. | | βœ… Text cleaning | Clean, formatted text ready for analysis. | | βœ… Error handling | Robust fallback mechanisms. | | βœ… Apify ecosystem | Seamless integration with other Actors, triggers, and webhooks. | ### πŸ“¦ Changelog - Initial release of Smart Article Extractor - Newspaper3k integration for NLP-based extraction - fallback for complex pages - Title, author, and publisher extraction - Full text content extraction - Publication date parsing - Keyword extraction and analysis - Article summary generation - Image URL extraction - Word count calculation - Homepage detection and filtering - Text cleaning and formatting - Dual extraction engine support - Error handling with detailed logging - Automatic dataset integration - Full Apify Actor integration *** ### πŸ§‘β€πŸ’» Support & Feedback - **Issues & Ideas:** Open a ticket on the Apify Actor issue tracker - **Contributions:** Pull requests are welcome via the GitHub repository - **Documentation:** Visit Apify Docs for comprehensive platform guides - **Community:** Join the Apify community forum for discussions and support - **Bug Reports:** Submit detailed bug reports through the issue tracker - **Feature Requests:** Suggest new features to improve the extractor *** ### πŸ’° Pricing - **Free** for basic usage on Apify platform - **Paid plans** available for higher limits and priority support *** **Disclaimer:** Smart Article Extractor is provided as-is for research and content analysis purposes. Users are responsible for ensuring their usage complies with website policies and applicable copyright laws. Always attribute content to original authors and publications. *** ### πŸŽ‰ Get Started Today **Begin extracting article content now!** Use Smart Article Extractor for: - πŸ“° Content Aggregation - πŸ“Š Text Analysis - πŸ” Research & Analysis - πŸ’‘ NLP Tasks - πŸ“š Database Building **Perfect for:** - Content Strategists - Researchers - Data Scientists - Content Curators - Analysts *** **Last Updated:** February 2025\ **Version:** 1.0.0\ **Status:** Active Development\ **Support:** 24/7 Customer Support Available\ **Platform:** Apify *** ### πŸ“š Related Tools For comprehensive content analysis and research, explore our full suite of tools: - Fast News Content Scraper - Google Search Results Scraper - Ranked Keywords Scraper with SEO Metrics - RAG Web Scraper - All-in-One Media Downloader # Actor input Schema ## `urls` (type: `string`): Article URLs to extract (comma-separated) ## `useNewspaper3k` (type: `boolean`): Use newspaper3k for better text extraction (slower but more accurate) ## `maxImages` (type: `integer`): Maximum images to extract per article (default: 20) ## `maxLinks` (type: `integer`): Maximum outbound links to extract per article (default: 30) ## `useApifyProxy` (type: `boolean`): Recommended to avoid blocks ## `apifyProxyGroups` (type: `array`): Select proxy type ## Actor input object example ```json { "urls": "https://example.com/article-1, https://example.com/article-2", "useNewspaper3k": true, "maxImages": 20, "maxLinks": 30, "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": "https://example.com/article-1, https://example.com/article-2" }; // Run the Actor and wait for it to finish const run = await client.actor("datapilot/smart-article-extractor").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": "https://example.com/article-1, https://example.com/article-2" } # Run the Actor and wait for it to finish run = client.actor("datapilot/smart-article-extractor").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": "https://example.com/article-1, https://example.com/article-2" }' | apify call datapilot/smart-article-extractor --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=datapilot/smart-article-extractor", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Smart Article Extractor", "description": "News Article Extractor Actor fetches article URLs and extracts structured content using Requests, , and Newspaper3k. It collects title, author, publish date, text, summary, keywords, images, and word count. Supports proxy use and outputs clean JSON results.", "version": "0.0", "x-build-id": "9zl6egHExoFCiEeJY" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/datapilot~smart-article-extractor/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-datapilot-smart-article-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/datapilot~smart-article-extractor/runs": { "post": { "operationId": "runs-sync-datapilot-smart-article-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/datapilot~smart-article-extractor/run-sync": { "post": { "operationId": "run-sync-datapilot-smart-article-extractor", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "urls" ], "properties": { "urls": { "title": "Article URLs", "type": "string", "description": "Article URLs to extract (comma-separated)" }, "useNewspaper3k": { "title": "Use Newspaper3k", "type": "boolean", "description": "Use newspaper3k for better text extraction (slower but more accurate)", "default": true }, "maxImages": { "title": "Max Images per Article", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum images to extract per article (default: 20)", "default": 20 }, "maxLinks": { "title": "Max Links per Article", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum outbound links to extract per article (default: 30)", "default": 30 }, "useApifyProxy": { "title": "Use Apify Proxy", "type": "boolean", "description": "Recommended to avoid blocks", "default": true }, "apifyProxyGroups": { "title": "Proxy Groups", "uniqueItems": true, "type": "array", "description": "Select proxy type", "items": { "type": "string", "enum": [ "RESIDENTIAL", "DATACENTER" ] }, "default": [ "RESIDENTIAL" ] } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```