# arXiv Paper Scraper — Abstracts, Authors & Metadata (`logiover/arxiv-paper-scraper`) Actor Scrape research paper metadata from arXiv.org the worlds largest open-access repository. Search by keyword across computer science physics mathematics biology. Returns titles abstracts authors categories PDF links and DOIs. No API key required. - **URL**: https://apify.com/logiover/arxiv-paper-scraper.md - **Developed by:** [Logiover](https://apify.com/logiover) (community) - **Categories:** Developer tools, Business - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## arXiv Paper Scraper — Research Metadata, Abstracts & Author Data (No API Key) ![Apify Actor](https://img.shields.io/badge/Apify-Actor-00A67E?logo=apify\&logoColor=white) ![No API key](https://img.shields.io/badge/No%20API%20key-required-2ea44f) ![Pay per result](https://img.shields.io/badge/Pricing-Pay%20per%20result-1C7ED6) ![Research Data](https://img.shields.io/badge/Category-Research%20%26%20Science-8B5CF6) ![Export](https://img.shields.io/badge/Export-JSON%20%7C%20CSV%20%7C%20Excel-F59E0B) Scrape research paper metadata from **arXiv.org**, the world's largest open-access preprint repository with **2.5M+ scholarly articles** across computer science, physics, mathematics, quantitative biology, economics, statistics and more. Search by **keyword, topic, author or category** and get back structured records with **titles, full abstracts, complete author lists, categories, submission dates, PDF links, DOIs and journal references** — clean JSON, straight from the **official arXiv API**. **Fast, no browser, no API key, no login.** > ### 🏆 Why this arXiv scraper? > > **15 fields per paper** · **up to 1,000 papers per query, unlimited queries** · direct HTTP against the official arXiv Atom API (no browser, no key) · datacenter-proxy friendly · export to JSON / CSV / Excel. The turnkey way to build a **research-paper dataset** for literature reviews, AI/tech intelligence, talent sourcing and NLP training data. *** ### ✨ What this Actor does / Key features - 📚 **Full-text abstracts** — the complete abstract for every paper, not just a snippet — ready for embeddings, summarization or topic modeling. - 👤 **Complete author lists** — every co-author name, perfect for talent sourcing and citation-graph work. - 🏷️ **Category filtering** — narrow to arXiv taxonomy codes like `cs.AI`, `cs.CL`, `cs.CV`, `stat.ML`, `physics.optics` or `q-bio.GN`, or search across all fields. - 🔎 **Multi-query search** — pass many search terms in one run; each paper is tagged with the `searchQuery` that found it. - 🗓️ **Date filtering & sorting** — keep only papers submitted after a `dateFrom` cutoff; sort by relevance, last-updated or submission date. - 🔗 **Direct links** — abstract-page URL and direct **PDF download URL** for every result. - 🧾 **Publication metadata** — `doi`, `journalRef` and author `comment` (accepted-venue notes, code links, page counts) when the authors provide them. - ⚡ **Official API, no browser** — queries arXiv's public Atom/OAI-compatible API at `export.arxiv.org`; no headless browser, no HTML parsing, no scraping tricks. - 🛡️ **Proxy support** — Apify Proxy integration; arXiv is proxy-friendly and datacenter proxies work fine. ### 🚀 Quick start (3 steps) 1. **Configure** — add one or more `searchQueries` (keywords, a topic, or an author name). Optionally add `categories`, a `dateFrom` cutoff and a `sortBy` order. 2. **Run** — click **Start**. The Actor pages through the arXiv API for every query and streams papers into your dataset. 3. **Get your data** — open the **Output** tab and export to **JSON, CSV, Excel or XML**, or pull it via the Apify API. ### 📥 Input Give the Actor at least one `searchQueries` value. Everything else is optional. #### Example — recent AI & NLP preprints (dataset building) ```json { "searchQueries": ["large language models", "retrieval augmented generation"], "categories": ["cs.AI", "cs.CL", "cs.LG"], "sortBy": "submittedDate", "dateFrom": "2026-01-01", "maxResults": 500 } ``` #### Example — track one author's output (talent sourcing) ```json { "searchQueries": ["Yann LeCun"], "sortBy": "submittedDate", "maxResults": 200 } ``` #### Example — broad topic sweep across fields ```json { "searchQueries": ["quantum computing", "diffusion models", "protein folding"], "sortBy": "relevance", "maxResults": 300, "proxyConfiguration": { "useApifyProxy": true } } ``` | Field | Type | Description | |-------|------|-------------| | `searchQueries` | array | Search terms — keywords, topics, paper titles or author names. Each term is queried independently. **Required.** | | `categories` | array | arXiv category codes to filter by (e.g. `cs.AI`, `cs.CL`, `cs.CV`, `stat.ML`, `physics.optics`, `q-bio.GN`). Leave empty to search all categories. | | `maxResults` | integer | Maximum total papers to return across all queries (1–1000). Default 200. | | `sortBy` | string | `relevance`, `lastUpdatedDate` or `submittedDate`. Default `relevance`. | | `dateFrom` | string | Keep only papers submitted on/after this date (`YYYY-MM-DD`). Empty = no date filter. | | `proxyConfiguration` | object | Apify Proxy settings. arXiv is proxy-friendly; datacenter proxies work fine. | > **Tip — category codes:** browse [arxiv.org/category\_taxonomy](https://arxiv.org/category_taxonomy) to find the exact code for your field. Combining a focused `searchQueries` term with one or two `categories` gives the cleanest, most on-topic dataset. ### 📤 Output One row per paper — **15 fields**, exportable to JSON, CSV, Excel or XML. Here is a trimmed sample record: ```json { "arxivId": "2401.12345", "title": "Scaling Laws for Retrieval-Augmented Language Models", "authors": "Jane Doe, John Smith, Alice Zhang", "abstract": "We study how retrieval augmentation changes the scaling behavior of large language models across compute, data and parameter budgets. We find that…", "categories": "cs.CL, cs.AI, cs.LG", "primaryCategory": "cs.CL", "publishedDate": "2024-01-22T18:00:04Z", "updatedDate": "2024-02-15T09:31:12Z", "pdfUrl": "https://arxiv.org/pdf/2401.12345", "arxivUrl": "https://arxiv.org/abs/2401.12345", "comment": "18 pages, 7 figures. Accepted at ICML 2024. Code: github.com/example/repo", "journalRef": "Proceedings of ICML 2024", "doi": "10.48550/arXiv.2401.12345", "searchQuery": "large language models", "scrapedAt": "2026-07-06T12:00:00.000Z" } ```
📋 Full field reference (click to expand) | Field | Description | |-------|-------------| | `arxivId` | Unique arXiv paper identifier (e.g. `2401.12345`) | | `title` | Paper title | | `authors` | Comma-separated list of all author names | | `abstract` | Full abstract text | | `categories` | All arXiv category codes assigned to the paper | | `primaryCategory` | The paper's primary arXiv category | | `publishedDate` | Original submission date (ISO 8601) | | `updatedDate` | Date of the most recent revision (ISO 8601) | | `pdfUrl` | Direct PDF download link | | `arxivUrl` | Abstract (landing) page URL | | `comment` | Author comments — page count, figures, accepted venue, code links | | `journalRef` | Journal reference, if the paper has been formally published | | `doi` | Digital Object Identifier | | `searchQuery` | The search term that surfaced this paper | | `scrapedAt` | Scrape timestamp (ISO 8601) |
### 💡 Use cases - **Literature reviews** — pull every paper matching your topic and categories, filter by date, and export structured metadata instead of copy-pasting from the website. - **AI & tech intelligence** — track emerging methods and competitor research output by monitoring the categories that matter to your team. - **Talent sourcing** — search by domain, then mine the `authors` field to find researchers publishing in your area — every author is a potential hire. - **NLP / ML dataset building** — assemble titles and abstracts for text classification, topic modeling, retrieval benchmarks or citation-graph construction. - **Trend & recency monitoring** — sort by `submittedDate` with a rolling `dateFrom` to keep only fresh preprints and detect new work as it lands. - **Bibliometrics** — join `doi` / `journalRef` with other sources to enrich a scholarly-publications database. ### 👥 Who uses it Researchers & PhD students running systematic literature reviews · VC and corporate-strategy teams tracking emerging tech · recruiters and engineering leaders sourcing research talent · ML/NLP teams building training and evaluation datasets · data journalists and bibliometrics analysts mapping the research landscape. ### 💰 Pricing This Actor runs on a simple **pay-per-result** model — you pay for the papers you extract, with no separate Apify platform fees to calculate. The arXiv API itself is free. Try it on the **free tier** first, then scale up. See the **Pricing** tab on this page for the current rate. ### ❓ Frequently Asked Questions **Is it legal to scrape arXiv?** This Actor uses arXiv's own **public, officially supported API** and collects only openly available metadata. arXiv explicitly provides this API for programmatic access. You are responsible for respecting arXiv's API terms of use (polite request rates) and using the data lawfully. The Actor includes built-in delays to stay within arXiv's polite-use guidance. **Does arXiv have a public API? Is this an API alternative?** arXiv *does* offer a free public API, and this Actor uses it directly (the Atom/OAI-PMH-compatible endpoint at `export.arxiv.org`). Think of this Actor as a **ready-to-run wrapper** around that API: it handles pagination, multi-query batching, date filtering, retries and clean field mapping, then hands you export-ready JSON/CSV — so you don't have to write and maintain the client yourself. **Do I need an API key or a login?** No. arXiv's API requires **no API key, no authentication and no login** — only an Apify account to run the Actor. **Can I scrape arXiv without an API key or login?** Yes. There is no key or account on the arXiv side. You supply your search queries, the Actor calls the public API over direct HTTP, and returns structured papers. **How much data can I get?** Up to **1,000 papers per query**, with **no limit on the number of queries** — so you can build datasets of tens of thousands of papers in a single run by combining many search terms. arXiv paginates large result sets; the Actor fetches pages sequentially with polite delays. **Can I download the actual PDFs?** The Actor returns the direct `pdfUrl` for every paper. You can download the PDFs separately from those links. Full-text extraction *from* the PDFs is not included in this Actor. **Can I search arXiv by author name?** Yes. Put the author's name in `searchQueries` and the Actor returns every matching paper with its full co-author list, abstract, categories and PDF link. #### How do I build a dataset of recent AI papers from arXiv? Set `searchQueries` to your AI topics, add `cs.AI` / `cs.LG` / `cs.CL` to `categories`, set `sortBy` to `submittedDate`, and use `dateFrom` to keep only recent preprints. Then export the dataset as CSV or JSON. #### How do I export arXiv data to CSV or JSON? Run the Actor, then export the resulting dataset as **CSV, JSON, Excel or XML** from the Apify console, or pull it programmatically via the Apify API. #### What's the difference between `publishedDate` and `updatedDate`? `publishedDate` is the original v1 submission date; `updatedDate` reflects the most recent revision. Sort by `lastUpdatedDate` to catch papers that were recently revised with new results. #### Why is `doi` or `journalRef` sometimes empty? Preprints often haven't been formally published yet, so they carry no DOI or journal reference. These fields populate once (and if) the authors add that metadata on arXiv. ### 🔗 More research & AI-intelligence scrapers by logiover Building a research or competitive-intelligence pipeline? Pair arXiv with the rest of the AI-research suite: | Actor | What it does | |---|---| | [Semantic Scholar Research Scraper](https://apify.com/logiover/semantic-scholar-research-scraper) | Peer-reviewed papers, citations & influence metrics | | [Hugging Face Hub Intelligence](https://apify.com/logiover/huggingface-hub-intelligence-scraper) | Models, datasets & spaces metadata | | [GitHub Repository Scraper](https://apify.com/logiover/github-repository-scraper) | Repo metadata, stars, topics & activity | | [GitHub Activity Stream](https://apify.com/logiover/github-activity-stream) | Real-time commits, releases & events | | [npm Package Intelligence](https://apify.com/logiover/npm-package-intelligence-scraper) | Package metadata, downloads & dependencies | | [Company Deep Research Scraper](https://apify.com/logiover/company-deep-research-scraper) | Full company dossier: tech stack, socials, contacts | | [AI Deep Research](https://apify.com/logiover/ai-deep-research) | Autonomous multi-source research agent | | [AI Web Search](https://apify.com/logiover/ai-web-search) | Structured web search results for agents | | [News Intelligence Scraper](https://apify.com/logiover/news-intelligence-scraper) | Multi-source, deduplicated, sentiment-scored news | | [Discussion Intelligence Scraper](https://apify.com/logiover/discussion-intelligence-scraper) | Reddit + HN + Product Hunt + Stack Exchange opinion | | [CVE Security Advisory Monitor](https://apify.com/logiover/cve-security-advisory-monitor) | Fresh CVEs & security advisories | | [AI Citation Source Finder](https://apify.com/logiover/ai-citation-source-finder) | Find citable sources for AI-generated claims | 👉 Browse all **[logiover scrapers on Apify Store](https://apify.com/logiover)** — 180+ actors across real estate, jobs, crypto, social media & B2B data. ### ⏰ Scheduling & integration Schedule this Actor on Apify to track new preprints in your field daily or weekly. Export results to JSON, CSV or Excel, sync to Google Sheets, or push to your database, BI tools and webhooks through the Apify API. Connect it to **Make, n8n or Zapier** to build automated research-monitoring and alerting pipelines — or wrap it in an **MCP server** so AI agents can pull fresh papers into their context on demand. ### ⭐ Support & feedback Found a bug or need an extra field? Open an issue on the **Issues** tab — response is usually fast. If this Actor saves you time, a **★★★★★ review** on the Store page genuinely helps and is hugely appreciated. 🙏 ### ⚖️ Legal This Actor extracts only publicly available metadata via arXiv's officially supported API, and is intended for legitimate research, analytics and dataset-building use. You are responsible for complying with arXiv's API terms of use, respecting polite request rates, and following any applicable local laws. *** ### 📝 Changelog #### 2026-08-01 - Completed the August 2026 full health check: verified empty/programmatic default, Console UI default, and two source-informed alternative inputs on Apify. - Confirmed successful live execution, non-empty structured output, dataset-field/type integrity, and logical sample quality within the 5-minute quality window. #### 2026-07-06 - ✨ README overhaul: keyword-rich hero, full 15-field output reference with a realistic sample, three ready-to-run example scenarios, high-intent FAQ, and cross-links to the wider AI-research scraper suite. #### 2026-07-01 - Maintenance pass: re-verified end-to-end on live data and confirmed successful runs within the 5-minute quality window on the default input. - Sharpened Store metadata (SEO title & description) and expanded the FAQ with high-intent, long-tail questions for easier discovery in Google and Apify Store search. - Added ready-to-run example tasks that cover common real-world use cases. # Actor input Schema ## `searchQueries` (type: `array`): Optional. Search terms to query on arXiv (author names, paper titles, keywords, topics like 'large language models'). You can also use arXiv field syntax such as 'ti:transformer' or 'au:hinton'. Leave EMPTY to browse the most recent submissions in the chosen category (or cs.AI by default). ## `category` (type: `string`): Optional. Restrict results to a single arXiv subject category. Leave empty to search across all categories. If no search query is given, this category is browsed newest-first. ## `categories` (type: `array`): Optional advanced field. Extra arXiv category codes to include (OR-combined with the category above), e.g. cs.AI, stat.ML. Most users only need the dropdown above. ## `maxResults` (type: `integer`): Maximum total number of papers to return across all search queries. ## `sortBy` (type: `string`): Sort field for results. ## `sortOrder` (type: `string`): Ascending or descending order (ignored when Sort By is Relevance). ## `dateFrom` (type: `string`): Optional. Only include papers submitted on/after this date (YYYY-MM-DD). Leave empty for no date filter. ## `proxyConfiguration` (type: `object`): arXiv is a clean public API. Apify Proxy (Automatic) is used with a direct-connection fallback on the final retry. ## Actor input object example ```json { "searchQueries": [ "large language models" ], "category": "", "categories": [], "maxResults": 200, "sortBy": "submittedDate", "sortOrder": "descending", "dateFrom": "", "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): All records extracted by this run. Open the Dataset tab to browse, filter, and export as CSV, JSON, or Excel. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "searchQueries": [ "large language models" ] }; // Run the Actor and wait for it to finish const run = await client.actor("logiover/arxiv-paper-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "searchQueries": ["large language models"] } # Run the Actor and wait for it to finish run = client.actor("logiover/arxiv-paper-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "searchQueries": [ "large language models" ] }' | apify call logiover/arxiv-paper-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=logiover/arxiv-paper-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/acts/x3lX21u1D5oUpK4tW/builds/psN54CEBLMh6rEUTX/openapi.json