arXiv Papers Scraper · Search, Metadata, Authors & Full Abstracts

An arXiv papers scraper and arXiv paper scraper to search and scrape academic research papers from arXiv. Export matching papers as structured rows: full abstract, complete author list, subject categories, submission and revision dates, DOI, journal reference and direct PDF link. It reads the official arXiv API, so there is no login, no API key, no proxy and no browser — just clean academic paper metadata for literature reviews, RAG pipelines, vector stores, and LLM training datasets.

Maintained by reapX. Every row cites the Apify run that produced it — nothing is inferred, modelled or filled in, and a field absent from the source is absent from the row. The extracted archive for this source is browsable at reapx.dev/data/arxiv-papers-scraper/ and mirrored as an open dataset on Hugging Face and Kaggle. Questions: reapxdev@proton.me

arXiv paper scraper features & metadata

What this arXiv scraper extracts

Every row carries 24 fields, all camelCase, all documented in the output schema:

Field	What it holds
arxivId, arxivBaseId, version	Versioned ID (2504.12342v2), stable join key, revision number
title	Paper title, whitespace normalised
abstract, abstractWordCount	The complete abstract, never truncated
authors, authorCount, firstAuthor	Full author list in arXiv's own order
authorAffiliations	Institutions, where the submitter declared them
primaryCategory, categories, isCrossListed	Where the paper sits in the arXiv taxonomy
publishedAt, publishedDate, updatedAt	First submission and latest revision
doi, doiUrl, journalRef	Publication status once a preprint reaches a journal
comment	The submitter's note - page count, conference acceptance, code links
absUrl, pdfUrl	Landing page and direct PDF
sourceQuery, scrapedAt	The exact arXiv query and collection time, so any result set is reproducible

Five named views ship with the dataset - Overview, Full abstracts, Authors and affiliations, Publication status and Subject categories - so you can export just the columns a given job needs.

How to search arXiv by subject category

Put arXiv subject class codes into arXiv subject categories: cs.AI, cs.LG, cs.CL, cs.CV, cs.CR, cs.DB, cs.SE, cs.RO, stat.ML, math.ST, q-bio.QM, econ.EM, physics.comp-ph, eess.SP, quant-ph - any code in the arXiv taxonomy works.

• Any of them (OR) gives you the union - a broad feed for one field.
• All of them (AND) gives you only papers filed under every code at once - the fastest way to find genuinely cross-disciplinary work, for example cs.CV and cs.CL for vision-language research.

How to get new arXiv papers every day

Set Rolling window: last N days to 1 and schedule the Actor daily. The window is recomputed on every run, so the same task keeps returning only what is new. 7 gives you a weekly reading list. Combine it with Date to filter on → Last updated to catch revised versions rather than first submissions.

How to find papers by author or institution

Authors takes full names - Yann LeCun, Yoshua Bengio - and keeps any paper with at least one of them. Affiliations or institutions matches an institution name anywhere in the arXiv record. arXiv does not index affiliation as its own field, so this behaves as a strong full-text filter across the affiliation tag, comments, title and abstract, not as a registry lookup.

How to separate cross-listed papers from a category's own output

arXiv papers are often filed under several subject classes. Two switches split them:

• Only papers whose PRIMARY category matches - the native output of a subject class, the way its listing page reads.
• Only cross-listed papers - work imported into your categories from somewhere else. Point it at physics.comp-ph or q-bio.QM to see exactly what machine learning those fields are absorbing.

Advanced arXiv API query syntax

Advanced: raw arXiv query string passes your text straight to the arXiv API's search_query, overriding the form fields. Full boolean grammar is available:

cat:cs.LG AND abs:"graph neural network" ANDNOT cat:cs.CV
(cat:stat.ME OR cat:math.ST) AND ti:"causal inference"
au:"Hinton_G" AND submittedDate:[202601010000 TO 202612312359]

Prefixes: ti: title, abs: abstract, au: author, co: comment, jr: journal reference, cat: category, all: everything.

Speed and rate limits

HTTP only, no headless browser. arXiv asks API clients to leave at least three seconds between requests, and this Actor enforces that floor - you cannot configure it lower. Papers arrive 200 per request, so 1,000 papers takes about five requests and under a minute of waiting. Requests that hit HTTP 429 or a 5xx are retried five times with exponential backoff, honouring Retry-After.

Input configuration

Every field is optional. The prefilled values below are what runs if you press Start without changing anything.

What to search for

Combine any of these filters. Everything you fill in is combined with AND, so more filters means fewer, sharper results.

Field	Type	Accepts	What it does
searchQuery	string	free text	Words or a phrase to search for. Multi-word input is matched as an exact phrase. Leave empty to browse a whole subject category by date. Prefilled with "large language models".
searchField	string	one of 6: all, ti, abs, au, ...	Which part of the arXiv record the keyword above is matched against. 'Everything' searches title, abstract, authors, comments and journal reference at once.
categories	list	one value per line	arXiv subject class codes, e.g. cs.AI (artificial intelligence), cs.LG (machine learning), cs.CL (computation and language), cs.CV (computer vision), cs.CR (cryptography and security), cs.DB (databases), cs.SE (software engineering), cs.RO (robotics), stat.ML, math.ST, q-bio.QM, econ.EM, physics.comp-ph, eess.SP, quant-ph, astro-ph.CO. Any code from the arXiv taxonomy works. Leave empty to search all of arXiv. Prefilled with ["cs.AI"].
categoryMatch	string	any, all	'Any' returns a paper filed under at least one of the categories. 'All' returns only papers filed under every category at once - useful for finding genuinely cross-disciplinary work.
authors	list	one value per line	Author names to match. A paper is kept if it has at least one of these authors. Full names work best.
affiliations	list	one value per line	Institution names matched anywhere in the arXiv record (author affiliation tag, comments, title or abstract). A paper is kept if it matches at least one. arXiv does not index affiliation as a separate field, so treat this as a strong text filter rather than a registry lookup.
arxivIds	list	one value per line	Fetch these exact papers instead of searching. Accepts new-style IDs (2504.12342), versioned IDs (1706.03762v7) and old-style IDs (cs/9510101). When set, all search filters above are ignored.

Cross-listing filter

arXiv papers can be filed under several subject classes at once. These two switches let you isolate the native feed of a category or, the opposite, only its imported work.

Field	Type	Accepts	What it does
primaryCategoryOnly	true/false	checkbox	Drop papers that were merely cross-listed into your categories. Gives you the native output of a subject class, the way its listing page reads.
crossListedOnly	true/false	checkbox	Keep only papers whose primary category is something else - work imported into your categories from another field. The fastest way to see what physics or biology is borrowing from machine learning.

Date window

Use the rolling window for a feed that stays fresh on a schedule, or fixed dates for a one-off archive slice. Leave all three empty for no date limit.

Field	Type	Accepts	What it does
dateField	string	submittedDate, lastUpdatedDate	Filter on the day the paper first appeared, or on the day its latest version was posted.
lastDays	integer	1 to 3650	Return only papers from the last N days, recomputed on every run. Use this for scheduled monitoring - 1 for a daily digest, 7 for a weekly review. Overrides the fixed dates below.
dateFrom	string	YYYY-MM-DD	Earliest date to include, inclusive. Ignored when a rolling window is set.
dateTo	string	YYYY-MM-DD	Latest date to include, inclusive. Ignored when a rolling window is set.

Sorting and how many

How results are ordered and where to stop.

Field	Type	Accepts	What it does
sortBy	string	submittedDate, lastUpdatedDate, relevance	Newest first is the right choice for monitoring. Relevance is the right choice for a one-off literature search.
sortOrder	string	descending, ascending	Descending puts the newest (or most relevant) paper first.
maxPapers	integer	1 to 30000	Hard stop on how many papers are returned and charged. Each paper returned costs $0.0025. 50 papers finishes in well under a minute. Prefilled with 50.
dedupeByPaper	true/false	checkbox	Collapse v1, v2, v3 of the same paper into a single row holding the latest version seen. Turn off if you want every revision separately.

Advanced

Only needed for query shapes the fields above cannot express, or to slow the scraper down further.

Field	Type	Accepts	What it does
rawSearchQuery	string	free text	Full arXiv API search_query syntax, passed straight through. Overrides every filter in sections 1 to 3. Example: cat:cs.LG AND abs:"graph neural network" ANDNOT cat:cs.CV
requestDelaySecs	integer	3 to 60	arXiv asks API clients to wait at least 3 seconds between requests. 3 is the minimum this Actor allows; raise it if you are running several jobs at once.

Pricing

Pay per event. The primary event is Paper returned (paper-returned), charged at $0.0025 per event on the free plan, falling to $0.00125 on the highest tier.

Charged once per arXiv paper written to the dataset, with its full abstract, complete author list, subject categories and links. Runs that are blocked, rejected by arXiv, or that match zero papers are never charged.

Other charged events: apify-actor-start (Actor Start).

Minimum charge cap per run: $0.05.

Tiered discounts apply automatically on every paid Apify plan. Platform usage is absorbed by the Actor, so the per-event price is the whole price.

Usage examples

The prefilled run, which is what Start does with nothing changed

{
  "searchQuery": "large language models",
  "searchField": "all",
  "categories": [
    "cs.AI"
  ],
  "categoryMatch": "any",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxPapers": 50,
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

Filter by keyword or phrase

{
  "searchQuery": "retrieval augmented generation",
  "searchField": "all",
  "categories": [
    "cs.AI"
  ],
  "categoryMatch": "any",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxPapers": 50,
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

Filter by search this field

{
  "searchQuery": "large language models",
  "searchField": "abs",
  "categories": [
    "cs.AI"
  ],
  "categoryMatch": "any",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxPapers": 50,
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

Filter by arxiv subject categories

{
  "searchQuery": "large language models",
  "searchField": "all",
  "categories": [
    "cs.LG",
    "stat.ML"
  ],
  "categoryMatch": "any",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxPapers": 50,
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

How to combine categories

{
  "searchQuery": "large language models",
  "searchField": "all",
  "categories": [
    "cs.AI"
  ],
  "categoryMatch": "all",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxPapers": 50,
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

Output example

One row, exactly as the actor wrote it to the dataset:

{
  "arxivId": "2607.28617v1",
  "arxivBaseId": "2607.28617",
  "version": 1,
  "title": "AISPA: User-Centric System Prompt Auditing for Large Language Model Applications",
  "abstract": "System prompts are instructions configured by developers to govern the behaviors of foundation models in AI applications. They are used throughout commercial AI products, but are rarely disclosed to the public or regu...",
  "abstractWordCount": 252,
  "authors": [
    "Xiangning Lin",
    "Shenzhe Zhu",
    "Shu Yang",
    "Zhenyu Zhang",
    "Haoqian Zhang",
    "Yipeng Zhao",
    "Chengxuan Qian",
    "Tianwei Wang",
    "Ziheng Zhang",
    "Zhenlong Yuan",
    "Dingcheng Wang",
    "Juncheng Wu",
    "Yuan Si",
    "Jiaxin Liu",
    "Baolong Bi",
    "Robert Mahari",
    "Tobin South",
    "Dazza Greenwood",
    "Zexue He",
    "Rishi Bommasani",
    "Sophia Kazinnik",
    "Andreas Haupt",
    "Samuele Marro",
    "Erik Brynjolfsson",
    "Alex Pentland",
    "Jiaxin Pei"
  ],
  "authorCount": 26,
  "firstAuthor": "Xiangning Lin",
  "authorAffiliations": [],
  "primaryCategory": "cs.AI",
  "categories": [
    "cs.AI",
    "cs.CL",
    "cs.CY",
    "cs.HC"
  ],
  "isCrossListed": true,
  "publishedAt": "2026-07-30T17:58:58Z",
  "publishedDate": "2026-07-30",
  "updatedAt": "2026-07-30T17:58:58Z",
  "doi": null,
  "doiUrl": null
}

That row carries 24 fields in total; 18 are shown.

Fields on every row

Field	Type	What it is
arxivId	string	The arXiv identifier including the version suffix, exactly as arXiv returns it. Unique per revision.
arxivBaseId	string	The arXiv identifier with the version suffix stripped. Stable across revisions - use this as the join key.
version	integer	Which revision of the paper this row describes. 1 is the original submission.
title	string	Full paper title with arXiv's line breaks and repeated spaces normalised to single spaces.
abstract	string	The complete abstract, never truncated, with line breaks normalised to single spaces.
abstractWordCount	integer	Number of whitespace-separated words in the abstract. Useful for filtering out stub records.
authors	list	Every author name, in the order arXiv lists them.
authorCount	integer	How many authors are on the paper. Large collaborations can run to hundreds.
firstAuthor	string	The first listed author, extracted for easy grouping and sorting.
authorAffiliations	list	Institutions declared in arXiv's affiliation tag. Populated on roughly one paper in forty - arXiv does not require submitters to fill it in, so an empty list means 'not declared', not 'independent'.
primaryCategory	string	The subject class the authors filed the paper under first. This is the category whose listing page the paper natively appears on.
categories	list	Every subject class the paper is filed under, primary first, including cross-lists.
isCrossListed	true/false	True when the paper carries more than one subject class, meaning it was imported into at least one field other than its own.
publishedAt	string	UTC timestamp of the original v1 submission, ISO 8601.
publishedDate	string	Calendar date of the original submission, YYYY-MM-DD, for grouping by day.
updatedAt	string	UTC timestamp of the most recent revision, ISO 8601. Equals publishedAt when the paper was never revised.
doi	string	Publisher DOI when the authors have registered one. Null while a paper is still an unpublished preprint, which is the large majority of recent submissions.
doiUrl	string	Resolvable doi.org URL for the published version. Null when there is no DOI.
journalRef	string	Free-text citation of the peer-reviewed venue, as supplied by the authors. Null for unpublished preprints.
comment	string	The submitter's own note - page count, figure count, conference acceptance, code links. Present on about half of all papers, null otherwise.
absUrl	string	Human-readable arXiv landing page for the paper.
pdfUrl	string	Direct link to the full-text PDF on arXiv.
sourceQuery	string	The exact arXiv API search_query string that produced this row, so a result set can always be reproduced or audited.
scrapedAt	string	UTC timestamp of the moment this row was collected, ISO 8601.

5 named dataset views ship with it: Overview, Full abstracts, Authors and affiliations, Publication status, Subject categories. They drive the Output tab in Console and the Output block on the Actor's .md page.

FAQ

Do I need an arXiv API key or account?

No. The arXiv API is completely open. There is no key, no login, no rate-limit token and no proxy required.

Does it return the full abstract or a truncated one?

The full abstract, every time. abstractWordCount is included so you can spot short or malformed records at a glance.

Can I get papers published in a specific date range?

Yes. Use From date and To date for a fixed archive slice, or Rolling window: last N days for a feed that stays current. Choose whether the window applies to first submission or to the latest revision.

How many papers can one run return?

Up to 30,000 - the arXiv API's own paging ceiling for a single query. For a larger harvest, split the work by subject category or by date window across several runs.

Why is doi empty on most rows?

Because most arXiv papers are preprints that have not been through a journal yet. doi, doiUrl and journalRef fill in only once the authors come back and register the published version. On a sample of 650 recent papers taken while building this Actor, about 2% carried a DOI and 52% carried a comment; on older, settled papers the DOI rate is far higher. Filter on doi != null in the Publication status view to see which preprints made it into a journal.

Why is authorAffiliations usually empty?

arXiv does not require submitters to fill in the affiliation tag, and most do not - expect it on about one paper in forty. An empty list means "not declared", not "independent researcher".

Can I fetch specific papers I already know the ID of?

Yes. Put them in Specific arXiv IDs. New-style (2504.12342), versioned (1706.03762v7) and old-style (cs/9510101) identifiers all work, as do full arxiv.org/abs/... URLs. All other filters are ignored when this is set.

Does it deduplicate versions of the same paper?

By default yes - one row per paper, holding the latest version seen. Turn One row per paper, not per version off to keep every revision as its own row.

Can I chain this into another Actor or an agent?

Yes. The dataset schema documents the title, description, type and an example for every field, so an agent can read the output contract without a sample run. arxivBaseId is the stable join key and sourceQuery records exactly how each row was found.

---

Unofficial - not affiliated with arXiv. Collects public data only. reapx. Contact reapxdev@proton.me.

The full published archive

• arxiv papers scraper archive — every entity this Actor has observed
• All reapx datasets — 20 sources, tens of thousands of pages
• llms.txt · feed.json · feed.xml — the machine-readable index, for agents

Each archive page carries Dataset JSON-LD, a canonical URL and the identifiers used, so an agent can resolve an entity without running anything. Nothing on those pages is estimated or modelled.

🧪 Example input

A real, runnable configuration — this is an actual input this Actor has run with.

{
  "searchQuery": "large language models",
  "categories": [
    "cs.AI"
  ],
  "maxPapers": 50,
  "searchField": "all",
  "categoryMatch": "any",
  "primaryCategoryOnly": false,
  "crossListedOnly": false,
  "dateField": "submittedDate",
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "dedupeByPaper": true,
  "requestDelaySecs": 3
}

📄 Sample output

One real row from a real run of this Actor, unedited.

{
  "arxivId": "2607.29602v1",
  "arxivBaseId": "2607.29602",
  "version": 1,
  "title": "FriendBench: Benchmarking Dyadic Familiarity Inference in Humans and Multimodal Large Language Models",
  "abstract": "Reading a social situation often depends on behavior, not words alone. We introduce FriendBench, a benchmark for inferring whether two people are already familiar or are meeting as strangers, from a 20-second clip of a dyadic ice-breaker conversation. Every pair answers the same type of prompt, so only the manner of interaction can reveal the answer. Across text, audio, and video, we compare 26 models from seven companies against matched human panels over 96 balanced dyads. The best model and the human crowd are statistically indistinguishable on accuracy in every modality, but reach it differently: humans stay balanced across the two answers, while the strongest models lean toward \"stranger\"---a difference in effective prior, not discrimination. Richer channels help both unequally, and only humans gain from visible behavior on top of speech. We release the stimuli, human ratings, and model predictions.",
  "abstractWordCount": 140,
  "authors": [
    "Jeffrey M. Girard",
    "Jason Z. Zheng",
    "Jacqueline R. Vertino",
    "Antony D'Avirro",
    "Benjamin Peloquin"
  ],
  "authorCount": 5,
  "firstAuthor": "Jeffrey M. Girard",
  "authorAffiliations": [],
  "primaryCategory": "cs.CL",
  "categories": [
    "cs.CL",
    "cs.AI",
    "cs.CV",
    "cs.HC"
  ],
  "isCrossListed": true,
  "publishedAt": "2026-07-31T16:33:39Z",
  "publishedDate": "2026-07-31",
  "updatedAt": "2026-07-31T16:33:39Z",
  "doi": null,
  "doiUrl": null,
  "journalRef": null,
  "comment": "15 pages, 3 figures",
  "absUrl": "https://arxiv.org/abs/2607.29602v1",
  "pdfUrl": "https://arxiv.org/pdf/2607.29602v1",
  "sourceQuery": "all:\"large language models\" AND cat:cs.AI",
  "scrapedAt": "2026-08-03T16:29:31Z"
}

How it works

1. You set the filters below, or none at all.
2. The Actor calls the source's own public endpoint and pages through the results.
3. Every row is pushed to the dataset as it is built, not buffered to the end — so a run that hits its time limit still returns everything it collected up to that point.
4. You are charged per row returned, so the maximum-results field is also your cost cap.

💬 Your feedback

Found a bug, or need a field this does not return yet? Open an issue on the Actor's Issues tab, or write to reapxdev@proton.me. Bugs get fixed and reasonable field requests get added.