# arXiv Paper Search Scraper (`fetch_cat/arxiv-paper-search-scraper`) Actor

Search arXiv papers by keyword, author, category, and date using public paper metadata.

- **URL**: https://apify.com/fetch\_cat/arxiv-paper-search-scraper.md
- **Developed by:** [Hanna Nosova](https://apify.com/fetch_cat) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 93.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 paper results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## arXiv Paper Search Scraper

Search public arXiv paper metadata by keyword, author, category, sort order, and publication date, then export titles, abstracts, authors, IDs, DOI, journal references, abstract links, and PDF links.

arXiv Paper Search Scraper is a lightweight research-data actor for literature review seed lists, paper monitoring, AI research feeds, newsletter curation, and dataset pipelines.

### At a glance

- **Input:** required `query`, optional `categories`, `author`, date filters, sorting, and `maxItems`.
- **Output:** paper title, abstract, authors, arXiv ID, categories, published/updated dates, DOI, journal reference, abstract URL, and PDF URL.
- **Controls:** filter by arXiv categories, author name, publication date range, sort field, and sort order.
- **Exports:** download JSON, CSV, Excel, XML, RSS, or HTML, or read results through API/MCP.
- **Best for:** literature search, recent-paper monitoring, research newsletters, lab alerts, and paper metadata enrichment.

### Who is it for?

This actor is for researchers, data analysts, AI teams, newsletter editors, product teams, and academic-monitoring workflows that need structured arXiv paper metadata without manually copying search results. It is especially useful when you need repeatable paper discovery, scheduled topic monitoring, or clean exports for spreadsheets, databases, dashboards, or AI-agent research summaries.

### Legality

This actor collects publicly available arXiv metadata returned by arXiv's public search endpoints. You are responsible for using the results lawfully, respecting arXiv policies and applicable copyright/database rules, and avoiding inputs that collect or process personal or sensitive data unnecessarily. The actor does not bypass logins, access private accounts, or download paywalled content.

### Input recipes

Use these concrete input patterns when starting a new run:

- **Recent LLM Papers:** `query: "large language models"`, `categories: ["cs.CL", "cs.AI"]`, `sortBy: "submittedDate"`, `sortOrder: "descending"`, `maxItems: 25`.
- **Graph Neural Network Review:** `query: "graph neural networks"`, `categories: ["cs.LG"]`, `sortBy: "relevance"`, `maxItems: 30`.
- **Author Watch:** `query: "representation learning"`, `author: "Yoshua Bengio"`, `maxItems: 20`.
- **Quantum Error Correction Scan:** `query: "quantum error correction"`, `categories: ["quant-ph"]`, `sortBy: "submittedDate"`, `maxItems: 25`.
- **Date-Bounded AI Safety Search:** `query: "AI safety"`, `categories: ["cs.AI"]`, `startDate: "2025-01-01"`, `endDate: "2025-12-31"`, `maxItems: 25`.

### What can it do?

- **Search arXiv metadata:** query public arXiv records by keyword, phrase, category, author, and publication date.
- **Export paper discovery rows:** save titles, abstracts, authors, arXiv IDs, categories, dates, DOI, journal references, abstract links, and PDF links.
- **Monitor recent research:** sort by submitted or updated date and schedule the same query for recurring paper feeds.
- **Build literature-review seed lists:** collect focused records for research notes, lab alerts, newsletters, or dataset enrichment.
- **Use as an arXiv data workflow:** export CSV/Excel/JSON, run through the Apify API, or expose the Actor to AI agents through Apify MCP.

### Input example

```json
{
  "query": "large language models",
  "categories": ["cs.CL", "cs.AI"],
  "sortBy": "submittedDate",
  "sortOrder": "descending",
  "maxItems": 25
}
```

### Output example

```json
{
  "atomId": "http://arxiv.org/abs/2401.00001v1",
  "arxivId": "2401.00001v1",
  "paperTitle": "Example paper title",
  "summary": "Paper abstract text...",
  "authors": ["Jane Doe", "John Smith"],
  "primaryCategory": "cs.CL",
  "categories": ["cs.CL", "cs.AI"],
  "publishedAt": "2024-01-01T00:00:00Z",
  "updatedAt": "2024-01-02T00:00:00Z",
  "comment": "12 pages",
  "journalRef": null,
  "doi": null,
  "absUrl": "http://arxiv.org/abs/2401.00001v1",
  "pdfUrl": "http://arxiv.org/pdf/2401.00001v1",
  "query": "large language models",
  "position": 1
}
```

### Input configuration

| Setting | JSON key | Description |
| --- | --- | --- |
| Search query | `query` | Required keyword or phrase searched across arXiv metadata. |
| arXiv categories | `categories` | Optional category codes such as `cs.AI`, `cs.LG`, `cs.CL`, `stat.ML`, or `quant-ph`. |
| Author name | `author` | Optional author filter, such as `Yoshua Bengio` or `Fei-Fei Li`. |
| Maximum papers | `maxItems` | Maximum number of paper rows to save, from 1 to 5,000. |
| Resume from result index | `startIndex` | Optional zero-based offset for continuing a time-bounded run. Copy `nextStart` from the `PENDING_WORK` record. |
| Sort by | `sortBy` | `relevance`, `lastUpdatedDate`, or `submittedDate`. |
| Sort order | `sortOrder` | `descending` or `ascending`. |
| Published after | `startDate` | Optional publication-date lower bound in `YYYY-MM-DD` format. |
| Published before | `endDate` | Optional publication-date upper bound in `YYYY-MM-DD` format. |

### Output fields

| Field | Description |
| --- | --- |
| `atomId` | Canonical arXiv Atom ID or abstract URL. |
| `arxivId` | Clean arXiv identifier. |
| `paperTitle` | Paper title. |
| `summary` | Abstract text. |
| `authors` | List of author names. |
| `primaryCategory` | Main arXiv category when available. |
| `categories` | All category tags returned for the paper. |
| `publishedAt` | Initial publication timestamp. |
| `updatedAt` | Latest update timestamp. |
| `comment` | Optional author comment. |
| `journalRef` | Optional journal reference. |
| `doi` | Optional DOI. |
| `absUrl` | arXiv abstract page URL. |
| `pdfUrl` | arXiv PDF URL when available. |
| `query` | Search query used for the run. |
| `position` | Result position after filtering. |

### Pricing

This Actor uses Apify pay-per-event pricing. The prices below come from the current Actor pricing configuration. Apify public plans map to Store discount tiers, so the table shows both the user-facing plan context and the pricing tier name. The final price shown in Apify depends on the user account plan and any custom agreement.

| Event | What is charged | Price |
| --- | --- | ---: |
| `start` | One-time fee charged when a run starts. Covers fixed startup cost (init, proxy warmup, first HTTP setup). | $0.005 |

| Event | What is charged | Free / no discount | Starter / Bronze | Scale / Silver | Business / Gold | Custom / Platinum | Custom / Diamond |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `result` | Per arXiv paper saved to the dataset | $0.575 / 1,000 | $0.5 / 1,000 | $0.39 / 1,000 | $0.3 / 1,000 | $0.2 / 1,000 | $0.14 / 1,000 |

Apify may also charge platform usage for compute, storage, proxies, or data transfer outside this Actor pricing. Check the Actor run and the Apify Pricing tab for the exact cost shown to your account.

### Common workflows

- **Literature review seed list:** search a focused topic and export titles, abstracts, authors, and PDF links to a spreadsheet.
- **Recent-paper monitoring:** sort by `submittedDate` descending and schedule the same query weekly.
- **Author tracking:** combine `query` and `author` to follow a researcher or lab topic.
- **Category feeds:** use arXiv category codes to keep AI, NLP, computer vision, statistics, or quantum searches focused.
- **Research enrichment:** send abstracts, authors, categories, and URLs to a database, vector index, or newsletter workflow.

### Tips for best results

- **Use precise terms:** `retrieval augmented generation` is easier to review than `AI`.
- **Add categories:** category filters reduce off-topic matches for broad topics.
- **Use date filters carefully:** `startDate` and `endDate` are applied to publication dates after arXiv returns results.
- **Respect arXiv pacing:** larger runs take longer because the actor paginates through public arXiv API results.
- **Check optional fields:** DOI, journal reference, and comments are present only when authors provide them.

### Limits and caveats

- **Public metadata only:** the actor exports arXiv metadata, not paywalled paper content.
- **No citation graph:** it does not collect citation counts or references.
- **Date filtering is post-fetch:** very broad queries with narrow dates may scan more rows than they save.
- **Normal workload design:** keep queries focused for repeatable monitoring and predictable run times.

### API usage

**Node.js example:**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/arxiv-paper-search-scraper').call({
  query: 'large language models',
  categories: ['cs.CL', 'cs.AI'],
  sortBy: 'submittedDate',
  sortOrder: 'descending',
  maxItems: 25
});
console.log(run.defaultDatasetId);
```

**Python example:**

```python
from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')
run = client.actor('fetch_cat/arxiv-paper-search-scraper').call(run_input={
    'query': 'large language models',
    'categories': ['cs.CL', 'cs.AI'],
    'sortBy': 'submittedDate',
    'sortOrder': 'descending',
    'maxItems': 25,
})
print(run['defaultDatasetId'])
```

**cURL example:**

```bash
curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~arxiv-paper-search-scraper/runs?token=YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"query":"large language models","categories":["cs.CL","cs.AI"],"sortBy":"submittedDate","sortOrder":"descending","maxItems":25}'
```

### MCP and AI agents

Use the official Apify MCP server for agent workflows. The default server can search and run Actors; the focused URL exposes only this Actor.

**Claude CLI add command:**

```bash
claude mcp add apify-arxiv --transport http "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"
```

**JSON config block:**

```json
{
  "mcpServers": {
    "apify-arxiv": {
      "url": "https://mcp.apify.com?tools=fetch_cat/arxiv-paper-search-scraper"
    }
  }
}
```

Example prompts for MCP usage:

- "Use arXiv Paper Search Scraper to find 25 recent large language model papers in cs.CL and cs.AI, then summarize titles, authors, categories, and PDF links."
- "Run an arXiv search for graph neural networks in cs.LG and return a CSV-ready table with arXiv IDs, authors, abstracts, and PDF URLs."
- "Monitor new AI safety papers in cs.AI from 2025 and list DOI, journal reference, and PDF link when available."

### FAQ

#### Does it require an arXiv account or API key?

No. It uses public arXiv metadata.

#### Can I export papers to CSV or Excel?

Yes. Apify datasets export to CSV, Excel, JSON, JSONL, XML, RSS, and HTML.

#### Can I search by author?

Yes. Use `author` along with a required `query`. The actor builds an arXiv search query with both filters.

#### Why are DOI or journal fields empty?

Not every arXiv paper has DOI or journal reference metadata. Missing optional values are saved as `null`.

#### Is this an arXiv API alternative?

It is a packaged Apify workflow around public arXiv metadata search, not an official arXiv API.

### Related actors

- [Google Scholar Profiles Scraper](https://apify.com/fetch_cat/google-scholar-profiles-scraper)
- [GitHub Repositories Search Scraper](https://apify.com/fetch_cat/github-repositories-search-scraper)
- [Hacker News Search Scraper](https://apify.com/fetch_cat/hacker-news-search-scraper)

### Support

If a run fails, returns no data, or a field looks wrong, open an issue from the Actor page.

Please include the Apify run ID or run URL, input JSON, one example public URL, query, or input item, what you expected, and what the dataset returned. Small reproducible inputs make parsing or site-layout issues much faster to fix.

### Privacy and data handling

This Actor runs with Apify limited permissions and only processes data needed for the documented run. It uses the inputs you provide and the public records needed to produce the documented dataset to produce the output dataset and sends requests to arXiv public search/API endpoints; results are stored in Apify run storage for your account. FetchCat does not use your inputs or outputs for advertising, does not use them for model training, and does not retain them outside the Apify run except for transient support debugging when you explicitly share run details. You are responsible for using the Actor lawfully, respecting the target site's terms, and avoiding unnecessary personal or sensitive data in inputs.

# Actor input Schema

## `query` (type: `string`):

Keyword or phrase to search across arXiv paper titles, abstracts, authors, comments, journal references, and categories.

## `categories` (type: `array`):

Optional arXiv category codes to include, such as cs.AI, cs.LG, stat.ML, math.OC, or quant-ph.

## `author` (type: `string`):

Optional author filter, for example Yoshua Bengio or Fei-Fei Li.

## `maxItems` (type: `integer`):

Maximum number of matching papers to save to the dataset.

## `startIndex` (type: `integer`):

Optional zero-based arXiv result offset. Use the nextStart value from PENDING\_WORK to continue a time-bounded run without repeating earlier pages.

## `sortBy` (type: `string`):

Sort order supported by arXiv search.

## `sortOrder` (type: `string`):

Choose descending for newest or most relevant first, or ascending for oldest first.

## `startDate` (type: `string`):

Optional publication date lower bound in YYYY-MM-DD format. Applied to the paper publication date.

## `endDate` (type: `string`):

Optional publication date upper bound in YYYY-MM-DD format. Applied to the paper publication date.

## Actor input object example

```json
{
  "query": "machine learning",
  "categories": [
    "cs.AI"
  ],
  "maxItems": 20,
  "startIndex": 0,
  "sortBy": "relevance",
  "sortOrder": "descending"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

## `pendingWork` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "machine learning",
    "categories": [
        "cs.AI"
    ],
    "maxItems": 20,
    "sortBy": "relevance",
    "sortOrder": "descending"
};

// Run the Actor and wait for it to finish
const run = await client.actor("fetch_cat/arxiv-paper-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "machine learning",
    "categories": ["cs.AI"],
    "maxItems": 20,
    "sortBy": "relevance",
    "sortOrder": "descending",
}

# Run the Actor and wait for it to finish
run = client.actor("fetch_cat/arxiv-paper-search-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "machine learning",
  "categories": [
    "cs.AI"
  ],
  "maxItems": 20,
  "sortBy": "relevance",
  "sortOrder": "descending"
}' |
apify call fetch_cat/arxiv-paper-search-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=fetch_cat/arxiv-paper-search-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/Ah1GpLM1iSFIeIIjY/builds/DtqiVuOqgC6L1STtA/openapi.json