# Research Paper Scraper (`moving_beacon-owner1/my-actor-77`) Actor

Gather information such as paper titles, authors, abstracts, categories, PDF links, DOIs, and additional relevant details. This process does not require an API key for access.

- **URL**: https://apify.com/moving\_beacon-owner1/my-actor-77.md
- **Developed by:** [Jamshaid Arif](https://apify.com/moving_beacon-owner1) (community)
- **Categories:** Automation, Developer tools, Integrations
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## 📚 arXiv Research Paper Scraper

**Scrape academic papers from [arXiv.org](https://arxiv.org)** — the world's largest open-access preprint repository with 2M+ papers in Physics, Mathematics, Computer Science, Biology, Finance, and Statistics.

Extract paper titles, authors, abstracts, categories, PDF links, DOIs, and more. No API key needed.

***

### What does this actor do?

This actor searches arXiv.org and returns structured data for academic papers. It uses the official arXiv API, which is free and open — no authentication required.

**Use it to:**

- Monitor new papers in your research field daily
- Build literature review databases for any topic
- Track a specific author's publication history
- Retrieve full metadata for known papers by their arXiv IDs
- Filter papers by date range, category, or keywords in the abstract
- Analyze research trends by category and year distribution

***

### How to use

#### 1. Choose a scrape mode

The actor supports **4 scraping modes**, selected via the `scrape_mode` input:

| Mode | When to use | Required input |
|------|-------------|---------------|
| **Keyword Search** | Find papers on a topic | `search_query` |
| **Author Search** | Find all papers by a researcher | `author_name` |
| **Category Browse** | Browse latest papers in a field | `category` |
| **ID Lookup** | Fetch specific known papers | `paper_ids` |

#### 2. Configure your search

Fill in the input fields in Apify Console. Only the fields relevant to your chosen mode are required — everything else is optional.

#### 3. Run and export

Click **Start** and wait for the results. Export your dataset as JSON, CSV, Excel, or connect it to Google Sheets, Slack, or any other integration.

***

### Input parameters

#### Core settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `scrape_mode` | Select | `keyword_search` | Which search mode to use |
| `search_query` | Text | `machine learning` | Keywords to search for (Keyword Search mode) |
| `search_field` | Select | `all` | Which field to search: `all`, `ti` (title), `abs` (abstract), `au` (author) |
| `author_name` | Text | — | Researcher name (Author Search mode). Example: `Yoshua Bengio` |
| `category` | Text | — | arXiv category code (Category Browse mode, or as a filter in other modes) |
| `paper_ids` | Text | — | Comma-separated arXiv IDs (ID Lookup mode). Example: `1706.03762,2005.14165` |

#### Filters

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `date_from` | Text | — | Only papers from this date onward (`YYYY-MM-DD`) |
| `date_to` | Text | — | Only papers up to this date (`YYYY-MM-DD`) |
| `abstract_contains` | Text | — | Abstract must contain ALL of these comma-separated keywords |
| `exclude_keywords` | Text | — | Exclude papers containing ANY of these keywords in title or abstract |
| `min_authors` | Number | `0` | Minimum number of authors (0 = no filter) |

#### Output settings

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `max_results` | Number | `50` | Number of papers to scrape (1–2000) |
| `sort_by` | Select | `relevance` | Sort order: `relevance`, `submitted_date`, or `last_updated` |
| `include_abstract` | Boolean | `true` | Include full abstract text |
| `include_pdf_links` | Boolean | `true` | Include direct PDF download URLs |

***

### Input examples

#### Example 1: Find recent NLP papers about transformers

```json
{
    "scrape_mode": "keyword_search",
    "search_query": "transformer attention mechanism",
    "search_field": "all",
    "category": "cs.CL",
    "max_results": 50,
    "sort_by": "submitted_date",
    "date_from": "2025-01-01"
}
```

#### Example 2: Get all papers by a specific author

```json
{
    "scrape_mode": "author_search",
    "author_name": "Geoffrey Hinton",
    "max_results": 100,
    "sort_by": "submitted_date",
    "include_abstract": true
}
```

#### Example 3: Browse latest Machine Learning papers

```json
{
    "scrape_mode": "category_browse",
    "category": "cs.LG",
    "max_results": 30,
    "sort_by": "submitted_date"
}
```

#### Example 4: Fetch 5 famous AI papers by ID

```json
{
    "scrape_mode": "id_lookup",
    "paper_ids": "1706.03762, 2005.14165, 1512.03385, 1406.2661, 2301.07041"
}
```

#### Example 5: Advanced — diffusion models in ML, 2024+, excluding "survey"

```json
{
    "scrape_mode": "keyword_search",
    "search_query": "diffusion model",
    "search_field": "ti",
    "category": "cs.LG",
    "max_results": 40,
    "sort_by": "submitted_date",
    "date_from": "2024-01-01",
    "abstract_contains": "generative",
    "exclude_keywords": "survey, review"
}
```

***

### Output format

The actor produces two types of records in the dataset:

#### Summary record (first item)

The first record is a summary of the entire scrape run:

```json
{
    "type": "summary",
    "total_papers": 50,
    "unique_authors": 187,
    "top_categories": [
        { "category": "cs.CL", "count": 32 },
        { "category": "cs.LG", "count": 18 },
        { "category": "cs.AI", "count": 9 }
    ],
    "year_distribution": {
        "2024": 12,
        "2025": 38
    },
    "query_used": "all:transformer AND all:attention AND cat:cs.CL",
    "scrape_mode": "keyword_search"
}
```

#### Paper records

Each paper is a flat JSON object:

```json
{
    "rank": 1,
    "arxiv_id": "2301.07041v1",
    "title": "LLaMA: Open and Efficient Foundation Language Models",
    "authors": ["Hugo Touvron", "Thibaut Lavril", "Gautier Izacard", "..."],
    "authors_short": "Hugo Touvron, Thibaut Lavril, Gautier Izacard et al.",
    "num_authors": 14,
    "published": "2023-02-27",
    "updated": "2023-02-27",
    "primary_category": "cs.CL",
    "categories": ["cs.CL"],
    "category_names": ["Computer Science"],
    "abstract": "We introduce LLaMA, a collection of foundation language models...",
    "abstract_length": 1247,
    "arxiv_url": "http://arxiv.org/abs/2301.07041v1",
    "pdf_url": "http://arxiv.org/pdf/2301.07041v1",
    "source_url": "https://arxiv.org/e-print/2301.07041v1",
    "doi": "",
    "journal_ref": "",
    "comment": "Submitted to NeurIPS 2023",
    "links": [
        { "title": "pdf", "href": "http://arxiv.org/pdf/2301.07041v1", "type": "application/pdf" }
    ],
    "scraped_at": "2026-04-04T12:00:00+00:00"
}
```

***

### arXiv categories quick reference

Use these codes in the `category` field:

#### Computer Science

| Code | Field |
|------|-------|
| `cs.AI` | Artificial Intelligence |
| `cs.CL` | Computation & Language (NLP) |
| `cs.CV` | Computer Vision |
| `cs.LG` | Machine Learning |
| `cs.NE` | Neural & Evolutionary Computing |
| `cs.RO` | Robotics |
| `cs.SE` | Software Engineering |
| `cs.CR` | Cryptography & Security |
| `cs.DB` | Databases |
| `cs.IR` | Information Retrieval |

#### Other popular categories

| Code | Field |
|------|-------|
| `stat.ML` | Statistics — Machine Learning |
| `math.OC` | Mathematics — Optimization & Control |
| `quant-ph` | Quantum Physics |
| `econ.GN` | Economics — General |
| `q-bio.NC` | Quantitative Biology — Neurons & Cognition |
| `q-fin.ST` | Quantitative Finance — Statistical Finance |

The full category list is at [arxiv.org/category\_taxonomy](https://arxiv.org/category_taxonomy).

***

### Advanced query syntax

In `search_query`, you can use arXiv's native query syntax for precise searches:

| Syntax | Meaning | Example |
|--------|---------|---------|
| `ti:keyword` | Search in title only | `ti:deep learning` |
| `abs:keyword` | Search in abstract only | `abs:reinforcement` |
| `au:name` | Search by author | `au:bengio` |
| `cat:code` | Search by category | `cat:cs.AI` |
| `AND` | Both terms must match | `ti:neural AND ti:network` |
| `OR` | Either term matches | `cat:cs.CL OR cat:cs.LG` |
| `ANDNOT` | Exclude a term | `ti:transformer ANDNOT ti:survey` |
| `"phrase"` | Exact phrase match | `abs:"language model"` |

**Example:** Find papers by Bengio about attention in NLP:

```
au:bengio AND abs:attention AND cat:cs.CL
```

***

### How much does it cost to run?

This actor is very lightweight because arXiv's API is free and doesn't require browser rendering.

| Papers | Estimated time | Apify platform credits |
|--------|---------------|----------------------|
| 50 | ~30 seconds | < $0.01 |
| 200 | ~2 minutes | ~$0.01 |
| 500 | ~5 minutes | ~$0.02 |
| 2000 | ~20 minutes | ~$0.05 |

The arXiv API requests a 3-second delay between requests, which the actor respects automatically.

***

### Integrations and scheduling

#### Schedule daily paper monitoring

Set up a scheduled run in Apify Console to scrape new papers in your field every day. Combine with the Google Sheets or Slack integration to get automatic notifications.

**Example: Daily cs.AI papers to Slack**

1. Create a scheduled task with `category_browse` mode and `cs.AI` category
2. Sort by `submitted_date`, limit to `max_results: 20`
3. Connect the Slack integration to post results to your `#research` channel

#### Export to Google Sheets

After each run, the dataset can be automatically synced to a Google Sheet for easy sharing with your research team.

***

### Tips and best practices

- **Start with a small `max_results`** (10–20) to preview your query, then scale up.
- **Use `category` as a filter** in Keyword Search mode to narrow results to your field.
- **Combine `abstract_contains` with broad searches** to find niche papers — for example, search `cat:cs.LG` but require "graph neural" in the abstract.
- **Use `exclude_keywords`** to filter out survey/review papers if you only want original research.
- **arXiv limits results to 2000** per query — for larger datasets, run multiple searches with different date ranges.
- **Sort by `submitted_date`** for monitoring new papers; sort by `relevance` for topic exploration.

***

### Changelog

#### v1.0.0 (2026-04-04)

- Initial release
- 4 scraping modes: keyword search, author search, category browse, ID lookup
- Advanced filtering: date range, abstract keywords, exclude keywords, min authors
- Summary statistics with top categories and year distribution
- Batched dataset output for large result sets

# Actor input Schema

## `scrape_mode` (type: `string`):

How to search arXiv.

## `search_query` (type: `string`):

Keywords to search for (used in 'Keyword Search' mode). Supports arXiv query syntax: 'ti:deep learning', 'abs:transformer', 'all:reinforcement learning'. Plain text also works.

## `search_field` (type: `string`):

Which field to search in (used in 'Keyword Search' mode).

## `author_name` (type: `string`):

Author name to search for (used in 'Author Search' mode). Example: 'Yoshua Bengio'.

## `category` (type: `string`):

arXiv category to browse (used in all modes as a filter). Examples: cs.AI, cs.LG, cs.CL, stat.ML, math.CO, physics, quant-ph, econ, q-bio.

## `date_from` (type: `string`):

Only include papers submitted on or after this date (YYYY-MM-DD). Leave empty for no lower bound.

## `date_to` (type: `string`):

Only include papers submitted on or before this date (YYYY-MM-DD). Leave empty for no upper bound.

## `paper_ids` (type: `string`):

Comma-separated arXiv paper IDs to look up (used in 'ID Lookup' mode). Example: '2301.07041,1706.03762,2005.14165'.

## `max_results` (type: `integer`):

Maximum number of papers to scrape (1–2000). arXiv API limit is 2000 per query.

## `sort_by` (type: `string`):

How to sort the results.

## `include_abstract` (type: `boolean`):

Include the full abstract text in results.

## `include_pdf_links` (type: `boolean`):

Include direct PDF download URLs.

## `abstract_contains` (type: `string`):

Optional: only include papers whose abstract contains ALL of these comma-separated keywords. Example: 'transformer,attention'.

## `exclude_keywords` (type: `string`):

Optional: exclude papers whose title or abstract contains ANY of these comma-separated keywords.

## `min_authors` (type: `integer`):

Minimum number of authors a paper must have (0 = no filter).

## `proxy_configuration` (type: `object`):

Proxy configuration (rarely needed — arXiv API is open).

## Actor input object example

```json
{
  "scrape_mode": "keyword_search",
  "search_query": "machine learning",
  "search_field": "all",
  "author_name": "",
  "category": "",
  "date_from": "",
  "date_to": "",
  "paper_ids": "",
  "max_results": 50,
  "sort_by": "relevance",
  "include_abstract": true,
  "include_pdf_links": true,
  "abstract_contains": "",
  "exclude_keywords": "",
  "min_authors": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "scrape_mode": "keyword_search",
    "search_query": "machine learning",
    "max_results": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("moving_beacon-owner1/my-actor-77").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "scrape_mode": "keyword_search",
    "search_query": "machine learning",
    "max_results": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("moving_beacon-owner1/my-actor-77").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "scrape_mode": "keyword_search",
  "search_query": "machine learning",
  "max_results": 50
}' |
apify call moving_beacon-owner1/my-actor-77 --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=moving_beacon-owner1/my-actor-77",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/acts/sh4fgfSSu64HeLfRI/builds/cUltQb7jGQBKAlV9y/openapi.json