# Academic Paper Scraper (`constant_quadruped/academic-paper-scraper`) Actor

Search arXiv and PubMed in one request. Returns unified paper data: titles, authors, abstracts, DOIs, and PDF links. Filter by keywords, authors, categories, and date range. Built-in rate limiting and cross-source deduplication. Export to JSON, CSV, or Excel.

- **URL**: https://apify.com/constant\_quadruped/academic-paper-scraper.md
- **Developed by:** [CQ](https://apify.com/constant_quadruped) (community)
- **Categories:** Developer tools, AI
- **Stats:** 12 total users, 3 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Academic Research Paper Scraper

Apify actor that scrapes academic papers from **arXiv** and **PubMed** with a unified output format.

### Features

- **Dual Source Support**: Search both arXiv and PubMed simultaneously
- **Unified Output**: Consistent paper format regardless of source
- **Smart Deduplication**: Remove duplicates by DOI across sources
- **Flexible Filtering**: Filter by title, author, categories, and date range
- **Rate Limit Compliance**: Built-in throttling and automatic retry with exponential backoff
- **PubMed API Key Support**: Optional API key for faster PubMed access
- **Open Access PDF Recovery**: Optional Unpaywall lookup for papers missing a PDF link

### Input Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `searchQuery` | string | Yes | - | Keywords, phrases, or terms to search |
| `sources` | array | No | `["arxiv", "pubmed"]` | Which databases to search |
| `titleFilter` | string | No | - | Filter papers by title keywords |
| `authorFilter` | string | No | - | Filter by author name |
| `categories` | array | No | - | arXiv categories (e.g., `cs.AI`, `physics.quant-ph`) |
| `dateFrom` | string | No | - | Start date (YYYY-MM-DD) |
| `dateTo` | string | No | - | End date (YYYY-MM-DD) |
| `maxResults` | integer | No | 100 | Max papers per source (1-10000) |
| `sortBy` | string | No | `relevance` | Sort order: `relevance`, `date_desc`, `date_asc` |
| `pubmedApiKey` | string | No | - | NCBI API key for faster rate limits |
| `unpaywallEmail` | string | No | - | Your email for Unpaywall API (free, no signup) |
| `includeAbstract` | boolean | No | true | Include full abstract text |
| `deduplicateByDoi` | boolean | No | true | Remove cross-source duplicates |

### Output Format

Each paper is pushed to the dataset as one record in a unified format. Not every
field is present on every record — several are source-specific (see the table).

Example arXiv record:

```json
{
  "id": "arxiv:2401.12345",
  "source": "arxiv",
  "arxivId": "2401.12345",
  "doi": "10.1234/example",
  "title": "Paper Title",
  "abstract": "Full abstract text...",
  "authors": [
    { "name": "John Doe" }
  ],
  "publishedDate": "2024-01-15",
  "updatedDate": "2024-01-20",
  "categories": ["cs.AI", "cs.LG"],
  "journal": "Nature",
  "abstractUrl": "http://arxiv.org/abs/2401.12345",
  "pdfUrl": "http://arxiv.org/pdf/2401.12345",
  "htmlUrl": "http://arxiv.org/abs/2401.12345",
  "comment": "10 pages, 4 figures"
}
```

Example PubMed record:

```json
{
  "id": "pmid:39876543",
  "source": "pubmed",
  "pmid": "39876543",
  "pmcid": "PMC1234567",
  "doi": "10.5678/example",
  "title": "Paper Title",
  "abstract": "Full abstract text...",
  "authors": [
    { "name": "Jane Smith", "affiliation": "University Hospital" }
  ],
  "publishedDate": "2024-02-10",
  "categories": ["Neoplasms", "Gene Expression"],
  "journal": "Journal Name",
  "abstractUrl": "https://pubmed.ncbi.nlm.nih.gov/39876543/",
  "pdfUrl": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/pdf/",
  "htmlUrl": "https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1234567/"
}
```

#### Output Fields

| Field | Type | Populated for | Description |
|-------|------|---------------|-------------|
| `id` | string | both | `arxiv:<arxivId>` or `pmid:<pmid>` |
| `source` | string | both | `"arxiv"` or `"pubmed"` |
| `doi` | string | when available | Digital Object Identifier |
| `arxivId` | string | arXiv only | arXiv identifier (e.g. `2401.12345`) |
| `pmid` | string | PubMed only | PubMed ID |
| `pmcid` | string | PubMed only | PubMed Central ID (when deposited in PMC) |
| `title` | string | both | Paper title |
| `abstract` | string | both | Abstract text; empty string if `includeAbstract=false` or none exists |
| `authors` | array | both | `{ name, affiliation? }`; `affiliation` is PubMed-only |
| `publishedDate` | string | both | Publication date `YYYY-MM-DD` (may be empty if the source omits it) |
| `updatedDate` | string | arXiv only | Last update date `YYYY-MM-DD` |
| `categories` | array | both | arXiv taxonomy codes, or PubMed MeSH terms + author keywords |
| `journal` | string | when available | Journal name / arXiv journal reference |
| `abstractUrl` | string | both | Link to the abstract / landing page |
| `pdfUrl` | string | when available | Direct PDF link (see PDF Link Availability) |
| `htmlUrl` | string | when available | HTML version of the paper |
| `comment` | string | arXiv only | Author comments |

If a run finds no matching papers, a single status record is pushed instead
(`{ "_type": "status", "status": "no_results", ... }`) so the dataset is never
empty. A run summary (counts, per-source errors, timestamp) is also written to
the default key-value store under the `summary` key.

### Example Usage

#### Search for AI papers

```json
{
  "searchQuery": "transformer attention mechanism",
  "sources": ["arxiv", "pubmed"],
  "categories": ["cs.AI", "cs.LG", "cs.CL"],
  "maxResults": 50,
  "sortBy": "date_desc"
}
```

#### Search by author

```json
{
  "searchQuery": "deep learning",
  "authorFilter": "Hinton",
  "dateFrom": "2020-01-01",
  "maxResults": 100
}
```

#### PubMed only with API key

```json
{
  "searchQuery": "CRISPR gene editing",
  "sources": ["pubmed"],
  "pubmedApiKey": "your-ncbi-api-key",
  "maxResults": 500
}
```

### arXiv Categories

Common categories you can filter by:

- **Computer Science**: `cs.AI`, `cs.CL`, `cs.CV`, `cs.LG`, `cs.NE`, `cs.RO`
- **Physics**: `physics.quant-ph`, `physics.comp-ph`
- **Mathematics**: `math.OC`, `math.ST`
- **Statistics**: `stat.ML`, `stat.ME`
- **Quantitative Biology**: `q-bio.BM`, `q-bio.GN`, `q-bio.NC`

Full taxonomy: https://arxiv.org/category\_taxonomy

### PDF Link Availability

**Understanding why some papers lack PDF links:**

| Source | PDF Availability | Notes |
|--------|------------------|-------|
| **arXiv** | ✅ Always available | arXiv is fully open access |
| **PubMed with PMCID** | ✅ Available | Paper deposited in PubMed Central |
| **PubMed without PMCID** | ⚠️ Often unavailable | Paywalled journal articles |

#### Unpaywall Integration

When you provide `unpaywallEmail`, the actor queries [Unpaywall](https://unpaywall.org) to find open access versions of papers that lack PDF links. This can recover PDFs from:

- Institutional repositories
- Author preprint servers
- Publisher open access copies

**Limitations:**

- **New papers (< 2-4 weeks old)**: Unpaywall may not have indexed them yet
- **Paywalled papers with no OA version**: No legal free PDF exists
- **Papers without DOI**: Cannot be looked up in Unpaywall

For recent papers without PDFs, the `abstractUrl` field always provides a link to the paper's landing page.

### Rate Limits

The actor respects API rate limits:

- **arXiv**: 3-second delay between requests
- **PubMed**: 3 requests/second (or 10/second with API key)
- **Unpaywall**: 10 requests/second

Get a free PubMed API key at: https://www.ncbi.nlm.nih.gov/account/

### Limitations

- **Metadata + links only, no full text.** The actor performs static HTTP calls
  to the arXiv and PubMed XML APIs (and optionally the Unpaywall JSON API). It
  parses metadata and link URLs; it does **not** render JavaScript, download PDF
  binaries, or extract full-text body content.
- **`maxResults` is per source.** It caps each of arXiv and PubMed independently
  (1–10000, default 100). With both sources enabled you may receive up to
  \~2× `maxResults` records before deduplication. Actual counts are lower when a
  source has fewer matches, and are bounded by what the upstream APIs return.
- **Upstream rate limits (built-in throttling):** arXiv ~1 request / 3 seconds;
  PubMed ~3 requests/second (10/second with an NCBI `pubmedApiKey`); Unpaywall
  \~10 requests/second. Large `maxResults` values therefore take longer to run —
  arXiv is paged at up to 2000 records/request, PubMed at up to 500/request.
- **PubMed sort:** both `date_desc` and `date_asc` map to PubMed's publication-date
  sort (newest first). True oldest-first ordering is only applied for arXiv.
- **PDF availability is not guaranteed.** arXiv records always include a PDF link.
  PubMed records only include a PDF link when a PMCID exists, or when Unpaywall
  finds an open-access copy. Paywalled articles with no OA version return no
  `pdfUrl` — use `abstractUrl` for the landing page.
- **Unpaywall enrichment is opt-in.** It runs only when you supply a valid
  `unpaywallEmail`, only for papers that are missing a `pdfUrl` and have a DOI.
  Very recent papers (< ~2–4 weeks) may not be indexed yet. No API key is
  required, but an email is mandatory per Unpaywall's terms.
- **DOI-based deduplication only.** `deduplicateByDoi` matches records across
  sources by DOI. Records without a DOI (common for arXiv preprints and some
  PubMed entries) cannot be cross-deduplicated.
- **Author affiliations are PubMed-only.** The arXiv API returns author names
  without affiliations.
- **No results is a success, not a failure.** If nothing matches (or both sources
  are transiently unavailable), the run still completes and pushes a single
  status record rather than failing; per-source errors are recorded in the
  run summary in the key-value store.

### Data Sources

- [arXiv API](https://info.arxiv.org/help/api/index.html) - Open access preprint server
- [PubMed E-utilities](https://www.ncbi.nlm.nih.gov/books/NBK25501/) - Biomedical literature database
- [Unpaywall API](https://unpaywall.org/products/api) - Open-access location lookup by DOI

### License

MIT License

# Actor input Schema

## `sources` (type: `array`):

Which academic databases to search

## `searchQuery` (type: `string`):

Keywords, phrases, or terms to search for in papers

## `titleFilter` (type: `string`):

Optional: Filter papers where title contains these keywords

## `authorFilter` (type: `string`):

Optional: Filter by author name

## `categories` (type: `array`):

arXiv subject categories (e.g., cs.AI, cs.LG, physics.quant-ph). See https://arxiv.org/category\_taxonomy

## `dateFrom` (type: `string`):

Start date for filtering papers (YYYY-MM-DD)

## `dateTo` (type: `string`):

End date for filtering papers (YYYY-MM-DD)

## `maxResults` (type: `integer`):

Maximum number of papers to retrieve per source

## `sortBy` (type: `string`):

How to sort the results

## `pubmedApiKey` (type: `string`):

Optional: NCBI API key to increase rate limit from 3 to 10 requests/second. Get one at https://www.ncbi.nlm.nih.gov/account/

## `unpaywallEmail` (type: `string`):

Your email address for Unpaywall API (free, no signup needed). Enables PDF lookup for paywalled papers via open access sources.

## `includeAbstract` (type: `boolean`):

Include full abstract text in results

## `deduplicateByDoi` (type: `boolean`):

Remove duplicate papers that appear in both sources (matched by DOI)

## Actor input object example

```json
{
  "sources": [
    "arxiv",
    "pubmed"
  ],
  "searchQuery": "machine learning",
  "maxResults": 100,
  "sortBy": "relevance",
  "includeAbstract": true,
  "deduplicateByDoi": true
}
```

# Actor output Schema

## `papers` (type: `string`):

Academic paper metadata from arXiv and PubMed

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "machine learning"
};

// Run the Actor and wait for it to finish
const run = await client.actor("constant_quadruped/academic-paper-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQuery": "machine learning" }

# Run the Actor and wait for it to finish
run = client.actor("constant_quadruped/academic-paper-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "machine learning"
}' |
apify call constant_quadruped/academic-paper-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=constant_quadruped/academic-paper-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3Wrw7nHersFIS9m7A/builds/Rg6gJH7J8dJ14Z6br/openapi.json