Crates.io Scraper — Rust Package Registry Data, Downloads & Metadata

Extract structured data from crates.io, the official Rust package registry, with no API key and no login required.

What does Crates.io Scraper do?

Crates.io Scraper connects directly to the official crates.io public REST API (https://crates.io/api/v1/) to extract package metadata at scale. The actor operates in three modes: list (paginate the full registry ranked by total downloads, recent downloads, newest, or alphabetically), search (full-text query against crate names and descriptions), and crateDetail (fetch precise metadata for a list of known crate names).

Each page request returns up to 100 crates; the actor paginates automatically until your maxResults limit is reached. Because crates.io's API is fully public and keyless, no credentials are needed. A descriptive User-Agent header is sent on every request to comply with crates.io's published API guidelines. Rate limiting is handled gracefully with polite inter-request delays and automatic retry on transient server errors. A single run can collect thousands of crate records in minutes.

Who is it for?

• Rust developers who want to benchmark their crate's download rank against similar packages.
• Open source analysts mapping the Rust ecosystem's growth, popular categories, and dependency trends.
• Security researchers building Software Composition Analysis (SCA) databases of known Rust dependencies.
• Data engineers building internal package registries or governance dashboards that require structured crate metadata.
• Competitive intelligence teams tracking which Rust libraries dominate specific domains (async, web, CLI, crypto, etc.).

Use cases

• Compile a ranked list of the top 1,000 Rust crates by total downloads for ecosystem research.
• Search for all crates matching "tokio" to find async runtime alternatives and related packages.
• Pull version history and repository links for a curated list of internal dependencies for auditing.
• Track weekly download velocity of competitor crates using sort=recent-downloads in a scheduled run.
• Feed crate metadata into an AI agent to generate dependency health reports or README summaries.

Why use Crates.io Scraper?

• Completely keyless — no API key, no account, no OAuth. Just run and get data.
• 13 fields per crate — name, description, downloads, recent downloads, versions, homepage, repository, docs URL, categories, keywords, timestamps.
• Bulk pagination — collect hundreds to thousands of crates in a single run.
• Three modes — list ranked crates, keyword search, or targeted detail fetch by crate name.
• Export to CSV, JSON, Excel — all standard Apify dataset formats out of the box.
• Pay-per-result — the actor's monetization is based on output volume, so you only pay for what you extract.

What data can you extract?

The actor extracts all publicly available metadata from the crates.io API for each crate.

Field	Type	Description
name	string	The crate's unique registry name (e.g. serde)
description	string	Author-provided short description
downloads	number	All-time total download count
recentDownloads	number	Downloads in the last 90 days
maxVersion	string	Highest published semver version
newestVersion	string	Most recently published version
homepage	string	Project homepage URL (may be null)
repository	string	Source code repository URL (GitHub, GitLab, etc.)
documentation	string	Docs.rs or custom documentation URL
categories	string	Comma-separated category slugs (e.g. web-programming, http-client)
keywords	string	Comma-separated author-defined keywords
createdAt	string	ISO 8601 timestamp of first publish
updatedAt	string	ISO 8601 timestamp of most recent update

Example output record

{
  "name": "serde",
  "description": "A generic serialization/deserialization framework",
  "downloads": 980000000,
  "recentDownloads": 42000000,
  "maxVersion": "1.0.197",
  "newestVersion": "1.0.197",
  "homepage": "https://serde.rs",
  "repository": "https://github.com/serde-rs/serde",
  "documentation": "https://docs.rs/serde",
  "categories": "encoding, no-std",
  "keywords": "serde, serialization, no_std",
  "createdAt": "2015-01-28T00:00:00Z",
  "updatedAt": "2024-03-25T15:00:00Z"
}

How to use

Option A — List top crates by downloads

Use mode=list to paginate the entire crates.io registry ranked by your chosen sort.

1. Open the actor's input tab.
2. Set Mode to list.
3. Set Sort to downloads (or recent-downloads, alpha, new-crates, updated).
4. Set Max Results to the number of crates you want (e.g. 1000).
5. Click Start.

{
  "mode": "list",
  "sort": "downloads",
  "maxResults": 1000
}

Option B — Search crates by keyword

Use mode=search to find crates matching a search query, ranked by relevance or downloads.

1. Set Mode to search.
2. Set Query to your search term (e.g. async http client).
3. Set Max Results to cap the number of results.

{
  "mode": "search",
  "query": "async http client",
  "sort": "downloads",
  "maxResults": 200
}

Option C — Fetch specific crates by name

Use mode=crateDetail when you have a known list of crate names and want precise metadata including resolved category and keyword labels.

{
  "mode": "crateDetail",
  "crateNames": ["serde", "tokio", "reqwest", "axum", "clap"]
}

Input parameters

Parameter	Type	Default	Description
mode	string	list	Operation mode: list, search, or crateDetail
sort	string	downloads	Sort order for list/search: downloads, recent-downloads, new-crates, updated, alpha
query	string	—	Search query (required when mode=search)
crateNames	array	—	List of crate names to fetch (required when mode=crateDetail)
maxResults	integer	500	Maximum records to return (1–10000)
proxyConfiguration	object	no proxy	Apify proxy settings (datacenter proxy is sufficient)

Full input JSON

{
  "mode": "list",
  "sort": "downloads",
  "query": "",
  "crateNames": [],
  "maxResults": 500,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Output example

{
  "name": "tokio",
  "description": "An event-driven, non-blocking I/O platform for writing asynchronous I/O backed applications.",
  "downloads": 750000000,
  "recentDownloads": 38000000,
  "maxVersion": "1.37.0",
  "newestVersion": "1.37.0",
  "homepage": null,
  "repository": "https://github.com/tokio-rs/tokio",
  "documentation": "https://docs.rs/tokio",
  "categories": "asynchronous, network-programming",
  "keywords": "io, async, non-blocking, futures",
  "createdAt": "2016-08-04T00:00:00Z",
  "updatedAt": "2024-03-01T12:00:00Z"
}

Tips for best results

• Use sort=recent-downloads when you want the currently trending crates rather than historical cumulative totals.
• Set maxResults to 100–300 for quick test runs before scaling to thousands.
• Use mode=crateDetail for detailed category and keyword arrays on a specific known list — the detail endpoint returns richer taxonomy data.
• Combine search + list: run a search to discover relevant crates, then run crateDetail on the found names for full metadata.
• Schedule weekly runs with sort=recent-downloads to track download velocity trends over time.
• Export to Google Sheets via Apify integration to share data with non-technical teammates.
• Filter by category in post-processing — categories are returned as comma-separated strings and easy to filter in spreadsheets or pandas.
• Watch for null fields — homepage and documentation are often null for newer or unmaintained crates; filter these out if you need complete records only.
• Paginate in chunks — if you want all 100,000+ crates on the registry, set maxResults=10000 and run multiple actors with page offset logic, or use the alpha sort to cover the full namespace.
• Use proxy for high-volume runs — crates.io uses Fastly CDN and rarely blocks, but enabling Apify datacenter proxy improves reliability for very large runs.

Integrations

Google Sheets — Connect the dataset to Google Sheets via the Apify integration to share crate rankings with your team in real time.

Slack — Use Apify webhooks to send a Slack message when your scheduled run completes, with a link to the new dataset.

Zapier / Make — Trigger downstream workflows when new crates matching your search appear, or when a crate's download count crosses a threshold.

Webhooks — Configure an Apify webhook to POST the dataset URL to your internal API on run completion.

Schedule — Use Apify Scheduler to run this actor weekly or daily to maintain a fresh snapshot of crates.io rankings for your dashboard.

API usage

cURL

curl -X POST \
  "https://api.apify.com/v2/acts/logiover~crates-io-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"list","sort":"downloads","maxResults":300}'

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('logiover/crates-io-scraper').call({
  mode: 'list',
  sort: 'downloads',
  maxResults: 500,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient('YOUR_TOKEN')

run = client.actor('logiover/crates-io-scraper').call(run_input={
    'mode': 'list',
    'sort': 'downloads',
    'maxResults': 500,
})

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item['name'], item['downloads'])

Use with AI agents (MCP)

This actor is available as a tool in Apify's Model Context Protocol (MCP) server, letting AI agents call it directly. For example, an agent can be prompted: "Use the crates-io-scraper to find the top 50 Rust HTTP client crates and summarize their download trends." The agent will invoke the actor, wait for results, and synthesize the dataset into a report — no human in the loop required.

FAQ

Does this actor require an API key or login?

No. Crates.io's API is fully public and keyless. The actor sends a descriptive User-Agent header as required by crates.io's usage guidelines, but no authentication is needed.

How many crates does the registry have?

As of 2026, crates.io has over 145,000 published crates. This actor can paginate the full registry by setting maxResults to 10,000 or more (run multiple times with different sort orders to cover the full set).

What is the difference between maxVersion and newestVersion?

maxVersion is the highest semver version (not counting pre-releases), while newestVersion is the most recently published version. For stable crates they are usually identical; for crates with active pre-release cycles (e.g. 1.0.0-beta.1), they may differ.

Why do some fields come back as null?

Fields like homepage and documentation are optional in crates.io's publishing spec. Many crate authors omit them, especially for small utility crates. categories and keywords may also be empty if the author skipped that step.

What does recentDownloads represent?

It is the download count for the last 90 days as reported by crates.io's API. This is a useful proxy for current adoption and trending status.

How do I export the data to CSV or Excel?

In the Apify dataset view, click Export and choose CSV, XLSX, or JSON. You can also use the Apify API to download the dataset programmatically.

How fast is the actor?

The actor fetches 100 crates per API request with a 500 ms delay between requests (to be polite to crates.io). A run of 1,000 crates takes approximately 10 requests × 500 ms = ~5 seconds of wait time plus network latency, typically completing in under 30 seconds.

Is there a rate limit on the crates.io API?

Crates.io does not publish a hard rate limit, but their guidelines ask bots to use a descriptive User-Agent and avoid hammering the API. This actor enforces a 500 ms delay between requests and retries on 429 (Too Many Requests) responses automatically.

How do I track a specific crate's growth over time?

Use mode=crateDetail with the crate name in a scheduled run (e.g. daily). Each run saves the current download count to a dataset, giving you a time series you can chart in Google Sheets or a BI tool.

Is this actor legal to use?

Yes. Crates.io is a public registry operated by the Rust Foundation. All data returned by this actor is publicly accessible via crates.io's documented public API. No authentication bypass, scraping of private data, or violation of crates.io's terms of service is involved.

What Rust categories can I filter by?

Crates.io uses a curated category taxonomy. Common ones include: web-programming, network-programming, asynchronous, cryptography, data-structures, command-line-utilities, database, encoding, algorithms, development-tools. Filter by category string in post-processing after extracting data.

Are download counts real-time?

Download counts are updated by crates.io periodically (typically daily). They reflect cumulative downloads served by the crates.io CDN and may lag by up to 24 hours.

Is it legal?

This actor accesses only publicly available data from crates.io's documented REST API. All requests are made without authentication and do not access any private or user-specific data. The actor complies with crates.io's usage guidelines by identifying itself with a descriptive User-Agent string. Use of public package registry data for research, analytics, and tooling is a standard practice in the software industry.

Always review crates.io's Terms of Service before large-scale data collection and ensure your use case is consistent with their crawler/bot policies.

Related scrapers

• npm Package Scraper — Extract package metadata from the npm registry for JavaScript ecosystem research.
• GitHub Activity Scraper — Scrape GitHub repository stats, commits, issues, and contributor data.
• CVE Advisory Scraper — Pull CVE security advisories from NVD for dependency vulnerability tracking.
• Tech Stack Detector — Identify which technologies (including Rust crates) power any website.

📝 Changelog

2026-08-01

• Completed the August 2026 full health check: verified empty/programmatic default, Console UI default, and two source-informed alternative inputs on Apify.
• Confirmed successful live execution, non-empty structured output, dataset-field/type integrity, and logical sample quality within the 5-minute quality window.