# USA Data.gov Dataset Scraper (`muhammadafzal/usa-data-gov-datasets-scraper`) Actor

Search Data.gov's current v4 Catalog API for U.S. government dataset metadata.

- **URL**: https://apify.com/muhammadafzal/usa-data-gov-datasets-scraper.md
- **Developed by:** [Muhammad Afzal](https://apify.com/muhammadafzal) (community)
- **Categories:** Developer tools, MCP servers, Automation
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 data.gov dataset returneds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## USA Data.gov Dataset Scraper

Search the U.S. Government's Data.gov catalog and return structured metadata for public datasets. Use it for government-data discovery, research pipelines, open-data monitoring, AI-agent retrieval, and finding the official download or API links published by federal, state, local, tribal, and territorial publishers.

### What it returns

Each dataset record contains:

| Field | Description |
| --- | --- |
| `title`, `datasetName`, `datasetId` | Human-readable title plus stable Data.gov CKAN identifiers |
| `description` | Catalog notes; this is metadata, not the underlying data rows |
| `publisher`, `publisherId` | Data.gov organization that publishes the dataset |
| `tags`, `formats` | Search tags and distinct advertised resource formats |
| `resources[]` | Download, API, and landing-page links with format and last-modified metadata |
| `metadataCreated`, `metadataModified` | Catalog metadata timestamps |
| `datasetUrl` | Canonical Data.gov catalog page |
| `sourceQuery`, `sourceApiUrl`, `scrapedAt` | Reproducibility and run provenance |

The actor uses Data.gov's current v4 Catalog API. Data.gov documents that this API returns dataset metadata and supports full-text search, organization/keyword filters, and cursor pagination; it does not serve the underlying data files. Resource links are returned for discovery, but linked files are not downloaded.

### When to use it

Use this actor when you need to search the Data.gov catalog and obtain structured dataset metadata. It is suitable for one-off discovery, batch collection, catalog audits, and AI agents that need a compact list of authoritative government data sources.

Do not use it to download CSV/JSON files, extract rows from a resource, or query a resource's domain-specific API. Take the returned `resources[].url` into a separate data-download or domain-specific actor for those jobs.

### Input

The default input searches for `climate` and returns up to 25 datasets. A useful search looks like this:

```json
{
  "searchQuery": "public health",
  "organization": "cdc",
  "tags": ["api"],
  "maxResults": 50,
  "sortBy": "last_harvested_date",
  "includeResources": true
}
```

`organization` is a Data.gov organization slug. `tags` are sent as exact keyword filters. Use `after` with the `nextAfter` value in `OUTPUT` to continue a large search without repeating the first page. `pageSize` controls API request size; `maxResults` controls the total number of dataset records emitted.

### Output and pagination

The default dataset has one record per matching Data.gov dataset. The `OUTPUT` key-value record contains `recordsReturned`, `catalogTotal` (currently `null` because the v4 API is cursor-based), `nextAfter`, `hasMore`, the applied filters, warnings, and the estimated PPE result cost. If `hasMore` is true, run again with the same filters and `after` set to `nextAfter`.

Example record:

```json
{
  "datasetId": "abc-123",
  "datasetName": "air-quality-data",
  "title": "Air Quality Data",
  "description": "Public air-quality measurements.",
  "datasetUrl": "https://catalog.data.gov/dataset/air-quality-data",
  "publisher": "Environmental Protection Agency",
  "publisherId": "epa",
  "tags": ["air quality", "environment"],
  "formats": ["CSV", "JSON"],
  "resources": [
    {
      "name": "CSV download",
      "format": "CSV",
      "url": "https://example.gov/data.csv",
      "description": null,
      "lastModified": null
    }
  ],
  "metadataCreated": "2024-01-10T15:20:00Z",
  "metadataModified": "2026-03-01T12:00:00Z",
  "accessLevel": "public",
  "license": "CC0 1.0",
  "sourceQuery": "air quality",
  "sourceApiUrl": "https://api.gsa.gov/technology/datagov/v4/search?...",
  "scrapedAt": "2026-08-01T12:00:00.000Z"
}
```

### Pricing

The actor uses predictable pay-per-event pricing:

| Event | Price |
| --- | ---: |
| Actor start | $0.00005 |
| Dataset returned | $0.005 per dataset |

For example, a 25-dataset run costs approximately $0.125 in result events, plus the actor-start event. The actor uses the `DATA_GOV_API_KEY` environment variable when configured; otherwise it uses Data.gov's documented `DEMO_KEY`, which has low rate limits and is intended for exploration. Set a personal Data.gov API key for production or repeated automation. Pay-per-use compute and proxy billing are enabled alongside PPE in Apify.

### Reliability and limits

The actor uses Data.gov's v4 Catalog API with a 30-second request timeout and three attempts with exponential backoff. Empty valid searches finish successfully with a truthful empty `OUTPUT`; API-key, rate-limit, and upstream failures include a diagnostic status message and fail only when no records were collected. The actor caps `maxResults` at 100 per run to keep agent calls predictable.

### Legal and source note

This actor accesses the public Data.gov Catalog API and returns catalog metadata and links. Review the terms, licenses, and access requirements of each linked dataset before downstream use. Data.gov's official documentation states that the Catalog API provides metadata about datasets rather than the actual dataset contents.

# Actor input Schema

## `searchQuery` (type: `string`):

Use this when you want Data.gov to search dataset titles, descriptions, tags, and publishers. Enter a phrase such as 'climate change' or 'public health'; the default is 'climate'. Leave blank to search the full catalog. This is not a direct dataset URL or a request to download the underlying data file.

## `organization` (type: `string`):

Use this when you want to restrict results to one Data.gov publisher organization. Enter its CKAN organization slug, for example 'epa' or 'department-of-energy'; leave blank for all organizations. This is a catalog filter, not a free-text organization name search.

## `tags` (type: `array`):

Use this when results must include one or more Data.gov tags. Enter tag values such as \['api', 'environment']; tags are combined as required filters. Leave empty to keep the search query broad; this is not a list of URLs.

## `maxResults` (type: `integer`):

Use this when you need to cap the number of dataset records returned. Set an integer from 1 to 100; the default is 25. This controls catalog metadata records, not the number of rows inside each linked resource.

## `pageSize` (type: `integer`):

Use this when you want to control Data.gov API pagination. Set an integer from 1 to 100; the default is 50. Smaller pages reduce each request payload, while maxResults still controls the total output.

## `after` (type: `string`):

Use this when continuing a previous Data.gov v4 search from its OUTPUT nextAfter value. Paste the opaque cursor exactly as returned; leave blank for the first page. This is cursor pagination, not a numeric dataset-resource row offset.

## `sortBy` (type: `string`):

Use this when result order matters. Choose relevance, popularity, distance, or last\_harvested\_date; the default is relevance. This changes ordering only and does not filter datasets.

## `includeResources` (type: `boolean`):

Use this when you need the downloadable or API resource links published with each dataset. The default is true and returns up to 20 resources per dataset; set false for a lighter metadata-only response. This does not download resource contents.

## `maxResourcesPerDataset` (type: `integer`):

Use this when a dataset has many distributions and you want to cap links per output record. Set an integer from 0 to 50; the default is 20. This only limits returned metadata links and never fetches the linked files.

## Actor input object example

```json
{
  "searchQuery": "public health",
  "organization": "epa",
  "tags": [
    "api",
    "environment"
  ],
  "maxResults": 25,
  "pageSize": 50,
  "after": "WzY5LjM0NDY5NiwwLCJiYmRhZGNmYi00NDM1LTQzZWUtYjhlMy0yMzZiZjBlZDEwODIiXQ==",
  "sortBy": "last_harvested_date",
  "includeResources": true,
  "maxResourcesPerDataset": 10
}
```

# Actor output Schema

## `status` (type: `string`):

SUCCEEDED when the Data.gov catalog request completed, including valid empty searches; PARTIAL when a PPE cap stopped output after some records.

## `recordsReturned` (type: `string`):

Number of Data.gov dataset metadata records written to the default dataset.

## `catalogTotal` (type: `string`):

The v4 cursor API does not return a total-count field, so this is null; use hasMore and nextAfter for continuation.

## `nextAfter` (type: `string`):

Opaque Data.gov v4 cursor to pass as after in a follow-up run when hasMore is true.

## `warnings` (type: `string`):

Non-fatal diagnostics, including truncation or payment-cap notices.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("muhammadafzal/usa-data-gov-datasets-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("muhammadafzal/usa-data-gov-datasets-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call muhammadafzal/usa-data-gov-datasets-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=muhammadafzal/usa-data-gov-datasets-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zjwxKLvZdy7ePvgW8/builds/2togUUGspgYJXfZbf/openapi.json