USA Data.gov Dataset Scraper

Search the U.S. Government's Data.gov catalog and return structured metadata for public datasets. Use it for government-data discovery, research pipelines, open-data monitoring, AI-agent retrieval, and finding the official download or API links published by federal, state, local, tribal, and territorial publishers.

What it returns

Each dataset record contains:

Field	Description
title, datasetName, datasetId	Human-readable title plus stable Data.gov CKAN identifiers
description	Catalog notes; this is metadata, not the underlying data rows
publisher, publisherId	Data.gov organization that publishes the dataset
tags, formats	Search tags and distinct advertised resource formats
resources[]	Download, API, and landing-page links with format and last-modified metadata
metadataCreated, metadataModified	Catalog metadata timestamps
datasetUrl	Canonical Data.gov catalog page
sourceQuery, sourceApiUrl, scrapedAt	Reproducibility and run provenance

The actor uses Data.gov's current v4 Catalog API. Data.gov documents that this API returns dataset metadata and supports full-text search, organization/keyword filters, and cursor pagination; it does not serve the underlying data files. Resource links are returned for discovery, but linked files are not downloaded.

When to use it

Use this actor when you need to search the Data.gov catalog and obtain structured dataset metadata. It is suitable for one-off discovery, batch collection, catalog audits, and AI agents that need a compact list of authoritative government data sources.

Do not use it to download CSV/JSON files, extract rows from a resource, or query a resource's domain-specific API. Take the returned resources[].url into a separate data-download or domain-specific actor for those jobs.

Input

The default input searches for climate and returns up to 25 datasets. A useful search looks like this:

{
  "searchQuery": "public health",
  "organization": "cdc",
  "tags": ["api"],
  "maxResults": 50,
  "sortBy": "last_harvested_date",
  "includeResources": true
}

organization is a Data.gov organization slug. tags are sent as exact keyword filters. Use after with the nextAfter value in OUTPUT to continue a large search without repeating the first page. pageSize controls API request size; maxResults controls the total number of dataset records emitted.

Output and pagination

The default dataset has one record per matching Data.gov dataset. The OUTPUT key-value record contains recordsReturned, catalogTotal (currently null because the v4 API is cursor-based), nextAfter, hasMore, the applied filters, warnings, and the estimated PPE result cost. If hasMore is true, run again with the same filters and after set to nextAfter.

Example record:

{
  "datasetId": "abc-123",
  "datasetName": "air-quality-data",
  "title": "Air Quality Data",
  "description": "Public air-quality measurements.",
  "datasetUrl": "https://catalog.data.gov/dataset/air-quality-data",
  "publisher": "Environmental Protection Agency",
  "publisherId": "epa",
  "tags": ["air quality", "environment"],
  "formats": ["CSV", "JSON"],
  "resources": [
    {
      "name": "CSV download",
      "format": "CSV",
      "url": "https://example.gov/data.csv",
      "description": null,
      "lastModified": null
    }
  ],
  "metadataCreated": "2024-01-10T15:20:00Z",
  "metadataModified": "2026-03-01T12:00:00Z",
  "accessLevel": "public",
  "license": "CC0 1.0",
  "sourceQuery": "air quality",
  "sourceApiUrl": "https://api.gsa.gov/technology/datagov/v4/search?...",
  "scrapedAt": "2026-08-01T12:00:00.000Z"
}

Pricing

The actor uses predictable pay-per-event pricing:

Event	Price
Actor start	$0.00005
Dataset returned	$0.005 per dataset

For example, a 25-dataset run costs approximately $0.125 in result events, plus the actor-start event. The actor uses the DATA_GOV_API_KEY environment variable when configured; otherwise it uses Data.gov's documented DEMO_KEY, which has low rate limits and is intended for exploration. Set a personal Data.gov API key for production or repeated automation. Pay-per-use compute and proxy billing are enabled alongside PPE in Apify.

Reliability and limits

The actor uses Data.gov's v4 Catalog API with a 30-second request timeout and three attempts with exponential backoff. Empty valid searches finish successfully with a truthful empty OUTPUT; API-key, rate-limit, and upstream failures include a diagnostic status message and fail only when no records were collected. The actor caps maxResults at 100 per run to keep agent calls predictable.

Legal and source note

This actor accesses the public Data.gov Catalog API and returns catalog metadata and links. Review the terms, licenses, and access requirements of each linked dataset before downstream use. Data.gov's official documentation states that the Catalog API provides metadata about datasets rather than the actual dataset contents.