# Kalshi Scraper (`mild_costume/kalshi-scraper`) Actor

A robust scraper for Kalshi.com prediction markets. Automatically extracts market names, prices, volumes, and more for real-time trading insights, analytics, and research applications.

- **URL**: https://apify.com/mild\_costume/kalshi-scraper.md
- **Developed by:** [Caelen Fry](https://apify.com/mild_costume) (community)
- **Categories:** News, Developer tools, Social media
- **Stats:** 53 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00/month + usage

To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## Kalshi Market Data Scraper

This Actor scrapes comprehensive market data from the Kalshi API including:
- Events (prediction market categories)
- Markets (individual betting markets with enhanced price analytics)
- Orderbooks (current bid/ask prices and liquidity depth)
- Recent trades (for historical price calculations)

### Enhanced Price Analytics

The scraper now calculates comprehensive price metrics to match professional trading datasets:

#### Core Price Data
- `last_price` - Current market price
- `yes_bid`, `yes_ask`, `no_bid`, `no_ask` - Current orderbook prices
- `bid_ask_spread` - Difference between best bid and ask

#### Historical Price Analysis
- `prev_day_price` - Price 24 hours ago
- `prev_hour_price` - Price 1 hour ago  
- `prev_week_price` - Price 7 days ago
- `day_change` - Absolute price change over 24 hours
- `hour_change` - Absolute price change over 1 hour
- `day_change_pct` - Percentage price change over 24 hours
- `hour_change_pct` - Percentage price change over 1 hour

#### Volume & Liquidity Metrics
- `volume` - Total contract volume
- `volume_24h` - Trading volume in last 24 hours
- `recent_volume` - Trading volume in last hour
- `dollar_volume` - Total volume in dollar terms
- `dollar_recent_volume` - Recent volume in dollar terms
- `dollar_open_interest` - Open interest in dollar terms
- `open_interest` - Total open contracts
- `liquidity` - Total orderbook depth

#### Market Mechanics
- `settlement_timer_seconds` - Time until settlement
- `can_close_early` - Whether market can close before expiration
- `close_unconfirmed` - Early closure status
- `risk_limit_cents` - Position risk limits
- `min_tick_size` - Minimum price increment
- `mutually_exclusive` - Whether market is mutually exclusive
- `settle_details` - Settlement mechanism details
- `description_context` - Additional market context

### Configuration

#### Input Parameters

```json
{
  "environment": "production",
  "includeEvents": true,
  "includeMarkets": true,
  "includeOrderbooks": true,
  "includeTrades": true,
  "maxEvents": 0,
  "maxMarketsPerEvent": 0,
  "enhancedPriceMetrics": true,
  "eventStatusFilter": "open"
}
````

- `environment`: "demo" or "production"
- `includeEvents`: Scrape event data (default: true)
- `includeMarkets`: Scrape market data (default: true)
- `includeOrderbooks`: Scrape orderbook data for price calculations (default: true)
- `includeTrades`: Scrape trade history for price analytics (default: true)
- `maxEvents`: Limit number of events (0 = no limit)
- `maxMarketsPerEvent`: Limit markets per event (0 = no limit)
- `eventStatusFilter`: Filter events by status - "open" (default), "closed", "settled", or "all"

#### Event Status Filter Options

- **"open"** (default): Only collect active events where trading is still happening
- **"closed"**: Only collect events where trading has ended but results aren't final
- **"settled"**: Only collect events with final results determined
- **"all"**: Collect all events regardless of status

The default "open" filter is recommended for most use cases as it focuses on actively tradeable markets.

#### Credentials

Set up your Kalshi API credentials in Apify secure storage:

- `KALSHI_API_KEY`: Your Kalshi Key ID
- `KALSHI_PRIVATE_KEY`: Your RSA private key in PEM format

### Output Format

The scraper outputs comprehensive JSON data that can be converted to CSV with all price analytics:

```json
{
  "type": "market",
  "scraped_at": "2025-01-17T10:30:00Z",
  "market_info": {
    "ticker": "EXAMPLE-24Q1",
    "last_price": 0.52,
    "yes_bid": 0.51,
    "yes_ask": 0.53,
    "bid_ask_spread": 0.02,
    "prev_day_price": 0.48,
    "day_change": 0.04,
    "day_change_pct": 8.33,
    "volume_24h": 1250,
    "dollar_volume": 650,
    "liquidity": 5000,
    "open_interest": 2500,
    "can_close_early": true,
    "settlement_timer_seconds": 86400,
    // ... additional comprehensive fields
  },
  "orderbook": {
    "yes": [...],
    "no": [...]
  },
  "recent_trades": [...]
}
```

### Comprehensive Data Collection

This enhanced version collects the same comprehensive data as professional trading platforms:

1. **Real-time pricing** with bid/ask spreads
2. **Historical price analysis** with change calculations
3. **Volume analytics** across multiple timeframes
4. **Liquidity metrics** from orderbook depth
5. **Market mechanics** and settlement details
6. **Trade history** for price discovery analysis

The output matches the structure of professional Kalshi datasets with 40+ data fields per market.

### Usage

For comprehensive data collection, use the included `input_comprehensive.json` configuration which enables all enhanced features.

The scraper automatically calculates derived metrics from orderbook and trade data, providing the complete picture needed for market analysis and trading strategies.

### Features

- **Comprehensive Data Collection**: Events, markets, orderbooks, and recent trades
- **Secure Credential Management**: Uses Apify secure storage for API credentials
- **Environment Support**: Works with both Kalshi demo and production environments
- **Robust Error Handling**: Automatic retries with exponential backoff
- **Rate Limiting**: Smart delays to respect API limits
- **Statistics Tracking**: Detailed progress and performance metrics
- **Flexible Configuration**: Customizable data collection options

### Setup

#### 1. Kalshi API Credentials

Get your API credentials from [Kalshi Account Profile](https://kalshi.com/account/profile):

1. Go to the **API Keys** section
2. Create a new API key if you don't have one
3. Note down your **Key ID** and download the **Private Key** file
4. The private key should be in PEM format starting with `-----BEGIN RSA PRIVATE KEY-----`

#### 2. Secure Storage Setup (Recommended)

For security, store your credentials in Apify's secure storage instead of hardcoding them:

##### Option A: Use the Helper Script

1. Put your credentials in `config.env`:

```bash
KALSHI_API_KEY=your-api-key-id
KALSHI_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----
your-private-key-content-here
-----END RSA PRIVATE KEY-----"
```

2. Run the setup script:

```bash
python3 setup_secure_storage.py
```

##### Option B: Manual Setup

In your Apify console or via API, set these key-value pairs:

- `KALSHI_API_KEY`: Your Kalshi API Key ID
- `KALSHI_PRIVATE_KEY`: Your RSA private key in PEM format

#### 3. Local Development Setup

For local testing and development:

1. Clone this repository
2. Install dependencies:

```bash
pip install -r requirements.txt
```

3. Set up your credentials in `config.env`:

```bash
## Copy the template
cp config.env.template config.env

## Edit with your credentials
nano config.env
```

4. Run locally:

```bash
python3 load_from_env.py
```

### Performance Notes

- **Events + Markets only**: ~2-5 minutes for 400+ events
- **With Orderbooks**: Significantly longer due to API rate limits
- **With Trades**: Additional time depending on market activity
- **Rate Limiting**: Built-in delays prevent API throttling

### Troubleshooting

#### Authentication Issues

```
❌ Authentication failed: Invalid signature
```

- Verify your private key is in correct PEM format
- Ensure no extra whitespace or formatting issues
- Check that you're using the correct API Key ID

#### API Rate Limits

```
⚠️ Rate limit encountered, waiting...
```

- The scraper automatically handles rate limits
- Consider reducing `maxEvents` or `maxMarketsPerEvent` for faster runs
- Disable orderbooks/trades for much faster scraping

#### Local Development Issues

```
❌ No credentials found in any source!
```

- Make sure `config.env` file exists with proper credentials
- Verify the file format matches the template
- Check that environment variables are set correctly

#### Missing Data

```
⚠️ Failed to fetch orderbook for market XYZ
```

- Some markets may not have orderbook data available
- Network timeouts are automatically retried
- Check market status (some may be closed)

### API Documentation

For more details about the Kalshi API:

- [Kalshi API Documentation](https://docs.kalshi.com/)
- [Trading API Reference](https://docs.kalshi.com/trading-api/)

### Support

If you encounter issues:

1. Check the troubleshooting section above
2. Verify your API credentials and permissions
3. Review the Apify Actor logs for detailed error messages
4. Ensure you're using the correct environment (demo vs production)

### License

This project is open source. See the Apify platform terms for usage guidelines.

### Changelog

#### v0.2.0

- Added retry logic with exponential backoff
- Enhanced error handling and recovery
- Added runtime statistics tracking
- Improved rate limiting
- Added environment variable support
- Enhanced logging and progress reporting

#### v0.1.0

- Initial release
- Support for events, markets, orderbooks, and trades
- RSA-based authentication
- Demo and production environment support
- Configurable data collection parameters

# Actor input Schema

## `environment` (type: `string`):

Kalshi API environment to use

## `eventStatusFilter` (type: `string`):

Filter events by their trading status

## `includeEvents` (type: `boolean`):

Collect event data

## `includeMarkets` (type: `boolean`):

Collect market data

## `includeOrderbooks` (type: `boolean`):

Collect orderbook data for enhanced price metrics

## `includeTrades` (type: `boolean`):

Collect recent trade data for price analysis

## `maxEvents` (type: `integer`):

Maximum number of events to collect (0 = no limit)

## `maxMarketsPerEvent` (type: `integer`):

Maximum number of markets per event (0 = no limit)

## `enhancedPriceMetrics` (type: `boolean`):

Calculate comprehensive price analytics and historical data

## Actor input object example

```json
{
  "environment": "production",
  "eventStatusFilter": "open",
  "includeEvents": true,
  "includeMarkets": true,
  "includeOrderbooks": false,
  "includeTrades": false,
  "maxEvents": 500,
  "maxMarketsPerEvent": 0,
  "enhancedPriceMetrics": false
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("mild_costume/kalshi-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("mild_costume/kalshi-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call mild_costume/kalshi-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=mild_costume/kalshi-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Kalshi Scraper",
        "description": "A robust scraper for Kalshi.com prediction markets. Automatically extracts market names, prices, volumes, and more for real-time trading insights, analytics, and research applications.",
        "version": "1.0",
        "x-build-id": "POs0UUJv3u5n3yvJ4"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/mild_costume~kalshi-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-mild_costume-kalshi-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/mild_costume~kalshi-scraper/runs": {
            "post": {
                "operationId": "runs-sync-mild_costume-kalshi-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/mild_costume~kalshi-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-mild_costume-kalshi-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "environment": {
                        "title": "Environment",
                        "enum": [
                            "production",
                            "demo"
                        ],
                        "type": "string",
                        "description": "Kalshi API environment to use",
                        "default": "production"
                    },
                    "eventStatusFilter": {
                        "title": "Event Status Filter",
                        "enum": [
                            "open",
                            "closed",
                            "settled",
                            "all"
                        ],
                        "type": "string",
                        "description": "Filter events by their trading status",
                        "default": "open"
                    },
                    "includeEvents": {
                        "title": "Include Events",
                        "type": "boolean",
                        "description": "Collect event data",
                        "default": true
                    },
                    "includeMarkets": {
                        "title": "Include Markets",
                        "type": "boolean",
                        "description": "Collect market data",
                        "default": true
                    },
                    "includeOrderbooks": {
                        "title": "Include Orderbooks",
                        "type": "boolean",
                        "description": "Collect orderbook data for enhanced price metrics",
                        "default": false
                    },
                    "includeTrades": {
                        "title": "Include Trades",
                        "type": "boolean",
                        "description": "Collect recent trade data for price analysis",
                        "default": false
                    },
                    "maxEvents": {
                        "title": "Max Events",
                        "minimum": 0,
                        "type": "integer",
                        "description": "Maximum number of events to collect (0 = no limit)",
                        "default": 500
                    },
                    "maxMarketsPerEvent": {
                        "title": "Max Markets Per Event",
                        "minimum": 0,
                        "type": "integer",
                        "description": "Maximum number of markets per event (0 = no limit)",
                        "default": 0
                    },
                    "enhancedPriceMetrics": {
                        "title": "Enhanced Price Metrics",
                        "type": "boolean",
                        "description": "Calculate comprehensive price analytics and historical data",
                        "default": false
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```