# Llm Benchmarks (`david_flagg/llm-benchmarks`) Actor

Unified LLM data — pricing, benchmarks, specs, and local deployment info for\
300+ models. Compare cost, Open LLM Leaderboard scores, Arena ratings, context
lengths, GGUF availability, and VRAM estimates in one dataset.

- **URL**: https://apify.com/david\_flagg/llm-benchmarks.md
- **Developed by:** [David Flagg](https://apify.com/david_flagg) (community)
- **Categories:** AI, Automation, Developer tools
- **Stats:** 3 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## LLM Benchmark Aggregator

The LLM Field Guide. Pricing, benchmarks, specs, and local deployment data for 300+ language models in one dataset.

### What you get

Every model includes up to 30 fields across four categories:

#### Cost

- **Input/output pricing** per million tokens (from OpenRouter)
- Compare across 300+ models from OpenAI, Anthropic, Google, Meta, Mistral, DeepSeek, and more

#### Capability

- **Open LLM Leaderboard scores**: Average, IFEval, BBH, MATH Lvl 5, GPQA, MUSR, MMLU-PRO
- **Chatbot Arena**: MT-bench (multi-turn conversation quality), MMLU
- Model type, architecture, parameter count

#### Specs

- Context window length and max completion tokens
- Input/output modalities (text, image, audio)
- Tokenizer type, instruction format
- Content moderation status

#### Local deployment

- **GGUF availability** on HuggingFace (can you run it locally?)
- **Estimated VRAM** at Q4\_K\_M quantization
- HuggingFace model ID for direct download

No API keys required. All data comes from public APIs.

### Data sources

| Source | Data | Method |
|--------|------|--------|
| OpenRouter | Pricing, context, modality, 300+ models | REST API |
| Open LLM Leaderboard | 6 benchmark scores, 4,500+ models | HuggingFace Datasets API |
| Chatbot Arena (LMSYS) | MT-bench, MMLU, 300+ models | HuggingFace Space CSV |
| HuggingFace | GGUF model availability | Models API |

Models from OpenRouter are automatically enriched with benchmark scores and GGUF availability where matches exist.

### Example output

```json
{
  "model_id": "meta-llama/llama-3.1-70b-instruct",
  "model_name": "Meta: Llama 3.1 70B Instruct",
  "huggingface_id": "meta-llama/Llama-3.1-70B-Instruct",
  "pricing_input_per_mtok": 0.52,
  "pricing_output_per_mtok": 0.75,
  "context_length": 131072,
  "open_llm_average": 42.8,
  "open_llm_ifeval": 83.6,
  "open_llm_bbh": 55.3,
  "open_llm_math": 26.4,
  "open_llm_gpqa": 18.7,
  "open_llm_mmlu_pro": 50.8,
  "arena_mt_bench": 8.42,
  "parameter_count_b": 70.55,
  "gguf_available": true,
  "estimated_vram_q4_gb": 43.8,
  "is_moderated": false,
  "license": "llama3.1",
  "sources": ["openrouter", "open_llm_leaderboard", "chatbot_arena"]
}
```

### Filtering

- **Model name** — Search by name or ID ('llama', 'claude', 'qwen')
- **Max price** — Only models under a price threshold ($/MTok)
- **Min benchmark score** — Only models above a quality threshold
- **Model type** — Pretrained, chat, fine-tuned, merged, MoE
- **Sort by** any field — price, benchmark, context length, parameter count

### Use cases

- **Model selection** — Find the best model for your budget and use case
- **Cost optimization** — Compare pricing across providers for the same capability level
- **Local deployment planning** — Which models can you run on your hardware?
- **API integration** — Feed structured model data into your own tools and dashboards
- **Market intelligence** — Track the LLM landscape as models and prices change daily

# Actor input Schema

## `modelFilter` (type: `string`):

Only include models whose name or ID contains this text (case-insensitive). Examples: 'llama', 'claude', 'qwen', 'gemini'.

## `maxPricePerMTok` (type: `number`):

Only include models with input pricing at or below this amount in dollars per million tokens.

## `minAvgScore` (type: `number`):

Only include models with Open LLM Leaderboard average score above this threshold (0-100). Models without benchmark data are excluded when set.

## `maxResults` (type: `integer`):

Maximum number of models to return.

## `sortBy` (type: `string`):

Sort results by this field (descending for scores, ascending for price).

## `includeTypes` (type: `array`):

Filter by model type from Open LLM Leaderboard. Leave empty for all types.

## `checkGGUF` (type: `boolean`):

Check HuggingFace for GGUF quantized versions (for local deployment). Adds ~5 seconds to run time.

## Actor input object example

```json
{
  "maxResults": 500,
  "sortBy": "pricing_input_per_mtok",
  "includeTypes": [],
  "checkGGUF": true
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("david_flagg/llm-benchmarks").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("david_flagg/llm-benchmarks").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call david_flagg/llm-benchmarks --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=david_flagg/llm-benchmarks",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/S1qU0shmiJDGTBzlD/builds/TleBFXwHGuedPgCzZ/openapi.json