llm

xk6-llm

A k6 extension for benchmarking OpenAI-compatible chat-completions servers. Streaming-first. Emits TTFT, ITL, TPOT, goodput, and token-throughput metrics; per-chunk timing matches vllm bench serve semantics.

1,470 requests, 0 errors, 100% goodput, $0.382 total cost. Three-turn conversations against ibm-granite/granite-4.1-30b on a single NVIDIA B300 SXM6. The "TTFT by turn" panel surfaces prefix-cache speedup across turns; the cost and energy panels are derived from server-reported token counts.

Recorded against Ollama on an M3 MacBook Air. The same script and dashboard work against a hosted API or a vLLM cluster; only the numbers change.

Build

go install go.k6.io/xk6/cmd/xk6@latest
xk6 build --with github.com/msradam/xk6-llm@latest --output build/k6
./build/k6 version

Example

import llm from 'k6/x/llm';

const client = new llm.Client({
  base_url: 'http://localhost:11434/v1',
  model:    'granite4.1:3b',
});

export default async function () {
  const res = await client.chat({
    messages:   [{ role: 'user', content: 'Write a haiku about Poisson arrivals.' }],
    max_tokens: 128,
    temperature: 0,
  });
  console.log(`ttft=${res.ttft_ms.toFixed(1)}ms tpot=${res.tpot_ms.toFixed(1)}ms tokens=${res.completion_tokens}`);
}

./build/k6 run examples/chat.js

Metrics

Every metric is tagged model. Errors are additionally tagged error_type. Per-request cache_state and arbitrary tags are propagated when supplied.

API

new llm.Client(opts)

{
  base_url?:   string,                          // default: http://localhost:11434/v1
  api_key?:    string,
  model?:      string,
  timeout_ms?: number,                          // default: 60000
  ignore_eos?: boolean,
  headers?:    Record<string, string>,
  slo?:        { ttft_ms?, tpot_ms?, e2el_ms? },
  cost?:       { usd_per_million_input_tokens?, usd_per_million_output_tokens? },
  energy?:     { j_per_input_token?, j_per_output_token?, idle_w? },
}

client.chat(req)

req accepts the OpenAI chat-completion fields (messages, max_tokens, temperature, top_p, seed, etc.) plus optional slo, cache_state, and tags. Returns a Promise resolving to:

{
  content:             string,
  ttft_ms:             number,
  itl_ms:              number[],
  tpot_ms:             number,
  duration_ms:         number,
  response_headers_ms: number,
  chunks:              number,
  prompt_tokens:       number,
  completion_tokens:   number,
  finish_reason:       string,
}

new llm.Dataset(opts)

Replays a JSONL prompt corpus. One line per request: {"messages": [...], "max_tokens"?: N, ...}. Loaded once per process and cached by absolute path.

{
  path:     string,
  seed?:    number,    // default: 42
  shuffle?: boolean,
}

Methods: dataset.size(), dataset.next(), dataset.at(i), dataset.reset().

A converter for ShareGPT V3 to this format lives at scripts/sharegpt_to_jsonl.py.

What you can simulate

k6's load generator is a JavaScript runtime, so a single VU can carry conversation state, branch on responses, and drive multi-call workflows. xk6-llm doesn't define a session or agent abstraction; instead, the patterns live in examples/ and use the existing Client plus k6's tags to make the workflow visible in the dashboard.

Quickstart with Grafana

A Docker Compose stack with a pre-provisioned dashboard is in quickstart/. See QUICKSTART.md.

Validation

Cross-validated against vllm bench serve on a real vLLM 0.21.0 server (Qwen2.5-72B-Instruct-AWQ, A100 80GB). TPOT/ITL/E2EL agree within 5% on identical workloads; token counts are bit-exact with vLLM /metrics. Numbers and methodology in docs/validation-parity.md and docs/validation-results.md.

Compatibility

Attribution

This codebase was developed with assistance from Claude Code. PRs are reviewed and merged by humans.

License

Apache-2.0