datalab

xk6-datalab

Scriptable test data orchestrator and provider for Grafana k6 and beyond.

xk6-datalab provides a framework for managing test data. As both a k6 extension and a standalone command-line interface (datalab command), it provides a versatile "laboratory" for crafting, acquiring, and transforming test data. It promotes a clear separation of concerns: data generation is handled in dedicated TypeScript scripts, and data consumption happens efficiently within your k6 test scripts, all with the safety and clarity of TypeScript.

Key Features

• Flexible Data Generation: Scriptably define and generate synthetic test data tailored to your exact needs. Faker (from the xk6-faker extension) is available for generating synthetic test data.
• Diverse Data Ingestion: Seamlessly read and process existing test data from local CSV and JSON files.
• API Data Sourcing: Fetch and integrate dynamic data directly from external REST APIs, with optional transformations.
• On-the-Fly Transformation: Apply powerful and scriptable transformations to all sourced and generated data.
• Iterator-Based Provisioning: Consume test data through an efficient, lazy-evaluated JavaScript API within k6, perfect for dynamic test scenarios and large datasets.
• Decoupled Workflow: Separate your data preparation and acquisition logic from your test execution, promoting cleaner and more maintainable k6 scripts.
• Standalone CLI: Generate, inspect, and test your data generation and transformation logic directly from your terminal, independent of k6.
• Data Testability: Easily validate the shape, content, and quality of your generated and transformed test data via the CLI before integrating it into your load tests.
• Type-Safe Test Data: Export TypeScript types from your generators to enable type checking, autocompletion, and more robust, maintainable performance tests in your k6 scripts.

Quick Start

xk6-datalab promotes a clear separation of concerns: data generation in dedicated generator scripts, and data consumption in your k6 test scripts. Both are written in TypeScript for type safety.

1. Create a Generator Script

Create a generator.ts file. This is where you define your data's structure and its generation logic. Each exported generator function becomes a named, iterable data pool.

generator.ts

// Define a type for your data for a clear, reusable contract.
export interface User {
  subject: string;
  given_name: string;
  family_name: string;
  name: string;
  email: string;
}

// Export a generator function. This creates the "user" data pool.
export function* user(): Generator<User> {
  // TODO: Implement your generator logic here.
  for (let i = 0; i < 100; i++) {
    yield {
      subject: `user-${i}`,
      given_name: `John ${i}`,
      family_name: `Doe ${i}`,
      name: `John ${i} Doe`,
      email: `john.${i}.doe@example.com`,
    };
  }
}

2. Use it in a k6 Test

In your k6 script, import datalab to connect to your generator, get an iterator, and consume the data.

script.ts

import datalab from "k6/x/datalab";
// Import type declarations for autocompletion and type safety
import type { User } from "./generator.ts";

// Create a data pool from the generator script.
const datapool = datalab("./generator.ts");

// Create a generator for the "user" data pool.
const users: Generator<User, User, User> = datapool("user");

export default function () {
  const user: User = users.next().value;

  console.log(user);
}

run-k6.sh

k6 run script.ts -i 5 -q --no-summary --log-format raw --log-output stdout

Output:

{"email":"john.0.doe@example.com","family_name":"Doe 0","given_name":"John 0","name":"John 0 Doe","subject":"user-0"}
{"email":"john.1.doe@example.com","family_name":"Doe 1","given_name":"John 1","name":"John 1 Doe","subject":"user-1"}
{"email":"john.2.doe@example.com","family_name":"Doe 2","given_name":"John 2","name":"John 2 Doe","subject":"user-2"}
{"email":"john.3.doe@example.com","family_name":"Doe 3","given_name":"John 3","name":"John 3 Doe","subject":"user-3"}
{"email":"john.4.doe@example.com","family_name":"Doe 4","given_name":"John 4","name":"John 4 Doe","subject":"user-4"}

3. Use the Standalone CLI

You can also generate data directly from your terminal using the datalab command. This is perfect for debugging your generator or creating sample data.

run-cli.sh

Generate 5 users from generator.ts and print to stdout

datalab -c 5 generator.ts

Output:

{"email":"john.0.doe@example.com","family_name":"Doe 0","given_name":"John 0","name":"John 0 Doe","subject":"user-0"}
{"email":"john.1.doe@example.com","family_name":"Doe 1","given_name":"John 1","name":"John 1 Doe","subject":"user-1"}
{"email":"john.2.doe@example.com","family_name":"Doe 2","given_name":"John 2","name":"John 2 Doe","subject":"user-2"}
{"email":"john.3.doe@example.com","family_name":"Doe 3","given_name":"John 3","name":"John 3 Doe","subject":"user-3"}
{"email":"john.4.doe@example.com","family_name":"Doe 4","given_name":"John 4","name":"John 4 Doe","subject":"user-4"}

4. Examples

You can find more examples in the examples directory. The *.data.[tj]s files contain the generators, the *.[tj]s files contain the k6 scripts that demonstrate their use, and the *.test.[tj]s files contain the integration tests.

Download

Building a custom k6 binary with the hackathon-13-datalab extension is necessary for its use. You can download pre-built k6 binaries from the Releases page.

Build

Use the xk6 tool to build a custom k6 binary with the hackathon-13-datalab extension. Refer to the xk6 documentation for more information.

Contribute

If you wish to contribute to this project, please start by reading the Contributing Guidelines.

How It Works

Datalab's core is the generator script, a TypeScript file housing generator functions and type declarations. This script executes within a dedicated JavaScript runtime, integrated into both the k6 extension and a CLI program. Test data from various sources is uniformly accessed in k6 scripts via an iterator API. The CLI program also enables independent access to test data, separate from k6.