Stop Eyeballing Your Prompts: Test Them Like Real Code with Promptfoo
promptfoo
*Part 1 of a series on testing LLMs and prompts. We start with the simplest thing that works: a deterministic assertion.*
— -
If you’ve shipped anything built on an LLM, you know the sinking feeling. You tweak one line of a prompt to fix a weird edge case, deploy it, and three days later discover you quietly broke four other things. There were no tests, so there was no alarm.
We test regular software constantly — unit tests, CI gates, the works. But prompts? Most teams “test” them by eyeballing a few outputs in a playground and hoping for the best. That doesn’t scale, and it doesn’t survive contact with production.
[promptfoo](https://promptfoo.dev) fixes this. It lets you treat a prompt as a testable unit: you define inputs, you define what a *good* answer looks like, and it runs everything and tells you what passed.
This is the first post in a series. We’ll keep each post to a single idea. Today: the setup, and the simplest possible test.
— -
## The thing we’re testing
Our running example is a support-ticket triage assistant. Give it a customer message, and it returns structured JSON: a category, a priority, and a one-line summary.
Here’s the prompt (`prompts/triage.txt`):
```
You are a customer-support triage assistant.
Classify the ticket below and reply ONLY with JSON in this shape:
{“category”: “billing” | “bug” | “account” | “feature_request” | “other”,
“priority”: “low” | “medium” | “high”,
“summary”: “<one short sentence>”}
Ticket:
{{ticket}}
```
That `{{ticket}}` is a template variable. Each test case fills it with a different customer message. This is the key mental shift: **the prompt is a function, and tests feed it arguments.**
— -
## The config that ties it together
promptfoo is driven by a single config file, `promptfooconfig.yaml`:
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: My first promptfoo setup
# The prompt template. {{ticket}} gets filled from each test’s vars.
prompts:
- file://prompts/triage.txt
# The thing we are testing: a real LLM from OpenAI (key comes from .env)
providers:
- id: openai:gpt-4.1-mini
label: openai-gpt-4.1-mini
# Tests live in the tests/ folder (one file per category).
tests:
- file://tests/*.yaml
```
Get Pranesh’s stories in your inbox
Join Medium for free to get updates from this writer.
Three things to notice:
1. **`prompts`** points at our template file.
2. **`providers`** is *what we’re testing* — here, `gpt-4.1-mini`. The API key lives in a `.env` file (never commit it).
3. **`tests`** is a glob — every file in `tests/` joins the suite.
### Getting set up
```bash
npm install -g promptfoo
# put your key in .env: OPENAI_API_KEY=sk-…
promptfoo eval # run the suite
promptfoo view # open the results in a browser UI
```
That’s the whole loop. Now let’s write one test.
— -
## The simplest test that works
Start with a check that doesn’t need another LLM to judge it — it’s instant, free, and never flaky. Here’s `tests/billing.yaml`:
```yaml
- description: billing — charged twice
vars:
ticket: I was charged twice for my subscription. Please refund ASAP.
assert:
- type: contains # PASS if the answer includes this exact text
value: billing
- type: not-contains
value: bug
```
Read it like a sentence: *given* this ticket, the output **must contain** `billing` and **must not contain** `bug`. If the model ever misclassifies a refund complaint as a bug, this test goes red and you find out immediately.
A test case is just two parts:
- **`vars`** — the input that fills `{{ticket}}`.
- **`assert`** — the checks on the output. You can stack as many as you want.
The two assertion types here are the bread and butter of deterministic testing:
- **`contains`** — a plain substring match. The output must include this text.
- **`not-contains`** — the inverse. Pinning down what the model must *not* do is just as valuable as pinning what it must do.
Run `promptfoo eval` and you get a clean pass/fail — no squinting at JSON required.
> **One gotcha:** `contains` is a literal substring match, not semantic understanding. If you ever renamed the `bug` category to `defect`, every `contains: bug` test would fail even with perfect output. That rigidity is a *feature* — it’s exactly what catches regressions — but it means you maintain these tests as your schema evolves.
— -
## Why this is the right place to start
Deterministic assertions like `contains` and `not-contains` are the unit tests of the prompt world: fast, free, and completely reliable. Before you reach for anything fancier, cover as much behavior as you can with these. They’ll catch the majority of real regressions for almost no cost.
**Next in the series:** the fuzzy stuff. How do you test a free-form `summary` sentence, where two totally different wordings can both be correct? That’s where *model-graded* assertions come in — we’ll tackle `g-eval` in Part 2.
— -
*Following along? Part 2 covers model-graded scoring with `g-eval`.*
