Test Fixtures for Evals

I worked inside an LLM eval suite for weeks before I understood what the fixtures were for. My weekly review from that stretch says "I almost understood what fixtures are." I kept seeing the word and scrolling past a folder with thousands of JSON files in it.

It clicked when I stopped looking at the files and started thinking about the experiment they were part of.

The word

"Fixture" is older than software. In carpentry and machining, a fixture holds a workpiece in place so a tool can make the same cut the same way every time. A clamp is a fixture. Software borrowed the word for the same job: fixed, known data that holds a test still. You'll see it as @pytest.fixture in Python and as YAML files seeded into the test database in Rails.

Fixtures in an LLM eval

The agent I was working on answered questions by searching a knowledge base first, then writing a reply grounded in whatever came back. In its eval suite, a fixture was a recorded response from that search API. A real call was made once, and the JSON it returned was saved to a file in the repo:

{
  "request_id": "b7c1e0d2",
  "status": "ok",
  "data": {
    "search_results": [
      {
        "score": 0.51,
        "content": "To reset your password, open Settings and choose Security...",
        "metadata": { "article_id": "1042" }
      }
    ]
  }
}

When an eval runs, the agent's search call never reaches the live service. Something intercepts it and hands back the recorded file.

Isolating the model

This is the note I wrote the day it made sense, lightly cleaned up:

Let's say that you want to test a new LLM version. You want to keep everything else in your eval the same. The eval should be the same, the utterance should be the same, and the knowledge retrieved to support the agent should also be the same. That's what the fixtures are. They let you isolate the variables so you can actually test the new model.

Retrieval is easy to forget as an input because it feels like part of the system. If retrieval hits a live service, its results can move between two runs: someone edits an article, the index gets rebuilt, ranking scores shift a little. Then a score changes and you can't say whether the model did it or the documents did. The fixture pins the retrieved knowledge in place so the model is the only thing left that can move the score.

Early in that project, a colleague asked what success looked like compared to where the agent was today. You can't answer that without a baseline, and a baseline is only useful if you can run it again next week against the same inputs. Fixtures are what make "the same inputs" true for anything the agent fetches.

They don't remove all the noise. The model still samples, so the same prompt on the same fixture can produce different answers. On that project, results that looked significant at a small trial count had a habit of disappearing once the count went up. Fixtures leave that noise alone and take away the retrieval noise, so what remains comes from the model, which is the part you're trying to measure.

Record and replay

Ruby's VCR library calls the recordings cassettes. The flow in the suite I worked in was:

  1. The eval runs and the agent calls the search API.
  2. A mock server intercepts the HTTP request.
  3. The handler hashes the full request body and turns the hash into a filename.
  4. If that file exists, it returns the recorded JSON.
  5. If it doesn't and record mode is on, it calls the real service, saves the response under that name, and returns it.
  6. If it doesn't and record mode is off, the lookup misses.

A minimal version in TypeScript:

import { createHash } from 'node:crypto'
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
import { join } from 'node:path'
 
type SearchRequest = { query: string; topK: number; filters?: Record<string, string> }
 
declare function callLiveSearch(request: SearchRequest): Promise<unknown>
 
function fixturePath(dir: string, request: SearchRequest): string {
  const hash = createHash('md5').update(JSON.stringify(request)).digest('hex').slice(0, 8)
  const slug = request.query.toLowerCase().replace(/[^a-z0-9]+/g, '_').slice(0, 40)
  return join(dir, `${slug}-${hash}.json`)
}
 
export async function replaySearch(dir: string, request: SearchRequest, record: boolean) {
  const path = fixturePath(dir, request)
  if (existsSync(path)) return JSON.parse(readFileSync(path, 'utf8')).response
 
  if (!record) throw new Error(`No fixture for this request: ${path}`)
 
  const response = await callLiveSearch(request)
  writeFileSync(path, JSON.stringify({ request, response }, null, 2))
  return response
}

The filenames come out looking like this:

fixtures/search/how_do_i_reset_my_password-3f9a2c1e.json

Nothing searches for the file. The name is computed from the request, so the handler only has to check whether it exists.

The hash

My first reaction was that this was overcomplicated. If the whole thing has to be deterministic anyway, why not name each fixture after its eval case and pass the filename in?

So I checked. In that repo, the readable query prefix at the front of the filename was already unique for almost every fixture. The hash only told apart a handful of files that shared a prefix.

What the hash does buy is invalidation. Add a field to the search request and every hash changes, so every fixture reads as missing and gets re-recorded against the new request shape. With hand-picked filenames you'd keep replaying responses that were recorded against a request that no longer exists, and nothing would warn you.

The cost is that a hash only goes one way. The suite I worked in saved only the response, so when a lookup missed, the error named the file it expected and said nothing about why the request had changed. New field or reordered key, you rebuilt the request by hand to find out. The invalidation is also all or nothing. A change that has no effect on retrieval still re-records every file, which means hammering a rate-limited staging service and opening a pull request full of churn in files you never meant to touch.

If I built it again, I'd keep the hash and write the request into the fixture next to the response, the way the code above does. Then a miss is quick to explain: open the fixture with the same query prefix and diff its saved request against the new one.

Stale fixtures

A fixture freezes the knowledge base as it was on the day you recorded it. That's what keeps a model comparison honest, and it also means your eval is testing last month's documents. I'd re-record on purpose, in its own pull request, so a model change and a knowledge change never land in the same diff.