What an eval bench actually looks like

The conversation about evals usually starts the same way. The team agrees that “evals matter.” The team agrees that they should have some. The team also agrees, slightly later in the conversation, that nobody has time to build them right now, that LangSmith and Braintrust and Phoenix exist, and that the existing prompt tests in the repository are “probably fine.”

Six months later the team is debugging a customer-visible regression in production and discovers that the prompt tests catch about 12% of the regressions a real eval suite would catch. By that point fixing the bench is harder than building it would have been at the start, because the team has shipped a quarter of unevaluated prompt iterations and there is no baseline to compare against.

We build the eval bench in week one of every agent engagement, before the prompt is touched. The pattern is mechanical enough to write down. The rest of this post is what is in the suite, what it costs to run, and how it gets wired into the work.

The four sources of cases

The bench is a collection of cases. A case is an input, an expected output, and a few pieces of metadata that say what counts as passing. The cases come from four places.

The customer-success team. We spend a Wednesday and a Thursday afternoon with the CS team, taking notes as they walk through the queries they handle manually every week and the questions they wish the assistant could handle reliably. Each query becomes a case, with the input, the response the CS team would consider correct, the response they would consider acceptable, and the failure modes they have seen before. This is the most useful source. The CS team has spent more time looking at real customer questions than anyone else in the company. We typically come out of two sessions with 60 to 100 cases.

Production logs. We sample inputs from the last 60 days, stratified across customer segments and query categories. For each one we tag the production response and a domain-expert response. The delta is the case. About a tenth of these cases are regressions: cases where the production response was a confident error. We mark these explicitly so the bench fails the build if a future change reintroduces them. This source typically gives 200 to 400 cases.

Adversarial. We work with the team’s security engineer to craft inputs that probe the boundaries. Prompt injection through user-uploaded data. Jailbreak attempts. Queries that combine a benign question with a poisoned context document. Queries that test whether the assistant will reveal its own prompt. The adversarial set is smaller (typically 40 to 80 cases) but the cost-per-case of finding a regression here is dramatically lower than catching it in production.

Team intuition. The team writes 30 to 60 cases that capture behaviours considered load-bearing for the product. The cases where, if the assistant got it wrong, the product’s value proposition would be in question. These cases are marked critical: if any of them fails, the build fails regardless of the overall pass rate.

By the end of week one the bench typically has 300 to 600 cases. Pass rate against the existing production system is the baseline we work from.

The shape of a case

A case has the input, the expected output (or a function that decides if a given output is acceptable), a category (so we can report pass rate per category), a criticality flag (critical cases fail the build alone), and a notes field for context the reviewer will need three months later when they are wondering why the case was added.

We keep the cases in YAML, in a separate folder of the repository, version-controlled like any other code. A new case is a PR. A deleted case is a PR. The team that owns the bench has the same review discipline as the team that owns the production code, because the bench has become part of the production system.

The harness, and what it costs

The harness is a script that loads the cases, runs the agent against each one, and scores the output against the case’s acceptance criteria. The scoring is a mix:

• Exact-match for outputs where there is one correct answer (citation lookups, structured extraction).
• Semantic similarity with an embedding model for outputs where there is a range of acceptable phrasings.
• LLM-as-judge for outputs that need a quality call (factuality, tone, refusal correctness). We use a different model family than the production agent, to avoid the obvious bias.

A typical bench of 500 cases runs end to end in 10 to 14 minutes and costs around USD 1.50 in inference. That cost is the price of catching a regression before it ships. On every PR. Permanently.

Wiring it into CI

The bench runs on every change to the prompt, the tool definitions, the retrieval logic, or the model selection. We write a guard that blocks merges if:

• The overall pass rate drops by more than 1.5% from the baseline.
• Any critical case fails.
• The average cost-per-request exceeds the configured budget.
• The p95 end-to-end latency exceeds the configured budget.

Cost and latency are not “extra concerns” sitting alongside the eval. They are part of the eval. A prompt that improves the quality score but pushes the cost up by 30% is, in practice, a regression. We surface this in the PR.

The first PR that hits the new CI is usually a prompt change the team has been wanting to ship for two weeks. We make sure that PR runs the bench, sees the result, and either passes or surfaces the trade-off the team needs to make. After that the discipline takes hold on its own.

The feedback loop into production

The bench is not a static asset. Every week, a sample of real production runs that the customer-success team flagged as off (low confidence score, customer escalation, manual handoff) gets reviewed. The reviews are short. Each one decides: was the agent’s output acceptable, or was it not? Cases where the answer was “not” get added to the bench as new cases, with the production input and the domain-expert response. Cases where the answer was “acceptable” are tagged and used to widen the acceptance criteria for similar cases.

This loop is what makes the bench compound. After three months, the bench has more cases that came from real production failure modes than it had at start. Each new prompt change is now being tested against a larger and more realistic set than the one the team had at week one.

What it does not do

The bench does not replace human review. The first three weeks of any AI engagement, we still read sampled production outputs by hand. The bench catches regressions; it does not invent edge cases the team has not thought of. The cases come from human judgment, and the judgment loop is what makes the bench live longer than the engagement.

The bench also does not eliminate hallucinations. It catches the ones that match the patterns in the suite. It does not catch the entirely new failure mode the model invents at 3am next Tuesday. The mitigation is the guardrail layer that returns “I don’t know” under load, and the audit log per run that lets the team add the new failure to the bench when it shows up.

The one paragraph version

An eval bench is a versioned YAML directory of input + expected + acceptance criteria + criticality flag, run by a harness that scores against exact match, embedding similarity, and an independent LLM judge. It is wired into CI on every PR with budgets for cost and latency as well as quality. Production failures flow back into the bench weekly. Built this way in week one, the bench costs about USD 1.50 per PR and is the thing that separates an AI feature you can rely on from a demo you cannot.