Building an eval bench around an LLM assistant that had shipped without one, and what we found.

The conversation started with a screenshot. A customer-success leader at a B2B SaaS analytics product had been demoing the in-product AI assistant to a new customer during onboarding. The customer had asked a routine question — what the average deal size had been in the previous quarter — and the assistant had answered with confidence, with authority, and with a number that was wrong by an order of magnitude. The customer-success leader had paused, recovered, and the call had ended pleasantly enough. She walked into the engineering lead’s office an hour later.

The assistant had shipped six months earlier. It had been built fast. The prompt had gone through several iterations during development and was now treated as a stable state, which the CTO later told us he had assumed was a finished state when it was actually a state of inertia. There was no eval bench. There was no observability beyond what the LLM provider exposed in their dashboard. The team’s confidence in the assistant’s accuracy was based on the engineer who had built it saying it had “looked right” during the build phase, and on the absence of customer complaints during the first three months. Both turned out to be weaker signals than the team had treated them as.

The engagement brief was twelve weeks. The team wanted to know what they did not know about the assistant’s behaviour, then fix it.

What we found

We started with three days of reading. The prompt was 1,400 tokens of careful instruction-following, plus a tool-calling layer that interfaced with the product’s analytics API. There were three tools: a database-query tool that read the customer’s dataset, a chart-rendering tool, and a citation tool that surfaced source references. Most user queries went through one or two of these. Some went through all three.

Then we sampled 200 random production queries from the previous 30 days and ran them through the assistant again. We brought in three domain experts — a former analyst, a former CS lead, and a data engineer who had built the underlying schema — and asked them to review the responses without knowing what the assistant had originally said.

The agreement rate between the experts and the production output was 82%. Eighteen percent of the responses contained at least one factual error a domain expert would consider material. Of those, six percent contained errors the expert would consider damaging if surfaced to a customer.

The team had not been measuring this. There was no baseline to compare against. The error rate could have been getting better or worse over the previous six months and the team would have had no way to tell. The CS leader’s screenshot was not an outlier; it was a representative sample of a class of failures the team had been shipping without knowing.

Building the bench

The first three weeks were the bench — the golden-set + eval harness pattern we ship in every agent engagement. We did not touch the production prompt. We did not touch the model. We did not change a single line of the assistant code. We built the bench in a separate repository and ran it against the production system as if it were an external dependency.

The cases came from four sources.

The first, and the most useful, was the customer-success team. We ran a Wednesday and a Thursday afternoon with them, taking notes as they walked us through the queries they answered manually every week and the kinds of questions they wished the assistant could handle reliably. We came out of those two sessions with 89 curated cases, each with the input, the response the CS team would consider correct, the response they would consider acceptable, and the failure modes they had seen before. They were patient with us. We bought them food.

The second was the production logs. We sampled inputs from the last 60 days, stratified across customer segments and query categories. For each one we tagged the production response and a domain-expert response. The delta was the case. This batch gave us 312 cases. About 40 of them were cases where the production response had been a confident error, which we marked as regression cases for the bench to fail on if a future change reintroduced them.

The third was the adversarial set. We worked with the team’s security engineer to craft inputs that would test the boundaries. Prompt injection through chart titles. Jailbreak attempts through user-uploaded data. Queries that asked the assistant to reveal information about its own prompt. Queries that combined a benign question with a poisoned context document. We landed 67 adversarial cases by the end of week two.

The fourth source was the team’s own intuition about what should never regress. We wrote 39 cases that captured the behaviours the team considered load-bearing — the cases where, if the assistant got it wrong, the product’s value proposition would be in question.

By the end of week three the bench had 487 cases. The pass rate against the production assistant was 81%, which roughly matched the 30-day sample we had run during the audit. We had a baseline.

Wiring it into CI

Weeks four and five were CI. The bench ran on every change to the prompt, every change to the tool definitions, and every change to the retrieval logic. The CI job took twelve minutes to run end to end and cost about $1.40 in inference per run. We wrote a guard that blocked merges if the pass rate dropped by more than 1.5% from the baseline, or if any of the regression cases (the ones we had marked as critical) failed.

We also wired in two budget gates. The cost-per-request gate failed if the bench’s average inference cost exceeded $0.014, the existing baseline. The latency gate failed if the bench’s p95 end-to-end exceeded 2.4 seconds, against an existing p95 of 2.3 seconds. Both gates were tight on purpose. We wanted any change that drifted these to be visible in the PR, not a month later in the invoice.

The first PR that hit the new CI was a prompt revision a team engineer had been wanting to ship for two weeks. The bench passed. The cost budget passed. The latency budget did not — the new prompt was 280 tokens longer than the old one and pushed the p95 to 2.51 seconds. The engineer trimmed the prompt by 220 tokens, regained the latency budget, and the PR landed. Before the bench, the prompt would have shipped, and the latency drift would have been invisible until customer support started reporting that the assistant felt slower.

The model gateway and the cost cut

Weeks five through seven was the gateway. The team had been calling the LLM provider’s API directly from the assistant’s request handler. We built a thin gateway service that exposed a stable internal API and translated to the provider’s API beneath. The application code changed in one place — the request handler started calling the gateway instead.

The gateway gave us three things. It made model selection a configuration change. It made the retry logic, the rate limiting, and the cost accounting visible in one place. And it gave us the ability to swap providers — or run shadow traffic against a different model — without changing application code. (This is the LLM gateway pattern we describe in the service page.)

We used the gateway to swap the model. The production assistant had been running on a higher-tier completion model since launch. We ran the bench against the same model’s mid-tier sibling and against a competitor model in the same price band. The competitor scored 79% on the bench against the production model’s 81%, but the mid-tier sibling of the production model scored 84% — better than what was in production, at a cost-per-request of $0.0083 instead of $0.014. We shipped the swap behind a feature flag and ramped over two weeks. There were no customer complaints, no regressions, and the monthly inference invoice dropped 41% the following month.

That was not the plan. We had expected the bench to catch regressions. We had not expected it to enable a cost cut at the same scale. The team’s CFO sent a thank-you note.

Guardrails

Weeks seven through ten was the guardrails layer. We added three things.

A deterministic input filter that caught known prompt-injection patterns. A PII detection model that ran on the input before the main completion and refused if the input contained customer data that should not have left the customer’s own dataset. And a post-completion check that verified the response did not include content from the system prompt itself (the assistant had been leaking its prompt in about 0.4% of responses, which we had not measured before the bench but caught the moment we wrote a test for it).

Each guardrail had its own evals. The PII detection model had a separate bench of 142 cases — half synthetic, half drawn from anonymized real inputs. The bench tracked both false positives (over-refusing safe queries) and false negatives (failing to catch unsafe ones). The first version of the PII model had an unacceptable false-positive rate at the threshold we wanted, so we replaced it with a smaller specialised classifier that hit the false-positive number we needed.

The post-completion check was the one that produced the most surprises during validation. Two patterns of prompt leakage that the team had not been aware of came out of the bench’s first run. Neither had been visible in customer-facing output before, because the prompt content that leaked was mostly innocuous formatting instructions. The third pattern was less innocuous: a small fraction of responses leaked the names of the internal tools the assistant had access to, which could have been used to fingerprint the system. The post-completion check intercepted the pattern. The team’s security engineer was glad we had built it.

The first big regression caught

It happened in week eleven. The team’s lead engineer was experimenting with a new prompt structure that put the tool-calling rules earlier in the system prompt. Local testing looked great. The PR hit CI. The bench passed at 87% — better than baseline. The cost budget passed. The latency budget passed. The regression cases passed.

Three of the adversarial cases failed.

The change had made the assistant more willing to comply with instructions embedded in user-uploaded data. The bench caught it before it shipped. The engineer redesigned the prompt structure over the next two days, hit the adversarial bench again, and the second version passed everything. The change shipped a week later. The customer who would otherwise have been the first to find that vulnerability in production has, as far as we know, no idea that it had ever been close to shipping.

What we would do differently

We would build the bench in two weeks, not three. The third week was spent expanding the case count past 400, which in retrospect was past the point of marginal value for the first version. The bench did not need to be at 500 cases to start catching real regressions; 300 would have been enough to find the prompt-injection vulnerability and the prompt-leakage rate. The bench grows from real production traffic over time anyway. We have learned to ship the first version smaller and let it scale.

We would also wire the cost gate to use a cheaper sibling of the production model for the routine bench runs, and only run the full production model on PRs marked as release candidates. On this engagement we ran the bench against the production model on every PR, which was fine but more expensive than necessary for prompt-structure iteration. On the next engagement we paid about a third as much in dev-cycle inference cost without losing the regression signal.

Where the assistant sits now

The bench has 1,108 cases at the time of this writing, three months after the engagement ended. The pass rate sits at 87% with the production model and 79% with the cheaper sibling, which is still ahead of where the team was at the start of the engagement. The cost-per-request is holding at $0.0083. The latency p95 is 1.4 seconds. The team has shipped 23 prompt changes since handoff, of which the bench rejected 6 and accepted 17.

If you have an LLM feature in production and the question “how would you know if it regressed tomorrow” does not have a clean answer, we run a 20-minute Architecture Review where we look at your existing instrumentation, your bench (or absence of one), and your cost telemetry, and send back a written diagnosis. No deck. Just a roadmap.