01Case studies
Anonymized engagement writeups.
What we were asked to do. How we approached it. The outcomes we can share with client permission. Numbers are real, identifying details stripped.
- SaaS / AI products Confidential B2B SaaS analytics product May 14, 2026
Building an eval bench around an LLM assistant that had shipped without one, and what we found.
An AI assistant inside a B2B SaaS analytics product had shipped to production without an eval bench, without observability, and without cost telemetry beyond the monthly invoice. The team noticed when a customer-success leader saw a confidently wrong answer and walked into the engineering lead's office. We built the bench in three weeks, wired it into CI, added a model gateway, and rolled out guardrails. The bench caught two prior regressions the team had been living with, found three more during the engagement, and cut inference cost by 41% in the same quarter.
- Eval bench grown from 0 to 1,108 curated cases across twelve weeks, sourced from CS team interviews, production sampling, and adversarial inputs
- Seven regressions caught in CI before reaching production in the first 90 days
- Inference cost per request: $0.014 → $0.0083, a 41% reduction enabled by bench-driven model selection
- End-to-end p95 latency: 2.3s → 1.4s without changing the underlying model
- FinTech / Payments Confidential payments operator May 14, 2026
Closing a client-side audit gap on a live payments console, before the regulator found it.
A live payments console had grown over four years into a system where the compliance team could not reliably answer the question a regulator was about to ask. The client-side audit instrumentation was partial, inconsistent across screens, and shipped through the same channel as the API. We rebuilt the audit layer in place, in parallel with the existing system, without disturbing the operations team's daily work. Cutover happened on a Tuesday. The first regulator query landed three months later. The new evidence answered it in four hours.
- Client-side audit coverage across six regulated surfaces: 62% → 100% of audit-sensitive actions instrumented
- First post-launch regulator query answered in under four hours, against an internal estimate of six weeks if the rebuild had not happened
- Zero lost audit events across nine recorded network drops in the first quarter, courtesy of the durable client-side queue
- Time-to-evidence for compliance queries: down from a quarter-long investigation to an SQL filter against the new audit store
02. Newsletter
More on engineering for serious teams, monthly.
Practitioner notes on the work behind real FinTech systems. One short essay or case study a month. No marketing, no sequences. Reply to talk.
Prefer RSS? Subscribe to the feed →