Evals are product analytics for probabilistic software

A traditional product ships on a loop you already know: build, release, look at analytics, iterate. An AI product needs a different one:

define behavior, evaluate, release, trace, analyse errors, evaluate again, iterate.

The new steps are define behavior and evaluate, and they come before release. With AI products, "what does good look like?" has to become an engineering artifact. Evals are how that artifact is written down.

What teams are actually doing

LangChain's State of Agent Engineering surveyed 1,340 people between November and December 2025. A few numbers stand out:

• 57.3% said they already had agents in production.
• Quality was the most-cited barrier, at 32%, ahead of latency at 20%.
• 89% had implemented some form of observability, and 62% had detailed tracing.
• But only 52.4% ran offline evaluations, and 37.3% ran online evaluations.

The survey comes from a vendor in this ecosystem, so read it as directional evidence rather than a census. Still, the gap tells a story: teams can see what their agents do much more easily than they can judge it. LangChain defines agent engineering as the iterative process of harnessing LLMs into reliable systems. Without evals, the "reliable" part is a hope.

Model evals versus product evals

The most useful framing I have read is in AI Evals: Everything You Need to Know by Hamel Husain and Shreya Shankar (September 2026).

Model benchmarks compare general-purpose models on shared tasks. They help you shortlist a model, but they say little about your application. A high score on a hard science benchmark tells you almost nothing about whether your order-cancellation agent picks the right order.

Product evals measure whether your specific system meets user and business needs, and "your system" means the whole stack: the model, prompts, retrieval, tools and application code. A bug in retrieval or in a tool wrapper is an eval failure even if the model is perfect.

Start with error analysis, not infrastructure

The guide's strongest claim is that error analysis is the most important activity in evals. The process is refreshingly low-tech:

1. Collect a representative set of real interactions.
2. Open coding: read them and write free-form notes about what went wrong.
3. Axial coding: group those notes into a taxonomy of failure modes.
4. Iterate, using an agent to help once you understand the failures yourself.

They recommend reading at least 30 traces personally before asking an agent for suggestions. The target is to avoid what they call eval-driven development, writing tests for failures you have imagined instead of ones you have observed. Your metrics should reflect failures that matter to users and the business, and you only discover those by reading the data.

Concrete rules that save time

From the same guide:

• Prefer binary pass/fail over 1–5 scores. Numeric scales produce inconsistent judgments and hide uncertainty.
• One domain expert reviewing 20–50 outputs in a notebook usually beats buying tools too early.
• Validate any LLM-as-judge against human labels. They suggest labeling roughly 100–200 examples per failure mode, splitting into training, development and test sets, and measuring true positive and true negative rates.
• Don't outsource the core analysis. Outside teams lack the tacit domain knowledge needed to spot the failures that matter.

A minimal loop for a small project

Say you run a retrieval-augmented chatbot (I wrote about the architecture in The evolution of RAG). A realistic loop for one person:

1. Log every question, retrieved chunks, tool calls and answer.
2. Each week, read 30 recent conversations and write one-line notes about every failure.
3. Group the notes. Typical buckets: wrong document retrieved, correct document but wrong answer, refusal when it should answer, answer when it should refuse.
4. For the biggest bucket, write 10–20 test cases with a pass/fail check.
5. Fix the system, rerun the cases, and keep them as a regression suite.

After a month you have something rare: a written definition of "good" for your product, backed by examples.

Why this is product analytics in disguise

In ordinary products, analytics tell you whether users behaved the way you hoped. In probabilistic software, the system's own behavior is the uncertain part, so you need the equivalent measurement on the model side too. Tracing shows you what happened. Evals tell you whether it was acceptable. Together they close the loop that the earlier posts in this series open: framing the problem and owning the result.

One caution

Evals are not a guarantee. They test the failures you have found so far, and they can be gamed or become stale. Treat them as a living artifact: when production surprises you, the surprise becomes a new test case.