What Are AI Evals?

>_Your demo is not evidence

Every AI feature starts with a demo that works. You type an input, the model produces something impressive, the room nods, and the feature ships on the strength of three lucky examples. Then real traffic arrives, the failures start, and nobody can say whether the fix you pushed on Friday helped or just moved the breakage somewhere you were not looking. An eval is what you build so you never have to guess about that again.

An eval is a repeatable measurement of how well an AI system does a job. At its simplest it is two things: a dataset of representative inputs, and a way to score the outputs they produce. Run the dataset, score the results, and you get a number. Change a prompt, a model version, or a retrieval step, run it again, and watch the number move. That number is the line between engineering and vibes.

>_A definition that survives contact

The word 'eval' gets stretched to cover everything from a spreadsheet of hand-checked answers to a full continuous-integration harness. The definition worth holding onto has three parts: a set of inputs you care about, a scorer that turns each output into a judgment, and an aggregate metric you track over time. Drop any one and you have something weaker. Inputs without a scorer is a demo. A scorer without a fixed dataset is a spot check. A metric with no history is a vanity number.

Think of it as the test suite for software that never returns the same answer twice. A unit test asserts that a function returns exactly what you expect. An eval accepts that the output will vary and asks a different question: across a representative sample, how often is the behavior good enough, and how bad are the misses when they land. That move, from a boolean to a distribution, is the mental shift at the heart of /resources/what-is-ai-engineering.

>_Offline and online

Evals live in two places and you need both. Offline evals run before you ship, against a fixed dataset you control, in CI. They tell you whether a change made the system better or worse with the model held still and the inputs frozen. Online evals run in production against real traffic you cannot predict, and answer what the offline set never can: is the system actually working for real users, right now, on inputs nobody thought to write down.

• -Offline: fast, cheap, repeatable, safe to run on every commit. This is your regression gate and your development inner loop.
• -Online: measures live quality on real distributions and catches the drift a frozen dataset cannot see.
• -Online signals worth capturing: thumbs up and down, user edits and retries, escalation to a human, task completion, and guardrail trip rates.
• -The two form a pump: online surfaces the surprises, offline freezes them so they can never surprise you twice.

>_Three ways to grade an answer

Every eval needs a scorer, and there are three kinds. Serious suites use all three, matched to what each piece of the output can bear; picking the wrong one is the most common way an eval ends up slow, expensive, or quietly meaningless. The finer mechanics of each I work through in /resources/how-to-test-llm-applications.

• -Programmatic: plain code checks the output with exact match, regex, JSON schema validation, enum membership, or numeric tolerance. Free, instant, perfectly repeatable. Use it anywhere you can force structure onto the output, which is more often than people assume.
• -Model-graded: a second model scores the output against a rubric you write, for qualities like faithfulness, tone, or completeness. This is LLM-as-judge, the only practical way to grade open-ended text at scale. It is also a nondeterministic system judging a nondeterministic one, so calibrate it against human labels before you trust it.
• -Human: a person reads the output and judges it. Highest fidelity, highest cost, and the ground truth the other two only approximate. Spend it where it counts: building the dataset, calibrating the judge, and reviewing the cases that carry real risk.

>_Build the dataset first

The dataset is the highest-leverage artifact in the whole effort and the one people most want to skip. A great scorer over a lazy dataset tells you nothing; a decent scorer over a well-chosen dataset tells you almost everything. Do not invent examples at your desk, because desk examples are always too clean and too kind. Mine them from reality: production traces, the failures your support queue forwards, and the adversarial inputs you know are coming.

Version the set in git next to the code, label every case with why it exists, and treat each incident as a permanent new entry. Cover a spread, not just the happy path.

• -Happy path: the ordinary requests that make up the bulk of real traffic.
• -Edge cases: empty input, oversized input, mixed languages, genuinely ambiguous asks.
• -Adversarial and safety: prompt injection, attempts to leak the system prompt, jailbreaks.
• -Regression cases: every past bug frozen as a case so it can never quietly return.
• -A held-out slice you never tune against, so you can detect overfitting to the eval itself.

>_Measure the right things

A single blended score hides more than it shows. If your eval collapses everything into one quality percentage, a gain on easy cases will mask a regression on the hard ones and you will ship it. Slice the metric by category and report per slice, so a two percent overall rise cannot bury a collapse on the safety set.

Then pick metrics that map to real failure: pass rate against a threshold for the general set, groundedness for anything retrieval-backed, refusal accuracy for safety, exact-match rate for structured output, and cost and latency attached to every case. And treat the result as a measurement, not a verdict. You are estimating a rate over a sample, so it comes with error bars: fifty cases is enough to feel a trend and far too few to trust two decimals. Prefer binary judgments over one-to-ten scales, and watch the direction over many runs rather than any single value.

>_Do not overfit the eval

The moment a number becomes a target, people optimize the number instead of the product. Goodhart's law is not a footnote here, it is the central hazard of running evals. Tune prompts long enough against the same fixed set and you will climb the score while the real system stagnates, because you have taught it to pass your test rather than do the job.

The defenses are simple and you actually have to run them. Keep a held-out slice you never look at while tuning and check it only occasionally, so a gap between it and your working set exposes the overfit. Rotate and grow the dataset so it never hardens into a fixed puzzle. Distrust any change that helps the eval but that you cannot explain in terms of user-visible behavior. And never let one model both generate and be the sole grader of its own output, or you are marking your own homework.

>_Close the loop

An eval is not a document you write once and file away. It is a loop. A failure appears in production, you capture the offending input, add it to the dataset with the correct expectation, fix the system, and the gate ensures that exact failure can never ship again. Every incident makes the eval stronger, and a stronger eval makes the next incident less likely. That flywheel is the whole point.

So start small and start now. Fifty real cases, three programmatic checks, one calibrated judge, and a number on a dashboard beat a grand framework you never finish. Wire it into CI, gate on the aggregate, and grow the set from every failure. Do that and quality stops being a feeling you defend in review and becomes a line you can watch. When the model underneath you can change overnight without asking, that line is the only thing between you and shipping blind.