How to Test LLM Applications

>_assertEquals gives up here

Every test you trust rests on one assumption: same input, same output. Language models break it on the first call. Set temperature above zero, or let the provider reroute your traffic, and the exact string you asserted against last week is gone. The feature still works. Your test is now lying to you.

Testing an LLM application is not testing harder, it is testing differently. You stop asking whether the output equals a value and start asking whether it is good enough, often enough, and how bad it is when it fails. That shift, from a boolean to a distribution, is the whole discipline. For the grounding, see /resources/quality-assurance-in-the-ai-era and /resources/what-are-ai-evals. This piece is the how.

>_Start with a golden dataset

Nothing works until you have a dataset. A golden dataset pairs representative inputs with what a good response looks like, and it is the most valuable asset in an AI codebase, because every other technique here scores against it.

Do not invent it at a desk; the examples you make up are the ones your system already handles. Mine the real distribution instead: sample production traffic, pull the transcript behind every incident, and add each new failure as a permanent row the moment it is diagnosed. Fifty cases from reality beat five hundred imagined ones. You cannot always store an exact expected answer, so store the next best thing: the facts the reply must contain, the claims it must not make, the tone it should hit. The set should grow every week.

>_Three ways to grade an answer

Once you have inputs, you need graders. There is no single way to score an LLM output, there is a ladder, and mature suites use every rung. Reach for the cheapest one that catches the failure you care about.

• -Exact and structural match: for anything with a right answer, a label, an extracted field, a JSON shape, assert it directly. Deterministic, instant, free. Use it wherever the output space is closed.
• -Semantic similarity: when wording varies but meaning should not, compare embeddings or check for required key facts instead of exact strings. It matches 'the account was closed' against 'we shut it down' without screaming at every reword, but it is weak on negation, so treat its score as a signal.
• -LLM-as-judge: for open-ended quality like faithfulness and tone, have a strong model grade the output against a written rubric. It scales judgment that would otherwise need a human on every row, but it drifts and flatters verbose answers, so validate it against human labels before you trust it.

>_Assert properties, not answers

Some of the strongest tests never look at a golden answer at all. Borrowed from property-based testing, the idea is to assert invariants that hold for any valid output, whatever the wording. You do not need the right answer to know an answer is wrong.

A support reply must never contain another customer's data. A summary must be shorter than its source. A grounded model must cite only the retrieved documents. A router must return an allowed label. None of these care about phrasing, which is why they survive nondeterminism and catch the loud, unacceptable failures no similarity score flags. Better still, the invariant you assert in a test is the one you enforce at the boundary in production. Write it once, run it in both places.

>_Turn every bug into a regression test

In an LLM app, three things change behavior without a line of your code changing: the prompt, the model version, and the retrieval index. A tweak that fixes one complaint quietly degrades ten others. A provider ships a new model overnight and last month's behavior evaporates. Neither turns the build red unless you built it to notice.

A regression suite is your golden dataset plus your graders, run on every one of those changes, with scores tracked over time rather than collapsed into pass or fail. You watch a distribution move: did faithfulness drop after that prompt edit, did the change that raised helpfulness quietly cut safety? The habit that makes it pay off is rarely followed: every confirmed bug becomes a permanent row before it is fixed, so a hallucination you fix once can never silently return.

>_RAG: test retrieval and generation apart

Retrieval-augmented generation has two failure surfaces, and if you only test the final answer you cannot tell which broke. A wrong answer might mean the retriever never fetched the right document, or that it fetched it and the model ignored it. Different bugs, different fixes, so measure them apart.

• -Retrieval quality: given a query, did the right chunks come back? Score it with classic metrics, recall, precision, and hit rate at k, against queries with known-relevant documents. If recall is low, no prompt will save you.
• -Generation faithfulness: given the retrieved context, is the answer supported by it, and only by it? This is where groundedness checks and an LLM judge earn their keep, flagging claims the sources do not back.
• -Answer relevance: does the response address what the user actually asked, not just recite adjacent facts that happened to retrieve well?

>_Agents: grade the trajectory

Agents raise the stakes because they act. Between the request and the answer sits a chain of decisions: which tool to call, with what arguments, how to react when a call fails, when to stop. A correct answer reached through a broken path is a test that passes today and pages you next week.

So test the trajectory, not just the destination. Assert that the agent called the expected tool with valid arguments, recovered from an injected tool failure instead of inventing a result, did not loop, and stopped inside a step budget. Mock the tools so a run is deterministic and cheap and so you can force the failures real dependencies only produce at 3am: a timeout, an empty result, a malformed response. And cap steps and cost hard in the test, or you will discover your API bill instead of a bug.

>_Gate CI on deltas, not perfection

An eval suite that runs only when someone remembers is a chore, not a test. Wire it into CI so no prompt, model, or retrieval change merges without a score. But you cannot gate it like a unit test, because there is no clean pass or fail and the run costs real money and time.

Gate on movement against a baseline, not on absolute perfection:

• -Block the merge if the aggregate score drops more than a set tolerance below the main-branch baseline, not if any single case fails. One flaky row should not stop a good change.
• -Run several samples per case and average them, so temperature noise does not flip the gate. Set the tolerance wider than the run-to-run variance.
• -Split the suite: a fast smoke subset on every pull request, the full dataset nightly and before release. Cache identical calls so reruns are near free.
• -Pin the judge's model version, and never let a judge grade output from its own generator, or your bar drifts with the grader instead of the product.

>_Start smaller than you think

None of this needs a platform or a six-month project. The teams that test LLM applications well did not start with a framework. They started with twenty real examples in a file and one script that scored them. That is enough to catch your next regression, and catching one is enough to justify the next twenty.

So begin there. Pull twenty inputs from real traffic, write three graders you believe, one exact, one property, one judge, and run them on your current prompt to set a baseline. Add the next bug the day it appears, and wire the script into CI the week after. Do that and within a month you will have what most teams shipping AI features still lack: the ability to say, with evidence, whether today's system is better or worse than yesterday's. That is the whole job.