Quality Assurance in the AI Era

>_The build is green, the product is wrong

A passing test suite has always meant something. The code did what we asked. That contract is quietly breaking. When a feature is powered by a language model, the build can be green, every assertion can hold, and the product can still be confidently, fluently wrong. The output is plausible prose, not a stack trace, so nothing turns red.

I have spent most of my career owning quality, and the AI era did not make that job easier. It made it harder. We traded deterministic systems we could pin down for probabilistic ones that behave differently on Tuesday than they did on Monday, on inputs no one ever wrote a test for. Quality did not disappear. It moved, and most teams have not moved with it.

>_The new failure modes

The bugs changed shape. A null pointer is loud and reproducible. The failures that hurt AI products are neither. They are quiet, contextual, and often invisible until a customer screenshots them.

• -Hallucination: the model invents a citation, a policy, or an API that never existed and states it with total confidence.
• -Drift: nothing in your code changed, but the provider updated the model behind the endpoint and last month's behavior is gone.
• -Prompt injection: untrusted text in a document or web page overrides your instructions and turns your assistant against you.
• -Silent regression: a prompt tweak that fixes one case degrades ten others, and no assertion notices because there was never a single right answer.
• -Tone and safety failures: answers that are technically correct but off-brand, badly worded, or unsafe, and that no unit test was ever written to catch.

>_Why traditional QA breaks

Classic QA rests on three assumptions, and generative systems violate all three. First, determinism: same input, same output. Set temperature above zero, or let a provider silently reroute your traffic, and that guarantee is gone. Second, a known-correct answer to assert against. For an open-ended summary or a support reply there is no single golden string, there is a space of acceptable answers and a much larger space of bad ones. Third, that coverage implies confidence. Line coverage tells you which code ran, not whether the model reasoned well.

So the old reflexes fail. assertEquals wants one truth. Snapshot tests scream on every harmless rewording. A coverage number tells you nothing about whether the thing hallucinates. Carry the old playbook into an AI product and you get a suite that is green and useless.

>_From assertions to evaluations

The core shift is from checking outputs to scoring them. Instead of asking whether the result equals an expected value, you ask how good it is on the dimensions you care about. That is what an eval is, and it is now the center of gravity for AI quality. I go deeper on the mechanics in /resources/what-are-ai-evals and /resources/how-to-test-llm-applications, but the shape is simple.

You build a dataset of representative inputs, ideally mined from real traffic and past incidents rather than invented at a desk. You define graders. Some are cheap and deterministic: does the JSON parse, is a banned phrase absent, is the number in range. Others are semantic and need judgment: is the answer faithful to the source, is it relevant, is the tone right. For the judgment calls, a strong model used as a judge against a written rubric scales far better than a human reading every row, as long as you validate that judge against human labels first and keep re-checking it.

Then you run that suite on every prompt change, every model swap, every retrieval tweak, and you track scores over time instead of a single pass or fail. The unit of quality becomes a distribution, not a boolean.

>_Offline evals are not enough

An eval set is a snapshot of the questions you thought to ask. Production is the full distribution, and it moves. Real users phrase things you never imagined, paste in messy documents, and probe edges you never tested. So evaluation cannot stop at the pipeline. It has to run in production too.

That means tracing every request end to end: the prompt, the retrieved context, the model version, the raw output, the token counts, the latency. It means sampling live traffic and scoring it with the same graders you use offline, so quality becomes a monitored metric with a dashboard and an alert instead of a gut feeling. And it means watching for drift: if faithfulness scores slide three weeks running, you want to hear it from your monitoring, not from your churn numbers.

>_Guardrails and adversarial testing

Some failures are not accidents, they are attacks, and some outputs are unacceptable regardless of intent. Those need a layer that does not rely on the model behaving. Treat the model as untrusted and wrap it.

• -Validate at the boundary: schema checks, allow-lists, length and format limits, and a refusal to act on instructions found inside user-supplied content.
• -Red team on purpose: keep a growing suite of injection payloads, jailbreaks, and known-bad prompts, and run it like a security regression test.
• -Separate the trusted system prompt from untrusted data, and never let retrieved text issue commands.
• -Add deterministic circuit breakers: when an output fails a hard check, fall back to a safe default rather than ship a confident mistake.

>_Version everything, roll back fast

In an AI system the prompt is code, the model is a dependency, and the retrieval index is state. All three can change the product's behavior without a single line of your application code changing. If you cannot say which prompt, which model version, and which index produced a given answer, you cannot debug it and you cannot reproduce it.

So treat them like what they are. Version prompts in the repo, under review. Pin model versions and handle upgrades as deliberate migrations, not surprises. Canary a new prompt on a slice of traffic behind your eval and monitoring stack, compare the score distributions, and keep rollback one switch away. The provider will change the model under you eventually. Your job is to notice on your terms, not theirs.

>_Where this leaves quality owners

None of this means QA is dead. It means the opposite. When the system is probabilistic, quality stops being a phase at the end and becomes a continuous discipline that spans product, engineering, and data. Someone has to own the evals, define what good looks like, and hold the line when a shiny demo scores worse than the boring version it wants to replace. That is leadership work, and I get into it in /resources/qa-leadership-principles.

If you are starting from scratch, the first moves are concrete:

• -Instrument first: trace and log real inputs and outputs before you write a single eval.
• -Build a small golden dataset from real traffic and past failures, and grow it every time something breaks.
• -Automate a handful of graders that actually matter and wire them into CI and production alike.
• -Pin and version your models and prompts, and rehearse a rollback before you need one.
• -Assign an owner. Continuous evaluation with no owner decays into a dashboard nobody reads.

>_The whole game now

The teams that win the AI era will not be the ones with the cleverest prompts. They will be the ones who can tell, at any moment and with evidence, whether their system is getting better or quietly getting worse. Build the machinery that answers that question and keep it pointed at production. That is the whole game now.