Building a QA Strategy for AI Products

>_A pile of evals is not a strategy

Most teams shipping AI features already have some evals. A notebook here, a golden dataset there, a judge prompt someone wrote in a hurry. What they do not have is a strategy: a decision, made on purpose, about how much quality each feature needs, how they will prove it before release, and how they will keep proving it after. Without that, evals become a ritual that runs when someone remembers and blocks nothing.

A QA strategy for AI products is the connective tissue between the concepts in /resources/quality-assurance-in-the-ai-era and the tooling. It answers who owns quality, what bar each feature must clear, and what happens the day production starts lying. This is the blueprint I use, in the order I build it.

>_Map the blast radius first

Start with a question no framework answers: what is the worst thing this feature can do? A model that suggests a playlist and one that summarizes a discharge note can fail the same technical way, a hallucination, but the consequences are not comparable. Strategy begins by naming that gap explicitly, feature by feature, before anyone writes a grader.

For each AI capability, list the failure modes and rank them by severity, the damage one bad output can do, and likelihood, how often the model produces it, the way an FMEA does for hardware. Weighted by blast radius, they tell you where to spend: a rare but catastrophic failure outranks a frequent but harmless one.

• -Tier 1, unacceptable: physical, legal, financial, or safety harm. A wrong dosage, a fabricated citation, leaked personal data. One instance is a company event.
• -Tier 2, damaging: brand and trust harm. Confidently wrong answers, off-policy promises, offensive tone. Survivable in ones, corrosive in bulk.
• -Tier 3, annoying: unhelpful, verbose, or clumsy output that wastes a user's time but misleads no one.

>_Turn each risk into a number

A quality bar that lives in someone's head is not a bar. Once you have tiers, convert them into measurable thresholds a build can check. Vague goals like 'be accurate' cannot gate a release. 'Faithfulness at or above 0.97 on the Tier 1 dataset, measured over at least 300 sampled cases' can.

Resist the urge to set one universal bar. The whole point of risk mapping is that a Tier 1 medical summarizer and a Tier 3 brainstorming helper should not answer to the same threshold. Over-constrain a low-risk feature and you waste effort; under-constrain a high-risk one and you end up in the postmortem. Each bar names four things: the metric, the target, the minimum sample size behind it, and the owner allowed to move it.

>_Build the harness once, reuse it everywhere

The offline eval harness measures a candidate against its bar without touching a single user, and I cover its mechanics in /resources/how-to-test-llm-applications: a golden dataset mined from real traffic and past incidents, a ladder of graders from exact match to an LLM judge, and a baseline for the current production version. Strategy is less about which grader you pick and more about making the harness a shared, permanent asset instead of a one-off notebook.

Two rules keep it honest. First, the dataset is versioned in the repo and grows every time something breaks, so it never silently loses a case. Second, the same harness runs in three places with the same graders: a developer's machine, CI, and sampled production traffic. When the offline number and the production number come from identical code, the gap between them becomes one of your most useful signals.

>_Gate on evidence, not on confidence

A pre-release gate is the strategy's moment of teeth. Nothing that changes model behavior, and that means a prompt edit, a model swap, or a retrieval-index rebuild, ships until it clears the gate for its tier. The gate is not a person nodding in a review; it is automated checks with recorded results.

• -Delta against baseline: aggregate score at or above the production baseline, within a tolerance wider than the run-to-run noise. Block on regression, not one failed row.
• -Bar for the tier: the feature clears the absolute threshold its tier demands, not just parity with a weak baseline.
• -Red-team pass: the standing suite of injection payloads and known-bad prompts runs clean, like a security regression.
• -Budget check: p95 latency and per-request cost stay inside their limits; a nine-second correct answer is still a regression.
• -Rollback rehearsed: the previous prompt and pinned model are one switch away, and someone has pulled that switch in staging.

>_Ship in slices, not all at once

Passing offline does not mean safe in production; your eval set is only the questions you thought to ask. So no high-tier change goes to a hundred percent of traffic on day one. You earn it in stages, watching live quality metrics and keeping the ability to stop.

• -Shadow mode: run the new version beside the old on real traffic, serve only the old answer, score both offline. Zero user risk, maximum signal.
• -Canary: route one to five percent of traffic to the new version and watch live graders and business metrics, not just latency, against a control.
• -Ramp: widen exposure in steps only while metrics hold, pausing long enough at each level to catch the slow failures a tiny canary missed.
• -Holdback: keep a small permanent control on the old version at full rollout, so you can measure what the change actually bought.

>_Close the loop back to the dataset

The strategy is not finished at rollout. That is where the longest phase begins. The provider will change the model under you, users will phrase things no one imagined, and the retrieval index will drift as content ages. You find out on your terms only if production is instrumented as carefully as your tests.

Trace every request end to end: the prompt, the retrieved context, the model version, the raw output, tokens, and latency. Sample live traffic and score it with the offline graders so quality becomes a monitored metric with an alert, not a gut feeling. Then close the loop, the part most teams skip: every low-scoring case, every thumbs-down, every complaint is routed back into the golden dataset as a new row. Production failures become tomorrow's regression tests, and the harness gets sharper exactly where reality proved it was blind.

>_Rehearse the 3am hallucination

AI incidents do not look like outages. The service is up, the dashboards are green, and the product is confidently wrong in a way a user's screenshot surfaces before your monitoring does. That failure shape needs its own runbook, written before you need it.

Detection comes from the production scoring and drift alerts you already built, plus a fast, obvious path for users to report a bad answer. Containment is a deterministic kill switch: fail closed to a safe default or roll back to the last known-good prompt and pinned model, no debugging required. Only then do you diagnose. Exactly three things change an AI system's behavior: the prompt, the model version, and the retrieval index. Version all three and the culprit is usually obvious within minutes.

Then make the next incident less likely: add the failing case to the golden dataset before you ship the fix, so the bug can never silently return, and run a blameless postmortem that asks why the gate let it through rather than who wrote the prompt. Every incident should leave the harness stronger than it found it.

>_The strategy is the product

None of this is a phase you finish. Risk maps get revised, bars get raised, the dataset grows, and the loop from production back to eval never closes. A QA strategy for an AI product is not a document you write once and file. It is a living system that answers one question on demand: is today's version better or worse than yesterday's, and can you prove it.

If you build only one piece this quarter, build the loop from a real production failure back into a gated eval, because that circuit turns every mistake into permanent progress instead of a recurring surprise. Start with your highest-tier feature, map its blast radius this week, and put one number in front of its next release. The rest of the blueprint compounds from there.