QA Leadership Principles

>_You cannot inspect quality in at the end

For decades the QA org sat at the end of the line. Code came in, we poked at it, we filed bugs, we signed off. It was never glamorous, but it was legible. You knew what done looked like, you knew what a bug was, and you could argue about severity with a straight face. AI features broke that arrangement, and a lot of QA leaders are still running a process that no longer describes their product.

When the thing under test is a language model, there is no last mile to inspect. The correct answer is a range rather than a value, and the system you signed off on Friday can drift by Monday because a provider retuned a model you never touched. Leading quality there is not a scaled-up version of the old job. It is a different job, and these are the principles I keep coming back to.

>_Everyone owns quality. You own the definition.

The first reflex under pressure is to grow the QA team so it can catch more of the model's mistakes. It does not scale, and it teaches engineers that quality is somebody else's problem. In an AI product the people writing prompts and tuning retrieval are the ones deciding how it behaves. Quality has to live with them, not in a gate they throw work over.

So the leadership move is to distribute the doing and centralize the standard. Every engineer writes evals for the behavior they ship, the same way they write unit tests. What you own, and cannot delegate, is the definition of good: the rubric, the bar a change must clear, and the authority to block a release that scores below it. When a shiny demo tests worse than the boring version it wants to replace, someone has to say so out loud and mean it. That is the job. I covered the discipline side in /resources/quality-assurance-in-the-ai-era; this essay is about leading the people who do it.

>_Define done before anyone writes a prompt

The fastest way to ship an unreliable AI feature is to start building before anyone agreed what done means. With deterministic software the spec is often implicit in the acceptance criteria. With a probabilistic feature, if you do not write the bar down, it becomes whatever the demo happened to do the day someone showed it to a VP.

I make teams answer a short list before they build. Not a document nobody reads, a working agreement that turns straight into the eval set:

• -What is the acceptable answer space, and what are the answers we will never tolerate no matter how rare?
• -What is the target score on each quality dimension, and who signs off if we ship below it?
• -What is the ground truth, where does it come from, and who owns keeping it current?
• -What happens when the model fails: retry, fall back to a safe default, or hand off to a human?
• -What latency and cost budget does the feature have to live inside while it does all of the above?

>_Spend your attention where wrong answers hurt

You will never test every input to a model, so stop pretending coverage is the goal. The useful question is blast radius: when this feature is wrong, who gets hurt and how badly. A model that miswords a marketing subject line and a model that miscomputes a medication interaction are not the same risk, and they should not get the same slice of your team's attention.

I triage features into rough tiers. High-stakes surfaces, anything touching money, health, legal, or safety, get the deepest eval sets, a human in the loop, hard deterministic guardrails, and the tightest rollback. Low-stakes surfaces get lightweight checks and monitoring, and I accept more variance there. This is unpopular with people who want one standard applied evenly, but even scrutiny on uneven risk is waste on one end and exposure on the other. Concentrate the expensive attention where a confident mistake actually costs something.

>_Make evals as normal as tests

Every team says it values evals. Most treat them as a project someone will get to. The difference between the teams that ship reliable AI and the teams that ship demos is whether writing an eval is a reflex or an event. Your job as a leader is to make it a reflex, and that is culture work, not tooling work.

Concretely: a prompt change does not merge without an eval, the same way code does not merge without a test. Every production incident ends with a new row in the golden dataset so that failure can never silently return. The suite runs in CI and on live traffic, and its results are visible to the whole team, not buried in a notebook on someone's laptop. I go deeper on the mechanics in /resources/what-are-ai-evals, but the leadership point is simpler: what you measure in public and reward becomes normal. Celebrate the engineer who caught a regression in an eval, not only the one who shipped the flashy feature.

>_Measure the things that predict pain

Probabilistic systems generate infinite metrics, and most of them are decoration. A dashboard full of green numbers that nobody would stake a release on is worse than no dashboard, because it manufactures false confidence. Pick the few metrics that actually predict a bad day and put them where the team sees them daily.

• -Quality score distribution over time, per feature, not a single average that hides a rotting tail.
• -Faithfulness and hallucination rate on the surfaces where a made-up fact is expensive.
• -Escaped defects: failures that reached a real user, because that is the only score customers feel.
• -Drift signals: eval scores trending down with no code change, which usually means the model moved under you.
• -Human override and fallback rates, which quietly tell you how much the model is actually trusted in production.

>_Hire for judgment, not for clicking

The old QA hire executed test plans and found reproducible bugs. That skill still matters, but it is no longer the center. The people who thrive on AI teams look more like analysts and product thinkers than manual testers, and if you keep hiring the old profile you end up with a team that cannot tell you whether the model is any good. What I look for now, and coach my people toward:

• -Comfort with ambiguity: can they define good when there is no single right answer, and defend that definition?
• -Data fluency: can they build a dataset and read a score distribution rather than a comforting average?
• -Adversarial instinct: do they reach for the input that breaks the thing, including prompt injection and jailbreaks?
• -Enough engineering to wire a grader into CI and read a trace, even if they never ship the feature themselves.
• -The spine to hold a quality bar against a deadline, and the judgment to know which battles are worth it.

>_Grow the testers you already have

You do not need to fire your manual testers to get that profile. The best move is usually to grow the people you have. A tester who learns to write evals, read traces, and think in distributions becomes exactly the risk-focused quality partner an AI team needs, and brings product knowledge a new hire would take a year to rebuild.

Pair them with the engineers writing prompts, not in a separate silo. The tester sitting next to the person tuning retrieval finds the failure mode nobody thought to test, and the engineer starts writing better evals by osmosis. That pairing does more for quality than any process document, and it is one of the highest-leverage moves a QA leader can make.

>_Own the question, not just the answer

Strip away the tooling and the job comes down to one thing. At any moment, can you say with evidence whether your AI product is getting better or quietly getting worse, and does your team feel that they own that answer. If yes, you are leading quality. If the only person who can answer is you, and only after an incident, you are still running the old playbook on a new game.

Start small and start now. Take your highest-stakes feature, write down what done means, build a golden dataset from real traffic and past failures, wire one honest metric onto a shared dashboard, and make the next prompt change ship with an eval attached. Do that, hold the bar in public, and the culture follows. That is what leading quality looks like when correct is fuzzy.