Quality Control on AI Output
Outcome: a QC layer of free formula checks plus a sampling routine, with a written rule for what happens when a check fires.
- Surface
- App
- Level
- Intermediate
- Uses
- Formula columns
- Credits
- 0
- Prerequisite
- Lessons 11–15
Two layers
Automatic — free formula checks that run on every row and catch shape problems: wrong format, missing evidence, banned phrases, out-of-set values.
Manual — a small sample read by a human, catching what formulas cannot: tone, factual wrongness, and the sentence that is technically valid and obviously machine-written.
You need both. Automatic checks scale and catch the mechanical failures; sampling catches the ones that matter most and cannot be expressed as a rule.
The automatic checks
| Check | Applies to | Fails when |
|---|---|---|
| Value in allowed set | Classification columns | A new value appeared |
| Not empty | Every AI column | Blank output — a failure, not an answer |
| Evidence present | Research verdicts | A verdict with no supporting quote |
| Length in range | Generated copy | Truncated, or a paragraph |
| No banned phrase | Generated copy | Prompt drift |
| Contains an evidence token | Generated copy | The line ignored its evidence |
Not SKIP / not Unknown | Export view | Rows that abstained |
| Verified email | Export view | Unverified contact data |
All free, all formula columns, all computed on every run. Build them once.
Run-level monitors
Beyond per-row checks, four numbers per run tell you whether anything has drifted:
Distinct value count per classification column
Should equal your written value set exactly.
Unknown rate per research column
Compared to the previous run. A jump means the source or the input changed.
SKIP rate on the copy column
Compared to previous. A jump means evidence quality fell — usually a targeting change.
Pass rate through the QC filter
The single headline number. Track it run over run.
A change in any of these is the earliest available warning that something upstream moved.
The manual sample
Twenty rows, read properly, on every run that will actually be sent.
| What you are looking for | Not catchable by formula |
|---|---|
| Is the fact true? | Formulas check presence, not truth |
| Does it sound like a person? | Fluency is not humanity |
| Is the tone right for this audience? | Context-dependent |
| Would you send this to someone you know? | The only test that really matters |
That last question is the one to actually ask yourself. It filters out more bad copy than every automatic check combined.
Sample the extremes, not the middle. Ten rows from the top of your score and ten from just above the gate. The top rows are the ones a human will work personally; the marginal rows are where quality problems concentrate.
When a check fires
Write the escalation rule down, because in the moment there is always pressure to ship anyway.
| Failure rate | Action |
|---|---|
| Under 5% | Exclude the failing rows. Ship the rest. |
| 5–15% | Exclude, ship, and investigate before the next run |
| Over 15% | Stop. Something upstream changed. Do not ship. |
| Any factual error found in the sample | Stop. Re-check the whole column against a labelled set. |
The last row has no threshold on purpose. One wrong fact found in a sample of twenty implies a rate you cannot ship.
Do this now
Build the eight automatic checks
As formula columns. One-time work.
Add a single qc_pass column
All checks combined. This is what the export view filters on.
Build the four run-level monitors
Distinct values, unknown rate, SKIP rate, pass rate.
Read twenty rows from the extremes
Ten top-scored, ten marginal.
Write the escalation rule
And put it where whoever runs the pipeline will see it.
Log the pass rate
Every run, in the same place. The trend is the signal.
Check your work
- Every AI column has at least one automatic check
- A single
qc_passcolumn gates the export - The four run-level monitors compute automatically
- You read twenty rows from the extremes
- The escalation rule is written down
Where this breaks
Automatic checks give false confidence about the failure they cannot see. Every check can pass on a line that references an event that never happened — correct length, no banned phrase, evidence token present, and completely untrue. Only a human reading the source catches that, which is why the sample is not optional and why “the checks passed” is not a reason to skip it.
Further automation
Log the QC pass rate alongside credits spent and rows produced. A falling pass rate with steady spend is the clearest possible signal that targeting or source quality has drifted — and it shows up weeks before reply rates do.
Next lesson
17 — Warm leads to Instantly, end to end, the first of the three build modules, where everything so far runs as one MCP chain.
Reference for this lesson: AI agents, Tables, Export, Outreach integrations.