What Confidence Threshold Should You Use to Review AI Answers?
There is no confidence number that works for every list. Check 50 answers by hand, find the confidence level where the wrong ones pile up, and make that your review line. Below it, a human looks. At or above it, rows go through.
Updated October 13, 2026.
A cutoff copied from a blog post is a guess. Research on confidence-gated AI says the useful threshold depends on the confidence signal, the model, the task, the population and the cost of an error. Your list and your question decide the line, so you measure it.
Pick the line from a 50-row sample, not from a number you read somewhere
Here is the whole method. Example: say you ran the column-header question "Is this a B2B SaaS company?" on a 5,000-row export, and Columns put a confidence number next to every answer.
- Add a random-number column and sort by it, or pull rows by hand. You want 50 rows spread across the full confidence range, about 10 from each fifth of the list sorted by confidence. Do not take the first 50 rows of the export. Exports are often grouped, so the first 50 may all look alike.
- Add a column called "Human check". Open the site, profile or record for each sampled row and mark it Right or Wrong.
- Sort the 50 by confidence, high to low.
- Find where the first Wrong shows up and where Wrongs start to come often. That is your review line, or just above it.
- Filter the full list. Below the line goes to a human. At or above goes through.
Write your tolerance down before you look at the sample. Something like "no more than 1 wrong in 20 accepted rows". If you set it after you see the results, you are tuning the rule to the sample and it stops being a test.
Copy this band table to see whether the score separates good from bad
Group the 50 checked rows into confidence bands and count how many were right in each. A well-calibrated score tracks accuracy: rows at about 80 percent confidence should be right about 80 percent of the time. Gaps show where a score is overconfident or underconfident.
Example (illustrative numbers):
| Confidence band | Rows checked | Right | Share right |
|---|---|---|---|
| 0.90 and up | 20 | 19 | 95 percent |
| 0.70 to 0.89 | 15 | 11 | 73 percent |
| Below 0.70 | 15 | 7 | 47 percent |
Reading it: accept 0.90 and up, send everything else to a human. Accuracy falls as confidence falls, so the score is doing its job for this question.
To build it in a sheet, put confidence in column B and your Right/Wrong marks in column C, rows 2 to 51. The top band is:
=COUNTIFS(B2:B51,">=0.9",C2:C51,"Right")/COUNTIFS(B2:B51,">=0.9")
The middle band adds a second condition, B2:B51,"<0.9", and ">=0.7" in place of ">=0.9". Copy the pattern for the bottom band.
If wrong answers show up at every confidence level, do not set a line
Say the 0.90 rows are right about as often as the 0.70 rows. Then the score tells you nothing for this question, and any cutoff you pick is decoration. The fix is not a higher number. Rewrite the column header into one narrower yes/no fact and run the sample again. "Does the site mention a free trial?" is easier to answer and easier to check than "Is this a good fit for us?".
Pick the line by what is scarce: your review time or your tolerance for errors
Raising the line sends fewer wrong answers through and more rows to a human. Higher thresholds reduce errors but increase manual review, and lower thresholds do the opposite. So decide which one drives the line before you start.
| If this is true | Do this |
|---|---|
| A wrong row costs more than a human look (outreach to the wrong account, a client deliverable) | Set the line from the error rate you will accept. Raise it until the error rate above it is tolerable. |
| A human look costs more than a wrong row (a rough first sort, a list you will filter again) | Set the line from the rows you can review. Lower it. |
| Your review capacity is fixed (say one VA handles 300 rows a day) | Set the line so the rows below it fit that capacity, then read the error rate above it from your sample. |
If the error rate above a capacity-driven line is too high, a higher number will not save you, because you cannot review the extra rows. Narrow the question or add capacity.
Two sums to repeat with your own list
Rows to review = total rows x share below the line.
Wrong answers let through = rows above the line x error rate above the line.
Example (illustrative): 5,000 rows, and 60 percent of them sit in the top band. That is 2,000 rows to review and 3,000 above the line. At a 5 percent error rate above the line, about 150 wrong answers go through. Raise the line until the error rate above it drops to 2 percent and about 60 go through, but more rows land in review. You are trading human hours for bad rows, and the sums show the price of each.
"50 out of 50 right" does not mean zero errors
Fifty rows is enough to find where wrong answers cluster. It is not enough to prove a precise error rate. If a classifier gets all 50 test examples right, the true error rate at the 95 percent confidence level could still be anywhere up to about 8 percent. On a 5,000-row list, that is hundreds of wrong rows going through.
So use the 50 rows to place the line and a second check to confirm it:
- Write the pass rule in the sheet first: "Accept above 0.85 if no more than 3 of 100 checked rows are wrong."
- Pull 100 more rows from just above the line.
- Mark them Right or Wrong and apply the rule.
Example (illustrative): you want to be fairly sure the error rate above your line is under 3 percent. A common shortcut, the rule of three, says that if 0 of n checked rows are wrong, the true error rate is plausibly up to about 3 divided by n. For 100 rows, that is 3 percent. For 300 rows, 1 percent. If 3 of your 100 rows are wrong, the line is too low. Move it up and check again.
If you want a tighter margin, the standard formula for sizing a sample for a proportion uses the expected rate, the confidence level and the margin of error you accept. Expect 5 percent errors and want plus or minus 3 percent at 95 percent confidence: 1.96 x 1.96 x 0.05 x 0.95 / 0.03 / 0.03 comes to about 203 rows. For most lead lists that is overkill. Spend it when a wrong answer is expensive.
Five mistakes that quietly wreck the line, and what each costs
| Mistake | What it costs you |
|---|---|
| Copying a fixed cutoff like 0.8 from a blog | Wrong rows slip through, or too many right rows go to review. The right line depends on your question and list. |
| Sampling the first 50 rows | The line comes from an unrepresentative slice, and the whole list inherits the error. |
| Checking only the low-confidence rows | You never learn whether the top rows are right, so the line is a guess. |
| Pooling every question into one accuracy number | A strong column hides a weak one, and the weak column's errors reach the list unchecked. |
| Setting the line once and never re-checking | Scores need calibration and monitoring on the current population, task and model version, so last month's line can drift. |
Keep one line per column header. A yes/no question about a website and a fuzzy fit judgment have different error patterns, so give each its own 50-row check. Measuring accuracy by field, not only in aggregate is the same advice from the document-processing world.
Run the review queue lowest confidence first
Sort the rows below the line by confidence, lowest first. Your reviewer then sees the least certain rows before they run out of time, instead of burning the afternoon on easy ones.
Also keep a small spot-check above the line every run: a few rows from every band, including the top one. High confidence is not "always right". The spot-check is how you notice when the score starts to drift. Re-run the full 50-row check whenever you change the question, the list source or the model.
If you want the background on spotting bad answers, read how to know which AI answers in your spreadsheet are wrong. For what the confidence column is, see the guide to confidence scores for each row.
Your next ten minutes
Open the last list you ran through an AI column. Add a "Human check" column and a random-number column, sort by the random numbers, and mark the first 10 rows Right or Wrong. That is one fifth of the sample. Tonight, do the other four fifths, and write your tolerance in one line at the top of the sheet before you look at the results.
Set your own line on a real list
The confidence number is what makes this method possible: every answer carries one, so you can sort, sample and draw a line. You start with 500 row-questions on the free plan, no card, which is plenty for a 50-row check on your own export. Start free, no card.
Frequently asked questions
- Is 0.9 a good confidence threshold?
- It might be, but only your sample can tell you. Check 50 rows by hand and see whether answers at 0.9 and up are right as often as you need. If they are not, raise the line or narrow the question.
- How many rows should I check by hand?
- Check 50 random rows to place the line, spread across the full confidence range. Then check 100 more rows just above the line to confirm it against a pass rule you wrote beforehand.
- If every row I checked was right, can I trust the rest?
- Not fully. With 50 of 50 right, the true error rate could still be up to about 8 percent. A second check of 100 rows above the line gives you a tighter answer.
- Should I review the high-confidence rows too?
- Review them in a small spot-check, not in full. Keep a few rows from every band, including the top one, so you notice if the score starts to drift.
- Do I need a separate threshold for each question?
- Yes. A yes/no question about a website and a fuzzy fit judgment have different error patterns, so check and set a line for each column header.
- What if wrong answers appear at every confidence level?
- Then the score is not separating good from bad for that question, so do not set a cutoff. Rewrite the header as one narrower yes/no fact and run the sample again.