# How to review an AI shortlist without rubber-stamping it

> A person who approves an AI tool's shortlist isn't necessarily judging it. Why review drifts into agreement, what meaningful review looks like, a routine for one job, a sample shortlist with reviewer notes, and what to write down.

Ashley at SortTheStack, November 2, 2026. Original: https://sortthestack.com/blog/review-an-ai-shortlist

**Reviewing an AI shortlist means forming your own view, not agreeing with the
tool's.** In practice that means four things. Decide on a few applications
yourself before you look at the tool's verdicts. Check the evidence behind its
verdicts, not just the labels. Spend your time on the applications where its
mistakes hide. Write down why you decided, especially where you disagreed.
Research on people working with AI recommendations shows that, without
deliberate effort, review tends to drift into agreement.

This article is about the review step itself: what a person does between the
tool producing results and anyone being shortlisted or turned away. For which
steps to automate in the first place, see
[what to automate in hiring](https://sortthestack.com/blog/what-to-automate-in-hiring). For checking a
single result in detail, see
[why an AI tool ranked a candidate](https://sortthestack.com/blog/why-ai-ranked-a-candidate).

## Approving is not judging

A person can be "in the loop" in two very different ways.

| Approving                                  | Judging                                                  |
| ------------------------------------------ | -------------------------------------------------------- |
| Reads the tool's label and summary         | Reads the evidence, and sometimes the application itself |
| Asks "is there a reason to disagree?"      | Asks "what would I have decided?"                        |
| Spends the same few seconds on everyone    | Spends time where the tool is most likely to be wrong    |
| Disagrees rarely, without a written reason | Disagrees when the evidence says so, and records why     |
| Can't say afterwards why someone advanced  | Can say what the decision rested on                      |

Both look the same in a log: a person clicked a button. Only the second one
catches the tool's mistakes.

European data protection regulators draw the same line. Guidance from the
Article 29 Working Party on automated decision-making, later endorsed by the
European Data Protection Board, says an organisation "cannot avoid the
Article 22 provisions by fabricating human involvement". Oversight must be
"meaningful, rather than just a token gesture", carried out by "someone who has
the authority and competence to change the decision", who should "consider all
the relevant data"
([WP251 rev.01](https://ec.europa.eu/newsroom/article29/items/612053)). That is
written for the GDPR, but it's a good test anywhere.

## Why review drifts into agreement

Three findings explain why "a person reviews it" is weaker than it sounds.
None of them studied hiring, so treat them as evidence about how people work
with AI recommendations generally.

**Explanations increase agreement, right or wrong.** In user studies by
[Bansal and colleagues (2021)](https://doi.org/10.1145/3411764.3445717), people
worked with an AI about as accurate as they were. Explanations didn't improve
the team's accuracy. Instead, they "increased the chance that humans will
accept the AI's recommendation, regardless of its correctness." A tidy reason
next to a verdict makes the verdict more persuasive, not more correct.

**Forming your own view first helps, and people dislike it.** In an experiment
with 199 participants,
[Buçinca, Malaya and Gajos (2021)](https://doi.org/10.1145/3449287) found that
people often accepted an AI suggestion even when it was wrong, and that
explanations alone didn't fix this. Designs that made people think before
accepting, such as deciding on their own before seeing the AI's suggestion,
"significantly reduced overreliance". The trade-off was that participants rated
the designs that worked best least favourably. The task was about nutrition,
not hiring.

**Requiring a human can give false comfort.**
[Green (2022)](https://doi.org/10.1016/j.clsr.2022.105681) surveyed 41
government policies that require human oversight of algorithms. He concluded
that evidence suggests people are unable to perform the oversight functions
these policies expect, and that such requirements can provide "a false sense of
security".

Taken together: a review step only works if it's designed to make the reviewer
think. The routine below is our recommendation, built on these findings. It is
not a standard.

## A review routine for one job

### 1. Decide on five applications before you look

Before opening the tool's results, pick five applications at random and read
them against your requirements. For each, write your own call: shortlist,
maybe or no, with one line of reason. This takes about 20 minutes.

Then compare with the tool. Where you agree, move on. Where you disagree, work
out who is right. Sometimes it will be the tool, because it noticed a line you
skimmed past. Sometimes it will be you. Either way, you now know how this tool
reads this job, which is the thing you need to know before trusting the rest.

### 2. Read in full where mistakes hide

A tool's errors aren't spread evenly. Read these applications yourself:

- **Every applicant marked unclear on a must-have.** "The application doesn't
  say" is a question, not an answer.
- **Applicants on the boundary** between two outcomes.
- **A sample of everyone turned away**, whether by the tool or by a knockout
  rule. These are the mistakes nobody would otherwise see. Our
  [screening audit](https://sortthestack.com/blog/audit-resume-screening) shows how to size the sample.

### 3. Check the evidence on the rest

For the applicants the tool put forward, you don't need to re-read everything.
Check the evidence for the requirements that decide the job. Is each quote in
the application? Does it say what the verdict claims? "Supported the month-end
close" is not "ran the month-end close".

### 4. Write down your decision in your own words

For everyone you shortlist or turn away, record the reason, in a sentence, in
terms of your requirements. Copying the tool's summary is not a reason. Where
you disagree with the tool, say so and why. That record is what lets you, or
anyone else, check the decision later. Our guide to
[writing a candidate recommendation](https://sortthestack.com/blog/how-to-write-a-candidate-recommendation)
covers the longer version for finalists.

### 5. Look at your own pattern

After a few jobs, look at how often you disagreed with the tool. A reviewer who
never disagrees is either working with a perfect tool or not really reviewing.
The five-application check in step 1 will tell you which.

## A worked example

Larkspur Partners, the fictional accounting firm in our guides, is hiring a
Senior Accountant. Must-haves include a CPA designation and owning a month-end
close end to end; familiarity with CaseWare and corporate tax preparation are
nice to have. The applicants, evidence and notes are invented for
illustration.

The hiring manager read five applications before opening the results, and
agreed with the tool on four. The fifth was an applicant the tool marked
unclear on the close. The manager had read "managed the close checklist" as
ownership. Re-reading, they decided the tool was right that it didn't say who owned the
close, so that became a phone screen question rather than a shortlist place.

The manager's shortlist review for three applicants:

| Applicant       | Tool's outcome | Evidence checked                                                                                            | Reviewer's note                                                                                                                           |
| --------------- | -------------- | ----------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Amara Nwosu     | Recommended    | Close: "Comfortable owning the close end to end and explaining the numbers to owners." Quote found, says it | Agree. Shortlist. Owns the close; CaseWare shown. Nothing to probe beyond the standard questions                                          |
| Sofia Marchetti | Recommended    | Corporate tax (nice to have): Not shown, "Prepare compilation engagements and supporting working papers"    | Agree. Shortlist. Compilation work isn't corporate tax, so Not shown is right. It's a nice-to-have; ask about it in the interview         |
| Tomás Herrera   | Possible       | Close: Unclear, "Present monthly financial packages to owners and advise on cash flow and budgeting"        | Agree it's unclear: presenting the numbers isn't owning the close. Two roles have overlapping dates. Phone screen on both before deciding |

None of these notes overrules the tool, and that's fine. Each says what the
reviewer checked and why they agreed. If Tomás's phone screen shows he ran the
close, they'll move him forward and record why the evidence changed.

## What to keep

For each job, keep:

- **The requirements as applied**, including any changes made during the
  review.
- **Your five-application check**: your calls, the tool's, and what you
  concluded where they differed.
- **A reason for each decision**, in your words, with who made it.
- **Every case where you overruled the tool**, and why.

That is enough to answer the questions that matter later: why was this person
turned away, who decided, and on what evidence.

## Where SortTheStack fits

SortTheStack is built to be reviewed rather than approved. Every application
gets an outcome (Recommended, Possible, Needs review or Requirements not met)
with each requirement marked met, not met or unclear and the text from the
application quoted. Every quote is checked against the resume before you see
it, and the compare view marks the rows where finalists differ. Setting
aside clear mismatches automatically is off by default, and no rejection is
sent unless someone chooses to send one.

When you change the product's recommendation, you add a reason, which is saved
to the candidate's decision record with who made the change, and team members
can comment on each candidate. For the five-application check in step 1, read
those applications from the original files before opening their results.
[See how the screening works](https://sortthestack.com/product).

## Sources

- Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T.,
  & Weld, D. (2021). Does the whole exceed its parts? The effect of AI
  explanations on complementary team performance. _Proceedings of the 2021 CHI
  Conference on Human Factors in Computing Systems_, 1–16.
  [doi.org/10.1145/3411764.3445717](https://doi.org/10.1145/3411764.3445717)
- Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think:
  Cognitive forcing functions can reduce overreliance on AI in AI-assisted
  decision-making. _Proceedings of the ACM on Human-Computer Interaction,
  5_(CSCW1), 1–21. [doi.org/10.1145/3449287](https://doi.org/10.1145/3449287)
- Green, B. (2022). The flaws of policies requiring human oversight of
  government algorithms. _Computer Law & Security Review, 45_, 105681.
  [doi.org/10.1016/j.clsr.2022.105681](https://doi.org/10.1016/j.clsr.2022.105681)
- Article 29 Data Protection Working Party,
  [Guidelines on automated individual decision-making and profiling for the purposes of Regulation 2016/679](https://ec.europa.eu/newsroom/article29/items/612053)
  (WP251 rev.01, adopted 6 February 2018; endorsed by the European Data
  Protection Board, Endorsement 1/2018), accessed 2 November 2026.