# How to audit your resume screening, whether a person or AI does it

> The candidates a screen wrongly rejects are invisible unless you go looking. A practical audit for small teams, in two parts: re-reading a sample of rejected applications, and checking whether rejections fall unevenly, with the four-fifths rule and its limits.

David at SortTheStack, October 26, 2026. Original: https://sortthestack.com/blog/audit-resume-screening

**To audit your resume screening, take a random sample of the applications
you turned down, have a second person read them against the same requirements
without seeing the original decision, and trace every disagreement to its
cause. Where you have the data, also compare how often different groups pass
the screen.** The first check finds qualified people you're losing. The second
finds whether the losses fall unevenly. Neither needs special software, and the
first takes about an hour.

This applies whoever does the first read: you, a colleague, a knockout
question or an AI tool.

## Why rejected applications need checking

A screen's mistakes come in two kinds. Letting through someone who doesn't fit
is visible: you meet them and find out. Turning away someone who did fit is
invisible: they never come back, and nothing tells you they were there.

These invisible mistakes are common. In a survey of 2,275 employers,
[Harvard Business School and Accenture (2021)](https://www.hbs.edu/ris/Publication%20Files/hiddenworkers09032021_Fuller_white_paper_33a2047f-41dd-47b1-9a8d-bd08cf3bfa94.pdf)
found that 88% of employers using software to filter or rank applicants said
it screened out qualified high-skills candidates at least some of the time.
Applicants worry about it too. In
[iHire's 2026 survey](https://www.ihire.com/resourcecenter/employer/pages/the-state-of-online-recruiting-2026)
of 1,054 U.S. job seekers, 44.3% viewed employers' growing use of AI in hiring
negatively, and of those, 83.3% worried that qualified candidates may be
unfairly screened out.

People make the same mistakes as software, more slowly. A reader who is tired,
skimming for familiar job titles or thrown by an unusual career path will turn
away people who meet every requirement.

## Part 1: Re-read a sample of rejections

### Step 1: Draw a random sample

From the applications that were turned down, take a random sample: every
fifth one, or those whose names fall on a random number list. Twenty is a
practical size for a small team. Don't choose which ones to check; the
interesting mistakes are in the applications nobody would think to pick.

Include the applications removed by knockout questions, and any marked
"unclear" that were never followed up.

### Step 2: Re-read them blind

Give a second person the requirements and the sampled applications, without
the original decisions. Ask them to mark each requirement met, not met or
unclear, with the evidence, exactly as the first reader should have done. Our
guide to [what counts as evidence](https://sortthestack.com/blog/evidence-a-candidate-can-do-the-job)
has a simple way to judge it.

### Step 3: Sort the disagreements

| Second reader's verdict | What it means                                         |
| ----------------------- | ----------------------------------------------------- |
| Agrees: clearly misses  | The screen worked                                     |
| Unclear on a must-have  | Should have been asked, not rejected                  |
| Meets every must-have   | A false rejection: a qualified person was turned away |

Contact anyone in the third group if the job is still open. Then work out why
each one happened.

### Step 4: Trace each mistake to its cause

The cause matters more than the count. Common ones:

- **A knockout question worded badly.** Qualified people answered "no"
  because the question was ambiguous. See
  [screening questions](https://sortthestack.com/blog/job-application-screening-questions).
- **A requirement that is really a proxy.** A degree or a years-of-experience
  threshold removed someone who can clearly do the work.
- **Title or keyword matching.** The evidence was there under a different job
  title or wording.
- **Reader fatigue.** Check whether the mistakes come from late in the pile,
  or from a long session of reading.
- **The tool's settings.** An AI tool set to reject when it was meant only to
  rank, or a rule switched on that nobody remembered.

Fix the cause, then re-check the rest of the rejected pile for the same
problem. If a knockout question was wrong, every applicant it removed needs a
second look, not only the ones in your sample.

### What a clean sample does and doesn't tell you

Finding no mistakes in twenty is good news, but it doesn't prove the screen is
perfect. A quick rule of thumb from statistics, the "rule of three", says that
if you find no errors in a sample, the true error rate could still be as high
as about three divided by the sample size, with 95% confidence. No errors in
20 is consistent with an error rate of up to about 14%. No errors in 50,
up to about 6%. A small audit catches big problems. It can't certify a small
one away.

## Part 2: Check whether rejections fall unevenly

The second question is whether your screen passes some groups of applicants at
a lower rate than others. That's called adverse impact, and it can be unlawful
even when nobody intended it.

### The four-fifths rule

In the United States, federal enforcement agencies use a rule of thumb from
the
[Uniform Guidelines on Employee Selection Procedures](https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607/section-1607.4):
"A selection rate for any race, sex, or ethnic group which is less than
four-fifths (4/5) (or eighty percent) of the rate for the group with the
highest rate will generally be regarded by the Federal enforcement agencies as
evidence of adverse impact."

An invented example: if 40 of 100 applicants from one group pass your screen
(40%) and 25 of 100 from another pass (25%), the ratio is 25 ÷ 40 = 0.63,
below 0.8.

The same section warns against treating the rule mechanically. Smaller
differences can still be adverse impact when they are significant "in both
statistical and practical terms", and larger ones may not be when "the
differences are based on small numbers and are not statistically significant".
For a small employer with a few dozen applicants per job, that second warning
applies almost every time. Pool several jobs, or several months, before
drawing conclusions.

### You need data most small employers don't have

You can't calculate selection rates by group without knowing applicants'
groups, and collecting that is its own decision. It has to be voluntary, kept
separate from the applications, and never seen by the people screening. In
Ontario, the Human Rights Commission's guidance says application forms should
not ask about protected grounds. For employers who want to increase the
diversity of their workforce, it suggests asking people to self-identify, with
appropriate safeguards. Get advice before you start
collecting it.

If you don't have the data, Part 1 is still worth doing, and look at the
causes it finds. A screen that removes people over unexplained gaps, unfamiliar
schools or foreign job titles is likely to fall unevenly on someone.

### If you hire in New York City with an automated tool

New York City's Local Law 144 goes further for automated tools. An employer
using an "automated employment decision tool" to assess candidates for jobs in
the city must have had it bias-audited by an independent auditor within the
past year, publish a summary of the results, and notify candidates. According
to the city's
[FAQ](https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf),
the audit must at a minimum calculate "selection or scoring rates and the
impact ratio across sex categories, race/ethnicity categories, and
intersectional categories". The law itself doesn't require any particular
action on the results, but anti-discrimination law still does.

### Using a vendor's tool doesn't move the responsibility

If an AI tool or a test is doing part of the screening, ask the vendor what
they've checked. But the
[EEOC's guidance](https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures)
is plain: "While a test vendor's documentation supporting the validity of a
test may be helpful, the employer is still responsible for ensuring that its
tests are valid." Our article on
[whether AI can identify the best applicants](https://sortthestack.com/blog/can-ai-identify-the-best-applicants)
has a test you can run on any tool before trusting it.

## How often

- **For the first few jobs** you screen with a new process or tool, after
  each one.
- **After you change anything:** a new knockout question, new requirements, a
  new tool or a new setting.
- **Otherwise, a few times a year,** pooling recent jobs so the numbers mean
  something.

Write down each audit: the date, the job, the sample size, what you found and
what you changed. If anyone ever asks how you know your screening is fair,
that record is your answer.

## Where SortTheStack fits

SortTheStack makes Part 1 straightforward, because every decision comes with
its reason. Each applicant is marked Recommended, Possible, Needs review or
Requirements not met, with each requirement marked met, not met or unclear and
the evidence quoted, and a knocked-out applicant shows the rule that removed
them. Setting aside clear mismatches automatically is off by default. When
someone overrides the product's recommendation, they add a reason, which is
saved to the candidate's decision record, and that record can be exported.

It can't do Part 2 for you. SortTheStack doesn't collect applicants' race,
sex or other demographic information, and its assessments are instructed not
to infer them, so it can't calculate selection rates by group.
[See how the screening works](https://sortthestack.com/product).

## Sources

- Fuller, J. B., Raman, M., Sage-Gavin, E., & Hines, K. (2021).
  [_Hidden workers: Untapped talent_](https://www.hbs.edu/ris/Publication%20Files/hiddenworkers09032021_Fuller_white_paper_33a2047f-41dd-47b1-9a8d-bd08cf3bfa94.pdf).
  Harvard Business School Project on Managing the Future of Work and Accenture.
- iHire (2026).
  [_2026 State of Online Recruiting Report_](https://www.ihire.com/resourcecenter/employer/pages/the-state-of-online-recruiting-2026).
  Survey of 318 employers and 1,054 job seekers in the United States, July and
  August 2026. Accessed 26 October 2026.
- New York City Department of Consumer and Worker Protection,
  [Automated employment decision tools: Frequently asked questions](https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf).
  Accessed 26 October 2026.
- Ontario Human Rights Commission,
  [Designing application forms](https://www.ohrc.on.ca/en/iv-human-rights-issues-all-stages-employment/4-designing-application-forms),
  _Human rights at work_. Accessed 26 October 2026.
- U.S. Equal Employment Opportunity Commission,
  [Employment tests and selection procedures](https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures).
  Accessed 26 October 2026.
- Uniform Guidelines on Employee Selection Procedures, 29 C.F.R. §
  [1607.4](https://www.ecfr.gov/current/title-29/subtitle-B/chapter-XIV/part-1607/section-1607.4).
  Accessed 26 October 2026.