# What to automate in hiring, and what to keep human

> Automate the steps where a fixed rule is the right answer, let software assist where the work is reading, and keep people on the decisions that turn someone away or choose a hire. A stage-by-stage guide, the one step most teams should automate but don't, and how to make human review real.

Sara at SortTheStack, November 1, 2026. Original: https://sortthestack.com/blog/what-to-automate-in-hiring

**Automate the steps where a fixed rule gives the right answer: messages,
scheduling, applying your own yes/no knockout questions, and combining
interview ratings by a formula. Let software assist, with a person checking,
where the work is reading applications. Keep people on the conversations and
on every decision that turns someone away or chooses a hire.** One step
surprises most teams: combining ratings is better done by a rule than by
discussion, and that is well supported by research.

This guide gives a test you can apply to any step, a stage-by-stage table,
and what it takes for "a human reviews it" to mean something.

## A three-question test

For any step in your hiring process, ask:

1. **Can you write down the rule?** If the right answer follows from a rule
   you could state in a sentence ("send a confirmation when an application
   arrives", "exclude applicants who answer No to the licence question"),
   a machine applies it more reliably than a busy person. If the answer takes
   judgement about a particular person, it can't be a rule.
2. **What does a mistake cost, and who would notice?** A late confirmation
   email costs a little goodwill, and the applicant notices. A wrongly rejected
   applicant costs you a candidate, and nobody notices, because rejected
   applications are rarely looked at again.
3. **Can a person see the result and reverse it?** Automation you can see and
   undo is a convenience. Automation that acts silently is a decision nobody
   made.

Those answers sort most steps into three groups: **automate**, **assist** (the
software does the work, a person checks it) and **keep human**. This is our
framework, built on the evidence below, not an industry standard.

## Stage by stage

| Stage                                     | Recommendation               | Why                                                                             |
| ----------------------------------------- | ---------------------------- | ------------------------------------------------------------------------------- |
| Posting the job to boards                 | Automate                     | Pure administration                                                             |
| Confirming receipt; status updates        | Automate                     | A rule, and silence loses candidates (see [ghosting](https://sortthestack.com/blog/candidate-ghosting)) |
| Scheduling interviews                     | Automate                     | Booking links remove the back-and-forth                                         |
| Knockout questions on yes/no facts        | Automate, rule reviewed      | A rule you wrote; keep it to genuine requirements                               |
| Reading applications against requirements | Assist                       | Reading at volume is where software helps, but it misreads; check the evidence  |
| Rejecting applicants                      | Keep human                   | Mistakes are invisible, and the rules increasingly expect a person              |
| Phone screens and interviews              | Keep human                   | Evidence on AI interviewers is early (see below)                                |
| Combining interview ratings               | Automate (by formula)        | A fixed rule predicts better than discussion                                    |
| The hiring decision                       | Keep human, one named person | Someone has to own it and be able to explain it                                 |
| References and the offer                  | Keep human                   | Conversations, negotiation and commitments                                      |

A few rows need more than a line.

### Knockouts: automate the rule, review the rule

A knockout question ("Do you hold a valid Class 1 licence?") is automation you
designed. It is safe when the question is a genuine requirement with a yes/no
answer, and it quietly costs you good people when it isn't. Review each one
before posting; our guide to
[screening questions](https://sortthestack.com/blog/job-application-screening-questions) covers which
make fair knockouts.

### Reading applications: assist, and check

Software is good at applying the same requirements to every application, the
140th as carefully as the first. It also misreads. Use a tool that shows the
evidence for each verdict, read the applicants it marks unclear yourself, and
spot-check the rest. Our article on
[why an AI tool ranked a candidate](https://sortthestack.com/blog/why-ai-ranked-a-candidate) has a
five-minute check, and
[can AI help you identify the best applicants?](https://sortthestack.com/blog/can-ai-identify-the-best-applicants)
covers the wider risks.

### Interviews: keep human for now

AI voice interviewers exist, and one large field experiment found they made
interviews more consistent in entry-level customer-service hiring. That is one
setting and a working paper. Our article on
[AI interviewers](https://sortthestack.com/blog/ai-interviewers) covers what it does and doesn't show.

## The step to automate that most teams keep human

Teams usually combine interview ratings by talking: everyone shares their
impressions and the group settles on a view. The research says a fixed rule
does better.

A meta-analysis by
[Kuncel and colleagues (2013)](https://doi.org/10.1037/a0034156) compared
mechanical combination (a formula or fixed rule) with holistic judgement in
selection and admissions decisions. It found "consistent and substantial loss
of validity when data were combined holistically", even when the people
combining were experts who knew the jobs. In predicting job performance, the
difference amounted to "an improvement in prediction of more than 50%".

"Automate" here can mean a spreadsheet column that averages the ratings. The
judgement stays where it belongs: in each interviewer's ratings against the
anchors, and in the discussion of what the formula can't capture. Our guides
to [consistent interview scoring](https://sortthestack.com/blog/consistent-interview-scoring) and
[making the final decision](https://sortthestack.com/blog/making-the-final-hiring-decision) show how.

## The step most often automated that should stay human

Rejection. Many tools can reject applicants below a threshold without anyone
looking. That turns every error in the tool into a decision nobody reviewed,
and the people affected are the ones you never see again. Our
[screening audit](https://sortthestack.com/blog/audit-resume-screening) is the way to find out how
often it happens.

The rules point the same way. They differ by place, change quickly and this is
not legal advice, but as of 1 November 2026:

- **European Union.** The GDPR gives people the right not to be subject to a
  decision "based solely on automated processing" that significantly affects
  them, and gives "e-recruiting practices without any human intervention" as
  an example (Article 22 and Recital 71 of
  [Regulation (EU) 2016/679](https://eur-lex.europa.eu/eli/reg/2016/679/oj)).
- **New York City.** Local Law 144 covers tools that substantially assist or
  replace discretionary decisions, including screening. Using one requires a
  bias audit within the past year and notice to candidates
  ([NYC DCWP](https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf)).
- **Ontario.** Employers with 25 or more employees must say in a publicly
  advertised job posting if they use AI to screen, assess or select applicants
  ([Ontario Ministry of Labour](https://www.ontario.ca/document/your-guide-employment-standards-act-0/requirements-related-publicly-advertised-job)).
- **United States generally.** The EEOC's position is that "the employer is
  still responsible for ensuring that its tests are valid", whoever built them
  ([EEOC](https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures)).

## Making human review real

"A person reviews every decision" means nothing if the person clicks Approve
three hundred times in an hour. People tend to over-rely on automated advice,
a pattern known as automation bias.

A systematic review by
[Goddard, Roudsari and Wyatt (2012)](https://doi.org/10.1136/amiajnl-2011-000089)
covered 74 studies, mostly in healthcare decision support. How much people
over-relied depended on workload, task complexity and time pressure, and on
how far they trusted the system. What helped included training, emphasising the user's
accountability, and providing information rather than a recommendation.
Clinical decisions aren't hiring decisions, but the conditions it describes,
a busy person approving a machine's verdicts at speed, are familiar.

In practice:

- **Show the reviewer the evidence, not just the verdict.** Checking a quote
  against an application is review. Agreeing with a label is not.
- **Name who is accountable** for each rejection and each hire. A decision
  with an owner gets looked at.
- **Make overriding easy, and record why.** If disagreeing with the tool takes
  five clicks and agreeing takes one, the tool decides.
- **Check the agreements too.** A reviewer who never overrides is either
  working with a perfect tool or not reviewing. Re-read a sample of what was
  approved.
- **Budget the time.** If review gets thirty seconds an application, decide
  in advance which applications get the full read (the unclear ones) and
  which get a spot-check.

Our guide to [reviewing an AI shortlist](https://sortthestack.com/blog/review-an-ai-shortlist) turns
these into a routine for one job, with a worked example.

## Where to start this week

If you're a small team doing most of this by hand, three changes cover most of
the benefit:

1. **Automate the confirmations.** Every applicant hears that you received
   their application, the same day.
2. **Use a booking link** for phone screens and interviews.
3. **Average your interview ratings in a spreadsheet** before you discuss
   anyone.

Then decide what to do about reading. If you get more applications than you
can read properly, that's where software earns its keep, as long as it shows
its evidence and doesn't reject anyone on its own.

## Where SortTheStack fits

Here is what SortTheStack does, sorted by the same three groups.

- **Automates:** a confirmation to applicants who apply through your
  SortTheStack careers page, and, when a job closes, a neutral notice to
  candidates still at the Interviewing stage that a decision has been made.
  Knockout rules you write are applied exactly as set, with no AI involved.
- **Assists:** it reads every application against your requirements and marks
  each requirement met, not met or unclear, with the evidence quoted. Every
  quote is checked against the resume before you see it.
  Setting aside clear mismatches automatically is off by default; when
  switched on for a job, it is logged and reversible.
- **Leaves to you:** no rejection is sent unless someone on your team chooses
  to send one, and the hiring decision is yours. When someone changes the
  product's recommendation they add a reason, saved to the candidate's
  decision record. You can send an interview invitation with your own
  scheduling link.

[See how the screening works](https://sortthestack.com/product).

## Sources

- Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013).
  Mechanical versus clinical data combination in selection and admissions
  decisions: A meta-analysis. _Journal of Applied Psychology, 98_(6),
  1060–1072. [doi.org/10.1037/a0034156](https://doi.org/10.1037/a0034156)
- Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A
  systematic review of frequency, effect mediators, and mitigators. _Journal of
  the American Medical Informatics Association, 19_(1), 121–127.
  [doi.org/10.1136/amiajnl-2011-000089](https://doi.org/10.1136/amiajnl-2011-000089)
- Regulation (EU) 2016/679 (General Data Protection Regulation), Article 22
  and Recital 71,
  [EUR-Lex](https://eur-lex.europa.eu/eli/reg/2016/679/oj), accessed 1 November 2026.
- NYC Department of Consumer and Worker Protection,
  [Automated Employment Decision Tools: Frequently asked questions](https://www.nyc.gov/assets/dca/downloads/pdf/about/DCWP-AEDT-FAQ.pdf)
  (29 June 2023), accessed 1 November 2026.
- Ontario Ministry of Labour,
  [Requirements related to publicly advertised job postings](https://www.ontario.ca/document/your-guide-employment-standards-act-0/requirements-related-publicly-advertised-job),
  accessed 1 November 2026.
- U.S. Equal Employment Opportunity Commission,
  [Employment tests and selection procedures](https://www.eeoc.gov/laws/guidance/employment-tests-and-selection-procedures),
  accessed 1 November 2026.