How to compare job candidates side by side

To compare candidates fairly, put them in one table: the job’s requirements as rows, the candidates as columns, and in each cell the evidence you have from every stage, not your overall impression. Then compare across each row, one requirement at a time, before you look at anyone’s total. Comparing person by person invites the most recent or most confident candidate to win; comparing row by row makes you look at the same evidence for everyone.

This guide walks through the method, how to handle candidates with very different backgrounds, what to do about a tie, and a free candidate comparison matrix (Excel) to do it in.

Scorecard or comparison matrix?

They do different jobs, and you’ll usually want both.

• A candidate scorecard is filled in by one interviewer, for one candidate, straight after one interview.
• A comparison matrix brings everything together at the end: what the screening found, the work sample result, and the interview ratings, for every finalist, in one place.

Step 1: Use the same rows for everyone

The rows are the requirements you screened on, plus anything a later stage was there to test. For the fictional Senior Accountant role at Larkspur Partners used in our guides:

Requirement	Where the evidence comes from
CPA designation (must-have)	Screening question, confirmed
Runs month-end close independently (must-have)	Screen, then interview question 1
Handles a close problem under time pressure	Interview question 2
Prepares audit working papers	Screen, then interview question 3
Learns new systems	Interview question 4
Reviews others’ work constructively	Interview question 5
Reconciliation work sample	Work sample, scored against anchors

If a candidate brings something impressive that isn’t on any row, note it underneath. Don’t add a row for one candidate. A row that only one person can score isn’t a comparison.

Step 2: Fill in each cell with a rating and the evidence

Each cell gets a rating from 1 to 5 against anchors you wrote in advance, a short note of the evidence behind it, and the stage it came from. If you don’t have the evidence, mark it unclear, not low. An unclear cell is a question you haven’t asked yet.

Where you have ratings from more than one interviewer, enter the average and note any gap of more than one point. A gap means two people saw different things, and it’s worth finding out what before you decide.

Step 3: Compare across rows, not down columns

This is the step that changes the outcome. Take one requirement and read across every candidate’s cell for it, side by side, before moving to the next row.

Why: when you read down one candidate’s column, your overall impression of that person colours each rating. When you read across a row, you are asking a narrower question: “on the month-end close, who showed the strongest evidence?” It’s harder for a confident manner or a memorable story to carry across rows it has nothing to do with. This is our recommended practice, built on the same principle that research on structured interviews supports: rate each part separately against a fixed standard, and only then combine. See structured interviews for that evidence.

While you read across a row, check one thing: is each rating based on the same kind of evidence? A 5 from a detailed interview answer and a 5 from a single line on a resume are not the same strength of evidence. Note it.

Step 4: Combine by a rule you set in advance

Add the ratings up the way you decided before you met anyone. Equal weights for every row are the sensible default: a review of research on combining scores found that equal weights have substantial predictive validity compared with statistically derived ones, and called them “a highly appropriate approach” under many circumstances (Bobko, Roth and Buster, 2007). A common exception is to count must-haves twice.

Combining by a fixed rule matters. A meta-analysis by Kuncel and colleagues (2013) found that combining information about candidates mechanically predicted performance substantially better than combining it by overall judgement, even expert judgement.

The matrix does this for you: you choose equal weights or must-haves counted twice, and it averages each candidate’s rated rows, ignores unclear cells, and counts how many must-haves are confirmed and how many cells are still unclear.

Step 5: Then discuss, and decide

The total is the start of the discussion, not the end of it. Use the discussion to catch what the numbers miss: a misheard answer, a rating given on thin evidence, something important that isn’t on any row. If the discussion changes the decision, write down which row it changed and why.

Comparing candidates with very different backgrounds

Finalists rarely have matching histories. One candidate has eight years at a public accounting firm; another ran the books for a 30-person manufacturer and taught themselves the cloud system the firm uses.

Compare the evidence against the requirement, not the background against another background. The question for the close row is not “who has the more impressive employer?” but “who showed they can run a close independently?” A candidate from an unfamiliar setting may describe the same skill in different words, or have done it in a smaller organisation with fewer people to lean on. Read for what they did. If you can’t tell, it’s unclear, and that’s a question for a short follow-up call.

Be wary of rows that quietly reward a particular background, such as “experience at a firm like ours”. If familiarity with your industry genuinely matters, write down what that familiarity lets someone do, and make that the row.

When it’s a tie

Close totals are common, and that’s useful information: the candidates are genuinely close on the evidence you have. Before reaching for a tie-breaker:

1. Look at the unclear cells. A tie with open questions isn’t a tie yet. Ask the questions.
2. Look at the must-haves. If one candidate’s strength is concentrated in must-have rows and the other’s in nice-to-haves, that’s a real difference.
3. Look at the strength of the evidence, as noted in step 3.
4. Ask for one more piece of evidence on the row that matters most: a short follow-up exercise or a reference check focused on a specific fact.

Avoid tie-breakers that have nothing to do with the job, such as who you’d rather have a drink with. That is how “fit” turns into hiring people like yourself. Our article on culture fit and other hiring assumptions covers the research.

Write down the decision

One or two sentences, tied to rows: “Chose Candidate A: stronger evidence on running the month-end close, a must-have. B was stronger on handling a close problem and on the work sample, and B’s audit row is still unclear.” It takes a minute, it helps you explain the decision to the people you didn’t choose, and it’s what you’ll want when you look back at the hire in six months. Our article on hiring metrics suggests a simple way to do that look back.

Using the matrix

The candidate comparison matrix is an Excel workbook with room for ten requirements and five candidates. Each candidate gets a rating column and an evidence column side by side, so reading across a row shows you the evidence as well as the numbers. Leave a rating blank when the evidence is unclear: the cell turns yellow and the average ignores it. A must-have rated below 3 turns red. Underneath, it calculates each candidate’s average of rated rows, rows still unclear, and must-haves rated 3 or above, with space for anything not on any row and for the decision itself. The example is the Larkspur hire above; clear it and add your own requirements.

Where SortTheStack fits

SortTheStack has a side-by-side compare view for finalists, showing how each one did against each requirement with the evidence from their application. It covers the screening rows of the matrix; interview and work sample ratings are yours to add. See how the screening works.

Sources

• Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98(6), 1060–1072. doi.org/10.1037/a0034156
• Bobko, P., Roth, P. L., & Buster, M. A. (2007). The usefulness of unit weights in creating composite scores: A literature review, application to content validity, and meta-analysis. Organizational Research Methods, 10(4), 689–709. doi.org/10.1177/1094428106294734