I reopened the file after we'd already made the hire

Open notebook with a printed email reply beside folded AI screening transcript pages, one phrase circled in pencil, warm morning light on cream paper

Last Thursday, around 4:30 in the afternoon, I was doing the thing I'd been putting off for two weeks: closing out the candidate files from a search we'd finished.

The role had been filled. A senior analyst position, one I'd been running since late March. Thirty-nine candidates screened, nine above the threshold, one offer accepted, start date confirmed. All the things that should make a search feel finished.

I let the archiving sit for two weeks anyway.

the email I hadn't answered

The sixth file I opened had a note I'd added to myself in early May: "reply with feedback." The candidate had scored 61 on our screening rubric. Standard decline had gone out. Two days later she'd written back. Not a challenge, not an appeal. A seven-line email asking if there was any specific area she could work on for future applications in this kind of role.

I flagged it to respond to. The finals week happened. Then onboarding. Then two weeks of something else.

She was one of 39 candidates we'd screened. Our threshold was 74. The other 30 below the cutoff didn't come back into focus after the initial pass. That's how the volume works. Nine candidates above the threshold means nine people to think carefully about. The others get a clean decline and that's usually the end of it.

I was opening her file to do the thing I'd meant to do in May.

what the transcript said

The AI had scored her across four areas in the rubric I'd built: SQL depth, cross-functional communication, analytical reasoning, experience owning ambiguous briefs. She'd done reasonably well on the first three. The fourth is where she'd lost ground. The model's summary read: "Demonstrated structured thinking in familiar contexts. Limited evidence of owning ambiguous problem definition."

Probably accurate.

Three-quarters of the way into the transcript, there was an exchange the AI had flagged as "moderate signal, independent judgment." On my first read I'd seen "moderate signal" and kept moving. Reading it now, without a shortlist to build: she'd been on a team that was three weeks from shipping a reporting dashboard when she pushed to delay.

Not because the build wasn't done. Because she'd noticed that the primary metric the dashboard would surface was correlated with a variable that would shift in Q2 in a way that would make the number look better without actually indicating any real improvement. She'd written a note. She'd taken it to her manager. She was overruled on timeline. The dashboard shipped. She'd been right. It took six months for anyone else to catch up to what she'd seen.

The AI summary of that section: "showed independent judgment in a data review context."

I've been thinking lately about what a reference call can surface that a transcript compresses away. This was the reverse of that problem. The transcript had the whole story. The summary had reduced it to eleven words.

the score and what it doesn't say

Sixty-one was defensible. We advanced candidates at 74 and above. I calibrated that threshold over two previous searches and it had held up. It's not an arbitrary number.

But the 13-point gap between her score and the cutoff doesn't mean what it sounds like it means. It might reflect a real difference in capability. It might just be how she structured her examples that particular afternoon compared to a candidate who'd been recently coached on giving clean, rubric-shaped answers. Or question four, which I've noticed some candidates can anchor on a recent project much more easily than others.

From inside the score, I can't tell which of those it is.

The rubric asked about experience owning ambiguous problem definition. The section I'd skimmed was about noticing a flaw in a metric before anyone else had seen it. Those are related but not the same thing. She answered the question she was given. Maybe I wrote the question wrong.

I might be rationalising this. Reading a transcript after the hire is already made is not a controlled experiment. You read it differently. She might be exactly who the score said she was, and I'm just doing the thing where the story makes sense once you know which scene you're looking at. I've probably done that before.

The feedback note is still unwritten.

What I would have said three weeks ago: "The role required strong ownership of ambiguous problem definition, and the candidates we advanced showed clearer examples of leading from an unclear brief."

Accurate. Also no longer quite what I believe.

What I'd write now is more complicated. And I'm not sure that's useful to someone who just wants something actionable to work on before the next application. She asked a specific question. I can give her an answer that's true. I'm just not sure it's the whole truth.

Maybe the right call is to send the straightforward version and leave out the part where I'm not sure the cutoff was measuring what I thought it was measuring. Maybe that's the easy call. I genuinely can't tell the difference from here.

Still working on that one.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you want to understand what your screening scores are and aren't telling you, book a call.