Forty-one above threshold and I'd read eleven of them

Open notebook with a long handwritten list of candidate names and a tall stack of printed transcript pages beside it, warm morning light on cream paper

Last Tuesday, somewhere around 4pm, I got a Slack message from the hiring manager on a senior customer success role we'd been running for just over two weeks.

She wanted to know where we were on the shortlist.

I looked at the queue. 41 candidates. All above 78.

The AI had processed 89 applications in three days. Every transcript run, every score calculated, every summary written. No 6pm review sessions where candidate thirty-seven gets nine seconds because you're also thinking about dinner.

I'd read 11 of the transcripts in any real detail.

how you end up here

The volume on a senior CS role is usually manageable. Two or three weeks of mid-funnel applications, maybe 20 or 25 candidates above a reasonable threshold. Enough to read carefully.

This one was different. LinkedIn Easy Apply enabled. The role had gone live and applications came in faster than I'd expected for a seniority level that usually filters itself.

By day three the AI had run the full batch. I'd asked it to flag everyone above 78. It flagged 41.

That number felt like a success for about one afternoon.

Then it became the actual problem.

A 78-plus threshold across 89 applications doesn't give you 41 strong candidates. It gives you the left edge of a distribution. The rubric stops distinguishing at a certain volume. Everyone in that range is "good enough against the criteria." That doesn't mean they're the same. It means the brief was precise enough to produce a crowded result.

I looked at the spread. Nine between 78 and 82. Fourteen between 83 and 87. Eleven between 88 and 92. Seven above 92.

Seven above 92 for a single customer success role, in two weeks. I'd been on three other searches this year where nobody cleared 88.

So I went back and read the JD. Whoever had written it was specific: named tools, named metrics, a defined scope. The AI's rubric had something real to calibrate against. A tight brief produces a generous shortlist. Not because the candidates aren't real. Because the criteria surface everyone who's touched the right things, and in a deep talent market for CS roles, that turns out to be a lot of people.

I hadn't accounted for that when I set 78 as the threshold.

what 41 above threshold actually means

Of the 11 transcripts I'd read in any real depth: maybe four felt genuinely strong. The kind where the candidate's thinking is visible, where they go sideways on a question because that's where they think the real answer is. The kind where I'd circle a section and write next to it: read this, just this.

The other seven of the 11 were fine. Above threshold, nothing wrong, hard to distinguish from each other on the page.

And then there were 30 I hadn't opened. Summaries. Scores. A few tags.

That's not a shortlist. That's a queue.

The hiring manager wanted to know where the shortlist was. July was approaching. From her side, the AI was supposed to be making this faster. The funnel was running. The pile at the bottom of it was just mine to work through.

Which, to be fair, is exactly right. It is mine to work through.

what I did, which I'm not sure was right

I raised the threshold to 84. Cut the queue from 41 to 22. Read those 22.

It felt like a decision. I'm not sure it was one.

The 19 I'd cut are still above 78 on my original rubric. I have no idea if the candidate who should be my top recommendation is somewhere in that group. I haven't read their transcripts. I trusted that a 6-point gap in the distribution mattered enough to skip them, without checking whether it did.

Maybe that's right. I think it's probably approximately right. I'm still uncomfortable with "approximately."

What keeps running through my head is that the AI changed what I'm reading, not how much. It shifted my categories, my default framing for a candidate, how I describe a person's strongest point. Now it's shifted the shape of the review pile itself. Before AI screening I had 25 CVs to skim and a few to read carefully. Now I have 22 transcripts, each three or four times as long as a CV, plus 19 more I've semi-consciously deprioritized without reading. The total review time hasn't gone down. It's redistributed. And somewhere in that redistribution I've been making choices about who to read carefully and who to summarize that I haven't been fully honest with myself about.

I told the hiring manager I had a shortlist of 11 by Thursday. She seemed satisfied.

The 19 in the queue are still there. I'll probably clear them this week. I might not.

None of this is a failure of the AI. The screening ran well. The scores made sense. The problem is operational: we built AI to clear the front end of the funnel, it did, and then the middle was still there, two people still need to read it, and the pile is now made of transcripts instead of CVs and takes longer per person.

Going faster at the top didn't make the middle go faster.

I don't have a fix yet. A higher default threshold, maybe. Or a second-pass filter on the transcript content itself, asking the model to distinguish within the above-threshold group. Something between "everyone above 78" and "nobody below 84."

Ask me again in June.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If your AI screening is moving faster than your team can review what it finds, book a call.