She was a completely different candidate by question four

Open notebook with brief notes on the left page and a longer denser paragraph on the right, beside printed AI screening transcript pages, warm morning light on cream paper

Last Thursday afternoon, around 2pm, I was clearing a screening queue for a mid-level product ops role we'd been running for three weeks. Eighteen transcripts, most already read, four still sitting in a yellow review pile.

The one I opened first had a score of 61. Low enough that I'd been putting it off. I have a habit of reading the high scorers when I'm moving fast and leaving the borderline ones for later, when I feel like being more careful about them. Not the most rigorous system, but there it is.

I opened it expecting to confirm the number and move on.

the first two answers

The first two answers were rough. Not wrong, exactly, but clipped. She answered question one in 47 words. Question two in 63. The kind of short that reads as either very efficient or not quite settled yet. The AI scored both sections below median, which was correct. There wasn't much to score.

I was eating a sandwich at my desk by this point, half-reading, clicking through. (The afternoon slot is genuinely the worst time for me to make careful decisions. I know this and I keep doing it anyway.)

Then question three.

Something shifted. She started answering a question about a product launch that had gone sideways and the answer just kept going. Not rambling. Specific in a way that requires you to have actually been there. She named the constraint that broke first. She described what she'd tried before the thing that worked, and why the first attempt had been wrong in retrospect.

The AI flagged that section as high signal. I read it twice.

By question four she was a completely different person in the transcript. If you covered questions one and two and started from question three, you would not guess the score was 61. The answers were detailed, gaps named honestly, tradeoffs specific. One section had her talking about a decision she disagreed with and what she did instead of pushing back openly. That kind of thing only sounds real when it is.

what the score was measuring

The score averaged the whole call. The nervousness in questions one and two dragged the number down and the model doesn't weight for trajectory. It reads what was said, scores it against the rubric, and produces a number. Which is mostly what you want. But the number compresses the arc into a single point.

When I run a live call, I can feel when someone settles in. Usually after the second or third question. The nervousness drops and they start talking like they'd talk to a colleague. I adjust for that instinctively. The transcript doesn't adjust. It holds whatever happened, early and late, with equal weight.

I've been thinking about this more since reading how screening transcripts capture some things and smooth over others. This is the clearest case I've seen of the gap. The transcript held the full arc. The score summarised it. Those are two different things and I'd been treating them as the same.

I went back through 11 other transcripts from the same search, ones where I'd already made decisions. I was looking for the same pattern. In three of them I found something like it: an early answer that was cautious and short, then a shift around question three or four where the person found their footing. In one case I'd already moved that candidate forward. In the other two, I'd passed.

I can't know if either of them would have been a good hire. But I'm less confident in those two calls than I was last week. That's not a comfortable thing to sit with.

the follow-up, and what I'm trying now

I flagged the 61-scorer for a live conversation before committing to a shortlist decision. She came in yesterday at 1pm. Settled almost immediately, which matched questions three and four in her transcript. Her answers in the live call were consistent with the second half of the screening, not the first.

She's going to the shortlist.

The thing I keep coming back to: I only noticed the arc because I'd been putting the transcript off. If I'd opened it on a busy Tuesday morning with twelve other files ahead of it, I might have looked at 61, decided that was enough to know, and closed the tab. I do that sometimes. Most recruiters do. The score is there to help you move faster and 61 usually means something.

Except when it means "nervous at the start."

If you've ever made a call on a borderline transcript without reading past the number, you know what I mean. It's not lazy. It's just the pace of the work. But something gets lost.

What I'm building into my process now: on any transcript under 70, I read the first scored section and the last scored section before deciding. Not the whole transcript. Just whether the arc goes in one direction or two. Takes maybe four minutes on a 25-minute call.

I might be over-indexing on one week. Ask me again in a month.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you want to understand what your AI screening scores are averaging over, book a call.