After the reference call, I read the transcript again

Open notebook with handwritten reference notes beside a printed AI screening transcript, warm morning light on cream paper

Last Tuesday, around 4pm, I was working through reference calls for three candidates advancing to final interviews for a senior ops lead search we'd had running since early March.

The second call was with a former manager at the logistics company where one of the candidates had spent three years. Fourteen minutes total. He was helpful and concrete. She was strong under deadline pressure, reliable across scope changes, good at managing up to stakeholders who didn't always know what they wanted.

Then near the end he said, almost as a closing summary: "She does really well once she knows what she's building toward. Once the direction is set, she moves fast."

I wrote it down. Wasn't sure what to do with it.

going back to the transcript

That evening I opened her screening transcript. She'd scored 84. I'd read it once when the scores came in, added a note ("strong signal, move to finals"), and kept moving. She was third in the pool. Clearly qualified. Nothing that required a second pass.

Reading it again, with that phrase in my head, the pattern appeared.

Every high-signal section the AI had flagged was a story about execution. Good execution. The kind the model rewards: domain-specific and concrete, clearly reasoned. But every example started the same way. "After the team had agreed on the direction..." "Once we had the brief in hand..." "When the roadmap shifted, I picked up the new priorities and..." Every story began after somebody else had decided what the story would be about.

Not one of them was about noticing something and deciding to do it before anyone asked.

I went back through 22 transcripts from the same search, scanning the opening phrases of the flagged high-signal sections. The pattern wasn't unique to her, but in her case it was unbroken. Every example faced the same way.

The AI had scored those sections correctly. Domain knowledge, clear communication. The kind that makes the model's job easy. But the model doesn't distinguish between "strong execution on clear asks" and "builds structure when there isn't one" unless I write that distinction into the rubric. I hadn't. I'd written "strong execution and cross-functional delivery." I got exactly what I'd asked for.

The AI wasn't failing. I'd written the wrong question.

what I asked the hiring manager

I sent him the reference note with my read attached. He came back in about two hours: "This tracks with something I noticed but couldn't name. Ask her, in the final, to walk you through a time she had to build the structure before she could execute in it."

Better question than anything I'd put in the rubric.

She answered it in the panel. Paused longer than she had in any of her AI screening answers before landing on an example. The example itself was real: a project she'd inherited mid-flight with no clear brief, two weeks she'd spent building the frame before any of the actual work could start. She described it with the slightly messy edges you get when someone is remembering something genuinely difficult rather than rehearsing an answer about something difficult. Not polished. I stayed with it longer than the clean answers.

The pause itself told me something. In the AI screening call she'd barely paused at all. That's part of what had given the transcript its even quality, the thing I'd noticed before in candidates who've learned to perform for what the rubric rewards. The panel question gave her nowhere to hide. She found something real instead.

She's through to the final stage. The outcome isn't decided yet.

the thing I'm carrying from this

The small change in how I read transcripts now: I look at which direction the examples face. Not just whether they're concrete and credible, which the AI rightly scores for, but whether the person in the story is the one who decided something or the one who executed after someone else decided. It shows up in language. "We were asked to..." vs. "I just started doing it before anyone said anything." Small distinction. Diagnostic one.

I'd been reading enough transcripts to catch patterns that have smoothed over time, the kind of coaching that flattens answers into a consistent shape. This one got past me until a reference call handed me the frame. I'm not sure the AI could have flagged it, because I hadn't asked the right question. That's the part I keep coming back to.

Maybe the rubric needs a new field. Maybe some things still have to come from a fourteen-minute phone call.

I genuinely don't know yet.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If your rubric might be selecting for the right skills but missing a few, book a call.