The role we posted in February isn't the one we're filling

Open notebook with handwritten candidate scoring columns beside a printed job description with a circled paragraph, warm morning light on cream paper

Three days ago, in the debrief for a senior product analyst role we'd been running since early April, the hiring manager said something that's been sitting with me.

It came up near the end of the call. Not flagged, not introduced. She was wrapping up her notes on the shortlist and she added, almost as an aside: "Oh, and depending on how Q3 goes, this role might pick up some GTM analytics coverage too. We haven't decided yet."

We had 47 candidates scored. Nine above 80. The top score was 91, from a candidate who'd spent six years at two companies building exactly what the job description called for: complex SQL, cross-functional reporting, full ownership of the metrics layer. Her screening call was clean. The AI summary and my own notes landed in the same place.

She had no GTM analytics background. The JD hadn't asked for any.

The JD was written in February.

what the score was measuring

The AI scores against the job description. That's the contract: give it a clear rubric and it produces a focused shortlist, consistently, without the slow drift that sets in when you're reviewing candidate thirty-three at 6pm on a Thursday.

But a job description is a frozen document. It captures what a role was when someone sat down and wrote it. Sometimes that's recent. Sometimes it's four months ago, and the team has since reorganized around a new product area, and the person who wrote the description no longer quite recognizes the thing they're hiring for.

In February, this role had a narrow scope: product analytics, internal reporting. Nothing customer-facing. The kind of description that produces a focused rubric and, downstream, a focused shortlist.

By May, something had shifted. The hiring manager had started thinking about adding a commercial angle. Nothing formal, nothing decided. Something like a 40% probability of becoming real by Q3.

The 91-scorer was right for the role as described. I don't have a rubric for the role as it might become.

The second-ranked candidate came in at 74. She'd spent most of her career at the intersection of product and go-to-market analytics. The AI scored her lower because her pure domain fit was narrower against the February description. If the role shifts by Q3, she might be the candidate I actually needed.

I'd moved her to the review pile after the initial shortlist pass. She wasn't going to surface again without something prompting me to go back.

the part the transcript holds

After the debrief, I went back through both transcripts.

The score is one layer. The actual conversation is another. I've been noticing this more lately, tracking what the gap between the two means in practice. The score tells you fit against what you asked for. The transcript holds something else, something harder to compress into a number.

The AI had flagged one section of the top scorer's call as "contextual adaptability, moderate signal." I'd skimmed it on my first read because everything else in her transcript was so clearly strong. The summary looked like a footnote.

Going back to actually read it: she described a quarter where her company changed direction mid-sprint and she'd rebuilt the reporting layer from scratch in something like three weeks. The delivery wasn't polished. Her phrasing was loose in places. But the thing she was describing was real, and she clearly understood the specifics of it in a way that doesn't come from having rehearsed an answer about adaptability.

That section of the transcript matters more now than it did in my first pass.

Not because it changed her score. Because it's evidence of how she handles the ground moving under a project. Which is, as of the debrief three days ago, now the question.

I sent her to the final round. That answer was most of the reason.

The 74-scorer I flagged for the hiring manager before the finals. Not as a push either way. Just: here's what I found, here's why she might become more relevant depending on what Q3 looks like, you decide if you want her in the room.

I think the thing I should have done, and didn't, is a brief check-in with the hiring manager four weeks into the search. Not about candidates. About the role. Has anything shifted? Is the description still right? Five minutes, written down. Because finding out the rubric has drifted during the debrief, when you've already run 47 candidates through it, is the worst possible time to find that out.

I didn't do it here. We posted the role in early April and I ran a tight, clean process against the February description. Got a precise result. The precision was real. I'm less sure the question was.

She might be exactly right. She might be exactly right for a role that Q3 reorganizes out from under her before her first review. I can't know that yet.

Ask me again in August.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you're screening candidates against a job description that hasn't been looked at in months, book a call.