She stopped to ask what I meant, and I realized I didn't know

Open notebook showing a handwritten rubric question with a question mark circled in pencil, beside printed screening transcript pages, warm morning light on cream paper

Last Wednesday, around 10 in the morning, I was at my desk working through a batch of transcripts for a product operations search that had been running for just over three weeks. The hiring manager had messaged the night before asking when she'd have a shortlist. I'd told her Wednesday morning. So I had about two hours to get through what was left and make some decisions.

I was on my fifth transcript of the morning when I nearly moved past her.

The candidate had scored a 71. High enough that I wouldn't drop her automatically, not high enough to feel like a sure advance. The model summary flagged "limited specificity in stakeholder examples." I opened the transcript to understand what that actually meant.

I got to question five.

the question that nobody had challenged

The question I use for product ops roles reads something like: "Tell me about a situation where you had to manage stakeholder expectations on a project where the requirements weren't fully defined. What was the hardest part?"

I wrote it four months ago. It had run in 47 transcripts since, across three different searches. It felt solid. People talked. They gave real examples. Some of those examples were very good.

She paused for about nine seconds before answering anything. The transcript shows it as a gap. Then: "Sorry, I want to check I understand this correctly. When you say stakeholders, are you asking about external clients or people internally on the team?"

I read that twice.

Nobody had done that before. Not in 47 transcripts.

The candidates who had scored well on question five had all done roughly the same thing. They picked external stakeholders, usually a client or a partner organisation, and built their answer around a shifting brief or a slipping timeline. Confident. Specific in the ways the rubric rewarded. Structurally very clean. I had read a lot of good answers on this question. I had not noticed that almost all of them were answering the same version of it.

She was the first person to notice the question had more than one version.

what going back through the transcripts showed me

I spent about 40 minutes reading through 31 transcripts from the last two months, looking only at the answers to question five.

Twenty-six candidates had defaulted to external stakeholders without acknowledging that the question could apply to anything else. Four had hedged, working in both interpretations mid-answer, which made their responses feel less focused than the confident ones. She was the only person who named the ambiguity before choosing an interpretation.

The confident 26 had scored well. The four hedgers had scored middling. Her clarifying pause had cost her time, shortened her actual answer, and gotten her flagged by the model for low specificity.

None of that was wrong on the model's part. She did give less of an answer. The score reflected that accurately.

One transcript from February stood out when I went back to it. A candidate who had eight years in enterprise SaaS. Her answer had been detailed and strong, well-structured around a specific client engagement that had gone sideways mid-sprint. I'd rated it highly at the time. Reading it again, I noticed she had said nothing about internal stakeholders. Not because she lacked the experience. Because the way I'd phrased the question had quietly pointed her outward, and she'd followed that direction without questioning it.

Half the question had been invisible.

the thing I was accidentally measuring

I thought I was testing how candidates navigate unclear requirements. That is what the question appears to test. But unclear requirements in a job context is not the same as an unclear question in a screening call, and I had been using one to stand in for the other for four months.

The candidates who answered fluently had either not noticed the ambiguity or had noticed it and decided to pick an interpretation and run. Both of those produce good-looking answers. Neither is the instinct I want from someone in product operations, where a large part of the role is slowing down and asking "what problem are we actually solving?" before anyone builds the wrong thing at speed.

I had been accidentally selecting for people who are comfortable performing on top of vagueness. Which is, in some settings, a useful trait. In this one it's the thing I'm trying to screen against.

There's something similar to what I found when I retired my best interview question last month. The questions that produce smooth, complete answers are often the ones where the candidate has been handed a clean interpretation without realising it, or has learned to impose one fast. The question stops measuring what you think it measures. The signal goes quiet. You don't notice because the answers still sound fine.

If you've written a screening question that always seems to produce strong answers, it's worth asking who the question is easy for.

She is moving to the next round.

I'm rewriting question five this week. I don't know what it will be yet. Probably another entry once I figure it out.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you want your screening rubric to actually measure what it's supposed to, book a call.