I retired the question I'd been asking for six years

Open notebook with a crossed-out line beside interview transcript pages — a question finally retired

The question I loved

For six years, I ended almost every screening call with the same thing.

"Tell me about a time you failed and what you learned from it."

I thought it was my best question. It felt like it cut through the CV polish, the rehearsed opener, the candidates who present well but say nothing. Get someone talking about failure and they either go real or they deflect — and both, I told myself, are information.

I used it without thinking. Like a good hammer.

Then last month I started going back through three months of AI screening transcripts. Not looking for anything in particular — just trying to understand which moments in the conversation the model was finding predictive. The AI surfaced scores and summaries, but I wanted to read the transcripts behind them.

I read seventy-something transcripts over two evenings.

What three months of transcripts showed me

The "failure" question generated, functionally, the same answer across almost every transcript I opened.

Some version of: a project ran over, or a team dynamic went sideways. Then: I hadn't communicated early enough. Then: now I check in more proactively and things have improved.

The specific details changed. The names of companies, the type of project, the length of the paragraph. The emotional structure was identical.

I went back further. I found the transcripts of candidates who'd been hired and who I knew had been performing well at the six-month mark. Then I found the transcripts of candidates who'd been hired and hadn't worked out. I tried to find a pattern in their "failure" answers that separated them.

I couldn't find one.

The strong performers and the weaker performers had equally polished, equally structured, equally scripted answers to this question. One group had better answers to other questions. Not this one.

The moments that actually separated them were elsewhere in the transcripts. A candidate who had spent ten minutes explaining a technical decision they'd argued against at their previous company — not just that they'd pushed back, but what they understood about the tradeoffs that the rest of the team hadn't fully weighed. A candidate who mentioned almost as an aside that they'd spent their evenings rebuilding an internal tool because it frustrated them. The AI had flagged both of those moments in its summaries. I'd skimmed them. Going back and reading properly, I stayed with them.

The "failure" question wasn't producing those moments. It was producing scripts.

That's not a flaw in the candidates. The question has been rehearsed into uselessness. It appears in every interview prep guide. Every career coaching service. Every "ten questions to prepare for" article online. The candidates who've made it to a screening call have almost all practised an answer. And once a question gets practised enough across a whole population of job-seekers, it stops producing conversations and starts producing performances.

This is something I'd half-noticed before, but only as a feeling. Reading how transcripts change what you actually hear in a screening helped me articulate why: the written record makes patterns visible that you miss when you're listening in real time and trying to evaluate simultaneously. I wasn't going to spot this pattern across seventy interviews from memory. The transcripts made it legible.

What I'm asking instead

I replaced it two weeks ago with a question I don't see in prep guides yet.

"What's something you spent time on recently that wasn't in your job description?"

It's not a perfect question. Some candidates find it disarming in a way that doesn't reflect their capability — the open-endedness makes some people reach for an answer they think I want to hear rather than an honest one. I've had to get comfortable with longer pauses before they land somewhere.

But the answers don't sound the same.

I get digressions. Real ones. Someone who's been reading about a domain adjacent to their work because a problem they were solving pointed them there and they kept going. A skill they've been building with no deliverable attached to it. A frustration with a tool or a system that they've been quietly working around — or quietly fixing. The AI summaries on these answers look different: less compressible, more varied, sometimes harder to score cleanly.

That variability is the signal.

I know this question will stop working eventually too. These things have a shelf life. The prep guides will catch up, and a new version of the script will emerge. When the transcripts start showing me the same arc again, I'll retire this one too.

That's the actual job now: keep reading the transcripts, keep noticing the patterns that have flattened over time, keep asking the questions nobody's written a guide about yet.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you're curious which parts of your screening process are still producing real signal, book a call.