He hadn't done the work but scored 15 points higher

Open notebook beside stacked screening transcript pages, two candidate scores written in pencil, warm morning light on cream paper

It was Tuesday morning, about ten minutes before my 9am team catch-up, and I was trying to finish a first pass on 23 transcripts from a data engineering search that had been running for six weeks.

I had 14 left. I was going faster than I should have been.

the transcript I almost closed

Candidate A had scored 79. CV was solid. Two relevant positions, four years of experience, the specific ETL tool listed under technical skills. The model summary read: "Strong technical vocabulary, confident answers, limited implementation detail on pipeline question." I skimmed that flag, assumed it was a rubric artefact, and opened the transcript to make a quick call.

I read the pipeline answer.

He described building the transformation layer first, then configuring the source connectors to feed into it.

I stopped.

Anyone who has worked with this tool, or most ETL platforms, would recognise the problem. You configure sources before you build transformations. The source schema tells you what you're transforming. Do it the other way and you're writing transformation logic against a structure you haven't seen yet. The sequencing isn't a preference. You configure sources first because that's how the system is designed to be used.

His vocabulary was right. The answer was 211 words and covered everything the rubric asked about. The model had scored it well because it matched the surface pattern of a strong answer. Nothing flagged a problem, because the model doesn't know the sequencing is backwards. Knowing that requires understanding how the tool actually works, not just what it's called.

I've started thinking about this as vocabulary without the work. You can absorb the right words from documentation, from conference talks, from a colleague who builds pipelines while you sit nearby. The vocabulary accumulates. The decisions you make when the schema doesn't match. The sequencing errors you run into. The moment when your assumptions about the data turn out to be wrong. Those stay with the person who actually did the work.

the 64 who was honest about the gap

Candidate B was at 64. No ETL tool on her CV. The model had docked her on the technical section before she'd said anything.

She opened her pipeline answer with: "I haven't worked with that platform specifically, so I want to flag that upfront. The closest thing I've done is a project at a logistics company where we were pulling from three source systems with inconsistent schemas."

Then she described what she'd built. She started with source configuration. Dealt with a schema mismatch on the third connector. Explained why the transformation logic had to be rebuilt twice when the upstream data didn't land the way they'd expected. At the end she asked which version the team was running, because the connector behaviour had changed between releases and she wanted to be accurate about what she'd actually do.

That question, in about 19 words, told me more about her real experience than the previous candidate's entire answer.

The rubric had penalised her for the upfront hedge. Lower confidence signal. Her answer was 143 words, shorter because she'd spent the first few seconds being honest about the gap instead of performing expertise she didn't have. The model rewarded fluency and penalised the hedge. She gave less of both.

what the rubric can't see

The model scored those two transcripts correctly, by its own rules. The 79 was fluent and complete. The 64 had a weak opening and a shorter answer. Both of those things are true.

The issue is that the rubric was measuring signals that a skilled bluffer can produce: fluency, structure, keyword coverage. What it couldn't check is whether the described process is physically possible in the order described. That requires domain knowledge the model doesn't have.

What surfaced the error was a small flag in the summary. "Limited implementation detail." It made me slow down and read, when I would otherwise have moved on. That is a narrow thing to depend on.

There's something adjacent to what I noticed when a candidate paused to ask what my rubric question actually meant. In both cases the rubric was producing scores that looked fine. The difference here is that I wasn't measuring the wrong thing. I was measuring the right thing in a way that a performance of expertise can pass.

I don't have a clean fix yet. Calibration sessions with the hiring manager before the search starts, working through what correct technical answers actually look like, would probably help most. It adds time I don't always have at the start of a search.

Candidate B is through to the next round. Candidate A is not.

The other 21 transcripts are still in another tab. I reviewed them faster than I reviewed these two. I'm not sure what I scored confidently that I shouldn't have.

Talk soon.

— the recruiter

---

The Diary of an AI Recruiter is a daily column from the team at Ployo. If you want your AI screening to surface expertise rather than vocabulary, book a call.