Hiring in the Age of AI

I’ve been interviewing and hiring for Software Engineering roles for most of my career. There’s always been concerns about candidates “cheating” their way through interviews. “I think this candidate is using google”, “This candidate seemed to already have known the questions we were asking”, and similar accusations have been common refrains at hiring huddles for as long as I can remember.

Making sure we as engineering managers and engineering leaders are fairly and accurately evaluating the qualifications for a candidate is essential to building functional engineering teams, and removing or detecting usage of interview assistance tools is a natural initial response after you catch your first candidate cheating their way through an interview. But this begins a forever game of cat and mouse with each side escalating the detection and stealth capabilities respectively until at some point we just give up and return to the days of flying candidates into an office, and interviewing them on a company laptop.

But I’d propose a different solution. A solution more well suited for the world we’re in now where we have hundreds of applicants for every posted role, and where the initial phone screens and take home questions provide less signal than ever before. Even if we move back to onsite interviews, the time and money we’d waste when a candidate can cheat their way through the initial screens would be a huge problem.

Before I get to that solution though, I’d ask a question for you to ponder before you read further. A question we should all be asking whenever we need to hire for any role.

And that question? “What am I hiring this person to do.”

Ok, take some time, and jot down a few things that come to mind for that question.

I bet your first thought was “I’m hiring them to write code” or some variance of it. Maybe you defined it a bit further and included a specific language, or a domain in which they work, or maybe even a project you need them to contribute to. But that’s not what you’re actually hiring them to do, that’s just the tool they’re going to use.

If you were going to hire someone in construction who’s framing 2x4 walls all day, are you going to interview them on how well they swing a hammer and tell them they can’t use a nail gun? No, you care about the final product. You care about if they can frame and install a wall. Swinging a hammer, marking a 2x4, and how well they use a chop saw are only about how well they know their tools and not how they use them in their craft.

So what are we hiring software engineers to do?

We’re hiring them to think. We’re hiring them to think in a very specific way, to break down real world problems as stated by customers, Product Managers, or the occasional flighty CEO. So how do we interview for that?

Well, first off, trash the “Implement double sort” and every similar algorithm focused question like it. They haven’t been the right question to ask since, well, ever. Unless you’re hiring for a university research lab where you’re developing new algorithms. Then, maybe, you can still start with that.

No, the questions you should be asking should be able to tell you if the candidate meets your hiring bar if they write nothing but pseudo-code, or that they fail to meet that bar if they write syntax-perfect C++, Java, or pick your language-of-the-day.

Instead you should ask questions that make the candidate think, that force them to get clarifications, that make them deconstruct the problem to its bare components, and then demonstrate they can translate those components back into the language of computers.

I’d go so far as to propose the idea, is it really so bad if they’re using AI? Google? Stack overflow? A well built interview question should give you signal about what you care about no matter what tools are used.

I remember a time years ago when Eclipse was not but a young upstart and when its users were accused of lazy-programming because of its auto completion features. Vim and Emacs (and the rare and scorned Notepad++) users looked with disdain at the GUI application that auto-completed method calls, and showed type hits when you used the (shudder) mouse.

But today? Today they’re using language server integrations in their beloved vim. They get the same benefit because the tooling made engineers more efficient at the thing anyone can do (looking up what you called that damn method in the other class you wrote last week) and gave them more time to do their actual jobs of solving the damn problem!

AI is likely no different. It could end up accelerating our teams, helping them implement what they’ve done hundreds or thousands of times before, and let them focus on what they as engineers love most: solving unique technical challenges.

Over the next few weeks I’ll post more diving into how we can craft these questions and how I’ve used this strategy when interviewing and hiring for both SWE and SRE roles.