Hiring in the Age of AI - 2

Last post I addressed some of the problems with many of the common engineering interviewing practices found in today’s tech scene and introduced a philosophy of interviewing that focuses on the outcome and product we expect from our hires instead of simply performing a skill assessment for how they use specific tools.   This week I want to dive into some specifics of how we as engineering interviewers can instead evaluate the outcome and product with an example problem, and how to ask our questions in ways that show a candidate’s overall cognitive process.

First, let’s dive into the day of a typical software engineer. Outside of the somewhere from 10% - 50% of their day spent in meetings, I think we can break down a typical day into something along these lines: 

Now, a lot of other tasks fit into these three categories, and the exact splits can change depending on the experience level and role of the engineer, but it captures the high level view.  If we model our interviews off of what an actual day looks like, writing code should be a much less substantial part of our interviews than it is! 

But how do we actually interview for this? 

First let’s think about our typical architectural interview questions.  Those can be used to assess engineering skills that fall in the “Thinking about how code would function” category.  Depending on your engineering practices this could be library structuring, code refactoring, micro-service architecture, or similar questions.  The two most important parts, aside from the architecture answer itself, for this type of interview question are:

1. How does the candidate respond to missing information? 
2. How does the candidate respond to changes to the provided or assumed requirements? 

If you think about this, it’s rare that our engineers get all of the requirements they need as part of their tickets, and it’s even more rare that the first solution (or the first set of requirements…) to a problem is the last. How they respond to curveballs can give you a great indicator on how they will solve similar issues on a day-to-day basis. 

Here’s a practical example of this

“I want you to implement a log collection agent. The agent should collect one or more logs from the system it’s running on and forward them to a centralized collection system.”

There’s only a few ways you’d typically see this implemented. There are also a few key follow-ups to expect from more senior engineers. I’m sure those of you who are engineers are already thinking through them and designing solutions. 

If we focus on the choices for the log collection side of this, one of the first implementation choices we have is Streaming or Polling. 

While the candidate is writing their code I like to ask questions about the tradeoffs for various parts. I do this intentionally during the implementation so I can see the responses to my questions in their implementation. Sometimes this causes a design or implementation change while they’re writing it. This also creates an opportunity to evaluate the candidate’s response to changes. Engaging in a conversation about the design and tradeoffs does (at least as of this writing…) create some obvious tells if the candidate is playing a game of telephone with them in the middle between you and an LLM.

“It looks like you’re implementing a streaming solution for this. What are the other ways to implement this?”

Get a discussion going. Ask for tradeoffs. Put new design limitations in place.  If you engage the candidate in enough conversation that they never finish the code but they demonstrate they can pinpoint every system level problem in the architecture, that should be considered a successful interview! 

“How would you change this code if you had 100,000 log sources to forward from?”

Ideally you want to clarify the requirements in ways that force them to reevaluate code they’ve already written, and see if they go back and fix it. At the simplest this might be changing the return type signature for a method, but if you have a particularly quick coder they might have several layers of implementation to get through. Sometimes you want to hint at this and see if they pick it up. This all plays back to the point of the last post. Get the candidate thinking, not just reacting. 

This also plays into another philosophy of mine I’ll save for another post, but I’ll leave some breadcrumbs. Ask questions that the candidate won’t be able to finish and use how far they get and how many insightful questions they ask to tell you how senior they are.

We can also jump around their code to help get more signal. Taking our previous example, once they’ve demonstrated what you want to see around log collection, dive straight into how they’d handle the network transmission mechanisms to send it back to a centralized collector. 

“Great! We’ve got to where I wanted on the log collection. Let’s talk about how you’d send the logs on to a collector now!”

This context switch will give you more opportunities to evaluate how well the candidate is holding the overall design of a system, and not just a single implemented component. I rarely use this to filter out a candidate, but it is a great indicator for their level of seniority.

Taken together, adjusting your interview to include these strategies lets you evaluate a candidate’s problem solving skills, ability to think on the spot while adjusting to changes in requirements, and how well they are able to keep an intended design and the components in mind along with how the components interact and depend on each other.