Updated 2/25/17
Author: Tim Fiola
There are two big factors as to whether a candidate is a good fit for your organization:
The technical screen/interview plays a key role in the latter item: determining if the candidate is a fit to help solve the engineering problems your organization is facing. Given your specific set of engineering challenges and goals, you need to ensure that your hire is the right one. Failure to get this right will have an impact, both immediate and medium-term. This article offers some advice on how to get this right and potential pitfalls in the technical screening process in some companies.
As many have in the techical field, I've been in my share of interviews. Often times I've set out on the interview trail with the specific goal of simply seeing what is out there in order to see what the job and techical landscape looks like as far as the problem sets and the skill sets in demand. As such, I've been in my share of technical screening interviews. I'd like to share the highlights of a recent experience that reflects a larger trend I've seen and offer some tips. To be clear, this article applies specifically to network engineering roles.
I had recently applied for a senior network engineering position at a company with a several datacenters connected via their own carrier-grade wide area network. The person doing the technical screening gave me a call. After some polite and friendly introductions, he started the formal technical screen. The first question he asked was how traceroute works. It'd been a while since I had studied those specific mechanics, but I still knew the main points: the series of packets sent with an incrementing TTL, with each host along the way sending a TTL expired message until the destination was reached. There were a few things I had forgotten, such as what type of message the destination host sends back and such, but I felt fine about that because as a senior contributor, the specific mechanics of traceroute aren't anything near a frequent concern for me: yes, it's a useful and interesting mechanic that makes it work, but I feel no need to memorize that and carry it with me.
The interviewer then asked me what specific IGPs I felt comfortable with. Good, I thought, we're off to the races now with something specific. I told him we could talk about OSPF if that was OK. The next question threw me off a bit: how does OSPF deal with a host sending malicious traffic? I politely explained that as far as I knew, OSPF doesn't deal with that: OSPF is an IGP, with one of the main purposes being to distribute next hops. There is no native capability in OSPF to deal with a malicious host. A malicious host is a security problem and it requires a security solution. We discussed that a bit but he again came back as to what I would do to neutralize the situation. I answered that without any specific knowledge of the topology or environment, I'd either shut the port facing the malicious host or implement a filter on the port closest to the host to eliminate the malicious traffic. After a bit of discussion on this, the interviewer made clear that the answer he was looking for was to inject a static route into OSPF to route traffic with the host's source address to Null0, because that was more scalable. To be honest, that solution had never occurred to me because it doesn't solve a security problem with a security solution; it deals with a security problem by implementing a routing solution that also bloats the IGP (and doing that is not scalable). We moved on a bit and talked about the differences between iBGP and eBGP and how reverse path forwarding works.
Finally, we talked about automation; this thrilled me. We talked about automation for as much time as we had for the previous topics combined and I really enjoyed it; if you ever have an hour to kill, buy me some good tequila and ask about my opinion on automation. So anyway, throughout the interview, I was waiting to get to the part where I am grilled on the strengths I list in my resumé; I figured he'd want to assess my depth of knowledge in these areas to see how well I can get into the nuts and bolts of technology I work with. Alas, that moment never came: after that series of questions, he asked me if I had any questions for him. I asked him if he had any questions about my resumé. He replied that he hadn't looked at my resumé: he liked to interview everyone with the same questions and then look at their resumé.
He asked if I had any other questions. Yes, I said, what are the skill sets you are looking for in the ideal candidate for this position? He replied that the skill sets he was looking for were covered in the questions he had asked. That answer floored me. It floored me so much that I need a whole web page to explain why.
Taking the technical interviewer at his word, the questions he asked me were indicative of the skill sets they were looking for to fill this senior network engineering role. Reviewing what he asked, then: this company was looking for a senior network engineer who
Of the five items in that list, the first four seem like qualifications/questions for an entry level network engineer: someone who'd recently completed some training of some type in networking basics but with little to no practical experience to talk about or draw upon. Those first four items are things that the junior level engineer is likely to have recently studied; assessing their understanding of those items gives insight as to what they have studied and how well. The fifth one was indeed legitimate for the senior engineer level interview: automation in the network context likely requires a mid to senior level engineer because it involves automating a network to allow it to scale. So if we take the technical interviewer at his word in this instance, he's screening for a senior level engineer by mainly asking questions for an entry level position and not examining the candidate resumé beforehand. I don't believe for one second that these questions really reflect the types of problems they need their senior level engineers to solve. Additionally, by not looking at a candidate's resumé beforehand, the technical screener loses a major opportunity do a deep dive in the candidate's areas of expertise. This deep dive is very helpful in determining a candidate's abilities to truly understand and then apply the technology they primarily work with.
This recent experience of mine is not indicative of every technical interview I've had, but it is very indicative of the type technical interviews conducted by more companies than you'd expect: generic/same questions for everyone, questions not indicative of the problems they really need to solve, and no attention paid to the resumé. Why do they do this? I have some ideas but none of them are satisfactory: they want every candidate to face the same questions in the name of fairness, the candidate's resumé doesn't mean anything because we have a specific set of problems to solve, assess how well can the candidates deal with questions outside of their area of expertise, determine how well they can problem solve and think on the fly. None of these justifications are valid for a process that will not get you the best candidate. In conducting themselves like this, the organization has turned their interview process into a quiz: a set of standardized questions with no regard for the candidate's actual body of work.
One method to understand how a candidate really solves problems is to look at their body of work and ask about specific instances in their experience that may relate to or be similar to a problem the hiring organization is facing. Relying solely on quiz questions to see how much random knowledge a candidate is carrying with them or seeing how they solve problems on the fly is lazy interviewing. For one thing, maybe the candidate is smart enough to understand that it's not necessary to memorize random stuff that they can look up; what's more important is that once they do have facts, can they smartly apply those facts and knowledge? Some candidates are not particularly good with problem solving with someone looking over their shoulder.
And another thing about problem solving on the fly: is that REALLY a needed qualification? Engineering and architecture are, by nature, very deliberate functions: no one should be making stuff up on the fly* in an engineering group; I've not once been walking down the hall and got mugged into a room by a VP and asked to design an algorithm for whatever that we need to deploy in the network in 10 minutes. That's not to say some of the knowledge uncovered in that type of interview is not interesting, because it can be valuable data as part of a set of data that includes other information in the technical screening process. It's also not to say that some brilliant people don't get hired on after these types of interviews: I know many brilliant people who got hired on to good companies by passing the quiz interview.
On the other hand, I also know of many brilliant people who've been turned away after the quiz style interview: those companies had no idea of the brilliance they turned away in these people. I'd argue that relying solely on this quiz style is a crapshoot as to whether this person can really help your organization or not. By doing so, you risk the right person being sent away and/or the wrong person being hired. Yes, I fully acknowledge that being able to think on your feet is an important and desirable trait, but screening out people who may not be strong in that particular area is a mistake, because that's not the only or necessarily most important way to contribute as a network engineer.
*I keep italicizing the on the fly term because it sounds sexy but in reality should not necessarily be a major criteria for hiring in a field that, by nature, is deliberate and measured. ---But can he do things on the fly?!--- OMG, shut UP, dipshit.
Looking at the situation above, a person I very much admire, Raymond Hettinger, would pound his fist on the table and exclaim there MUST be a better way!! Then he'd go code something in python core that would kick ass. In our situation here, I'd offer this guidance for conducting a technical screen, if you are ever asked to do so:
This entire preparation process can be done in 10-20 minutes. That's a reasonable investment in time to make sure you find the best candidate.
Why aren't more technical interviews conducted this way? Maybe they just don't know how to interview people, maybe it's a policy handed down from on high, maybe there's some institutional laziness that allows engineers to get away with this type of interview, maybe my ideas aren't as good as I think they are. There are numerous reasons why this may not be happening at a given company. None of them are justified, and one is just plain crazy (hint: that's the one where I suggest my ideas may not be good. How in the world could THAT be the case? amiright?! . . . nevermind . . . don't answer that, the look on your face says it all). Regardless, I know one size doesn't fit all, and there may be some valid criticisms of the suggestions here. But there's one thing for certain: putting some thought, innovation, and effort into crafting your interview style will provide a much better spectrum of data points by which to judge the applicant than a process that simply amounts to nothing more than a technical quiz. Thoughts? I'd love to hear your feedback!