How I've Learned To Hire Developers - Part II; The Tiered Interview ~ 2015-07-08

First Contact

Okay, so you've got applications piling up. How do you know to whom to reach out for an interview, and how quickly do you respond?

Some conventional wisdom says it's okay to let applications stream in for a while, but this is neglectful of your applicants and risky. Truly excellent applicants may be off the hiring market in less than two weeks. Chances are if a candidate has applied to your position they've also applied to several others. They may even be somewhat far along in the interview process with another employer and nearing an offer. If they're knocking on your door, and they look like a good fit on paper, why wait to talk to them? Get on it! There's an added bonus here that when an applicant hears back quickly it makes you, as an employer, look like you're both organized and invested in them as a candidate. All good things.

As for what to look for in an application: that's tricky. Since development work requires a level of attention to detail such that a single misplaced character can break an application it's perfectly reasonable to marginalize or reject candidates whose applications are riddled with spelling and grammatical errors. Candidates who have a poor grasp of how to present their skills and experience effectively in a clean document with good use of white space are also perfectly valid to write off. Geographic location isn't a protected class, so if you're only interested in candidates that don't need to relocate that's your prerogative.

For a skills match I recommend retaining as much control over the screen as possible. Non-technical folks, usually those from Human Resources, might play a keyword-matching game that will likely weed out some of your best candidates. See Jason Bock's If Carpenters Were Hired Like Programmers. You should be savvy enough to know that a candidate who knows a heaping pile of languages, many being object oriented, scripted, or both, will be able to pick up javascript quite fast despite that keyword not appearing on their resume. Think critically about what your applicants have done and how that might translate into what they can do, and embrace the two critical notions that:

1. No candidate will have the panacea of skills you're looking for
2. Both you and your new employee will want the job to be challenging, and that means leaving room to learn some new things on the job

So, general rules for selecting candidates to interview:

• Be strict in terms of written communication quality (application legibility, spelling, etc.)
• Be lenient in terms of existing skills (incomplete overlap is a best-case scenario)
• If an applicant looks good contact them immediately

Tier 1: The Phone Screen

No matter how great an applicant looks on paper never start things off with a face-to-face interview. The worst feeling in the world is talking to an applicant for two minutes in person and realizing they're not a good fit. They came all the way out to see you, and you don't want to rudely cut off the interview that short, so you agonizingly stretch it out and waste everyone's time. I've been there, it sucks, and it can be easily avoided with a graded (or tiered) interview process.

The first tier is the phone screen. When I reach out to applicants this is what I'm setting up. Initial phone screens can be lightly or heavily structured but at the very least should be time-limited (to respect the schedules of all parties on the call). Here's the general structure of my initial phone screens:

• Limited to 30 minutes (explicitly stated, leniently followed)
• The first ten minutes are for me to ask general questions about skills and experience
• The second ten minutes involves a brief scripted technical assessment
• The final ten minutes allows the applicant to ask any questions they may have

This simple approach crams a lot of information bandwidth into a short time frame. Remember that applicants are interviewing you as much as you're interviewing them, and certainly for the best applicants out there, you may be directly competing with other employers for one person's time and talent. Information bandwidth in conversations is important to max out because information should be flowing both ways. You want your applicants learning as much about you in these conversations as you are about them.

As to that time limit, I say explicitly stated, leniently followed because the applicant should know up front how much time they should set aside and provide you an out to cut it off at a half hour if it's obvious there's no reason to continue. However sometimes the conversation is really constructive so it's perfectly fine to let it stretch a bit to 45 minutes or so. Just don't let it stretch too long and remember it's your responsibility as the interviewer to keep things moving.

The scripted technical assessment is a rather tricky thing to get right. For this I like to keep a document that may change between hiring stretches but be applied consistently during the course of a single hiring phase. Questions should be quick to answer but not be "gotchas" (esoteric details that even seasoned developers may not have memorized). This, again, is difficult to dial in but in general I look for aspects of our tool chain that are pretty basic or universal with unambiguous descriptions or answers. My script evolved some over the years but here's a sampling of what's been on there lately:

My script will usually have more questions than I can ask or have answered in ten minutes just in case an applicant is really on their game and flies through them. I also try to cover a lot of areas—the example above touches on comparative programming languages, web vulnerabilities, object oriented programming, and relational databases. This also affords some flexibility if a candidate is just bombing one section (something that is not necessarily outright grounds for rejection).

As you listen to a candidate work through their answers it should become apparent where lie their strengths and weaknesses with respect to the subject matter. Cherry-pick questions from the script that play to their strengths. Don't be the interviewer who tries to stump your candidates. This makes you look like a petty jerk who wants to just feel smart. It works against well-qualified applicants that may just be blanking or nervous from the pressure of interviewing. Root for your candidates and go out of your way to feed them problems/questions at which they're likely to succeed but are nonetheless challenging. This will tell you far more about your candidates than forcing them to stumble through an area they've already demonstrated limited knowledge thereof.

While this method for a phone screen provides plenty of opportunities for efficiently collecting subjective and objective assessments of a candidate's fit it also provides opportunities for red flags to come up. Certainly if a candidate bombs several of the technical questions or worse—makes up bullshit answers—they can be pretty swiftly rejected. But perhaps more important is the final third of the phone screen where applicants ask questions. If a candidate has no questions to ask, or asks nothing of substance, consider this a massive warning. Good candidates will be interested in all sorts of nuanced aspects of the dev shop and the company, so no questions or questions solely about salary and benefits may point to an applicant that's not invested in the work. To them it may just be a job, and that may signal an early departure, lackluster work, or all kinds of other problems down the line.

When it comes time to close things I usually tell an applicant I'll be in touch within the next few days and then follow up on that expectation with religious conviction. If an applicant is particularly good or bad it may be a time saver for all involved to invite them to the next tier or reject them over the phone, but those are extreme cases. Usually I'll try to have a follow-up email sent out the next business day, no matter what.

Tier 2: Homework

If you're hiring developers than sooner or later you're going to need to see their code. There are a few ways to go about this. One could just ask for code samples submitted via email. One could set up a coding test of sorts by asking applicants to build something, or outsource this task to any of the handful of sites like Codility that offer code testing services. Waiting until a face to face interview and getting your first look at a candidate's coding ability on a whiteboard, however, is going to land you in that same awkward position of knowing an applicant isn't worth your time while you've got a day of interviews already booked. Always see a candidate's code before you see them in person.

Regardless of the approach consider your applicant's time to be a precious and finite resource. They are humans, after all. A monumental homework assignment may force them to reassess whether they want to even go through the trouble, choking off your applicant pool at a critical point in the interview process. Even the best candidates will want to be compensated for a sufficiently large amount of work. Certainly don't give applicants real work to do expecting to get some free labor from your hiring process. That's just being cheap and inconsiderate.

Also consider what you want to test. When I was dialing in a good homework assignment I wanted to allow applicants to showcase whatever code samples they wished—understanding that programming skills are language-agnostic—but I also wanted to assess version control and collaboration skills since those are day-to-day aspects of my dev shop. The natural choice to test these skills while calling for code samples was to use a GitHub repository.

In case the link above fails or directs to something markedly different from what I'm about to describe here, this link points to the last commit I made to the sparkfun/hacker-application repo. Essentially the repository had one top-level README with current job descriptions and instructions to follow the READMEs in each subdirectory. There were then three subdirectories:

• code - A place to hang code samples
• dotfiles - A place to hang custom configurations
• etc - A place for a few additional questions for the applicant to answer

Applicants were encouraged to fork the repository, fill the directories with appropriate files and responses, and commit their completed work back upstream. They could do this publicly by submitting a pull request to the SparkFun repository or by submitting a pull request to their own private fork while inviting myself and other managers as collaborators to keep their responses private. Given that this particular dev shop emphasized open source throughout our operations I found the vast majority of applicants were happy to go the open route, but one should never require applicants to apply publicly.

This approach afforded applicants a lot of flexibility in what they submitted while also testing their ability to work with Git and GitHub. We used both every day and emphasized the pull request / rebase model for development so the homework doubled as a form of on-boarding. Also, aside from learning the work flow and going through the process there are no great hurdles to clear in this task. An applicant could conceivably complete their pull request in fifteen minutes. An applicant with no prior knowledge of Git or GitHub could pick up the tool chain and complete the assignment in under an hour. Applicants could elect to spend more time on it if they chose, but it would be spent figuring out what they wanted to showcase, not because of some tricky problem they couldn't crack.

In general, applicants that completed the homework almost always got an invite to a face-to-face interview. They usually warranted it, and I think this is a combination of the flexibility of the assignment itself but also the effectiveness of the first-tier phone screen in weeding out applicants who didn't warrant a second look. Utilizing the same repository to later hire my own replacement as Director of IT was honestly the first time I encountered folks who just didn't get the assignment and submitted incomplete or just unacceptably poor pull requests, which was a bit of a surprise. But then these are managers we're talking about. =)

Tier 3: The Face-to-Face

Okay, so you've got an applicant who made it through the phone screen and submitted a quality homework assignment. You've seen the code they chose to show you and it looks good. You're ready to bring them in.

I never had a particularly rigorous process for the final interview phase. Preparing a general script of questions in advance to pitch to each candidate consistently is important, but they have to be quality questions. Things like "where do you see yourself in five years?" and "what is your biggest weakness?" are stupid questions that simply aren't constructive or useful. As you prepare your in-depth questions for interview consider that each question should be a jumping-off point into a larger conversation. Also consider what your thought process might be if posed the same question by, say, existing developers on your team or your current boss. Here's a sampling of the ones I used:

All of these questions warrant well more than one word or one sentence answers and are potential entrances to deeper conversations that may have a very real impact on what the candidate will actually do if they arrive on the job in the near future. That's the ultimate goal of the final interview tier: you and the candidate picturing them as a part of the team, in the job, doing the work beginning soon. Does that picture make sense?

This is also the critical time to expose a candidate and the existing dev team to each other. Do not assume that existing developers know the first thing about interviewing candidates. Interviewing is a skill that requires practice and a fair bit of foreknowledge to know what are good questions and what's a waste of time. You may have devs on your team with prejudices against protected classes that could catch you by surprise, or devs who are extremely uncomfortable to be in an interviewer role of any kind.

Train your developers on how to interview ahead of time. Share all of your questions with them and discuss why they're good questions. Encourage them to write and share their own questions too. There's no reason for anyone on the team to keep their questions private—you're all on the same team! Also, interview a given candidate with the entire team all at once. Unless certain developers express interest in one-on-one interviewing it's often the case that developers would rather develop. Minimize the time they need to step into the managerial role and do it all together so repeat questions don't get asked. This can be a bit intimidating for the candidate but forcing a developer into an interviewer situation can be just as intimidating to them and they already got the job. In addition, by interviewing as a group you can be present as a mediator. Your developers can lob questions to get the answers they may be looking for but you've likely already heard most of what will be asked and answered, giving you time to focus on the social interactions with the team.

Just as important as the full-team interview is the full-team debrief. Keep some time open for the team following the interview (twenty minutes or so should be plenty) and give the applicant their send-off in short order. Get back to your team to immediately collect their thoughts and reactions. Especially if many candidates are being paraded through developers on the team will have a hard time distinguishing them if too much time goes by between meeting and assessment. Get the conversation going immediately but steer it away from common pitfalls such as lack of complete conformity with the current mindset of the team. Mediate this conversation just as you did the full-team interview itself, keeping the focus on the skills and experience the candidate could bring to the team and how it would mesh with current haves and needs.

Finally, I recommend operating under the construct that anyone can veto a candidate but everyone must approve a candidate. This empowers existing developers to take ownership of their role in the interview process, because ultimately they have to work with the applicant in question. I encourage my dev teams to express their veto if they feel strongly against a candidate but they must provide concrete reasons for that veto. No hunches or bad vibes. But if one trusted member of the dev team expresses a well-reasoned argument against an applicant that's enough, even if the rest of the dev team is a resounding yes. The advantage here is that offers only get extended to candidates with unanimous support. There are no interpersonal tensions seeded before their first day. And, if you've gone to good lengths to write and post a good job post, hopefully your applicant pool will be broad enough that you can afford to lose some candidates that just couldn't make the unanimous support level.

Graceful Rejections

Rejecting candidates is never fun but it doesn't have to be hard. All it takes is honesty and timeliness.

Put yourself in your candidate's shoes and imagine how much it would suck to sit in limbo not knowing what's next. From there it should come naturally to reject candidates within 24 hours of whatever tier they failed to clear in your book.

If a candidate fell out of favor early in the process then a canned rejection is fine. I usually go for something like this:

This is appropriate when rejecting an application before or immediately after a phone screen. And that brings up an important point: always let applicants know they're rejected even if they've only just applied. Too many employers assume silence to be an adequate stand-in for application rejection, but this is just lazy and inconsiderate. If you have a canned response ready to go like this then take the thirty seconds to fire off an email so the applicant isn't waiting for weeks to hear back when nothing is ever going to show up. Not only can this improve your favor and reputation as an employer but you will actually get thank-yous. So few employers extend this basic courtesy to applicants that some are genuinely surprised and appreciative when they see it promptly.

When rejecting a candidate who's completed a homework assignment or a face-to-face interview the canned response can act as a template but it alone just won't cut it. Using a canned response with no context for the rejection at that stage in the game is almost as bad as rejecting a tier 0 or tier 1 applicant by just not responding. Be specific, honest, and constructive. Maybe the candidate lacks certain technical skills, or is overly invested in a conflicting methodology, or has some other technical or cultural mis-match. While in truth you don't owe your candidates anything these people did invest time an energy to make it this far, so the right thing to do is to summarize as best you can why you believe they're not a good fit.

The vast majority of candidates will accept this criticism and move on. Only rarely will you see a candidate protest, but ultimately you both know that you are the keeper of the position and if you wish to terminate the conversation after sending the one rejection email then that's the end of it. In my experience many candidates have been appreciative at learning just where they didn't measure up because it either gives them ideas of areas for professional development or makes them think differently about what flavors of development positions they should be applying for.

In one particularly incredible turn of events I rejected a candidate after the face-to-face because he lacked a base level of systems knowledge required for the full stack position. He gracefully accepted the rejection and quickly picked up another developer position at a different company with different needs. A little over a year later I was hiring again (from expansion, not turnover) and he applied again. In that year, perhaps in part because of the exposure he got to systems level concepts he lacked when interviewing the first time, this candidate had attained a breadth of advanced systems-level skills such that he was now a shoe-in for the job. He worked for me for another two years and was a singularly gifted developer that brought a unique and valuable skill set to the team—a noteworthy hire all around. Honest, constructive rejection had paid off for all parties.

The Final Negotiation

You're ready to extend an offer to a well-qualified candidate. Fantastic!

Offers typically come in the form of offer letters that detail several things such as the base salary and start date. Salary is almost always the main sticking point but work with your HR people to determine if other benefits are flexible/negotiable. For instance, if your company offers equity to new hires then maybe you can adjust salary and equity with respect to one another to dial in something that works for both parties. Maybe you can negotiate paid time off or other benefits that can be monetized. In general, though, salary is where all the fights take place.

Have a salary range with an absolute cap in mind before you even post your job. If you offer a starting salary at your cap then you've got to stick to your guns, so you may as well be clear up front that the offer is non-negotiable. If you shoot for somewhere in the middle of your range then obviously your offer is negotiable, but you don't need to spell that out. Many candidates do not have experience or just don't feel comfortable negotiating and so won't engage in it if not given an explicit invitation to do so.

When figuring out your salary range consider working with your supervisor or HR people to get a feel for what equitable salaries are for your use case. For this exercise you'll want to work with a regional or local business/employer organization that aggregates salary information. For example, in Colorado we worked with the Mountain States Employers Council to obtain data showing salary ranges based on job type, business sector and size, geographic location, and a handful of other attributes. This is how I could determine definitively that, for example, a $60,000 base salary for a full stack developer at an e-commerce company with 100-200 employees is in the 20th percentile of similar salaries (i.e. not that great). You can use such data to work backwards to a salary too. Let's say you wanted your developer salaries to hit the 50th percentile (e.g. your salary matches or beats 50% of other comparable positions in the region) then you can use the same data to figure out what a 50th percentile salary should be for your business.

However possible just don't be cheap. As one employer put it to me once: don't step over a dollar to pick up a dime. Good developers aren't cheap and they increase in value over time as they grow subject matter expertise in your particular technology stack and applications but also stay connected to the fast evolving world of programming and computer science. When a developer is underpaid they may spend more brain cycles caring about how they're underpaid and that can negatively affect their performance over a long time scale. Underpaid developers may also conclude that they could do better elsewhere and a pattern of underpaying your dev team could result in faster turnover which is way more expensive in the long run. Use these arguments when working with your superior(s) and/or HR to push those percentiles as high as possible. Seek to make your dev shop a destination that developers really want to be a part of and compensate them fairly when they get in. All taken together these factors can generate a healthy and highly productive development team with low turnover.

Conclusion / Continuation

To summarize some of the main points in this article:

• Good applicants will be off the hiring market fast. Contact good looking applicants immediately.
• Define tiers for the interview process:
  • A first tier phone screen, time limited, with an initial technical assessment and opportunity for the applicant to ask questions. Never start with a face-to-face interview.
  • A second tier homework assignment to see an applicant's coding skills and maybe other skills (like version control) too. Always see code samples before a face-to-face interview.
  • A third and final tier of face-to-face interviews with both you and the entire dev team. Always let as much of your team be exposed to a candidate before making an offer.
• If an applicant asks no questions about the work itself or the technology involved that's a huge red flag.
• Don't be the interviewer who tries to stump your candidates. Root for your candidates and keep conversations and challenges open and constructive (i.e. much as it would be if they were already working for you).
• Do not assume that existing developers know the first thing about interviewing candidates. Train your dev team on how to interview effectively and avoid forcing devs into 1-on-1 interview situations; that's not their job.
• Anyone can veto a candidate but everyone must approve a candidate.
• Reject candidates within 24 hours of them failing to measure up in a given tier.
• Silence is not an acceptable form of rejection. Even if an application is completely disconnected from the job the considerate thing to do is to still fire off a rejection email so they know where they stand.
• Work with a regional business/employer organization that aggregates salary information so your salary ranges are tied to actual data. Fight internally for the resources to pay your people well to keep turnover and discontent as low as possible.

The third and final installment of this series of articles will be from the role as a developer looking to get hired. Having just recently transitioned from Director of IT with hire/fire power at one company to Senior Developer with no hiring responsibilities at another company I definitely have a bevy of thoughts on how to look like an attractive candidate to hiring managers. Check back soon!

---