Take Home Tests

I recently came across a discussion on Reddit concerning take-home tests as part of an interview process. It basically boiled down to:

Is it acceptable for companies to ask candidates to work on a problem in their own time, so that their submitted solution helps decide whether to move forwards to an on-site interview?

The level of antipathy towards take-home tests in that discussion was an enormous surprise. The majority of commenters were not just against the idea, but virulently so, railing against it as another example of unreasonable demands made on candidates. Most people viewed these tests as a waste of time, rude, disrespectful, or a combination of all three. I think there was only one person whose opinion wasn't negative, and even that was ambivalent at best.

I have been asked to do take-home tests in the past, and did see that if poorly conceived and executed they can be a bad experience for the candidate. However, if they're approached in a thoughtful and careful way, they're one of the best tools for effective interviewing. Here's what I think does – and doesn't – work.

What doesn't work

A poorly implemented take-home test can alienate candidates or fail to be a useful indicator of job performance – or both. Here's what to avoid:

Using problems unrelated to the company or the role

It's a well-worn adage that interviews are as much about the candidate assessing the company as the company assessing the candidate. For engineers especially, one of the key indicators for on-the-job satisfaction is whether they feel they're doing challenging, impactful work. As such, why would you give them anything else as part of a take-home test?

Some "off the shelf" question you found on the Internet and pose as a take-home challenge isn't going to give the candidate any insight whatsoever into the kinds of challenges and problem domain they may end up working in if they do eventually join your team.

Not only this, but generic questions absolutely feed the feeling of the process being yet another hoop the disinterested company need them to jump through, rather than a meaningful exchange and chance to learn. Perhaps this fuels the negativity surrounding the idea of such tests at all.

Checking for specific pieces of knowledge

Do you really care whether the candidate has esoteric bits of knowledge at her or his fingertips? Are there APIs and algorithms they simply have to know off by heart?

Not only is the answer to those questions "No" for most roles, a take-home test is a poor way to check for such knowledge anyway.

When all human knowledge is effectively available to us 24/7, it becomes much more important that we can find, filter, and apply information and ideas, rather than them being sat in our heads. Any take-home test which requires the candidate to regurgitate specific knowledge is probably a waste of time.

Of course, for some roles, familiarity with a particular framework or language is crucial, but such familiarity can be tested by asking the candidate to solve a more general question. If they use the tools poorly – or fail to use the right ones at all – that's a better indicator than asking pop quiz-style questions about the tool itself.

No time limit

We know that every piece of an interview process is extremely stressful for a candidate, so why not avoid a time limit to help make this piece less intense?

The answer is fairness and diversity.

We must remember that not every candidate has the same background situation. Perhaps a recent college graduate is willing and able to pull an all-nighter, fitting thirty-six hours of Monster-fuelled work into a weekend, but that's not true of most people. Perhaps the candidate has a family to care for, a health condition, a full-time job they can't take vacation from, ...

We would hate to disadvantage people in a situation like that – and so should you. A time limit helps to ensure a level playing field between candidates.

What works at Teespring

So, with all these pitfalls, why can take-home tests still be one of the best tools for effective interviewing?

An open-ended question without a single, clear solution lets the candidate show their "personality" in their solution. By personality, I mean their values and what they prioritise. If there's a time limit, do they still write tests? Is there documentation or some comments? Which language and/or frameworks did they think were the best fit for the job? As an interviewer you get to choose which traits or values are important to you and your team.
By the end of the time limit the candidate has become very familiar with the challenge: it means that we can dive right into the heart of a non-trivial problem if they do come to an on-site interview. We care a lot about technical communication and analytical skills, and starting the interview with a common understanding of a shared problem lets us exercise those skills much more than if we had a cold start.

Putting it all together, at Teespring we:

Use a take-home challenge based on a real piece of Teespring work. It is boiled down into something approachable in a time limit, but in essence it's very similar to a challenge we faced in our day-to-day work.
Set a three-hour time limit, so that it's fair across all candidates, and can be done in an evening or weekend without being too much of a burden.
Allow the candidate to use any language, framework, or tool that they want (we even had one solution in MiniZinc!).
Care much more about the approach the candidate took rather than the raw algorithmic skill. It's relatively easy for us to learn new skills if we're faced with a daunting challenge, but it's really difficult and time-consuming to change someone's patterns of behaviour.