How about sitting the candidate down in front of an actual computer, or ask the candidate to bring in their own laptop? Then if you want to see how they do, pick out some open source projects that need a feature or two added, or a bug fixed, and have the candidate walk through some solution? This way, you are getting to see what their style is, and the random open source projects gets a potential improvement. And you won't get accused of having the candidate programming for you without pay, because contributing to random open source projects is kind of like charity or community volunteer work.

Or, here's a fun project. Have the candidate write up (in the language of their choice, or preferably multiple languages) a program that generates solutions for the Cracker Barrel peg board game (or variations thereof).

So instead of giving them an algorithm problem that they might have seen in the past (or seen something similar) you give them a random open source project -- along with all the idiosycracies it certainly contains -- that they are 100% unfamiliar with and ask them within a few minutes to make sense of the entire codebase and produce a meaningful contribution? Sorry, but that sounds even more cruel and unusual than asking a very obscure algorithm question.

This might be viable if you do it remotely: let them take their time to understand the project, understand the task, find the relevant section of the project, find the relevant section of code, understand the change they need to make, set up the development environment, and make the change.

That's at minimum a day's work for most tasks, and sometimes two if setup or debugging turns out to be more difficult than expected. It definitely doesn't scale as an in-person interview, but by doing it asynchronously you lose a lot of the live communication and thought-process data that current interviews provide…

…then again, you could take this opportunity to measure their ability to give asynchronous status updates instead: can they clearly document their progress, the challenges they faced, and how they overcame them?

I personally wouldn't be willing to spend 8 or even 16 hours working on an interview question for the chance to receive an offer. I'd take the one hour whiteboard dance.

What if you were compensated for your time?

My three favorite evaluation processes so far (from the POV of the interviewee):

1. Quick phone screen with someone technically competent enough to call me on my BS if it were painfully obvious, followed by a "come in and let's get you working as a contractor".

I went through this scenario twice, and both times it worked out rather well. Of course, there was risk involved with both parties, but it seemed to work fine for both a 5-person startup and 100-employee agency.

2. Phone call to talk generalities and big picture, followed by a lunch meeting that doubled as a conversational technical interview, followed by a take-home exercise on a paid-for-time-if-no-offer basis.

I went through this scenario once, and liked it even better. The idea behind paying me if no offer follows was that I could be (and was) given a real problem the company needs solved (small, fairly standalone feature in my case) that I could work out for them for a reasonable fee. They'd get full rights to the work, I wouldn't feel like it's a waste of my time if nothing comes of it, and it would all be nil and void if I were offered a job in the end.

3. Quick 30-minute phone call with a few team members to get a sense for what the position, team, and I are all about, followed by a 5-hour marathon in-person interview, but broken up into more digestible chunks: 45-min chat with one of the leads, 45-min chat with someone more on par with the position, 45-min whiteboard problem, 45-min session of actually solving problems at a real computer (while being encouraged to do it the way I would, using Google, Stack Overflow, asking them things as I would ask coworkers, etc.), etc.

This last one was at a BigCo in the Valley and I felt was a pretty damn solid way to go about it. I actually basically bombed the whiteboard problem, but didn't feel like it was an unfair question to ask me, as the single interviewer present was more interested in my process than my reciting a memorized answer.

People with visas cannot be compensated like this, which is a large part of your hiring pool.

I like this idea for small firms, but how does this scale to the level that the parent comment is asking about? If we want to interview a few hundred candidates this month, who finds the few hundred open-source project tasks and audits them for appropriateness?

And, for what it's worth, the peg game problem sounds to me like standard interview fare: an interesting puzzle that bears little resemblance to real-world software challenges. Why is it a better alternative?

It's actually not that hard -- invite them to work with you on a small project. Pay them reasonably. See how they work in RL. Then decide, and -- by all means -- give them honest feedback even when not hiring.

We did that one time. On the one had it worked very well. The person wasn't a good fit on the team so we declined. On the other hand it was emotionally draining because the guy was a really nice guy. It's really, really hard to work with someone, get to know them a bit and then at the end say, "Sorry, you're just not for us". I think it was even harder for the candidate who didn't clue in that they weren't doing very well. They met the team, got to know people a bit, got to feel the atmosphere and started to imagine what it would be like to work with us.

In the end it was such an upsetting experience for everybody that we agreed never to do it again. I think there are probably ways to make it work, but you need to be really careful to maintain some distance... But then, will you get the result you are looking for if you do?

In Germany this is called "Probearbeiten" and is fairly common for software or design jobs: when you have passed the (mostly theoretical) interview, you're invited to work a day or up to one week to evaluate your practical skills.

Generally you tend to be given a few tasks that are representative for the work you will be doing if you get hired. This is similar to spec work (e.g. the result will typically be discarded unless it is exceptionally good and solves real-world problems) but typically you will be reimbursed if you are not hired.

For a programming job you may be given a task that doesn't require intimate knowledge of any of the company's codebases but should give some insights into how you work, followed by a short review.

It's important to note that this doesn't scale well. It works best if the candidate works on site and can work alongside future team mates, which may impact the team's productivity for the duration. This approach works best for small to medium scale companies with a small pool of viable candidates. It's beneficial for the company to only send a candidate through this process if they're very likely to hire them.

I actually prefer this approach. By the time you're invited for "Probearbeiten" both sides are fairly confident you're going to be hired and it gives both sides a chance to determine whether it's a good fit.

It should also be noted that even in companies that don't do "Probearbeiten" there's a trial period ("Probezeit") after you're hired of up to six months where you're pretty much employed "at will" and can be fired on the spot.

I don't think that was the question, though. Small firms can afford to do this, but how does this scale to companies with who want to interview a few hundred candidates this month? Where do the hundreds of real-world projects come from?

This method doesn't work well for hiring people that already have a full-time job elsewhere.