Category: Process

The Sorting Hat

I recently interviewed some programmers for a couple of available positions at CTS, the startup crazy enough to take me as its CTO. The interviews started with me giving the candidate a quick overview of the company and the software that we’re hiring people to implement, after which it would be the candidate’s turn to tell his or her story. With the introductions out of the way — usually at the ten- or fifteen-minute mark of the hour-long session, I ran each candidate through the now-infamous “FizzBuzz” programming test:

Write a program that prints out the numbers from 1 through 100, but…

• For numbers that are multiples of 3, print “Fizz” instead of the number.
• For numbers that are multiples of 5, print “Buzz” instead of the number
• For numbers that are multiples of both 3 and 5, print “FizzBuzz” instead of the number.

That’s it!

This was the “sorting hat”: the interview took different tacks depending on whether the candidate could or couldn’t pass this test.

I asked each candidate to do the exercise in the programming language of their choice, with pen and paper rather than on a computer. By requiring them to use pen and paper rather than letting them use a computer, I prevented them from simply Googling a solution. It would also force them to actually think about the solution rather than simply go through the “type in some code / run it and see what happens / type in some more code” cycle. In exchange for taking away the ability for candidates to verify their code, I would be a little more forgiving than the compiler or interpreter, letting missing semicolons or similarly small errors slide.

The candidates sat at my desk and wrote out their code while I kept an eye on the time, with the intention of invoking the Mercy Rule at the ten-minute mark. They were free to ask any questions during the session; the most common was “Is there some sort of trick in the wording of the assignment?” (there isn’t), and the most unusual was “Do you think I should use a loop?”, which nearly made me end the interview right then and there. Had any of them asked to see what some sample output would look like, I would have shown them a text file containing the first 35 lines of output from a working FizzBuzz program.

The interviews concluded last week, after which we tallied the results: 40% of the candidates passed the FizzBuzz test. Worse still, I had to invoke the Mercy Rule for one candidate who hit the ten-minute mark without a solution.

The Legend of FizzBuzz

I was surprised to find that only one of the candidates had heard of FizzBuzz. I was under the impression that that it had worked its way into the developer world’s collective consciousness since Imran Ghory wrote about the test back in January 2007:

On occasion you meet a developer who seems like a solid programmer. They know their theory, they know their language. They can have a reasonable conversation about programming. But once it comes down to actually producing code they just don’t seem to be able to do it well.

You would probably think they’re a good developer if you’ld never seen them code. This is why you have to ask people to write code for you if you really want to see how good they are. It doesn’t matter if their CV looks great or they talk a great talk. If they can’t write code well you probably don’t want them on your team.

After a fair bit of trial and error I’ve come to discover that people who struggle to code don’t just struggle on big problems, or even smallish problems (i.e. write a implementation of a linked list). They struggle with tiny problems.

So I set out to develop questions that can identify this kind of developer and came up with a class of questions I call “FizzBuzz Questions” named after a game children often play (or are made to play) in schools in the UK.

Ghory’s post caught the attention of one of the bright lights of the Ruby and Hacker News community, Reg “Raganwald” Braithwaite. Reg wrote an article titled Don’t Overthink FizzBuzz, which in turn was read by 800-pound gorilla of tech blogging, Jeff “Coding Horror” Atwood.  It prompted Atwood to ask why many people who call themselves programmers can’t program. From there, FizzBuzz took on a life of its own — just do a search on the term “FizzBuzz”; there are now lots of blog entries on the topic (including this one now).

In Dave Fayram’s recent article on FizzBuzz (where he took it to strange new heights, using monoids), he observed that FizzBuzz incorporates elements that programs typically perform:

1. Iterating over a group of entities,
2. accumulating data about that group,
3. providing a sane alternative if no data is available, and
4. producing output that is meaningful to the user.

My FizzBuzz Solution

I can’t go fairly judge other programmers with FizzBuzz without going through the test myself. I implemented it in Ruby, my go-to language for hammering out quick solutions, and in the spirit of fairness to those I interviewed, I did it with pen and paper on the first day of the interviews (and confirmed it by entering and running it):

I consider myself a half-decent programmer, and I’m pleased that I could bang out this solution on paper in five minutes. My favourite candidates from our interviews also wrote up their solutions in about the same time.

A Forty Percent Pass Rate?!

Okay, the picture above shows a 33% pass rate. Close enough.

How is it that only 40% of the candidates could write FizzBuzz? Consider that:

• All the candidates had some kind of college- or university-level computer programming education; many had computer science certificates or degrees.
• The candidates had varying degrees of work experience, ranging from years to decades.
• Each candidate could point to a sizeable, completed project on which they worked.
• All the candidates came to us pre-screened by a recruiting company.

A whopping sixty percent of the candidates made it past these “filters” and still failed FizzBuzz, and I’m not sure how. I have some guesses:

• Moving away from programming: Some of them may have started as developers initially, but over time, their work changed. They still hold some kind of programming-related title and their work still has something to do with the applications they were hired to develop, but these days, most of their job involves tasks outside the realm of actually writing code.
• “Copy and paste, adjust to taste”-style programming: Also known as “Coding by Googling”,  this is when a developer writes applications by searching for code that’s similar to the problem that s/he’s trying to solve, copy-and-pastes it, and then tweaks it as needed. I get the feeling that a lot of client-side JavaScript is coded this way. This is one of the reasons I insisted that the FizzBuzz exercise be done on paper rather than with a computer.
• Overspecialization: It could be that some candidates have simply been writing the same kind of program over and over, and now do their work “on autopilot”. I’ve heard that FizzBuzz is pretty good at catching this sort of developer unaware.

If you’re interviewing prospective developers, you may want to hit them with the FizzBuzz test and make them do it on paper or on a whiteboard. The test may be “old news” — Imran wrote about it at the start of 2007; that’s twenty internet years ago! — but apparently, it still works for sorting out people who just can’t code.

Be sure to read the follow-up article, Further Into FizzBuzz!