Interviewing is a controversial topic in Silicon Valley. Should you attract and filter talent using puzzles? Should interviewers avoid puzzles and brain teasers altogether? Of course I don’t have the answer. But I have an opinion.

I always use exactly one puzzle when I interview candidates for tech positions (such as software engineering and security R&D). I believe the ability to solve this puzzle strongly indicates that the candidate thinks scientifically, which is very important in tech positions. I rarely recommend moving forward with a candidate unless he or she passes this puzzle.

The puzzle tests the candidate for a specific cognitive bias and works like this:

I will give you a sequence of three numbers (a triple). This triple adheres to a particular rule. Your goal is to figure out what this rule is. You will figure out what the rule is by proposing new triples to me. For each triple you give me, I will respond by telling you whether that triple adheres to the rule, or breaks the rule. You can give me as many triples as you like, but you only get one chance to guess the rule.

An example triple that follows the rule is 2, 4, 6 √. (The check mark indicates that the triple follows the rule).

Confirmation bias

This puzzle tests candidates for their susceptibility to the confirmation bias.

When given the puzzle, people tend to develop an initial hypothesis such as “ascending even numbers” (since it seems to nicely generalize the example triple 2, 4, 6 ). Once candidates have an initial hypothesis, they can go two routes in their investigation: confirmation and falsification.

Confirmation

Scientists have studied the confirmation bias using this exact puzzle, and the research has shown that the average Joe tends to confirm hypotheses, rather than attempt to refute them. For example, if your initial hypothesis is “ascending even numbers” you could confirm your hypothesis by giving triples that follow the rule, such as:

• 4, 6, 8 √
• 108, 116, 140 √
• -8, -6, -4 √

In the particular puzzle that I give, all of the above triples follow the rule. And if you fall victim to the confirmation bias, you might convince yourself that your hypothesis is correct based on the above evidence.

But in fact, the correct rule is not “ascending even numbers.”

So how do you solve the puzzle?

Falsification

To correctly solve the puzzle you must attempt to falsify your hypothesis (in addition to confirming it). For example, if your initial hypothesis is “ascending even numbers” you could refute your hypothesis by giving triples that attempt to break the rule, such as:

• 6, 4, 2 X
• 3, 2, 1 X 
• 1, 2, 3 √

In the above cases the last triple (1, 2, 3) follows the rule, while the first two break the rule. This evidence refutes the hypothesis (because if the hypothesis were true, 1, 2, 3 would break the rule).

Once you’ve falsified a hypothesis, you need to develop a new hypothesis and repeat the process.

Solution

The correct rule is “ascending numbers.” However, guessing the correct rule is not enough to get credit for solving the puzzle. To solve the puzzle you must scientifically attempt to confirm and refute your hypotheses until you’ve gathered enough evidence to confidently guess the correct rule.

Relevance

As I mentioned before, research has shown that most people fail to solve this puzzle. What then is the relevance of identifying whether or not someone can pass this puzzle? If they fail, you might argue “So what? Everyone has cognitive biases.”

To succeed in a technical job you must be able to think scientifically, and overcoming the confirmation bias is a cornerstone of scientific thinking.

For example, when debugging faulty software, you might be inclined to stop as soon as the faulty behavior disappears. But what you really need to do is figure out what is causing the faulty behavior? Don’t trick yourself into thinking you’ve figured it out until you’ve honed in on the problem by rigorously testing your hypotheses.

Or for example say you’re writing unit tests for your software. You carefully wrote the software under test and you’re feeling good that it’s correct. You might be tempted to take it easy on the unit tests, but really you must try as best you can to write unit tests that break your code.

So there it is.  Use this puzzle and gain confidence that the people you hire think scientifically!