Iyfjky

Question

Many companies - particularly in the tech sector - use puzzles of various kinds (including riddles, lateral-thinking questions, and guesstimation questions) in job interviews for potential new recruits.

To what extent is this practice effective or ineffective? Is there any correlation between puzzle-solving performance and job effectiveness?

If there is evidence of correlation, then for what types of roles? And what types of puzzles?

And if (as I currently understand to be the case) there is little or no evidence to support the effectiveness of puzzle-based job interviews, then why do companies persist with this practice?

Update: Thank you to everyone who has posted answers so far. Mostly what we have at the moment though is opinion, personal experience and anecdote - are there any actual data on this? I.e. studies?

I think it is rather telling that many big software companies are phasing out the use of puzzles. I believe both Google and Facebook have done away with it. — Sep 1 '15 at 19:05
Possible duplicate of workplace.stackexchange.com/questions/36580/â€¦ — Sep 2 '15 at 7:01
Comments are not for extended discussion; this conversation has been moved to chat. — Sep 2 '15 at 20:55

Kate Gregory 104k40230332 · Answer 1 · 2015-09-01 11:41:48Z

First, a disclaimer: when I interview people I do not set them puzzles. I used to and I stopped, because they gave me false positives and false negatives. I never used a puzzle from a book or the internet though. Second, some perspective: chefs, nannies, accountants, heads of HR, and university presidents are not asked these things. It's mostly software developers.

Now, why set a puzzle? Many people think it's a proxy for many software development tasks. A bug is a bit like a puzzle or a riddle (except that nobody deliberately set it up for you to solve) and by demonstrating in a job interview how you systematically approach such a problem, the theory is that you're demonstrating what a good problem-solver you are in general. Even when the solver says things like "it's basically lanterns-and-bridge but with pigs" they're showing how they can recognize one problem in another and apply previous knowledge. That seems like it would be all good. People who can see patterns (all these numbers are one more than a power of two, or only letters with even ASCII codes are coming out wrong) do tend to solve bugs faster than people who don't. People who can quickly generate a hypothesis and test it are also often good debuggers.

The biggest problem is the false positives. People who aren't able to reason logically, generate hypotheses, spot patterns, figure out the weak link, make a leap, or even do basic math can nonetheless memorize, and these are the absolutely worst people to hire for software development or other creative jobs. Some of them can learn how to pretend they are working out the solution on the spot, and fool you.

A smaller problem, but still real, is false negatives. People who are excellent debuggers but just think it's silly that a group of such a size would not have enough light left for people to cross the bridge however they want, or feel that once one person is over the bridge the others can cross in the dark, and they just won't let go and get playful and try to solve the underlying math problem (arrange these numbers into pairs using the following constraints.) People who write great code and are offended by being asked to do something they're not good at in order to get a coding job. That sort of thing. There's also a real risk that the solver produces an answer that is better than the asker had in mind, and is rejected for not "getting it right."

I ask people if they like puzzles, and why. But I don't ask them to do a puzzle. And I would not study puzzles as part of a job-interview prep process.

**unless the job interview is for a Puzzle Solver, right? Someone's gotta QA all those bits in the paper next to the funnies! ;) — Ian MacDonald, Sep 1 '15 at 12:15
@Dukeling I think someone's refusal to accept premises about lanterns with precisely 17 minutes of light and such is not a good indicator about their refusal to accept premises like what programming language we'll be using on this project or what platform to target. — Sep 1 '15 at 19:08
@Chan-HoSuh easy to say, but most of these things have a simple trick (eg weighing balls three at a time, or sending the two fastest people together and the two slowest people together) that cannot be "dug deeper" into. You get it or you don't. Once you get it, you get it, and there's really no way I can test whether you get it because you thought it through yourself, or because someone else explained it to you before. — Sep 1 '15 at 19:10
That's a very interesting perspective WRT puzzles as "a proxy for many software development tasks." When interviewing for my current job, they simply did away with the proxy: they asked me to code a solution to a moderately difficult problem, and then presented me with an IDE containing a broken program, showed me how to reproduce the bug, and asked me to fix it. In both cases the interviewers were watching what I did the whole time so they could observe my development process and see how I worked my way through things. — Sep 1 '15 at 19:39
I once had a question on getting a PC working where they were fishing for the fact that DHCP in windows defaults to one particular subnet when a DHCP server is unavailable. The way I went about solving it jumped right past that and IRL would have solved the problem but 'failed' their test cos it wasn't the thing they were focused on. Once they told me the 'right' answer I was put in the awkward position of appearing wrong or making them look foolish. — Sep 2 '15 at 9:35

score 41 · Answer 2 · 2015-09-03 09:42:13Z

To what extent is this practice effective or ineffective? Is there any
correlation between puzzle-solving performance and job effectiveness?

If there is evidence of correlation, then for what types of roles? And
what types of puzzles?

And if (as I currently understand to be the case) there is little or
no evidence to support the effectiveness of puzzle-based job
interviews, then why do companies persist with this practice?

Interviewers will often tell you that they use these questions to "see how you think", or "see how you think out of the box", or "see how you think under pressure" or some variant of that.

I don't believe any evidence exists to support such a claim. And I know of no company that can say "we have found that better puzzlers make better employees" with a straight face (puzzle-producing companies excepted).

Puzzle questions are a fad whose time has come and gone. At one time, companies like Microsoft used to ask questions like "why is a manhole cover round?" or "how many ping pong balls would it take to fill a jetliner?". Since folks equated Microsoft with "smart" and everyone wants to seem smart, they often chose similar questioning methods.

I once had a Developer ask a QA candidate I was recruiting "How would you find a needle in a haystack?". When I questioned the Developer if he thought that was a useful QA question, he said it would "tell him how the candidate thinks". When I pressed further, the said it was asked of him once in an interview and he thought it was a "good question". I never asked him to help with interviews again.

Apparently, Google no longer believes such questions are useful, and has some idea of the real purpose of these sorts of questions: http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=0

According to Laszlo Bock, senior vice president for people operations at Google: "They donÃ¢Â€Â™t predict anything. They serve primarily to make the interviewer feel smart."

I agree with Laszlo.

Hiring and interviewing are hard. Coming up with relevant questions to discern the fit of a potential employee is hard. Asking puzzle questions is just lazy, in my opinion, and wastes valuable interview time. Unfortunately, most folks aren't trained on how to conduct an effective interview, so they just go with what they have heard is "smart" or "a best practice" and don't put any more thought into it.

After you are hired and advance in your career such that you are the hiring manager - choose better interview questions! No more "What is the air-speed velocity of an unladen swallow?" please.

Absolutely agree - "How would you find a needle in a haystack" - get a big f***ING magnet! — Sep 1 '15 at 23:08
"how many ping pong balls would it take to fill a jetliner?" <- This is a Fermi estimation problem, not a puzzle. These have no "right" answer, the point is to see how the candidate tackles the estimation process. en.wikipedia.org/wiki/Fermi_problem — Sep 2 '15 at 3:37
@TheWanderingDevManager: My favourite is "Throw the hay in a clear box of water and mix." - hay floats, needles don't :) — Sep 2 '15 at 8:36
"How would you find needle in a haystack?" Get a big fan, blow all the hay away. even better Set the whole haystack on fire, and there you have it ... your precious needle. — Sep 2 '15 at 8:59
Google claim to have stopped doing the brain teasers, but some of their interviewers still spend a large fraction of time on Fermi estimation questions. I'm not convinced that "how many dogs are there in London?" (repeat 5 times with different interviewers, using different nouns and cities) has any stronger correlation with work performance than "guess the weight of the fruitcake". — Sep 2 '15 at 13:22

Kingrames · Answer 3 · 2015-09-01 12:35:51Z

I'm gonna say no.

Puzzles are specifically designed to have a "best" solution and oftentimes, especially in programming jobs, you run into problems where there IS no solution.

Using a puzzle in the job interview process is more than a little condescending, especially since you're talking about someone else's career or potential career and certainly their livelihood, and yet you're treating it like a game. It's very, very disrespectful.

That being said, sometimes it's not the go-to thing for businesses but rather just a "best option" for places hiring. A LOT of managers, HR reps, CEO's, and sometimes even IT leads have no clue what the actual programmers do (and worse, they think they do) and they use these puzzles to put someone in the state of mind they think you'll be in while you're programming. It's less of a skills test and more of an "attitude under stress" test.

But you'll find that most people who hire employees based off of the results of the puzzle part of the test are going to hire terrible people. (You should focus on the resume!)

While I agree with you about the puzzles, I would never focus just on the resume, way too many people lie on resumes about their actual skill level. — Sep 1 '15 at 17:31
You could perhaps also call something like "How many windows are there in New York?" a puzzle - the final answer is not really important at all, the important part is the way of thinking to get there (so that's not something which has a "best" solution). Purely from the condescending / respectfulness viewpoint, I don't see how asking someone to solve a puzzle is that different from making a joke during an interview, having a more relaxed and open interview or even asking them to solve a toy programming problem (which happens in just about every programmer interview). — Sep 1 '15 at 19:11
@Dukeling - the important part is the way of thinking to get there - so like every other puzzle, the solution is "I'd google it"? Because everything else is contrived bullshit and we all know it. — Sep 1 '15 at 20:35
@Dukeling that is a Fermi problem (the classic form is "How many piano tuners are there in Chicago?" though I've also seen "how many gas stations in the United States?"). If you know how to approach it, it isn't a puzzle at all - just a matter of multiplying guesses and knowing which ones to guess at. — user10042, Sep 1 '15 at 20:56

keshlam 41.5k1267144 · Answer 4 · 2015-09-01 22:15:38Z

Programming puzzles wind up, at best, being the equivalent of compiler micro-benchmarks. They're much more likely to tell you whether the programmer happens to have previously solved that problem or a similar one than to tell you how good they are at problem solving generally.

This. A good software engineer would consider multiple approaches, evaluate them and work in an iterative way to come up with the right solution. You don't solve everything in one hit off the spot. — Sep 1 '15 at 22:51

BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â› 1,4501524 · Answer 5 · 2015-09-02 08:27:36Z

up vote
8
down vote

I work as a senior software engineer in Germany. In the last 6 years, I had at least 25 interviews. I was asked only once a puzzle question. Needless to say, whole interview was quite odd.

My experience is that people do not ask puzzle questions anymore. At least not in IT sector. For example, in google Ã¢Â€ÂœWhat works well are structured behavioral interviews, where you have a consistent rubric for how you assess peopleÃ¢Â€Â.

Even microsoft, where the puzzle questions were born, stopped asking such questions since they realized it is a waste of time. There is this famous article "What would Feynman do?", which discredit such questions in quite a funny way.

answered Sep 2 '15 at 8:27

BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›

1,4501524

The practice is still alive and well here in London, unfortunately.
â€“Â A E
Sep 2 '15 at 9:29

5

Loved the Feynman link.. reminded me of Bohr's barometer question - to measure the height of a building using a barometer, ask the janitor and offer him the barometer if he tells you the height!
â€“Â gbjbaanb
Sep 2 '15 at 10:14

1

Incidentally, some guy once complained to me how math story problems often are like "imagine a drinking straw lying along the diagonal of a box, compute its length..." when the point is to just have you compute the diagonal's length. Story problems are full of artificial things like this. He said the problem is undefined because the circular diameter of the straw or its thickness was not given. I wonder who here would take the point as far as this guy. (Incidentally this guy won a Fields Medal, the "Nobel Prize" of mathematics, so it's not as if he couldn't do these story problems)
â€“Â Chan-Ho Suh
Sep 2 '15 at 12:48

1

@Chan-HoSuh You obviously missed the point. On the interview I had, the guy asked me to design a trash can. What kind of moron would ask that for a software engineering position? Regarding a guy like Feyman, how did you conclude that he "probably wouldn't be suited for industry", if he didn't answer idiotic puzzle the way interviewer wanted? The article I linked tells that such questions can give false positive and negative conclusions.
â€“Â BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›
Sep 2 '15 at 12:51

1

@Chan-HoSuh When you mentioned "drinking straw", first I thought of this ;) and "really. how?" :)
â€“Â BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›
Sep 2 '15 at 13:27

Â |Â
show 5 more comments

jamesdlin 1604 · Answer 6 · 2015-09-02 08:08:42Z

I sometimes (but rarely) ask puzzles. When I do, the puzzles I ask are not ones that require an a-ha moment (a flash of insight) but are ones that can be solved with logical deduction: can the candidate break down the problem into a smaller problem, solve that, and then use that result to solve a larger problem?

Or sometimes I'll ask non-puzzle questions, and if I have time leftover, I sometimes might chew up some time with puzzles. But when I do, the point isn't to try to use it as some proxy for gauging their programming ability; the point is to find a data point for culture fit because programmers I work with tend to like puzzles too. And maybe the candidate has heard the problem before, but that can happen with non-puzzle programming questions too. If they have (and say so), maybe the information I gather from it is that the candidate likes puzzles, knows how to get answers, and is honest.

I think that they can be sometimes useful, but you have to be careful with what puzzles you use and think about what you're actually trying to measure.

Probably the most reasonable answer here.
â€“Â Chan-Ho Suh
Sep 2 '15 at 12:37 — Sep 2 '15 at 12:37

Chan-Ho Suh 1,94111011 · Answer 7 · 2015-09-01 19:05:34Z

Besides software development, one sector where puzzles / brainteasers / quantitative problem-solving is tested as part of the job interview process is in technical roles in finance.

I've heard a couple reasons given for testing interviewees in such a manner:

1) it's a form of stress test. Since there are a lot of jobs where a person has to be very quick on their feet and be able to apply their quantitative skills at a moment's notice, this should be tested somehow.

2) It's an efficient test of basic skills. Finance tends to attract people of diverse academic backgrounds. How do you verify that someone that says they are "good at probability" is really so? Does having a math PhD suffice? There are plenty of people with only basic undergrad math that have better probabilistic reasoning skills than many math PhDs.

@JoeStrazzere I don't think asking a more complicated question with more to worry about is necessarily going to shed light on that. Separate questions can be asked on financial concepts. — Sep 1 '15 at 20:10
If you want to verify CV claims, ask the claims. Tell me about projects A and B. What issues did you experience? What did you do about it? What lesson have you taken with you? — Sep 2 '15 at 8:49
@Gusdor What you propose only works for certain claims. Certainly it won't help filter out the candidates with good basic technical skills from bad. — Sep 2 '15 at 12:34
@Chan-HoSuh Are you sure? Candidate "I can write some Python". Interviewer: "Cool, what is your favourite thing about the language? What don't you like?" Those sorts of questions help assess levels of experience. A detailed answer can show depth of knowledge, balanced analysis and passion for a skill or a past job. — Sep 2 '15 at 12:47
I certainly agree that they are commonly used in quant finance. I even wrote a book that's compilation of such puzzles and it's been very successful. One of the reasons I think that they are popular is that it's an attempt to level people from very different backgrounds. The questions vary from very technical eg write a routine in C++ to do X, to silly brainteasers to basic financial probability questions. — Sep 3 '15 at 5:33

JeffC 21716 · Answer 8 · 2015-09-02 17:43:04Z

I would say that it depends on the field. In some cases, like programming, candidates (generally) have a degree and some work history or at least school projects. They can be asked specific questions around programming so, to me, it makes less sense to ask them puzzle questions.

My specific experience is around Software QA. When I interviewed at Microsoft years ago for a QA position, I was straight out of graduate school and had no relevant work experience. There were no degrees in Software QA. So what should they ask? People that haven't done QA (and some that have) generally don't understand QA so it's hard to ask non-testers tester questions and expect to get good answers.

They asked me a lot of puzzle questions because they wanted to see how I went about solving puzzles. It wasn't so much about the answer, it was the thought process behind it. They also asked me simple testing questions... how would you test a date field? As I progressed through the interviews, the next person would ask questions based on what I was previously asked to see if I could learn and apply that knowledge.

You can learn a lot about a person by how they solve a problem. Do they give up immediately when they can't figure it out? Do they ask me lots of questions to try to gain more insight into how to solve it? Do they give one answer and get stuck? These skills are relevant to many jobs, not just QA.

I think the new hotness (well, not so new but anyway...) in interviews is the experience based question. Tell me about a time when you did X. The thinking is that the way you approached an issue in the past is likely the way you will approach it the next time it comes up. That's great... but how do you ask questions like those for a field that the candidate has no experience in? Not saying it can't be done but it's much harder... but, good interview questions are not easy to come up with.

Some people are having issues with the "contrived" puzzles. Yes, they are "contrived" it's on purpose. You are a dev on a project and someone asks you how you would fix X issue. Well, I would do Y... well Y would mean a complete rewrite of the project so Y is not an option. That's not contrived... it's real world and it happens all the time. You have to be able to work within constraints whether you like them or not. A contrived puzzle (at least the good ones) is a parallel to real life. — Sep 2 '15 at 17:47

Cort Ammon 2,7121713 · Answer 9 · 2015-09-03 03:40:37Z

All scientific evidence I have seen fails to demonstrate puzzles as effective. However, I believe they have some potential in non-scientific approaches. Consider that everything in the interview process is a tool. They're tools to help you and the company determine if they are good matches for each other (mostly to help the company, but it goes both ways). Accordingly, the interviewer uses any tools at their disposal. Some interviewers may find that asking a puzzle question is an effective tool in their toolbox to learn more about you, and then ask more pertinent questions. In such a scenario, I think puzzles are very useful. Its not the answer to the puzzle that matters, per se, but all of the subtle details surrounding it

I would like to switch over to fiction for a moment, specifically Frank Herbert's Dune. Fiction is often in a good position to explore the extreme version of a situation, and sometimes that can make the less extreme version more clear. In Dune, one of the groups has something known as the "pain box," or "agony box." You put your hand in it, and it uses nerve induction to generate the most excruciating pain in the universe. It is part of a test to determine if you are sufficiently human to go forward with their "interview." It is explained that it is not the pain that matters, it is how the interviewer observes your reactions, and interacts with you while you feel your hand melting off. In the hands of someone who does not know how to use it, the box is useless as anything besides a torture device. However, in the hands of someone who knows how to use it to accomplish their goals, it peels away layers of defenses so that they can conduct the real portion of the interview without interference from shallow things like memorization or bravado.

Now, after describing that interview, aren't you glad that they're really just asking you to solve a few puzzles?

Val 1,64051218 · Answer 10 · 2015-09-02 17:00:26Z

up vote
-2
down vote

There is a very interesting article about this topic on The Daily WTF.
The main problem is that although lateral thinking puzzles are meant to measure the way you arrive to the solution, most of the interviewers will expect the exact same answers they have on their scoring sheets. So a candidate who thinks well and has a lot of good ideas, might be dropped in favor of someone who happens to have heared this puzzle before.

[the one who solves the puzzle in a brilliant and innovative way] will not get the
job. Despite the complete absurdity of the design request, and the
complete practicality of his answer, the job will go to a candidate
who manages to answer the question by designing an extremely
overcomplicated solution for a completely non-existent problem. And
that candidate will be the same person who designs their software.

answered Sep 2 '15 at 17:00

Val

1,64051218

The tool is only as good as the person using it. If idiots ask the "perfect" question(s) and misjudge the answers, then ... bad things happen. Don't fault the questions, fault the askers.
â€“Â JeffC
Sep 2 '15 at 17:11

3

I would not call the proposed solution a solution in any real sense. Somehow the person writing that article concluded the wrong thing from the purported incident. The person failed the interview for good reason. Probably the interviewer was flabbergasted by the interviewee's insensitivity to the visually disabled and the intent of the question, which is to engineer a solution to a quality of life issue (which the interviewee then appeared callously indifferent to).
â€“Â Chan-Ho Suh
Sep 2 '15 at 17:34

2

There is a difference between a quality of life issue and a completely non-existent problem. I think you also concluded the wrong thing from the example. The point was that someone who can engineer an over-complicated solution to a problem which does not even exist, will not become a better developer than the one who identifies the service to be provided as unnecessarily. A better example I've seen is about moving Mount Fuji, while the interviewer refused to answer any of the questions like who is the consumer and how could we satisfy him without moving the mountain (the classical XY problem)
â€“Â Val
Sep 2 '15 at 17:52

2

@vsz I don't see how it's a nonexistent problem. Visually-disabled bikers exist, and they would like to do so more safely. Speaking of becoming better developers, if you are given a problem, you need to look to address it. If you think the business is mistaken and it's not a real issue, you need to do research to prove your case, not just dismiss it (as you and the other person did).
â€“Â Chan-Ho Suh
Sep 2 '15 at 17:58

2

The practical solution is a tandem.
â€“Â A E
Sep 2 '15 at 20:58

Â |Â
show 2 more comments

Kate Gregory 104k40230332 · Answer 11 · 2015-09-01 11:41:48Z

First, a disclaimer: when I interview people I do not set them puzzles. I used to and I stopped, because they gave me false positives and false negatives. I never used a puzzle from a book or the internet though. Second, some perspective: chefs, nannies, accountants, heads of HR, and university presidents are not asked these things. It's mostly software developers.

Now, why set a puzzle? Many people think it's a proxy for many software development tasks. A bug is a bit like a puzzle or a riddle (except that nobody deliberately set it up for you to solve) and by demonstrating in a job interview how you systematically approach such a problem, the theory is that you're demonstrating what a good problem-solver you are in general. Even when the solver says things like "it's basically lanterns-and-bridge but with pigs" they're showing how they can recognize one problem in another and apply previous knowledge. That seems like it would be all good. People who can see patterns (all these numbers are one more than a power of two, or only letters with even ASCII codes are coming out wrong) do tend to solve bugs faster than people who don't. People who can quickly generate a hypothesis and test it are also often good debuggers.

The biggest problem is the false positives. People who aren't able to reason logically, generate hypotheses, spot patterns, figure out the weak link, make a leap, or even do basic math can nonetheless memorize, and these are the absolutely worst people to hire for software development or other creative jobs. Some of them can learn how to pretend they are working out the solution on the spot, and fool you.

A smaller problem, but still real, is false negatives. People who are excellent debuggers but just think it's silly that a group of such a size would not have enough light left for people to cross the bridge however they want, or feel that once one person is over the bridge the others can cross in the dark, and they just won't let go and get playful and try to solve the underlying math problem (arrange these numbers into pairs using the following constraints.) People who write great code and are offended by being asked to do something they're not good at in order to get a coding job. That sort of thing. There's also a real risk that the solver produces an answer that is better than the asker had in mind, and is rejected for not "getting it right."

I ask people if they like puzzles, and why. But I don't ask them to do a puzzle. And I would not study puzzles as part of a job-interview prep process.

**unless the job interview is for a Puzzle Solver, right? Someone's gotta QA all those bits in the paper next to the funnies! ;) — Ian MacDonald, Sep 1 '15 at 12:15
@Dukeling I think someone's refusal to accept premises about lanterns with precisely 17 minutes of light and such is not a good indicator about their refusal to accept premises like what programming language we'll be using on this project or what platform to target. — Sep 1 '15 at 19:08
@Chan-HoSuh easy to say, but most of these things have a simple trick (eg weighing balls three at a time, or sending the two fastest people together and the two slowest people together) that cannot be "dug deeper" into. You get it or you don't. Once you get it, you get it, and there's really no way I can test whether you get it because you thought it through yourself, or because someone else explained it to you before. — Sep 1 '15 at 19:10
That's a very interesting perspective WRT puzzles as "a proxy for many software development tasks." When interviewing for my current job, they simply did away with the proxy: they asked me to code a solution to a moderately difficult problem, and then presented me with an IDE containing a broken program, showed me how to reproduce the bug, and asked me to fix it. In both cases the interviewers were watching what I did the whole time so they could observe my development process and see how I worked my way through things. — Sep 1 '15 at 19:39
I once had a question on getting a PC working where they were fishing for the fact that DHCP in windows defaults to one particular subnet when a DHCP server is unavailable. The way I went about solving it jumped right past that and IRL would have solved the problem but 'failed' their test cos it wasn't the thing they were focused on. Once they told me the 'right' answer I was put in the awkward position of appearing wrong or making them look foolish. — Sep 2 '15 at 9:35

score 41 · Answer 12 · 2015-09-03 09:42:13Z

To what extent is this practice effective or ineffective? Is there any
correlation between puzzle-solving performance and job effectiveness?

If there is evidence of correlation, then for what types of roles? And
what types of puzzles?

And if (as I currently understand to be the case) there is little or
no evidence to support the effectiveness of puzzle-based job
interviews, then why do companies persist with this practice?

Interviewers will often tell you that they use these questions to "see how you think", or "see how you think out of the box", or "see how you think under pressure" or some variant of that.

I don't believe any evidence exists to support such a claim. And I know of no company that can say "we have found that better puzzlers make better employees" with a straight face (puzzle-producing companies excepted).

Puzzle questions are a fad whose time has come and gone. At one time, companies like Microsoft used to ask questions like "why is a manhole cover round?" or "how many ping pong balls would it take to fill a jetliner?". Since folks equated Microsoft with "smart" and everyone wants to seem smart, they often chose similar questioning methods.

I once had a Developer ask a QA candidate I was recruiting "How would you find a needle in a haystack?". When I questioned the Developer if he thought that was a useful QA question, he said it would "tell him how the candidate thinks". When I pressed further, the said it was asked of him once in an interview and he thought it was a "good question". I never asked him to help with interviews again.

Apparently, Google no longer believes such questions are useful, and has some idea of the real purpose of these sorts of questions: http://www.nytimes.com/2013/06/20/business/in-head-hunting-big-data-may-not-be-such-a-big-deal.html?pagewanted=all&_r=0

According to Laszlo Bock, senior vice president for people operations at Google: "They donÃ¢Â€Â™t predict anything. They serve primarily to make the interviewer feel smart."

I agree with Laszlo.

Hiring and interviewing are hard. Coming up with relevant questions to discern the fit of a potential employee is hard. Asking puzzle questions is just lazy, in my opinion, and wastes valuable interview time. Unfortunately, most folks aren't trained on how to conduct an effective interview, so they just go with what they have heard is "smart" or "a best practice" and don't put any more thought into it.

After you are hired and advance in your career such that you are the hiring manager - choose better interview questions! No more "What is the air-speed velocity of an unladen swallow?" please.

Absolutely agree - "How would you find a needle in a haystack" - get a big f***ING magnet! — Sep 1 '15 at 23:08
"how many ping pong balls would it take to fill a jetliner?" <- This is a Fermi estimation problem, not a puzzle. These have no "right" answer, the point is to see how the candidate tackles the estimation process. en.wikipedia.org/wiki/Fermi_problem — Sep 2 '15 at 3:37
@TheWanderingDevManager: My favourite is "Throw the hay in a clear box of water and mix." - hay floats, needles don't :) — Sep 2 '15 at 8:36
"How would you find needle in a haystack?" Get a big fan, blow all the hay away. even better Set the whole haystack on fire, and there you have it ... your precious needle. — Sep 2 '15 at 8:59
Google claim to have stopped doing the brain teasers, but some of their interviewers still spend a large fraction of time on Fermi estimation questions. I'm not convinced that "how many dogs are there in London?" (repeat 5 times with different interviewers, using different nouns and cities) has any stronger correlation with work performance than "guess the weight of the fruitcake". — Sep 2 '15 at 13:22

Kingrames · Answer 13 · 2015-09-01 12:35:51Z

I'm gonna say no.

Puzzles are specifically designed to have a "best" solution and oftentimes, especially in programming jobs, you run into problems where there IS no solution.

Using a puzzle in the job interview process is more than a little condescending, especially since you're talking about someone else's career or potential career and certainly their livelihood, and yet you're treating it like a game. It's very, very disrespectful.

That being said, sometimes it's not the go-to thing for businesses but rather just a "best option" for places hiring. A LOT of managers, HR reps, CEO's, and sometimes even IT leads have no clue what the actual programmers do (and worse, they think they do) and they use these puzzles to put someone in the state of mind they think you'll be in while you're programming. It's less of a skills test and more of an "attitude under stress" test.

But you'll find that most people who hire employees based off of the results of the puzzle part of the test are going to hire terrible people. (You should focus on the resume!)

While I agree with you about the puzzles, I would never focus just on the resume, way too many people lie on resumes about their actual skill level. — Sep 1 '15 at 17:31
You could perhaps also call something like "How many windows are there in New York?" a puzzle - the final answer is not really important at all, the important part is the way of thinking to get there (so that's not something which has a "best" solution). Purely from the condescending / respectfulness viewpoint, I don't see how asking someone to solve a puzzle is that different from making a joke during an interview, having a more relaxed and open interview or even asking them to solve a toy programming problem (which happens in just about every programmer interview). — Sep 1 '15 at 19:11
@Dukeling - the important part is the way of thinking to get there - so like every other puzzle, the solution is "I'd google it"? Because everything else is contrived bullshit and we all know it. — Sep 1 '15 at 20:35
@Dukeling that is a Fermi problem (the classic form is "How many piano tuners are there in Chicago?" though I've also seen "how many gas stations in the United States?"). If you know how to approach it, it isn't a puzzle at all - just a matter of multiplying guesses and knowing which ones to guess at. — user10042, Sep 1 '15 at 20:56

keshlam 41.5k1267144 · Answer 14 · 2015-09-01 22:15:38Z

Programming puzzles wind up, at best, being the equivalent of compiler micro-benchmarks. They're much more likely to tell you whether the programmer happens to have previously solved that problem or a similar one than to tell you how good they are at problem solving generally.

This. A good software engineer would consider multiple approaches, evaluate them and work in an iterative way to come up with the right solution. You don't solve everything in one hit off the spot. — Sep 1 '15 at 22:51

BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â› 1,4501524 · Answer 15 · 2015-09-02 08:27:36Z

up vote
8
down vote

I work as a senior software engineer in Germany. In the last 6 years, I had at least 25 interviews. I was asked only once a puzzle question. Needless to say, whole interview was quite odd.

My experience is that people do not ask puzzle questions anymore. At least not in IT sector. For example, in google Ã¢Â€ÂœWhat works well are structured behavioral interviews, where you have a consistent rubric for how you assess peopleÃ¢Â€Â.

Even microsoft, where the puzzle questions were born, stopped asking such questions since they realized it is a waste of time. There is this famous article "What would Feynman do?", which discredit such questions in quite a funny way.

answered Sep 2 '15 at 8:27

BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›

1,4501524

The practice is still alive and well here in London, unfortunately.
â€“Â A E
Sep 2 '15 at 9:29

5

Loved the Feynman link.. reminded me of Bohr's barometer question - to measure the height of a building using a barometer, ask the janitor and offer him the barometer if he tells you the height!
â€“Â gbjbaanb
Sep 2 '15 at 10:14

1

Incidentally, some guy once complained to me how math story problems often are like "imagine a drinking straw lying along the diagonal of a box, compute its length..." when the point is to just have you compute the diagonal's length. Story problems are full of artificial things like this. He said the problem is undefined because the circular diameter of the straw or its thickness was not given. I wonder who here would take the point as far as this guy. (Incidentally this guy won a Fields Medal, the "Nobel Prize" of mathematics, so it's not as if he couldn't do these story problems)
â€“Â Chan-Ho Suh
Sep 2 '15 at 12:48

1

@Chan-HoSuh You obviously missed the point. On the interview I had, the guy asked me to design a trash can. What kind of moron would ask that for a software engineering position? Regarding a guy like Feyman, how did you conclude that he "probably wouldn't be suited for industry", if he didn't answer idiotic puzzle the way interviewer wanted? The article I linked tells that such questions can give false positive and negative conclusions.
â€“Â BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›
Sep 2 '15 at 12:51

1

@Chan-HoSuh When you mentioned "drinking straw", first I thought of this ;) and "really. how?" :)
â€“Â BÃÂˆÃÂ¾ÃÂ²ÃÂ¸Ã‘Â›
Sep 2 '15 at 13:27

Â |Â
show 5 more comments

jamesdlin 1604 · Answer 16 · 2015-09-02 08:08:42Z

I sometimes (but rarely) ask puzzles. When I do, the puzzles I ask are not ones that require an a-ha moment (a flash of insight) but are ones that can be solved with logical deduction: can the candidate break down the problem into a smaller problem, solve that, and then use that result to solve a larger problem?

Or sometimes I'll ask non-puzzle questions, and if I have time leftover, I sometimes might chew up some time with puzzles. But when I do, the point isn't to try to use it as some proxy for gauging their programming ability; the point is to find a data point for culture fit because programmers I work with tend to like puzzles too. And maybe the candidate has heard the problem before, but that can happen with non-puzzle programming questions too. If they have (and say so), maybe the information I gather from it is that the candidate likes puzzles, knows how to get answers, and is honest.

I think that they can be sometimes useful, but you have to be careful with what puzzles you use and think about what you're actually trying to measure.

Probably the most reasonable answer here.
â€“Â Chan-Ho Suh
Sep 2 '15 at 12:37 — Sep 2 '15 at 12:37

Chan-Ho Suh 1,94111011 · Answer 17 · 2015-09-01 19:05:34Z

Besides software development, one sector where puzzles / brainteasers / quantitative problem-solving is tested as part of the job interview process is in technical roles in finance.

I've heard a couple reasons given for testing interviewees in such a manner:

1) it's a form of stress test. Since there are a lot of jobs where a person has to be very quick on their feet and be able to apply their quantitative skills at a moment's notice, this should be tested somehow.

2) It's an efficient test of basic skills. Finance tends to attract people of diverse academic backgrounds. How do you verify that someone that says they are "good at probability" is really so? Does having a math PhD suffice? There are plenty of people with only basic undergrad math that have better probabilistic reasoning skills than many math PhDs.

@JoeStrazzere I don't think asking a more complicated question with more to worry about is necessarily going to shed light on that. Separate questions can be asked on financial concepts. — Sep 1 '15 at 20:10
If you want to verify CV claims, ask the claims. Tell me about projects A and B. What issues did you experience? What did you do about it? What lesson have you taken with you? — Sep 2 '15 at 8:49
@Gusdor What you propose only works for certain claims. Certainly it won't help filter out the candidates with good basic technical skills from bad. — Sep 2 '15 at 12:34
@Chan-HoSuh Are you sure? Candidate "I can write some Python". Interviewer: "Cool, what is your favourite thing about the language? What don't you like?" Those sorts of questions help assess levels of experience. A detailed answer can show depth of knowledge, balanced analysis and passion for a skill or a past job. — Sep 2 '15 at 12:47
I certainly agree that they are commonly used in quant finance. I even wrote a book that's compilation of such puzzles and it's been very successful. One of the reasons I think that they are popular is that it's an attempt to level people from very different backgrounds. The questions vary from very technical eg write a routine in C++ to do X, to silly brainteasers to basic financial probability questions. — Sep 3 '15 at 5:33

JeffC 21716 · Answer 18 · 2015-09-02 17:43:04Z

I would say that it depends on the field. In some cases, like programming, candidates (generally) have a degree and some work history or at least school projects. They can be asked specific questions around programming so, to me, it makes less sense to ask them puzzle questions.

My specific experience is around Software QA. When I interviewed at Microsoft years ago for a QA position, I was straight out of graduate school and had no relevant work experience. There were no degrees in Software QA. So what should they ask? People that haven't done QA (and some that have) generally don't understand QA so it's hard to ask non-testers tester questions and expect to get good answers.

They asked me a lot of puzzle questions because they wanted to see how I went about solving puzzles. It wasn't so much about the answer, it was the thought process behind it. They also asked me simple testing questions... how would you test a date field? As I progressed through the interviews, the next person would ask questions based on what I was previously asked to see if I could learn and apply that knowledge.

You can learn a lot about a person by how they solve a problem. Do they give up immediately when they can't figure it out? Do they ask me lots of questions to try to gain more insight into how to solve it? Do they give one answer and get stuck? These skills are relevant to many jobs, not just QA.

I think the new hotness (well, not so new but anyway...) in interviews is the experience based question. Tell me about a time when you did X. The thinking is that the way you approached an issue in the past is likely the way you will approach it the next time it comes up. That's great... but how do you ask questions like those for a field that the candidate has no experience in? Not saying it can't be done but it's much harder... but, good interview questions are not easy to come up with.

Some people are having issues with the "contrived" puzzles. Yes, they are "contrived" it's on purpose. You are a dev on a project and someone asks you how you would fix X issue. Well, I would do Y... well Y would mean a complete rewrite of the project so Y is not an option. That's not contrived... it's real world and it happens all the time. You have to be able to work within constraints whether you like them or not. A contrived puzzle (at least the good ones) is a parallel to real life. — Sep 2 '15 at 17:47

Cort Ammon 2,7121713 · Answer 19 · 2015-09-03 03:40:37Z

All scientific evidence I have seen fails to demonstrate puzzles as effective. However, I believe they have some potential in non-scientific approaches. Consider that everything in the interview process is a tool. They're tools to help you and the company determine if they are good matches for each other (mostly to help the company, but it goes both ways). Accordingly, the interviewer uses any tools at their disposal. Some interviewers may find that asking a puzzle question is an effective tool in their toolbox to learn more about you, and then ask more pertinent questions. In such a scenario, I think puzzles are very useful. Its not the answer to the puzzle that matters, per se, but all of the subtle details surrounding it

I would like to switch over to fiction for a moment, specifically Frank Herbert's Dune. Fiction is often in a good position to explore the extreme version of a situation, and sometimes that can make the less extreme version more clear. In Dune, one of the groups has something known as the "pain box," or "agony box." You put your hand in it, and it uses nerve induction to generate the most excruciating pain in the universe. It is part of a test to determine if you are sufficiently human to go forward with their "interview." It is explained that it is not the pain that matters, it is how the interviewer observes your reactions, and interacts with you while you feel your hand melting off. In the hands of someone who does not know how to use it, the box is useless as anything besides a torture device. However, in the hands of someone who knows how to use it to accomplish their goals, it peels away layers of defenses so that they can conduct the real portion of the interview without interference from shallow things like memorization or bravado.

Now, after describing that interview, aren't you glad that they're really just asking you to solve a few puzzles?

Val 1,64051218 · Answer 20 · 2015-09-02 17:00:26Z

up vote
-2
down vote

There is a very interesting article about this topic on The Daily WTF.
The main problem is that although lateral thinking puzzles are meant to measure the way you arrive to the solution, most of the interviewers will expect the exact same answers they have on their scoring sheets. So a candidate who thinks well and has a lot of good ideas, might be dropped in favor of someone who happens to have heared this puzzle before.

[the one who solves the puzzle in a brilliant and innovative way] will not get the
job. Despite the complete absurdity of the design request, and the
complete practicality of his answer, the job will go to a candidate
who manages to answer the question by designing an extremely
overcomplicated solution for a completely non-existent problem. And
that candidate will be the same person who designs their software.

answered Sep 2 '15 at 17:00

Val

1,64051218

The tool is only as good as the person using it. If idiots ask the "perfect" question(s) and misjudge the answers, then ... bad things happen. Don't fault the questions, fault the askers.
â€“Â JeffC
Sep 2 '15 at 17:11

3

I would not call the proposed solution a solution in any real sense. Somehow the person writing that article concluded the wrong thing from the purported incident. The person failed the interview for good reason. Probably the interviewer was flabbergasted by the interviewee's insensitivity to the visually disabled and the intent of the question, which is to engineer a solution to a quality of life issue (which the interviewee then appeared callously indifferent to).
â€“Â Chan-Ho Suh
Sep 2 '15 at 17:34

2

There is a difference between a quality of life issue and a completely non-existent problem. I think you also concluded the wrong thing from the example. The point was that someone who can engineer an over-complicated solution to a problem which does not even exist, will not become a better developer than the one who identifies the service to be provided as unnecessarily. A better example I've seen is about moving Mount Fuji, while the interviewer refused to answer any of the questions like who is the consumer and how could we satisfy him without moving the mountain (the classical XY problem)
â€“Â Val
Sep 2 '15 at 17:52

2

@vsz I don't see how it's a nonexistent problem. Visually-disabled bikers exist, and they would like to do so more safely. Speaking of becoming better developers, if you are given a problem, you need to look to address it. If you think the business is mistaken and it's not a real issue, you need to do research to prove your case, not just dismiss it (as you and the other person did).
â€“Â Chan-Ho Suh
Sep 2 '15 at 17:58

2

The practical solution is a tandem.
â€“Â A E
Sep 2 '15 at 20:58

Â |Â
show 2 more comments

Search This Blog

Iyfjky

Are puzzles an effective part of the recruitment process?

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

10 Answers
10

protected by Communityâ™¦ Sep 2 '15 at 17:43

10 Answers
10

10 Answers
10

protected by Communityâ™¦ Sep 2 '15 at 17:43

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

List of Gilmore Girls characters

Confectionery

Category

Random preview

Are puzzles an effective part of the recruitment process?

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

migrated from puzzling.stackexchange.com Sep 1 '15 at 16:12

10 Answers 10

protected by Communityâ™¦ Sep 2 '15 at 17:43

10 Answers 10

10 Answers 10

protected by Communityâ™¦ Sep 2 '15 at 17:43

Comments

Post a Comment

Popular posts from this blog

What does second last employer means? [closed]

List of Gilmore Girls characters

Confectionery

10 Answers
10

10 Answers
10

10 Answers
10