Killer Interview Questions

I have been interviewing and hiring testers since 1999. I like to think that I am pretty good at finding testers who can think, communicate, and fit into the team. I have three questions that I like to ask that really help me determine if the person can think and communicate. I don’t have any questions regarding “fitting into the team”, that is just based on how well I (and any other interviewers) interact with the person during the interview. In the 13 years that I have been interviewing testers I have hired roughly 120 and only regretted the hiring of three of them:

• One of them quit just before I was going to put him on a PIP (Personal Improvement Plan). This tester was the fourth person I hired and he was the husband of a friend. He did not answer my three questions well but I hired him because of the friend connection (drat),
• One was a very good tester but she did not really “fit in with the team”. She transferred from another division and I only did a phone interview with her. I am quite confident that if I had been able to interview her “in person” then I likely would not have hired her – but that is pure speculation on my part.
• One was let go during a round of layoffs. This tester is the only “anomaly” to my otherwise explainable hiring record. In other words, I can explain why I mistakenly hired the other two – but I can’t explain this one. She answered the questions well and then was just a really bad tester – not willing to learn, change the way she approached problems, not a good problem solver, seemed to like documentation over testing.
• So, I think that my record is pretty good (a >97% success rate – and >99% if you remove the one who didn’t answer the questions and the one I didn’t interview in person).

How did I achieve such a high success rate? Well, first off I did not search for domain experts. I often had to hire for roles that were very specific (e.g.: Physical layer tester for DSL lines – there are not very many testers with that type of experience). I found out fairly early on that in my context it was much easier to find a critical thinker, problem solver that could then learn the domain instead of a domain expert who would not be able to think his way out of any simple issue in the lab.

In my last 4 years at Alcatel-Lucent there were three test managers. John Hazel and I worked very well together and he is the author of one of the questions that I use. Whenever possible, John and I would interview together and we would enjoy the entire process. The other manager never really followed what John and I were doing. I’ll call the other manager Michel (because that is his real name). Michel had a position to fill and he went about hiring a new University graduate without asking for help from John or me. The tester that he hired was very “book smart” but was scared to do anything in the lab. He would ask other testers to pull cards, run cables, and reprogram SIM cards. It was not a surprise to us when he transferred to a different role within 6 months of joining Michel’s team.

One important aspect of my hiring practice is to focus my hiring on people directly out of University (or College) with a good working knowledge of computers. I have found that I am FAR more likely to find a superstar this way than to look at people who are floating around in the industry with many years of bad habits (er, experience). These people will have graduated, and if they have a computer science or electrical engineering degree then they should be able to code tools or automation to a satisfactory level. I honestly just make the assumption, and if I find out that I am wrong, I can either move them into a manual test role, or just let them go. I have never had to do this though.

A trick in the hiring process is to make sure that you are hiring people who not only want to be testers, but who will also probably be good at it. We all know that many people are just not capable of being good testers – I typically refer to this group of narrow-minded people as “software designers” :-P.

It is also very important that the testers in your organization get treated with respect and similar pay as the S/W designers. If this is not the case, then you have a much harder battle to fight with your management organization. It also helps a lot if the new tester is not “the” tester, but a member of a testing team (even a small team of 2 or 3 other people), so they have a opportunity to see that “test” is a viable career. If they are the only tester (or just one of two), then they can see that they have no career path, no mentor, no where to go –> except over to the “dark side”.

Reader: “Enough history and boring background crap. Cut to the chase, Paul. I’m about to start to skim for the questions.”

Okay. Okay. Here are my three interview questions that I like to ask in the interviews. John really like to ask other (fairly useless) background questions like “What was your biggest mistake? What was the result? and What did you learn from it?” He also liked to do the resume crawl. “So, tell me what are most proud of from your last job?” I can’t say that it was a complete waste of time as there were some interesting nuggets that got exposed but I found that more often than not they did not really add to my decision whether or not to hire that person.

The questions that I ask are fairly long (in situation) but the answers are fairly simple:

Question 1: I describe a hypothetical situation to them. I let them know that the setup is fairly long but the answer I am looking for is fairly
simple. I tell them they are in charge of daily automated sanity testing of new S/W builds. It takes about an hour to load the S/W onto the system and perform the test. The run the test on Friday and everything is fine. They send out an email to the team telling them to promote this code. If they had found an error, then they would have had to send an email to the owner of the failing section of code to have them fix it. They are going to take a week off, and they ask me to perform the testing while they are away. “No problem”, I reply. When they return from vacation 10 days later they discover that the load does not work. They ask me how the testing had gone while they were away, and I tell them that I completely forgot to perform the tests. They quickly send an email off to the owner of the failing module and she says that there many check-ins throughout the week including weekends. She needs to know the first load that failed. Oh, she is going on vacation in 4 hours. So, they have a load that worked last Friday, and now, 10 loads later (we do builds on the weekends too), they have a load that does not work. All loads are available on the server. We need to find out which load first broke so the SI (system integrator in charge of the builds) can review the many check-ins on from that day and locate the problem. The question is: How would they determine which load last worked and which one first broke in four hours or less. They only have one setup (if the person asks if they have multiple setups – I like that, but the answer is “no”).

What to look for in the answer: Analyze their thought process. If they are trying to review the check-ins, talk to designers about when they thought the code might have broken, or look at code themselves – that is a red flag to me – usually indicates that they are thinking more like a S/W designer. If they want to check the loads sequentially (from either end) – this is also a red flag – but less so than the previous one. You can reiterate at this point that you need to know quickly (restate the 4 hour time limit).
Hopefully, they will then realize that they need to do a binary (or bisection) search. I have found that about half the people get the binary search right away. It often surprises me how often this situation arises in testing. For example, I last tested feature A about 4 months ago and everything was fine. Just before a major release I go to test it again and it is broken. The designers have no idea what could be wrong as no one has touched that code in months. They need to know the load where it first broke to be able to find the interaction that is causing the problems. If you don’t do a binary search, you will likely be trying an awful lot of loads before you find the answer.

Question 2: This question involves a simplified network diagram. Your own computer (A) and another computer (B) in the next (empty) cubicle are on the same subnet, a computer (C) in the lab (a five minute walk away) is on another subnet, and the mail server on a third subnet (I usually draw the main subnet routers (X,Y, and Z) as well). You get a phone call from my boss asking you to go into work. There is something critical she just sent to you via email. Her cell phone battery dies at this point so you can’t talk to her anymore. I am not accessible either (drat). You come into work even though it is a statutory holiday – there is no one else is around to help you. You really need to check their email, but the problem is that you can not access the mail server (“mail server is not responding” message is being displayed). What would you do?

Answer to #2: Their initial response is usually quite interesting. Some people just say that they would call the “help desk” or go home and wait until the next working day. This indicates to me that they may not be willing to investigate problems fully (especially considering that was their answer during an interview). Let them know that they “need” to get their email, or at least exhaust their possibilities.
I let the problem change depending on their answer and their experience level. I let them tell me what they would try, and I give them the result. As an example, they might say they would try and ping router “X”, or workstation “B”. I tell them “X” is alive, or no response. I also usually start with a simple “the cleaner knocked the cable out” problem and then tell them that they have now come in on another stat holiday and start the question over again. You have to be careful that you are not assessing their networking knowledge (unless required for the job) so much as their problem solving abilities. In some cases, I have had to let applicants
know that there is “ping” command, or that they can try and telnet to “C”. Another interesting thing to analyze is what they do once they can get their mail. For instance, if they try from “A” and it doesn’t work.
Then they try from “B” and it does work. Some people have been happy that they got their email, and they do not show any indication of looking for the cause of why their own computer is not functioning (I find this behavior quite scary). I usually make them work it out until they discover a good probable cause like router “Z” is down, or that the mail server application is down (but sometimes they can still ping the server itself). I let them guide the answer, and the more they know about networking, the harder I make the failure.

Question 3: I would like to take credit for this last question but this one is by John Hazel.

The question presents a brief (and flawed) functional specification to the interviewee. The FS describes the billing rules of a small business called “Al’s Delivery Service”. I verbally describe Al’s business as a small business (with only one office) with dedicated clients and limited to no competition. He delivers only single bricks to any customer (the size, shape and approximate weight of a white board eraser). The actual composition of the brick can be anything you like (I usually imply an illegal substance – to explain the dedicated customers and unreasonably high rates charged by Joe).

What I write on the board:

Al’s Delivery Service

1. If the delivery is < 10 km from Al’s office the base cost of the delivery is $10.00
2. If the delivery is > 10 km from Al’s office the base cost of the delivery is $25.00
3. If there are any stairs involved in the delivery path, there is an additional $10.00 charged
4. If the delivery occurs on the weekend the cost of the delivery is a flat rate $50.00 (verbally clarify: regardless of distance and/or stairs)

I let them know right away that there are two parts of the question:
Al had a friend of his write the software that does the billing to the above spec. He has asked you as a professional tester, to test the math in the billing software before he starts using it. You can stress that you do not want them testing the UI – just test the math portion.

3a. What are the test scenarios that you would suggest for Al to verify that the math is correct in the typical situations? Sometimes I say that the UI is limited to two radio buttons for the <10km or >10km and two check boxes: one for stairs and one for weekend. There is a “go” button. That is it so they can not get into “I would enter negative 1 for the distance”, etc.
3b. What corrections, clarifications, and/or limitations would you make to the existing FS that will help Al’s business succeed?

Answer to question 3a: I am only looking for the following answer for the scenarios:

< 10 km
> 10 km
< 10 km w/stairs
> 10 km w/stairs
weekend (either on its own, or with the above 4 conditions)

So, essentially, I am looking for either 5 or 8 scenarios as the “correct” answer as long as they have an acceptable reason for going with 5 or 8. My preferred answer is 8, but I accept 5 as it was John’s original answer, yet he claimed to have knowledge of the implementation. I also am willing to listen to any plausible context for different answers.
Frequently I have to put limitations on the presented user interface to prevent the literally countless possible inputs. I tell them that they have two radio buttons for “<10 km” and “>10 km”, and two check boxes (weekend and stairs). If they are being a bit more thorough, then I let them know that the default is <10 km, and both check boxes unchecked. To be clear, the radio buttons are mutually exclusive,, and the check boxes are independent.
What I am looking for in the answer (besides either of the two answers above) is the thought process that they display coming up with the list. Some people fumble around and have no clear thought process, while others come across as much more organized and clear in their method. Honestly, I have not seen much difference in the overall effectiveness of either type, but I have only ever hired one person who messed up this question (both parts), and they were mentioned at the beginning of this post – they quit after 4 months.

The second question provides much more interesting information for the hiring decision.
Once again, I am looking for some particular answers, yet there are MANY others that I have heard and appreciated as very valid (although sometimes obscure). The “main” answers are:

What about = 10 km (more of a design issue than a FS issue as it would be almost impossible to measure that far exactly)
What, if any, is the maximum distance?
What, if any, is the maximum number of stairs?
What about statutory holidays?

A small sample of other possible answers:

• What time do weekends/holidays start?
• What about evenings?
• What about the minimum number of stairs?
• What if the delivery starts on a Friday, but is a long distance and ends on the weekend?
• Offer a discount to repeat customers.
• Charge by mile traveled (similar to main point 2)
• Have more divisions in the distance (5, 10, 15, 20, 25 miles) to smooth $10 to $25 change.
• Charge by flight of stairs (similar to main point 3)
• What about parking?
• Does he do international deliveries? What about custom charges?
• There are many other possibilities – you can easily use your own judgment. Until they get the main 4 points,
• I usually allow them to sweat it out with occasional small hints (normally just pointing out which of FS lines
• needs clarification/limiting/fixing). After they get the 4 main points, I will let them finish on their own pace.

Well, those are the three main questions that I ask in my interviews. I hope they serve you as well as they have served me over the past 13 years.