During my last few years as a test manager at Alcatel-Lucent, I decided to try a slightly different approach to the development of test ideas.
Before this experiment my testers would be given a feature to test and they would start by reading the functional specification (user stories if you are in an agile environment). They would develop a list of potential test ideas based on the specification document and THEN add any other ideas they could think of.
I found that when I reviewed their list of test ideas that I would often think of fairly creative test ideas that were missing from their list. Now, I would like to think that the root cause of my creativity was my highly endorsed “general awesomeness” (see my LinkedIn profile for all endorsement bombings I have received) but I am not that self-confident. I thought I would try an experiment. The experiment went very well and it changed how my team approached new testing situations.
When my team was first presented with a new feature I would allow them time to explore the feature and develop test ideas BEFORE I let them look at the functional specifications. I tried this as an experiment on one feature and I found that the creativity shown by the team was noticeably higher than when they had started with the functional specifications. I have labelled this “Functional Specification Blinders”. By reading the information in the functional specification I have found that it becomes very difficult to be able to think of testing ideas that are not included in the functional spec document. I think the reason behind this goes back to how we were exposed to learning as children. We are asked to think of explanations of how something might work (or react or behave, etc.) and then we are told how it works by a teacher (or other “expert”). Once we are told “the truth” we are expected to accept that and move on with our new knowledge. The problem with this approach is that often “the truth” is wrong or at least incomplete or misleading. I have found that the same phenomena happens when thinking about how software might be tested (or used or implemented). By allowing my team to use their creativity before reading the functional spec they have the freedom to think of test ideas that are more creative, unusual and potentially more powerful. In one class I was teaching of Rapid Software Testing, during an exercise on creating models, the students were asked to think of how to factor an object (identify all the dimensions of it) or in other words try to identify a thorough list of anything that we might want to test. One student discovered a published standard for the object before he had tried to think of any dimensions on his own. For the remainder of the exercise (roughly 5-10 minutes) he was completely incapable of thinking of ANY dimension that was not mentioned in the standard. I was pleased that he was able to show me a bit more proof in my “Functional Specification Blinders” theory.
So, if you have the freedom to investigate your feature in whatever method you choose, then I encourage you to start with other methods of learning about the feature than reading the specifications. You can any of the methods from this incomplete list:
- Play with the feature. If the feature is available (in any form) then interact with it.
- Talk to the product owner/product manager
- Talk to marketing
- Talk to customers
- Talk to the developers (or their manager)
- Talk to customer support
- Read marketing materials, claims, web sites, etc.
- Investigate competitor’s products
- Have discussions with other testers
- Talk with the architects/designers
By exploring some of these venues BEFORE reading the specifications then you can allow your creativity to blossom. The specification will still be there, waiting for you, once you have tried thinking for yourself.
Instructor of Rapid Software Testing courses. Context-driven software testing consultant. 17+ years of experience in software testing
Hello Paul,
Like the name Functional Specification Blinders. I called it, “Testing with Blinkers On” when I first saw through this.
A while ago, I ran an experiment on similar lines.
Group 1
Testers were given a spec, asked to list down test ideas, add many more by pairing and brainstorming and test the product. Testers in this group behaved just like horses with blinkers – looking at what was given to them. Rarely, one or two testers went beyond the spec and would back track because they had this mental block that testing for spec was *The Expectation* and they *MUST NOT* wander.
Group 2
Testers were given the product to test, asked to explore, brainstorm ideas and test the product. Since there was not much information available, they went out of their way to gather information, talk to more people in different teams and having varying backgrounds. This helped them build a mental model of the product way better than testers in Group 1. Note that Group 1 testers could have done the same thing, but the spec blocked them. Rather, they allowed the spec to block them by defining a non-sensical boundary.
In some cases, testers in this group spoke to real users, invited a few random users to use this product while they noticed, added the creators of the product on Linked In and sent them questions on the Vision and Purpose of the product, what was the revenue model and so on. Every single information gained was useful in testin
Group 3
Group 3 was an expert level Group 2 in that given a specific project context, they worked like a SWAT team who turned around things in no time. Such teams are powerful in changing business contexts and add great value without losing out on time and effort.
These days, if have to pick a project, I tell them not to share any specs or demos upfront leaving them to scratch their heads. I only ask for a better view of the project context and the scope of testing. I let my team explore the product for first couple of days, build their own mental model, come up with a list of queries we can shoot them with. All along, my team collaborates constantly with the stakeholders to get more information that might be useful for testing.
I thought I’ll share this story here!
Regards,
Parimala Hariprasad
Sounds similar to my RST class where one of the students got so imprisoned in his own mental model that he simply could not think of any other ideas.
I like this approach, but I’m often given these documents before I get the product… and therefore find myself reading them first (can’t really justify sitting around and not doing anything). Then I need to try and ‘forget’ what I’ve read when I do finally get the product, so I’m not biased by it.
Thx for sharing.
Hi Paul,
Your experience reminds me an analogy to a sentence of James in his “The case against test cases” article.
He raised the “is it (the test case) limiting my imagination…?” question. In my interpretation it was a warning, that a set of tests (or even a list of expected results for a single test) preliminary created by others may limit my creative thinking. It may give ideas as well – but also frames my mindset.
It made me reinforced, because in practice, when e.g. I was transferred some work I always put the tests (if existed) aside, created my own ones and only thereafter compared my tests to the existing ones.
Perhaps I fell to another psychological trap – I understood James’ note the way I wanted to understood, because that supported my approach anyway. 🙂 But I think I was not far from the truth.
And I believe both of you points to the same cognitive behavior of the human mind.
BR,
Zoltan
Fantastic article Paul and from a psychological viewpoint what you have prevented is a great many biases from being formed that could hinder creativity. One to take away and try and apply.