I live in Mountain View, California and work for SOASTA. I have worked in testing for 16 years, and been engaged with the CDT community for 11.
Of the work I’ve done in the testing community, I am most proud of being a Speak Easy mentor. I’m also an organizer of a long-running peer workshop (WOPR, the Workshop on Performance and Reliability), and a member of the Community Advisory Board for STPCon. I have been a member of AST for several years, first attending CAST in 2010.
After last year’s CAST, I set out to form the AST Committee on Standards and Professional Practices to help AST engage with issues affecting our profession as a professional trade organization. This work, and my experience working with AST to get this Committee started led to me accepting a nomination to run for the AST Board.
I am asking for your vote because there are things I hope to accomplish for our organization, our community, and our profession as an AST board member. I believe I have a track record of getting things done, by digging in and doing them myself. I think that’s what we need more of on the AST board. With urgency, focus, humility, and willingness to iterate, we can accomplish a lot.
The ideas of our community’s founders and leaders changed testing forever. I want to publicize our school’s approach to testing as viable, respectable, and highly effective to the testing AND software development worlds, giving our membership support and resources for implementing modern testing principles. I also want to help solidify a launchpad for the next generation of our community’s leaders to lift off from. We have a bright, passionate, and diverse crop in our community that will shine very brightly when they get their turn.
I am sorry I won’t see so many of my friends and colleagues in Grand Rapids next month. During CAST, I will be speaking at an Agile Development conference (Agile2015). I plan to talk with that community about testing, and how “automating everything” is sure to miss many bugs. From where I sit in the heart of Silicon Valley, this desire to turn testing into checking is an even greater risk to our profession’s future than factory certifications and testing standards – and is not an empty house of cards like these economic tactics.
I will push AST forward as an organization by challenging the organization to improve as an advocate for all testers, whether they identify as a member of our community or not. AST should continue to proudly be a Context-Driven organization, but the business and profession of testing has been defined by large commercial interests for too long. We must present a credible, experience-based alternative to the obsolete, ineffective command and control processes recommended by those seeking to profit from marketing McTesting process and “expertise”. We will win with our superior ideas, demonstrating that in a world of faster development iterations with smaller teams, it’s our community that is moving the practice forward.
I hope to help AST do more for its membership. My first set of tasks are to help members to find and collect tools and links for self-learning, aid them with job posting and searches, help them connect with peers for advice, and provide them with practice references when they set out to improve testing in their organizations. A key task for helping this happen is to revitalize the AST website so that others can easily contribute their research, experiences, and voices to a common body of knowledge. I will ask the board to allow me to revamp the AST website towards community engagement, and I will personally do much of the work to make it happen.
I want to help AST expand its footprint internationally. While much of our membership is American, there are thriving communities in Europe, Australia, New Zealand, and even Asia. AST can support and sponsor decentralized, loosely affiliated CDT conferences in places where there is demand and people on the ground.
Lastly, I will continue to work with the STP board to welcome CDT content and speakers into those conferences. If you take a look at our fall schedule, you can see that is going quite well.
To learn more about me, you could visit my blog here at contextdrivenperformancetesting.com, follow me @ericproegler, or chat with me sometime. I’d love to hear more ideas about how to increase AST’s accountability to the community, so that it can better earn the community’s attention and participation.
ISO/IEC/IEEE 29119 supports dynamic testing, functional and non-functional testing, manual and automated testing, and scripted and unscripted testing.The processes defined in this series of international standards can be used in conjunction with any software development lifecycle model. Each process is defined…and covers the purpose, outcomes, activities, tasks and information items of each test process.
Please remember I am criticizing the standard (and the idea of a testing standard), not the people who worked on it. I believe that smart, experienced people attempted to lay out their view(s) of testing, hoping to help people test effectively. I think that in the right discussion about the many contexts software might be tested in, they might concede that no prescriptive standard can be relevant and useful in every context. In fact, some of them are already doing that. Whatever the shortcomings of 29119 (and there are plenty) it could never possibly satisfy its mission, even if it was a better standard than it actually is.
My best-practice, conform-ational approach is to summarize my primary conclusions at the top of my blog posts, sparing tens of readers the post’s full brilliance. Here are my “above the fold” takeaways from analyzing ISO 29119-2:
29119 literally puts process (Part 2) before technique (promised in Part 4, still not published)
29119 claims to be applicable to testing in *all* software development lifecycle models, despite heavy documentation and compliance burdens
29119-2 has Conformance on page 1. To claim Conformance, there are 138 “Shalls” to conform to in this document. To claim “Tailored Conformance” without meeting every “Shall”, “justification shall be provided…whenever a process defined in…29119 is not followed”
Part 2’s vocabulary section has conflicts, revisions, and pointers to new terms relative to Part 1. This is not a “gotcha” – but is worth remembering when someone claims that with a test standard “At least there is a common vocabulary for testing”.
Conformance is driven by fear. Fear is the mind-killer.
Some of the “shalls” are highly specific. Some are vague and hard to understand. Some, through reference, contain multitudes. Some are nonsense.
The standard is not detailed enough to be very useful to someone who doesn’t already understand a fair amount about testing, yet an experienced tester would waste a lot of time and effort attempting to comply with it.
29119-2 goes to Conformance very early – Page 1. Either Full or Tailored conformance can be claimed for the standard.
“Full conformance is achieved by demonstrating that all of the requirements (i.e. shall statements) of the full set of processes defined in this part of ISO/IEC/IEEE 29119 have been satisfied.”
“Tailored conformance is achieved by demonstrating that all of the requirements (i.e. shall statements) for the recorded subset of processes have been satisfied. Where tailoring occurs, justification shall be provided (either directly or by reference), whenever a process defined in…ISO 29119 is not followed. All tailoring decisions shall be recorded with their rationale, including the consideration of any applicable risks.”
I can find no guidance on what “the recorded subset of processes” means. Not just what the various nesting levels of “process” are in the standard, either. Are these the processes that reference record-keeping and documentation? I bet I can find a consultant to help not-interpret that…
There is a “Reference” example given for exclusion from the requirement for providing direct justification:
“Where organizations follow information item management processes in standards such as ISO 15489… ISO 9001…or use similar internal organizational processes, they can decide to use those processes in place of the information item management tasks defined in this part of ISO/IEC/IEEE 29119.”
So, no exclusion from the requirement to document and describe the justifications – just an exclusion from the requirement to provide a separate document including these justifications for ISO 29119, as long as they are in another document somewhere else.
After 10 months, the only defense raised thus far by the authors of the standard to the questions about difficult compliance is to claim it is more flexible than what is actually said in the standard:
… and that’s the last message in the conversation. I suppose we could take the word of a standard author over the standard itself, which says with little ambiguity under Intended Usage: “The organization shall assert whether it is claiming full or tailored conformance to this part of ISO/IEC/IEEE 29119”.
Section 2 spells out definitions for some terms. There is overlap with Section 1 – and some disagreement with what was found there.
For example, in Section 1, Feature Set meant “collection of items which contain the test conditions of the test item to be tested which can be collected from risks, requirements, functions, models, etc.” Section 2: “logical subset of the test item(s) that could be treated independently of other feature sets in the subsequent test design activities”. Additional differences, revisions, and pointers to new terms are found. This is not a “gotcha” – but is worth remembering when someone claims that with a test standard: “At least there is a common vocabulary for testing”, ISO 29119 already has divergence in critical definitions between its first two parts.
At least these terms are interesting to think about. It’s far less interesting to trace the relationships between test activity, test item, test condition, rest requirement, test phase, test plan, test policy, test planning process, test procedure, test procedure specification, test condition, test process, test sub-process, test script, test set, test models, test technique, test specification, and test type. Yes, these are all separate things, but time spent debating their boundaries is time not spent “testing”.
Exploratory testing is again defined as “spontaneously designs and executes”, not “simultaneously” as we define it.
Process and Hierarchy
This diagram shows a hierarchy of test processes. It doesn’t actually cover all the processes referenced in the standard, despite the caption’s claim. The diagram does demonstrate the standard’s insistence on separating control processes from execution processes.
It is intended to illustrate that the vertical layers define each other downwards. First is the organizational process that defines process for organizational test policies, which dictate policy, strategy, process, procedure, and “other assets”. Test Management Processes are defined at the project level, Dynamic Test Processes are said to control a phase or particular type of testing.
This seems tailored for adoption by the mid-level executive who wants to put their stamp on an organization’s entire testing practice. Over and over again, the standard lays out separate process nodes for each possible step of testing. This exhaustive documentation of the steps involved in one view of testing is way too much for an experienced tester, who would rather provide useful information to stakeholders. It’s still not enough to arm someone with no testing experience to plan and supervise good testing. So who is it for?
When Fear Drives Testing
Software testing is frequently perceived as a high-risk, low-reward activity by people who aren’t testers. It’s thought of as a cost center (“there is no ROI in testing”) and if anything goes wrong, someone’s in trouble. Over and over again, testing is blamed for poor quality, despite the fact that most people who work in software engineering know “you can’t test quality into the product”. Testing is often thought of as less intellectually rigorous than other parts of software engineering, frequently is not a prestigious area to work in, sometimes is led by people without real training, experience, and/or skill in testing, and is often a convenient scapegoat for quality issues – particularly by people who should know better.
Many people that work in testing (rightfully) fear the buck stopping on their desk after a quality failure, and for good reason. If you are likely to have blame imposed for a bug escape, the most rational response by a skilled person might be to interrogate the context and demand the tools and latitude to gather the most comprehensive and useful set of information about the system under test.
If you are controlled by fear, you might shy away from the responsibility, and look for some cover under best practices. After all, if you faithfully observed and obeyed someone else’s plan, you can’t be blamed if the plan fails, right? It wasn’t you, it was the plan!
If you don’t know what you are doing, you might be even more likely to seek the comfort of an externally defined standard that removes your responsibility to decide what to do. If you don’t trust your team (and yourself), you hand off control to someone or something else. Like a prescriptive standard, full of “shall statements” to replace “you thinking”.
The standard is still not detailed enough to be very useful to someone who doesn’t already understand a fair amount about testing, yet an experienced tester could waste a lot of time and effort trying to comply with it. Any discussion of actual techniques seems to be waiting for 29119-4 – at one point promised for late 2014, currently late in the approval process.
There are 138 instances of “shall” in this document. Some of them are highly specific. Some, by reference, contain multitudes. Some are simply nonsense. Some of them are too vague to be useful, though that may make them more applicable in multiple contexts. Some real wisdom can be found in here.
I spent some time pulling apart the various processes, sub-processes, dependencies, and circular references. Rather than try to further sketch out the overall shape of process (and documentation) requirements, I present my 10 most entertaining/concerning/Kafkaesque “Shall Statements” in ISO 29119-2:
The person responsible for organizational test specifications shall implement the following activities and tasks in accordance with applicable organization policies and procedures with respect to the Organizational Test Process.
The organizational test specification requirements shall be used to create the organizational test specification.
Appropriate actions shall be taken to encourage alignment of stakeholders to the organizational test specification.
The traceability between the test basis, feature sets, test conditions, test coverage items, test cases and test sets shall be recorded.
The testing of the feature sets shall be prioritized using the risk exposure levels documented in the Identify and Analyze Risks activity (TP3).
Any risks that have been previously identified shall be reviewed to identify those that relate to and/or can be treated by software testing.
Each required test activity in the Test Strategy shall be scheduled based on the estimates, dependencies and staff availability.
Those actions necessary to implement control directives received from higher level management processes shall be performed.
Readiness for commencing any assigned test activity shall be established before commencing that activity, if not already done.
The test coverage items to be exercised by the testing shall be derived by applying test design techniques to the test conditions to achieve the test completion coverage criteria specified in the Test Plan…
NOTE 2 Where a test completion criterion for the test item is specified as less than 100% of a test coverage measure, a subset of the test coverage items required to achieve 100 % coverage needs to be selected to be exercised by the testing.
It’s not all baffling. Here’s a richly meaningful shall statement that demonstrates something about the depth necessary to understand context:
A Test Strategy (comprising choices including test phases, test types, features to be tested, test design techniques, test completion criteria, and suspension and resumption criteria) shall be designed that considers test basis, risks, and organizational, project and product constraints…
NOTE 3 This takes into consideration the level of risk exposure to prioritise the test activities, the initial test estimates, the resources needed to perform actions (e.g. skills, tool support and environment needs), and organizational, project and product constraints, such as:
a) regulatory standards; b) the requirements of the Organizational Test Policy, Organizational Test Strategy and the Project Test Plan (if designing a test strategy for a lower level of testing); c) contractual requirements; d) project time and cost constraints; e) availability of appropriately-skilled testers; f) availability of tools and environments; g) technical, system or product limitations.
The last third of 29119-2 is an Annex mapping clauses of other standards (ISO 12207, ISO 15288, ISO 17025, ISO 25051, BS 7925, and IEEE 1008) to 29119-2. Rather than critique these other standards, I will simply question the value and purpose of this exercise. Is it to justify the standard, or to prove that it equals or even supersedes the others?
We still have parts 3 (and 4? soon?) of 29119 to go. Having processes defined before considering what we want to accomplish will guarantee we end at our desired results (whatever that might be), right?
It’s the most coherent and together I’ve ever seen my community on an issue. There are long-standing, deep divides between my community and others who work in software quality on standards and certification. We disagree on issues that have a significant impact on our profession. Tone policing doesn’t address the actual disagreement. Intellectuallyeviscerating the content and arguments is simply waved away. We could talk more about compliance, pathetic and otherwise, but it seems we are finding a collective voice, and hopefully, we will no longer be dismissed as a lunatic fringe.
Rather than try to talk about all of the issues or re-echo some of what has already been said, I want to talk about why these issues matter to me enough to seek action. What are the practical impacts of standards and certifications on people working in testing?
I am against the testing standards and certifications I’ve seen because they I think their goals lead towards making testing a shitty job for a lot of people, and amplify the idea that testing and testers can be commoditized. When the experimentation and learning of testing is reduced to a series of reproducible steps, it is easier to “manage” (hire/fire/outsource) the people that do it, and to pay them less. Let’s not be mistaken: skilled testers are not desired in many of these situations. They are expensive, difficult to replace, hard to scale – and hard to commoditize.
@AntonR348@QualityFrog My point, as any project manager will know, is that special skills requirements can be a constraint & a project risk
I found a citation of 350,000 US testers in 2007. I heard Laurent Bossavit’s soft but steely voice in my head as I pursued this number (intersecting with the infamous NIST $59.5 billion annual cost of bugs legend), and found talk of calculations based on how many developers there were (~1.2m in 2007). The bucketed classification scheme I mentioned previously has only 183,000 for 2007 in the US. If we choose to base this on some custom calculation of available data, we could choose a fraction of the 1.4m developers as previously guessed at, or maybe even the 3.6m in IT.
ISTQB claims they have certified 336,000 testers through the end of 2013, with about 50,000 certifications issued in 2013. 8.1% of these in the Americas is claimed; that’s about 27,000 in the Western Hemisphere.
In any case, this gets us to a number somewhere between several hundred thousand and a few million testers in the world – and only a tiny percentage of those are flying the CDT flag.
@Oliver_NZ There seem to be, roughly, a thousand of us. 20 years of spreading the word pays off.
My testing job – and the jobs of most of the people I know that work in testing – are decent jobs, but that’s because we have generally found our way to situations where we can apply judgement, experience, and skill to interesting and challenging work. These days, that circle is selected to the people I know from testing conferences and Twitter, which is certainly not the testing community at large.
Many – and I nearly said most – of the jobs I’ve seen in testing outside of this circle have not been good jobs, particularly in larger organizations. They have the “lowest-rung” IT job, sometimes considered just above first-level help desk, and sometimes below. Many testers use these jobs as stepping stones to get to something that pays better and is more interesting. Others see it as a path to management. Very few people see testing as a viable career, particularly when faced with what the testing jobs in their organization are. These testing jobs are always about to be reorganized/outsourced/sacrificed as budget markers because of how people react to issues reaching deployments, and because these jobs are difficult to tie to revenue.
One reason why these testers are not as effective is because of how they are forced to work; they spend a lot of time producing test case/test execution documentation – the primary criticism of (and prescribed “output” from) ISO 29119. Producing test case documentation might be interesting, and is an opportunity to find issues the same way writing automated checks is; you can find a problem while exploring the software in order to create test cases. Repeating the subset of activities that were actually documented from this process as manual checks is not the most effective way to find software problems, and usually doesn’t find new ones. It is checking to see if old problems reappear, and isn’t really testing. Documenting this low-value activity in detail is not useful, and lowers whatever value and effectiveness the “testing” effort was going to have even more by increasing the time spent not testing. These distinctions are either hard for factory testers to understand – or deliberately avoided.
Perhaps more importantly, checking and documenting checking is shitty, boring work that is not enjoyable and has significant handicaps to finding bugs baked in. It might give someone who has never seen the software before a guided tour, but testers who work with the software every day are not learning anything that will help them improve the software – or themselves. Testers stuck in these jobs spend their days doing uncreative, ineffective work, increasing the justification for and ease of implementation in outsourcing or automating their jobs away, leaving them without marketable skills and prospects the next time a desperate executive pulls out his spreadsheets, looking for something to squeeze costs on.
Developers seized their profession and remade it so they could be more effective, stop wasting their time on low-value, irrelevant tasks, recover their craft from industrial thinking, and made their job more rewarding, in multiple senses. Couldn’t and shouldn’t we do that, too?
Sometimes when things go very wrong, there is an insistence that liability land on someone. Sometimes not. One example of scapegoating people for failing to prove a negative is the six-year jail sentences, and $10m in costs and fines assigned to 7 scientists in Italy who were held “responsible” for not warning the public about an earthquake. My experience is that testers are easy scapegoats for quality problems; the public sees bug escapes as testing failures instead of engineering failures.
My community’s resistance to ISO29119 and other attempts to control how others test could be critical to some future ally tester trying to defend intelligent testing – and themselves. In the US legal system, the rules for evaluating whether a technical or scientific methodology is relevant is directly tied to consensus, and maintenance of standards.
Standard used by a trial judge to make a preliminary assessment of whether an expert’s scientific testimony is based on reasoning or methodology that is scientifically valid and can properly be applied to the facts at issue. Under this standard, the factors that may be considered in determining whether the methodology is valid are: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication; (3) its known or potential error rate; (4) the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community. See Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). The Daubert standard is the test currently used in the federal courts and some state courts. In the federal courts, it replaced the Frye standard.
The rigor our community applies to evaluating claims is a good fit here. We need to keep loudly and consistently calling bullshit on hand-wavy best practice and standard assertions, or they will become the *standards* our work is *judged* by.
Many people throw up their hands about government activities, considering it a waste of their time to follow, frustrated by its arbitrary, uninformed, and frequent ineffectualness. Others rage over the sinister motivations assigned to the side they don’t agree with. There are always plenty, living in libertarian fantasies (redundant), who want to call the whole thing irrelevant.
Meanwhile, in the real world, government makes decisions every day, careening from one crisis to another, trying to solve every problem with lobbyist-authored, detailed instructions on how everything must work. Seems like a standards or certification board? Or at least the reality of what they want?
Here is my US Representative (lower Federal House) – the representative for all of Silicon Valley – trying to ask a question about a “Security Wall” for healthcare.gov. She is brushed off with the weakest of doubletalk.
It seems clear that we can choose to engage with this issue, or let other people define the terms for everyone. I want us to avoid becoming to subject to standards and certifications chosen for us by people who don’t understand the issues involved. Can we educate them and the public? Or will we have them respond reactively in the heat of a moment?
The Future: Algorithms, Automation, and Robotics
People get excited about hardware, but the future arrives in software.
I see these amazing test challenges all the time, mostly because this picture was taken within a mile of my house, and even closer to where these are being developed. Old-timey automakers are working on this now, too. There are other vehiclesstanding by.
The economic drive to automate shows no sign of letting up. Economic drives do not have consciences – they require people to pay attention and check abuses.
Our present already runs on algorithms that can cut communications, shut down power, crash the stock market, or even kill us when there are bugs. Our access to water, food, health care, education, employment, housing, and transportation are all subject to computer systems working correctly – even when people operate them “correctly”. This is a very good place for a reminder that software and testing are still very young fields, with a lot left to learn and a lot farther to go.
In the future, more and more complex tasks will be automated, to the extent that the important decisions can be automated. A few humans will be needed for exception cases, but software will be expected to handle more and more routine tasks. The economic effects of this trend are troubling enough. The risks of autonomous software issues in medicine, transportation, economics, energy, and “defense” need to be met by engaged, expert testers. We need thinking, exploring testers with time and space to do their best in order for our future to be safe.
The Bottom Line
I want people who work on software to demand skilled testing because of its superior risk mitigation. Modern testing is the way forward because it is more effective for exposing bugs. Our natural allies in the Agile Community will be happy to have us step forward and take control of the testing narrative. They are no more interested in us wasting time generating proof of compliance documentation than we are in doing it.
I will speak up for my values, and the values of my community. I will help amplify the ideas of my community, and look for ways we can influence how the world at large thinks about testing. I will push for our community’s treasures to be acknowledged as serious testers who are the very best in the world at what they do.
I want testing to be a skilled, well-paid profession. I want testing to be a profession that bright people can learn from, improve at, and advance in, not just something they pass through on their way to something else – unless they want to. As well-trained critical thinkers, they should be successful wherever they go.
Enough sitting out, being polite, and waiting for someone else to step forward. I will not stand by while consultants sell out our craft as a marketing exercise. We must rescue testing from compliance and documentation, and make it so that skilled testers are expected to choose the appropriate test methods and documentation for the stakeholders and context they find themselves on.
So far, I have helped set up and maintain professionaltestersmanifesto.org, which I hope you’ll consider signing. Karen Johnson has done a great thing for our community in helping articulate our principles.
I have talked about these issues in a podcast interview, though it has not posted yet. I have some writing to do here, and other places. I want to speak clearly and loudly about the benefits of modern testing, and how standards and certifications limit testers and testing.
Longer term, I have other work to do on this. Shortly after CAST, I made a formal proposal to AST for a Standards and Certification SIG. More on that soon.
I made the mistake of clicking through. The article is titled “Best Practices of Context-Driven Testing”, and it is delicious, savory troll bait.
I had to leave a comment, which follows. Still too shallow – but that is appropriate for the subject:
(Update: a month later, comment not approved. As always, beware putting effort into comments and commenting interfaces.)
I’m not hung up on words. It’s just these two words really suck.
I understand that some people get tired of the CDT community digging in their heels on “Best Practices”, but it needs to be understood that this isn’t about being schoolmarms who pedantically correct “Who” with “Whom”. It’s about rejecting the idea that there is one correct answer in any situation, and that “Best Practices” are inextricably tied to the idea that you can replace judgment, skill, and experience with process weenery.
“Best Practices” can and will never accomplish their goal: replacing skill and experience with canned knowledge and prescriptive recipes. A skilled practitioner can explain when a practice makes sense, and when it doesn’t. An expert can evaluate a situation and solve a problem. An amateur risks believing they are an expert because they read a Wikipedia page – or can regurgitate a “Best Practice”.
People that think the “Best Practice” label is harmless have probably never run into a situation where someone is using that particular appeal to a nebulous, non-existent authority to assert control and dismiss disagreement. Many people have seen and experienced how dangerous it is to allow anyone to wave around whatever practice that sounds good that they read about last and call it “Best”.
Good testing requires examining what you know, where it came from, how much you trust it, and how you might falsify it. When a professional tester dismisses criticism of “Best Practices”, they are suggesting that they are probably not very good at testing. Not because they don’t have a prescribed reaction to a blasphemous term, but because they seem dismissive of the deep thinking required to be good at testing.