Defending Against Standards and Certification

In the few weeks since James Christie scared the crap out of CAST 2014there have been a number of blog posts, a hashtag, a slow and large skeet target, a petition, and other actions taken. We now know about rent-seeking, some have properly contextualized ISO 29119, some of the lions in our field have spoken, and some of us have laid down markers for longer engagement with these issues.

Tough questions continue to be posed, and strong analyses of who is involved and what they’ve said in the past continues. A usual suspect has oscillated between poking fun at these concerns, saying that standards don’t really matter, and attacking our motives and how we identify ourselves (a rotation “Greatest Hit” rather than anything new, admittedly). Interesting how people who complain about “disrespectful and antagonistic rhetoric” have such constructive things to say.

Someone else is now unpopular (or, newly – James Bach has attempted to engage with him on multiple issues in the past), and there is enough of that to go around. There are sincere people who think standards can be helpful. Some have said that what has happened is an overreaction, but I think of it more as an overdue one.

The Uninteresting Fits of Capitalist Dreams

It’s the most coherent and together I’ve ever seen my community on an issue. There are long-standing, deep divides between my community and others who work in software quality on standards and certification. We disagree on issues that have a significant impact on our profession. Tone policing doesn’t address the actual disagreement. Intellectually eviscerating the content and arguments is simply waved away. We could talk more about compliance, pathetic and otherwise, but it seems we are finding a collective voice, and hopefully, we will no longer be dismissed as a lunatic fringe.

Rather than try to talk about all of the issues or re-echo some of what has already been said, I want to talk about why these issues matter to me enough to seek action. What are the practical impacts of standards and certifications on people working in testing?

I am against the testing standards and certifications I’ve seen because they I think their goals lead towards making testing a shitty job for a lot of people, and amplify the idea that testing and testers can be commoditized. When the experimentation and learning of testing is reduced to a series of reproducible steps, it is easier to “manage” (hire/fire/outsource) the people that do it, and to pay them less. Let’s not be mistaken: skilled testers are not desired in many of these situations. They are expensive, difficult to replace, hard to scale – and hard to commoditize.

Who Does This Affect?

For starters, everyone that works as a software tester. It’s difficult to know how many software testers there are in the world, and if that number is growing or shrinking. The US Bureau of Labor Statistics has a job code for testing, but seems to lump testers into a group called “Computer Occupations, All Other” with many other classifications. There were 196,000 US employees in that bucket in 2013.

I found a citation of 350,000 US testers in 2007. I heard Laurent Bossavit’s soft but steely voice in my head as I pursued this number (intersecting with the infamous NIST $59.5 billion annual cost of bugs legend), and found talk of calculations based on how many developers there were (~1.2m in 2007). The bucketed classification scheme I mentioned previously has only 183,000 for 2007 in the US. If we choose to base this on some custom calculation of available data, we could choose a fraction of the 1.4m developers as previously guessed at, or maybe even the 3.6m in IT.

ISTQB claims they have certified 336,000 testers through the end of 2013, with about 50,000 certifications issued in 2013. 8.1% of these in the Americas is claimed; that’s about 27,000 in the Western Hemisphere.

In any case, this gets us to a number somewhere between several hundred thousand and a few million testers in the world – and only a tiny percentage of those are flying the CDT flag.

The Testing Profession

My testing job – and the jobs of most of the people I know that work in testing – are decent jobs, but that’s because we have generally found our way to situations where we can apply judgement, experience, and skill to interesting and challenging work. These days, that circle is selected to the people I know from testing conferences and Twitter, which is certainly not the testing community at large.

Many – and I nearly said most – of the jobs I’ve seen in testing outside of this circle have not been good jobs, particularly in larger organizations. They have the “lowest-rung” IT job, sometimes considered just above first-level help desk, and sometimes below. Many testers use these jobs as stepping stones to get to something that pays better and is more interesting. Others see it as a path to management. Very few people see testing as a viable career, particularly when faced with what the testing jobs in their organization are. These testing jobs are always about to be reorganized/outsourced/sacrificed as budget markers because of how people react to issues reaching deployments, and because these jobs are difficult to tie to revenue.

One reason why these testers are not as effective is because of how they are forced to work; they spend a lot of time producing test case/test execution documentation – the primary criticism of (and prescribed “output” from) ISO 29119. Producing test case documentation might be interesting, and is an opportunity to find issues the same way writing automated checks is; you can find a problem while exploring the software in order to create test cases. Repeating the subset of activities that were actually documented from this process as manual checks is not the most effective way to find software problems, and usually doesn’t find new ones. It is checking to see if old problems reappear, and isn’t really testing. Documenting this low-value activity in detail is not useful, and lowers whatever value and effectiveness the “testing” effort was going to have even more by increasing the time spent not testing. These distinctions are either hard for factory testers to understand – or deliberately avoided.

Only Until Automation Is Cheaper
Only Until Automation Is Cheaper

Perhaps more importantly, checking and documenting checking is shitty, boring work that is not enjoyable and has significant handicaps to finding bugs baked in. It might give someone who has never seen the software before a guided tour, but testers who work with the software every day are not learning anything that will help them improve the software – or themselves. Testers stuck in these jobs spend their days doing uncreative, ineffective work, increasing the justification for and ease of implementation in outsourcing or automating their jobs away, leaving them without marketable skills and prospects the next time a desperate executive pulls out his spreadsheets, looking for something to squeeze costs on.

Developers seized their profession and remade it so they could be more effective, stop wasting their time on low-value, irrelevant tasks, recover their craft from industrial thinking, and made their job more rewarding, in multiple senses. Couldn’t and shouldn’t we do that, too?


Liability Musical Chairs - Who's The Least Cunning?
Liability Musical Chairs – Who’s The Least Cunning?

Sometimes when things go very wrong, there is an insistence that liability land on someone. Sometimes not. One example of scapegoating people for failing to prove a negative is the six-year jail sentences, and $10m in costs and fines assigned to 7 scientists in Italy who were held “responsible” for not warning the public about an earthquake. My experience is that testers are easy scapegoats for quality problems; the public sees bug escapes as testing failures instead of engineering failures.

My community’s resistance to ISO29119 and other attempts to control how others test could be critical to some future ally tester trying to defend intelligent testing – and themselves. In the US legal system, the rules for evaluating whether a technical or scientific methodology is relevant is directly tied to consensus, and maintenance of standards.

The Daubert Standard, from

Standard used by a trial judge to make a preliminary assessment of whether an expert’s scientific testimony is based on reasoning or methodology that is scientifically valid and can properly be applied to the facts at issue. Under this standard, the factors that may be considered in determining whether the methodology is valid are: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication; (3) its known or potential error rate; (4) the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community. See Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). The Daubert standard is the test currently used in the federal courts and some state courts.  In the federal courts, it replaced the Frye standard.

The rigor our community applies to evaluating claims is a good fit here. We need to keep loudly and consistently calling bullshit on hand-wavy best practice and standard assertions, or they will become the *standards* our work is *judged* by.


Many people throw up their hands about government activities, considering it a waste of their time to follow, frustrated by its arbitrary, uninformed, and frequent ineffectualness. Others rage over the sinister motivations assigned to the side they don’t agree with. There are always plenty, living in libertarian fantasies (redundant), who want to call the whole thing irrelevant.

Meanwhile, in the real world, government makes decisions every day, careening from one crisis to another, trying to solve every problem with lobbyist-authored, detailed instructions on how everything must work. Seems like a standards or certification board? Or at least the reality of what they want?

Here is my US Representative (lower Federal House) – the representative for all of Silicon Valley – trying to ask a question about a “Security Wall” for She is brushed off with the weakest of doubletalk.

It seems clear that we can choose to engage with this issue, or let other people define the terms for everyone. I want us to avoid becoming to subject to standards and certifications chosen for us by people who don’t understand the issues involved. Can we educate them and the public? Or will we have them respond reactively in the heat of a moment?

The Future: Algorithms, Automation, and Robotics

Once we solve for potholes, and road debris, and weather, and construction…

People get excited about hardware, but the future arrives in software.

I see these amazing test challenges all the time, mostly because this picture was taken within a mile of my house, and even closer to where these are being developed. Old-timey automakers are working on this now, too. There are other vehicles standing by.

The economic drive to automate shows no sign of letting up. Economic drives do not have consciences – they require people to pay attention and check abuses.

Our present already runs on algorithms that can cut communicationsshut down powercrash the stock market, or even kill us when there are bugs. Our access to water, food, health care, education, employment, housing, and transportation are all subject to computer systems working correctly – even when people operate them “correctly”. This is a very good place for a reminder that software and testing are still very young fields, with a lot left to learn and a lot farther to go.

This is not Sci-Fi. And if it were better, it would have bio-hacking. Still!

In the future, more and more complex tasks will be automated, to the extent that the important decisions can be automated. A few humans will be needed for exception cases, but software will be expected to handle more and more routine tasks. The economic effects of this trend are troubling enough. The risks of autonomous software issues in medicine, transportation, economics, energy, and “defense” need to be met by engaged, expert testers. We need thinking, exploring testers with time and space to do their best in order for our future to be safe.

The Bottom Line

I want people who work on software to demand skilled testing because of its superior risk mitigation. Modern testing is the way forward because it is more effective for exposing bugs. Our natural allies in the Agile Community will be happy to have us step forward and take control of the testing narrative. They are no more interested in us wasting time generating proof of compliance documentation than we are in doing it.

I will speak up for my values, and the values of my community. I will help amplify the ideas of my community, and look for ways we can influence how the world at large thinks about testing. I will push for our community’s treasures to be acknowledged as serious testers who are the very best in the world at what they do.

I want testing to be a skilled, well-paid profession. I want testing to be a profession that bright people can learn from, improve at, and advance in, not just something they pass through on their way to something else – unless they want to. As well-trained critical thinkers, they should be successful wherever they go.

Enough sitting out, being polite, and waiting for someone else to step forward. I will not stand by while consultants sell out our craft as a marketing exercise. We must rescue testing from compliance and documentation, and make it so that skilled testers are expected to choose the appropriate test methods and documentation for the stakeholders and context they find themselves on.

So far, I have helped set up and maintain, which I hope you’ll consider signing. Karen Johnson has done a great thing for our community in helping articulate our principles.

I have talked about these issues in a podcast interview, though it has not posted yet. I have some writing to do here, and other places. I want to speak clearly and loudly about the benefits of modern testing, and how standards and certifications limit testers and testing.

Longer term, I have other work to do on this. Shortly after CAST, I made a formal proposal to AST for a Standards and Certification SIG. More on that soon.

One Comment

Comments are closed.