Immediately after CAST 2014, I entered a proposal for an AST Special Interest Group (SIG). I’ve put the contents of that proposal below, as a dialogue with what AST presents as the process for creating a SIG.
I have not yet received feedback from the AST board on this proposal. In any case, I consider it a declaration of principles and intention. I hope to work with the AST board on formalizing positions that the AST will take on the important issues our community faces. I look forward to seeing AST’s response to my proposal.
AST SIG on Standards, Certification, and Regulation
What is the focus, need, and purpose of this SIG? (An example mission or charter is helpful)
What activities will be done by this SIG? (publications, workshops, multimedia)
I have included a charter below that describes these activities.
Who will lead the SIG? (They must be an AST member in good standing)
Eric Proegler, initially. Fiona Charles, James Christie, Lee Copeland, Iain McCowatt, Huib Schoots, and Mark Tomlinson are the other founding members.
Is there a need for funding? What will it be used for and how much is needed?
Yes. We are asking for about $600 USD, for purchasing ISO 29119-1 through 3 (~$200 each). Sections 4 and 5 are not yet published. AST will own these documents.
Charter: AST SIG on Standards, Certification, and Regulation
Our industry and our craft thrives and grows when professional testers can apply their skills and experience to solving testing problems. The quality of our work will decline if constrained by standards and certifications, which will mean buggier software released into the wild and increased risk to the public.
Resistance in our community to testing standards and certifications is broad, but somewhat diffused. In this environment, a small number of people have taken the opportunity to create systems where they can present themselves as the arbiters of how testing should be conducted, and as the gatekeepers to the profession, creating a direct financial benefit for themselves. These standards and certifications are frequently marketed in terms of liability and risk incurred by organizations who do not employ and require them – and insist that their service providers do the same.
Our world essentially already runs on software. In the coming years, unmanned vehicles, robotics, advanced medical devices, and other complex algorithm-driven control systems are going to make this even more obvious to a public that does not have a deep understanding of how software engineering and testing are related. Quality issues such as healthcare.gov are typically seen by the public as testing failures, not software engineering failures. If we allow standards and certification vendors to define the dialog, we will someday soon find ourselves in a regulatory climate reacting to the latest Very Bad Thing with an imperative to Do Something – with vendors who stand to profit ready to describe what that should be.
By engaging directly with standards, certification, and regulation, we hope to defend our craft, protect our fellow testers, and educate the public on testing. We must provide detailed critiques, describe alternatives, and call attention to what works. We will publish commentary and criticism on Testing Standards, Tester Certification, and the Regulation of Software Testing and Software Quality. We will directly engage with and influence these issues by contacting standards bodies, legislative staff, and the press. We will advance AST as a credible, trusted, cited, and authoritative source of information for people who do not work in software engineering when they think, write, and legislate about software testing and quality.
We will work with the board to articulate official AST positions on Testing Standards and Tester Certifications, reflecting the mission and values of AST.
We will publish critiques of specific standards and certifications, starting with ISO 29119 and ISTQB. We will describe where they fall short, and making it clear who is advancing them.
We will publish short background pieces on how modern software testing is performed for audiences outside of software testing, in order to inform and influence how they think about testing and verification.
We will publish about testing intelligently and effectively in regulated environments, such as FDA regulated testing, the JPL’s experience with Exploratory Testing on spacecraft software, and other examples that attest to the value and trustworthiness of modern testing techniques and approaches.
We will provide timely commentary when software issues become news stories.
We will present our positions and rhetoric at testing conferences, sharing our findings and inviting feedback to improve them.
We will publicly seek seats on standards bodies such as ISO, enter comments on standards under development, and contribute to new standards.
We will conduct workshops to finalize and gather support and signatories for the positions and critiques we develop.
We will monitor legislative activity pertaining to software testing, quality, and engineering, report on what is happening to AST at large, and attempt to intervene/engage with legislative investigations and actions.
We will cultivate contacts with media organizations to encourage them to contact AST Members for information about software testing and quality.
It’s the most coherent and together I’ve ever seen my community on an issue. There are long-standing, deep divides between my community and others who work in software quality on standards and certification. We disagree on issues that have a significant impact on our profession. Tone policing doesn’t address the actual disagreement. Intellectuallyeviscerating the content and arguments is simply waved away. We could talk more about compliance, pathetic and otherwise, but it seems we are finding a collective voice, and hopefully, we will no longer be dismissed as a lunatic fringe.
Rather than try to talk about all of the issues or re-echo some of what has already been said, I want to talk about why these issues matter to me enough to seek action. What are the practical impacts of standards and certifications on people working in testing?
I am against the testing standards and certifications I’ve seen because they I think their goals lead towards making testing a shitty job for a lot of people, and amplify the idea that testing and testers can be commoditized. When the experimentation and learning of testing is reduced to a series of reproducible steps, it is easier to “manage” (hire/fire/outsource) the people that do it, and to pay them less. Let’s not be mistaken: skilled testers are not desired in many of these situations. They are expensive, difficult to replace, hard to scale – and hard to commoditize.
@AntonR348@QualityFrog My point, as any project manager will know, is that special skills requirements can be a constraint & a project risk
I found a citation of 350,000 US testers in 2007. I heard Laurent Bossavit’s soft but steely voice in my head as I pursued this number (intersecting with the infamous NIST $59.5 billion annual cost of bugs legend), and found talk of calculations based on how many developers there were (~1.2m in 2007). The bucketed classification scheme I mentioned previously has only 183,000 for 2007 in the US. If we choose to base this on some custom calculation of available data, we could choose a fraction of the 1.4m developers as previously guessed at, or maybe even the 3.6m in IT.
ISTQB claims they have certified 336,000 testers through the end of 2013, with about 50,000 certifications issued in 2013. 8.1% of these in the Americas is claimed; that’s about 27,000 in the Western Hemisphere.
In any case, this gets us to a number somewhere between several hundred thousand and a few million testers in the world – and only a tiny percentage of those are flying the CDT flag.
@Oliver_NZ There seem to be, roughly, a thousand of us. 20 years of spreading the word pays off.
My testing job – and the jobs of most of the people I know that work in testing – are decent jobs, but that’s because we have generally found our way to situations where we can apply judgement, experience, and skill to interesting and challenging work. These days, that circle is selected to the people I know from testing conferences and Twitter, which is certainly not the testing community at large.
Many – and I nearly said most – of the jobs I’ve seen in testing outside of this circle have not been good jobs, particularly in larger organizations. They have the “lowest-rung” IT job, sometimes considered just above first-level help desk, and sometimes below. Many testers use these jobs as stepping stones to get to something that pays better and is more interesting. Others see it as a path to management. Very few people see testing as a viable career, particularly when faced with what the testing jobs in their organization are. These testing jobs are always about to be reorganized/outsourced/sacrificed as budget markers because of how people react to issues reaching deployments, and because these jobs are difficult to tie to revenue.
One reason why these testers are not as effective is because of how they are forced to work; they spend a lot of time producing test case/test execution documentation – the primary criticism of (and prescribed “output” from) ISO 29119. Producing test case documentation might be interesting, and is an opportunity to find issues the same way writing automated checks is; you can find a problem while exploring the software in order to create test cases. Repeating the subset of activities that were actually documented from this process as manual checks is not the most effective way to find software problems, and usually doesn’t find new ones. It is checking to see if old problems reappear, and isn’t really testing. Documenting this low-value activity in detail is not useful, and lowers whatever value and effectiveness the “testing” effort was going to have even more by increasing the time spent not testing. These distinctions are either hard for factory testers to understand – or deliberately avoided.
Perhaps more importantly, checking and documenting checking is shitty, boring work that is not enjoyable and has significant handicaps to finding bugs baked in. It might give someone who has never seen the software before a guided tour, but testers who work with the software every day are not learning anything that will help them improve the software – or themselves. Testers stuck in these jobs spend their days doing uncreative, ineffective work, increasing the justification for and ease of implementation in outsourcing or automating their jobs away, leaving them without marketable skills and prospects the next time a desperate executive pulls out his spreadsheets, looking for something to squeeze costs on.
Developers seized their profession and remade it so they could be more effective, stop wasting their time on low-value, irrelevant tasks, recover their craft from industrial thinking, and made their job more rewarding, in multiple senses. Couldn’t and shouldn’t we do that, too?
Sometimes when things go very wrong, there is an insistence that liability land on someone. Sometimes not. One example of scapegoating people for failing to prove a negative is the six-year jail sentences, and $10m in costs and fines assigned to 7 scientists in Italy who were held “responsible” for not warning the public about an earthquake. My experience is that testers are easy scapegoats for quality problems; the public sees bug escapes as testing failures instead of engineering failures.
My community’s resistance to ISO29119 and other attempts to control how others test could be critical to some future ally tester trying to defend intelligent testing – and themselves. In the US legal system, the rules for evaluating whether a technical or scientific methodology is relevant is directly tied to consensus, and maintenance of standards.
Standard used by a trial judge to make a preliminary assessment of whether an expert’s scientific testimony is based on reasoning or methodology that is scientifically valid and can properly be applied to the facts at issue. Under this standard, the factors that may be considered in determining whether the methodology is valid are: (1) whether the theory or technique in question can be and has been tested; (2) whether it has been subjected to peer review and publication; (3) its known or potential error rate; (4) the existence and maintenance of standards controlling its operation; and (5) whether it has attracted widespread acceptance within a relevant scientific community. See Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). The Daubert standard is the test currently used in the federal courts and some state courts. In the federal courts, it replaced the Frye standard.
The rigor our community applies to evaluating claims is a good fit here. We need to keep loudly and consistently calling bullshit on hand-wavy best practice and standard assertions, or they will become the *standards* our work is *judged* by.
Many people throw up their hands about government activities, considering it a waste of their time to follow, frustrated by its arbitrary, uninformed, and frequent ineffectualness. Others rage over the sinister motivations assigned to the side they don’t agree with. There are always plenty, living in libertarian fantasies (redundant), who want to call the whole thing irrelevant.
Meanwhile, in the real world, government makes decisions every day, careening from one crisis to another, trying to solve every problem with lobbyist-authored, detailed instructions on how everything must work. Seems like a standards or certification board? Or at least the reality of what they want?
Here is my US Representative (lower Federal House) – the representative for all of Silicon Valley – trying to ask a question about a “Security Wall” for healthcare.gov. She is brushed off with the weakest of doubletalk.
It seems clear that we can choose to engage with this issue, or let other people define the terms for everyone. I want us to avoid becoming to subject to standards and certifications chosen for us by people who don’t understand the issues involved. Can we educate them and the public? Or will we have them respond reactively in the heat of a moment?
The Future: Algorithms, Automation, and Robotics
People get excited about hardware, but the future arrives in software.
I see these amazing test challenges all the time, mostly because this picture was taken within a mile of my house, and even closer to where these are being developed. Old-timey automakers are working on this now, too. There are other vehiclesstanding by.
The economic drive to automate shows no sign of letting up. Economic drives do not have consciences – they require people to pay attention and check abuses.
Our present already runs on algorithms that can cut communications, shut down power, crash the stock market, or even kill us when there are bugs. Our access to water, food, health care, education, employment, housing, and transportation are all subject to computer systems working correctly – even when people operate them “correctly”. This is a very good place for a reminder that software and testing are still very young fields, with a lot left to learn and a lot farther to go.
In the future, more and more complex tasks will be automated, to the extent that the important decisions can be automated. A few humans will be needed for exception cases, but software will be expected to handle more and more routine tasks. The economic effects of this trend are troubling enough. The risks of autonomous software issues in medicine, transportation, economics, energy, and “defense” need to be met by engaged, expert testers. We need thinking, exploring testers with time and space to do their best in order for our future to be safe.
The Bottom Line
I want people who work on software to demand skilled testing because of its superior risk mitigation. Modern testing is the way forward because it is more effective for exposing bugs. Our natural allies in the Agile Community will be happy to have us step forward and take control of the testing narrative. They are no more interested in us wasting time generating proof of compliance documentation than we are in doing it.
I will speak up for my values, and the values of my community. I will help amplify the ideas of my community, and look for ways we can influence how the world at large thinks about testing. I will push for our community’s treasures to be acknowledged as serious testers who are the very best in the world at what they do.
I want testing to be a skilled, well-paid profession. I want testing to be a profession that bright people can learn from, improve at, and advance in, not just something they pass through on their way to something else – unless they want to. As well-trained critical thinkers, they should be successful wherever they go.
Enough sitting out, being polite, and waiting for someone else to step forward. I will not stand by while consultants sell out our craft as a marketing exercise. We must rescue testing from compliance and documentation, and make it so that skilled testers are expected to choose the appropriate test methods and documentation for the stakeholders and context they find themselves on.
So far, I have helped set up and maintain professionaltestersmanifesto.org, which I hope you’ll consider signing. Karen Johnson has done a great thing for our community in helping articulate our principles.
I have talked about these issues in a podcast interview, though it has not posted yet. I have some writing to do here, and other places. I want to speak clearly and loudly about the benefits of modern testing, and how standards and certifications limit testers and testing.
Longer term, I have other work to do on this. Shortly after CAST, I made a formal proposal to AST for a Standards and Certification SIG. More on that soon.