On Calling People Out

Hello again! It’s been way too long. I’ve been very busy at The AST, which has been taking every spare moment I have (and some I don’t). I’m revving up for WOPR25, and have been hard at work at my day job.

My blogging career has suffered greatly. So, what’s brought me back (to fighting with Twitter embed code)? Events at a Testing Conference this week.



I wasn’t there. What’s been seen on Twitter is supplemented with some accounts from people who were there. I don’t claim to have full knowledge of what occurred, what was said, or the history.

I still know enough. I don’t need to know any more than I do to state unequivocally that this is not OK. I am writing here to state that loudly and clearly.

I am absolutely writing about James Bach, as clearly as I can. I admire his work and his accomplishments. I feel differently about his conduct as a leader in our community, and I am calling him out for that.

This problem is bigger than Tuesday and Maaret. Other women have had similar experiences. It’s time for our community to decide what we will tolerate and what we won’t. Will we look out for each other? Or continue to look the other way?

The world is awakening, or at least being woke. We must do better.

Principles and Framing

There are a couple of things that need unpacking before we continue.

First, I am a member of the AST Board of Directors. I am not writing in this capacity today – I am speaking only for myself, as someone involved in the Context-Driven community. I will still carry these opinions as part of a group that makes decisions about conducting the AST’s business.

Secondly, I wasn’t present. My information is second-hand, except for the majority of it that occurred on Twitter where everyone can see. I think I’ll still be fine, given what I want to talk about.

Third, I know Maaret a little. I greatly respect her, I like her and her husband, and am pleased to share the Introvert’s Nod when we see each other. I have not spoken to her about this. I won’t speak for her, just myself.

Fourth, I know James a little better. I admire and respect his work, his intellect, and his accomplishments. He is truly a giant in our field, and has done a great deal for the practice and profession of testing. I also know he can be an asshole sometimes, and gets plenty of practice.

Despite seeing this over the years, and despite sympathizing with people he’s crushed, I’ve hired him for workshops. I’ve even publicly defended him as worth it on the whole, within the last 60 days. I have made business decisions, and I have some regrets. I write today about what I am thinking today. I want to believe he is a good person, and people that I respect insist that he is.

Fifth, this is not the first time that James has been publicly confrontational towards Maaret. I won’t rehash here, but suffice it to say, James has pressed her in similar ways before. While the behavior and results are consistent, the existence of this history casts Tuesday’s events in a different light.

Sixth, I love debate. Statements that can’t withstand scrutiny or debate aren’t worthy of acceptance. It is good to deconstruct, fact check, turn over, pressure test, and evaluate statements, in proportion to the utility they claim to possess. Sometimes, statements are casual, and not worth deep examination. Sometimes they are offered very seriously, and should be vetted thoroughly. Like most testers, I enjoy this process.

Finally and most importantly, I refuse to engage with this as some referee who needs to “see both sides”. Very few things actually work that way – some fake balance or false equivalence doesn’t bring us closer to truth. It more often props up something that’s just wrong. There is global warming, and humans have significantly contributed to it. You should vaccinate your kids. The toilet paper should go over the top, not down the back. Hillary won the debate. Budweiser sucks. James was wrong to do what he did.

What Happened?

The story as I have it goes something like this: Maaret gives a keynote address at the conference. A few minutes into her keynote, James interrupts her to correct some point of phrasing.


This is at least just rude in any circumstances. When the full context is considered – the inescapable, somewhat subjective (but only somewhat) combination of who said it, to whom, why, where, and how – it looks worse. An older man took it on himself to talk over a woman delivering a keynote address.  This is a woman he has previously felt he needed to publicly correct then, too. Much of the audience is nodding along right now, particularly women who’ve been treated this way.

I have no information about what the particular subject was, how that exchange went, or much else in the way of detail other than at the end of Maaret’s talk, there was the opportunity for questions, and James asked none.

What I do know is that at the start of *his* talk, with Maaret in the room, James started with The Slide. Apparently, Maaret objected in real time, and started responding. Which seems to be what James wanted.

As I’ve heard the story, after 10 or 15 minutes, the other people in the room wanted to hear the advertised talk, and asked for that. I hear that eventually occurred, and the rest of the incident happened on Twitter, where it is easily followed.

There are a number of problems here. Let’s take a look:

Naming Maaret: Did Maaret consent to being part of James’ talk? Not as far as I can tell. Why would you start your talk by calling out a specific person? James seems to feel not only that he is justified in doing so, but he is proud of it.

I disagree. Making a discussion of our craft about specific individuals does not make the world better. It means we ask people to choose sides. We make it personal, in every sense.

James could have made whatever reactionary points from an earlier talk he wanted to make without making it personal. He chose to make it personal. This is the single worst thing he did, from my perspective. Personally, I’d have gone with the talk I prepared. Doing otherwise seems a little cavalier with other people’s time and attention.

Coming out against “nice”: A favorite inside joke with my wife and I is that the world often mistakes politeness for niceness, implying that we are thinking and judging others worse than they know. So let’s call it polite. Grownups are generally polite to each other, and that’s a good thing. Being polite and pleasant to each other makes the world a better place. Not being polite hurts people needlessly. If you disagree with someone, you can still do so respectfully. That’s not what happened here.

By contrasting niceness with authenticity, James strongly implies that people that are nice are inauthentic.

Treating people with respect is the minimum requirement for being a decent person. The United States has a vein of people right now who are frustrated with “political correctness”, which they would describe as inauthentic and performative (if they knew those words). I recommend substituting “treating people with respect” every time you hear that phrase – it is enlightening to think about “treating people with respect is destroying our country” or whatever.

Compassion is a great place to be, though. The world is awfully short on it. I know that every person I meet is struggling with things they can’t control but still suffer from. When I am frustrated with people who don’t get something, are boring me, are wasting energy and time – I am not proud of myself. I then shudder to think of how people judge me. That’s why you should be polite…or, “nice”. Because why make this world any worse, if you can help it?

Also, in case you missed it – since the slide is about Maaret, isn’t James essentially calling her inauthentic?

What about anyone else in that room who wants to be nice – or polite? Isn’t that also saying they are inauthentic?

Suggests Debate is Critical: I pretty much agree with this. Who wouldn’t? We need our bullshit detectors turned up high, particularly in testing. If we see or hear something we disagree with, we need to be able to process that…productively.

What I disagree with is the contention that James gets to unilaterally set the rules of engagement, which seem to be that he can challenge anyone at any time about anything he wants. There was a question and answer period after Maaret’s keynote, where James could have asked whatever challenging question he wanted. I hear he did not avail himself of that opportunity. Outside of that window, he needs her consent to have a debate. It’s only a “debate” if both parties want to participate.

If you write that you disagree with someone’s public statements, then no worries, as long as you are reasonably polite about it. If you physically direct your criticism at an unwilling participant, it’s just harassment. Twitter is in person, for some definition of in person, because of @’s. It’s not as confrontational as speaking directly to someone, but it still makes your criticisms known to their subject, usually in real time, and challenges them to defend themselves.

If a person doesn’t want to debate with you any more, that could be because they feel overwhelmed by your superior logic. Or, it could be because they feel they’ve explained their point of view already, and you’re not getting it. Or, they could feel overwhelmed by being criticized by someone they admire. Or, they may have something else they need to do right then and don’t have time to discuss it.

Or, they’ve seen this movie before and know how it ends. Not all “debate” or criticism is honorable in intent. In fact, much of it isn’t, and says much more about the needs of the person doing the criticizing than anything else. If you’ve been close to a person who is emotionally abusive, you’ve experienced criticism for the sake of criticism, designed to hurt, crafted to cut deep, deployed to destroy confidence. It is very difficult to hear criticism and process it as face value, even if you haven’t been abused (many people have been abused).

Even if we grant the positive intention and somehow make it so that everyone can easily assume good intentions, we’re not going to get there with just arguing. We’ll just figure out who is good at arguing in public. If you have to resort to bullying someone into debate as combat sport, then it isn’t about ideas anymore.

Others defending James cited academia. Of course, in academia, people are allowed to assemble their arguments calmly, taking their time to reflect deeply so that they can best represent themselves. A process of real-time showdown does not lend itself to deep reflection.

And sometimes, people are just tired of arguing with YOU.

Deciding that you are entitled to criticize someone in person, whenever you want, however you want, is bullshit. That is demanding that people absorb you being an asshole whenever you think you’re entitled – which is exactly the right word here.

And again, by “defining (his) position vs. Maaret’s”, he is implying she is the opposite. What, she is against examining statements and discussing them? Or is it that she doesn’t grant James his terms of debate – verbal combat any time, any where he feels like it?

Double Negative Something Something: Again, the formulation here seems to imply that since James is contrasting himself with Maaret, she must believe excellence is achievable without focusing attention and energy. Not James’ best work.

In Summary

The way James behaved Tuesday was poor enough that I feel some responsibility to my current and future colleagues in testing to publicly say something about it. Other people tried to tell James he had gone too far. Instead, we got belligerence and admonitions that we simply didn’t understand his points. Let’s bring this home with a “public debate”:

Trying to deal with the problem, though I have a different opinion about what exactly the problem is. I am speaking up here, and saying that this was too much and too far.


Hopefully, it is clear that it was some specific things, and some of it was context. It got personal, and it seems to have got personal enough that the arguments got flimsy. Just like you assume we’ll trust your good intentions, assume we really mean what we say.

Any uninvited, in-person debate, yes. That’s exactly right. The other words you might use here are harassment or abuse.

While some of my criticism here is based on the context of an older man speaking at a younger woman, rest assured that James has behaved in a similar fashion to people of different sexes, ages, and ethnicities. When I note the sexist subtext, it’s because when we tolerate this level of disrespect in public discourse, we signal that our community is not a safe place. I’m still not convinced James would be equally likely to interrupt and confront a man.

We understood. You were clear, in what you said, and in your choices around who/what/where/how. The why? We can give you credit for just wanting truth out. But when someone is emotional, it’s hard to take their word that there aren’t other motivations, whether they know them or not.

James, you’re being called out. Your behavior is not acceptable, and it’s time for people to tell you to stop. 

You’ve been a pillar of our community, with roots that go back farther than most of us have been in testing. Our work and our craft are immensely better because of the work you’ve done. But imagine how much farther we’d be without the handicap of our brightest light also “leading” us into infighting and extremism.

You greatly influence the intellectual climate of the community, and therefore have a lot to do with its size and current condition. You’ve built a history of treating people poorly. Much of what has happened is perceived as related to the age and gender of the people you have “debated” on your own terms, regardless of what they were comfortable with. I think there is a gender issue here, too. In any case, one-sided debates are just abuse.

People fear you, and not just because of imposter syndrome. Whether you realize it or not, some good people have been lost to our community because of your behavior. Some were chased away directly by you, and others quietly left after seeing what happens if anyone gets too far ahead of themselves around you.

Sure, you’ve embarrassed some people who “deserved it”. But you’ve also shut down plenty who have not. I won’t publicly list the people who won’t engage with you any more, but I think you can think of a few pretty smart people – mostly women – without trying too hard.

Worse, there are acolytes who model your behavior, and feel entitled to treat people the same way you do. When people feel they have the right to challenge anything anyone ever says in public at any time, they’re getting that from you. I am far more ready to point to the gender implications in this source of abuse.

And it is abuse – it should be clear that no one owes you a debate duel when and wherever you demand it. Our community creates appropriate environments to encourage the easy flow of debate/dialog/talking about stuff. Outside of these, you are back to the expectations of society at large – basic human decency and politeness. Or if that seems inauthentic: not being an asshole, particularly to young women. 

Old white guys all over the place are finding out that behavior people used to tolerate isn’t tolerated any more. It was always wrong, but it’s not being allowed to pass by any more. Time to evolve.

WOPR24 Experience Report

This blog post relays the findings of a peer workshop on performance and reliability testing and monitoring. I will not provide the usual TL; DR summary this time – if you want this gold, you will have to sift it yourself.

On 22-24 Oct, I attended and facilitated WOPR24, a LAWST-inspired workshop for performance engineers. I have been involved in organizing WOPRs for about 6 years now, and have attended 18(!) WOPRs over the last 11 years.

The attendees of WOPR24 were Ajay Davuluri, Oliver Erlewein, James Davis, Andy Hohenner, Yury Makedonov, Eric Proegler, Mais Tawfik Ashkar, Andreas Grabner, Doug Hoffman, John Meza, Michael Pearl, Ben Simo, and Alon Girmonsky. We were hosted by BlazeMeter in their Mountain View, California office, which was a fantastic venue. Adi BenNun of BlazeMeter took great care of us, making us very comfortable, feeding us very well, and making sure we had everything we could want.


I could talk for a long while about WOPR, WOPR’s history, our mission of advancing the practice and community building, how WOPRs are put together, and how Ross Collard changed the trajectory of my career and life by inviting me to WOPR3. At some future date, I will. But for now, so that we can talk about the great content of WOPR24, I will just drop a link about WOPR and let you surf that site.

The Workshop Theme

Each WOPR has a Theme, to help focus the experience reports and lay out what we want to explore. From http://www.performance-workshop.org/wopr24/:

Production is where performance matters most, as it directly impacts our end users and ultimately decides whether our software will be successful or not. Efforts to create test conditions and environments exactly like Production will always fall short; nothing compares to production!

Modern Application Performance Management (APM) solutions are capturing every transaction, all the time. Detailed monitoring has become a standard operations practice – but is it making an impact in the product development cycle? How can we find actionable information with these tools, and communicate our findings to development and testing? How might they improve our testing?

Content – Experience Reports

The primary format of WOPR is Experience Reports (ERs), which are narratives supported by charts, graphs, results, and other relevant data. Each ER is followed by facilitated discussion triggered by the ER. 7 attendees presented ERs over 3 days. Systems we discussed included the following:

  1. An online advertising auction system to algorithmically price and purchase ads for specific user profiles in < 40ms. Presenter used Splunk to model and characterize log events, web hits, application instrumentation, business metrics, and other content.
  2. An automotive parts sourcing SaaS application. Presenter discussed supplementing regular human-conducted load tests with CI automated tests. Lively discussion about thresholds and test environment control/resetting ensued.
  3. A mapping application company’s efforts to test with virtualizing high end video cards (https://www.cdw.com/shop/products/NVIDIA-GRID-K2-graphics-card-2-GPUs-GRID-K2-8-GB/3126398.aspx).
  4. An application that collects, rolls up to big data sets, and displays back to the system’s owner detailed operational metrics from a very large number of embedded systems, distributed around the world.
  5. A order-taking website for a large retailer. Discussion was about launch of a site, and the difficulty of enrolling reluctant development stakeholders in performance testing projects.
  6. A non-profit transportation industry service that publishes rate tables multiple times per day to and from all the vendors in that sector. Discussion of concurrency bugs and how they were reproduced.
  7. A large financial services SaaS provider shared some of the issues around testing mobile in-app video and chat. Generating traffic and evaluating results were some of the technical issues we discussed. 

Content – Exercises

Between ERs, we conducted several guided discussions to explore specific areas of interest. Findings from those are relayed here:

What should we alert on?

In this exercise, we just started calling out thresholds and notes. We definitely didn’t finish.

Monitor What Threshold Notes
CPU > 75% Web/App, > 50-60% DB User + System CPU: For Vus with 4 or less and hyperthreaded, Warning 50 alarm 75, critical 95. With > 4/metal with HT disable warning 75 critical 99 Context-dependent Physical and virtual CPUs Per core and overall 1 minute monitor – number of sequential observations to trigger (two at which level? 3 alarm? 1 critical)? One minute? 5 Minute?
Order Rate Dynamic Baseline (Time/Day/Etc) Oliver: Tues is Max, Friday is Min revenue, order rates, conversion rate, bounce rate Business spend: Advertising money out
Failure Rate http error rate against yesterday Need to have a baseline. Beware of bots, synthetic measures, etc
Queue Length: Threads Web/App, detectable at App? Should have threads available, 2threads/core, beware of thread contention/concurrency events Alert on contention/concurrency?
Connection Pool Utilization JMX: DB Connections, Outgoing Requests DB Connections, external web service calls, depends on app server. Is app waiting for a connection?
TCP Inbound Queue Anything increasing in subsequent samples
Message Q Messages creates/sec, size (no growth), message age, expiring messages present?
CPU Q Load Average/CPU Queue Length > 2 per core Load Average: 4x number of cores
MIPS/Second 3500/second, threshold 2900
Response Time Several times higher than SLA TCP/Response Time can indicate app server overload distribution and median/thresholds vs baseline
Errors (Particular) Class/Types, Rates, and severity Error clusters
CPU Ready Time Any significant percentage
GC Time Percentage of time – what’s impact? Over 10% Full collection? Partial? Age of objects leaving generation
Averaging http status Compare to baseline average literal numbers By request (type), or by page. Watch for dramatic changes Watch for bots > 50% Big difference, prod vs. test. Remove synthetics in test. Load balancer, keep-alive, etc
Throughput Cloud $ rate, watch for bottleneck Network throughput + context switch over Cpu = ratio for sanity check, first bottleneck
GPU utilization in virtualization 75-85% utilization = efficient virtualization
Redirects Count/rate. Redirects on keep-alives Endless loops? Redirects per user, check for max
SAN I/O count
I/O latency Write latency Queue lengths let me see problem faster
Thread context switches
Wait time (DB) Latches, locks, just look for increases Also wait states
Log Volume (and by type)
Cache hit ratio Page life expectancy
DB Connections
Virtual active mem vs real memory Check for Disk swapping
Physical memory free 66% Managed memory Leave small amount free
Network I/O Based on network link rating
Network errors/packet discards
objects generated
Induced GCs
Live Sesions
Disk utilization – space and I/Os
Average SQL Statements/Request Data driven pattern detection problem n+1 query results
Disk space rate consumption
Starting/stopping monitoring agents
Connection Pool: Available
Monitoring Boxes
# restarts
Rate of change in connection pool count
recycling of worker process/app pool
Log messages by severity
thread status – deadlocks?
Business transaction rates – whatever those are
Cloud node thrashing Spin up spin down
Functional audit log/application log Specific events
Transaction span lifecycles
Scale of code change on updates This jar file is x times previous verision
Page size (html) Delta vs previous
User/system ratio
Web Server Request Queue: IIS
Batches/sec 200
%c 1/2/3 VMs in power-save mode? Detected measuring against baseline
Response time total for all requests (SUM) Saw things that were not obvious Correlate against load

What Would an Ideal Dashboard Look Like?

For this activity, we broke up into groups to talk about different contexts. There are some metacomments in each section, reflecting when they came out.

SaaS Company Dashboard


This dashboard has horizontal bars to show different metrics for different consumers. The audience is the whole company. The Checkbox/Xs on the left are to provide a two state indicator to easily, rapidly, and broadly detect when something is wrong, by functional area. Here is what each had:

  • CEO – For each metric, the current time period against a previous time period. Demos, Signups, subscriptions, Retain/Drop Rates, and some support metric such as call count
  • CFO – Cost per lead, cost per user, and a sparkline for Revenue
  • CMO – Social media engagement, Mailing opens/click rates, ad conversion
  • CIO – Availability, error rate, Response time, throughput
  • CTO – Deployments, key resource metrics

Each one of these bars is designed for drill-down. Remember, this is the top level.


For E-Commerce, they decided not to draw a single dashboard. They defined five different functional areas they care about, and drew a Venn diagram to show that there are overlaps of interesting information, but very little that everyone wanted, needed, or would benefit from knowing.

Consider this continuum, generic to specific, from “easy for everyone to understand” to “doesn’t have to be understood except by the individual who uses the dashboard”. You could also think of these as spokes from the center outwards.

Company -> Department -> Team -> Individual

As the questions were posed by them:

  • Who is looking at the dashboard?
  • What is their role?
  • What do they want to see?
  • What actions would they take if they did see something?

The four areas they defined, and the notes on each:

  1. Performance Testing
  2. Business – Adoption Rate, Active/passive campaigns, usage by feature, dollars per aggregate
  3. Development – Separate dashboards for each scrum team
  4. OPS – Active support ticket counts

BigCorp’s Dashboard


Here is a dashboard imagined for a very large company. It is intentionally light on detail to avoid having visitors/contractors get information they shouldn’t have. Some of the resulting discussion was about dysfunctions, but these are real safety issues in some organizations.

Consider the following characteristics in deciding what to have in the dashboard:

  • Tooling – what can you show
  • Time – both for measuring effort, and for displaying data in context
  • Security – Don’t show something you shouldn’t – review content with Security officer?
  • Liability – Knowing specific things – or being responsible for exposing specific things – might put a person in a position they don’t want to be
  • Cost – Screens, software, etc.
  • Political dimensions of having potentially out-of-context data be embarrassing to specific executives or individuals. Is this public shaming? Are people stuck with metrics that will never go green? Is that deliberate? What if we move the goalposts so that someone can see green – are they still worth tracking? Should we turn off our sales dashboard when we are clearly going to miss goals, so that we don’t depress morale?
  • NO STOCK PRICES. Or other metrics that are not actually connected to current, actionable data.

Some specifics about this dashboard:

The “Christmas Tree” or dragstrip pattern is for a number of areas to indicate green/yellow/red. This is designed only to communicate whether something interesting is happening.

Other visualization methods – use vertical meters, or speedometers.

The other primary visual is a graph comparing a current time period against a previous, such as this day’s revenue against last year’s/quarter’s/week’s. The interesting suggestion here was to change the background color as an indicator.

Final Metacomments

Focus in the information needs: What is actionable? What is relevant? What is helpful?

Go to people consuming data – what are they needing? Answer that question, instead of starting with what is easy to provide.

Revenue metrics are key


What metrics do we talk about? What metrics do we present? Those metrics already have a life, so we should try to reuse them.

Test for information completeness – establish that the dashboards are truthful and accurate. Make sure the team knows what the data means and where it comes from.

Dashboards provide transparency into current states – but require interpretation. They are starting place for discussions across teams.

Leave space for temporary/rotating items

Bonus Screenshot of an Ops Dashboard Someone Built

Unclear what the news presenter sees on the dashboard that concerns her ;-p


What Would We Want to Monitor for Mobile Users?

In this exercise, we listed the attributes that would be interesting to capture for mobile monitoring.

On the device:

  1. Network conditions: latency, bandwidth, jitter, packet loss
  2. Geographic location
  3. Screen size/device
  4. OS version
  5. Battery State
  6. Memory – free/available
  7. Number (list?) of apps running
  8. Powersave mode
  9. Chaperoned?
  10. Carrier
  11. Is the screen cracked or broken?
  12. Signal strength (wireless/cellular)
  13. Accessories
  14. Recent movement/direction/accelerometer data
  15. Contention? Drops/retransmits
  16. Carrier Plan status?
  17. Time: last restart/power cycle
  18. Rooted/Jailbroken?
  19. Date in Service
  20. Accessibility Options
  21. Geographically similar to other devices? People travelling together in a train, for example

In the app:

  1. Screen load RT
  2. Memory footprint
  3. Data transfer
  4. CPU cycles
  5. Gestures captured
  6. Round trips/waterfall
  7. Ad activity
  8. Freemium status?
  9. Known customer
  10. Time of day
  11. Demographics
  12. Date of Install
  13. Version
  14. Date last used
  15. Other app requests
  16. Phone connection right now?

My Stump Speech – Asking for your Vote for AST Board

Hello! My name is Eric Proegler, and I am Context-Driven.

I am writing today about my candidacy for the Association for Software Testing Board of Directors. I’d like to tell you a little more about me, and why I’m running.

I live in Mountain View, California and work for SOASTA. I have worked in testing for 16 years, and been engaged with the CDT community for 11.

Of the work I’ve done in the testing community, I am most proud of being a Speak Easy mentor. I’m also an organizer of a long-running peer workshop (WOPR, the Workshop on Performance and Reliability), and a member of the Community Advisory Board for STPCon. I have been a member of AST for several years, first attending CAST in 2010.

After last year’s CAST, I set out to form the AST Committee on Standards and Professional Practices to help AST engage with issues affecting our profession as a professional trade organization. This work, and my experience working with AST to get this Committee started led to me accepting a nomination to run for the AST Board.

I am asking for your vote because there are things I hope to accomplish for our organization, our community, and our profession as an AST board member. I believe I have a track record of getting things done, by digging in and doing them myself. I think that’s what we need more of on the AST board. With urgency, focus, humility, and willingness to iterate, we can accomplish a lot.

The ideas of our community’s founders and leaders changed testing forever. I want to publicize our school’s approach to testing as viable, respectable, and highly effective to the testing AND software development worlds, giving our membership support and resources for implementing modern testing principles. I also want to help solidify a launchpad for the next generation of our community’s leaders to lift off from. We have a bright, passionate, and diverse crop in our community that will shine very brightly when they get their turn.

I am sorry I won’t see so many of my friends and colleagues in Grand Rapids next month. During CAST, I will be speaking at an Agile Development conference (Agile2015). I plan to talk with that community about testing, and how “automating everything” is sure to miss many bugs. From where I sit in the heart of Silicon Valley, this desire to turn testing into checking is an even greater risk to our profession’s future than factory certifications and testing standards – and is not an empty house of cards like these economic tactics.

I will push AST forward as an organization by challenging the organization to improve as an advocate for all testers, whether they identify as a member of our community or not. AST should continue to proudly be a Context-Driven organization, but the business and profession of testing has been defined by large commercial interests for too long. We must present a credible, experience-based alternative to the obsolete, ineffective command and control processes recommended by those seeking to profit from marketing McTesting process and “expertise”. We will win with our superior ideas, demonstrating that in a world of faster development iterations with smaller teams, it’s our community that is moving the practice forward.

I hope to help AST do more for its membership. My first set of tasks are to help members to find and collect tools and links for self-learning, aid them with job posting and searches, help them connect with peers for advice, and provide them with practice references when they set out to improve testing in their organizations. A key task for helping this happen is to revitalize the AST website so that others can easily contribute their research, experiences, and voices to a common body of knowledge. I will ask the board to allow me to revamp the AST website towards community engagement, and I will personally do much of the work to make it happen.

I want to help AST expand its footprint internationally. While much of our membership is American, there are thriving communities in Europe, Australia, New Zealand, and even Asia. AST can support and sponsor decentralized, loosely affiliated CDT conferences in places where there is demand and people on the ground.

Lastly, I will continue to work with the STP board to welcome CDT content and speakers into those conferences. If you take a look at our fall schedule, you can see that is going quite well.

To learn more about me, you could visit my blog here at contextdrivenperformancetesting.com, follow me @ericproegler, or chat with me sometime. I’d love to hear more ideas about how to increase AST’s accountability to the community, so that it can better earn the community’s attention and participation.

Analysis of ISO 29119-2: Test Processes

This is the second post in a series following and analyzing the ISO 29119 standard. Most of the essential context references were covered in the first post, Analysis of ISO 29119-1. One thing that has changed since that first post is the AST Committee I proposed has been formalized. Watch for more from us soon!

So, what can we expect in Part 2 of the Standard?

ISO/IEC/IEEE 29119 supports dynamic testing, functional and non-functional testing, manual and automated testing, and scripted and unscripted testing.The processes defined in this series of international standards can be used in conjunction with any software development lifecycle model. Each process is defined…and covers the purpose, outcomes, activities, tasks and information items of each test process.

Can’t wait!

Please remember I am criticizing the standard (and the idea of a testing standard), not the people who worked on it. I believe that smart, experienced people attempted to lay out their view(s) of testing, hoping to help people test effectively. I think that in the right discussion about the many contexts software might be tested in, they might concede that no prescriptive standard can be relevant and useful in every context. In fact, some of them are already doing that. Whatever the shortcomings of 29119 (and there are plenty) it could never possibly satisfy its mission, even if it was a better standard than it actually is.


My best-practice, conform-ational approach is to summarize my primary conclusions at the top of my blog posts, sparing tens of readers the post’s full brilliance. Here are my “above the fold” takeaways from analyzing ISO 29119-2:

  • 29119 literally puts process (Part 2) before technique (promised in Part 4, still not published)
  • 29119 claims to be applicable to testing in *all* software development lifecycle models, despite heavy documentation and compliance burdens
  • 29119-2 has Conformance on page 1. To claim Conformance, there are 138 “Shalls” to conform to in this document. To claim “Tailored Conformance” without meeting every “Shall”, “justification shall be provided…whenever a process defined in…29119 is not followed”
  • Part 2’s vocabulary section has conflicts, revisions, and pointers to new terms relative to Part 1. This is not a “gotcha” – but is worth remembering when someone claims that with a test standard “At least there is a common vocabulary for testing”.
  • Conformance is driven by fear. Fear is the mind-killer.
  • Some of the “shalls” are highly specific. Some are vague and hard to understand. Some, through reference, contain multitudes. Some are nonsense.
  • The standard is not detailed enough to be very useful to someone who doesn’t already understand a fair amount about testing, yet an experienced tester would waste a lot of time and effort attempting to comply with it.


29119-2 goes to Conformance very early – Page 1. Either Full or Tailored conformance can be claimed for the standard.

  • “Full conformance is achieved by demonstrating that all of the requirements (i.e. shall statements) of the full set of processes defined in this part of ISO/IEC/IEEE 29119 have been satisfied.”
  • “Tailored conformance is achieved by demonstrating that all of the requirements (i.e. shall statements) for the recorded subset of processes have been satisfied. Where tailoring occurs, justification shall be provided (either directly or by reference), whenever a process defined in…ISO 29119 is not followed. All tailoring decisions shall be recorded with their rationale, including the consideration of any applicable risks.”

I can find no guidance on what “the recorded subset of processes” means. Not just what the various nesting levels of “process” are in the standard, either. Are these the processes that reference record-keeping and documentation? I bet I can find a consultant to help not-interpret that…

There is a “Reference” example given for exclusion from the requirement for providing direct justification:

“Where organizations follow information item management processes in standards such as ISO 15489… ISO 9001…or use similar internal organizational processes, they can decide to use those processes in place of the information item management tasks defined in this part of ISO/IEC/IEEE 29119.”

So, no exclusion from the requirement to document and describe the justifications – just an exclusion from the requirement to provide a separate document including these justifications for ISO 29119, as long as they are in another document somewhere else.

After 10 months, the only defense raised thus far by the authors of the standard to the questions about difficult compliance is to claim it is more flexible than what is actually said in the standard:


… and that’s the last message in the conversation. I suppose we could take the word of a standard author over the standard itself, which says with little ambiguity under Intended Usage: “The organization shall assert whether it is claiming full or tailored conformance to this part of ISO/IEC/IEEE 29119”.

Clashing Definitions

Section 2 spells out definitions for some terms. There is overlap with Section 1 – and some disagreement with what was found there.

For example, in Section 1, Feature Set meant  “collection of items which contain the test conditions of the test item to be tested which can be collected from risks, requirements, functions, models, etc.” Section 2: “logical subset of the test item(s) that could be treated independently of other feature sets in the subsequent test design activities”. Additional differences, revisions, and pointers to new terms are found. This is not a “gotcha” – but is worth remembering when someone claims that with a test standard: “At least there is a common vocabulary for testing”, ISO 29119 already has divergence in critical definitions between its first two parts.

Do-not-think-it-meansAt least these terms are interesting to think about. It’s far less interesting to trace the relationships between test activity, test item, test condition, rest requirement, test phase, test plan, test policy, test planning process, test procedure, test procedure specification, test condition, test process, test sub-process, test script, test set, test models, test technique, test specification, and test type. Yes, these are all separate things, but time spent debating their boundaries is time not spent “testing”.

Exploratory testing is again defined as “spontaneously designs and executes”, not “simultaneously” as we define it.

Process and Hierarchy


This diagram shows a hierarchy of test processes. It doesn’t actually cover all the processes referenced in the standard, despite the caption’s claim. The diagram does demonstrate the standard’s insistence on separating control processes from execution processes.

It is intended to illustrate that the vertical layers define each other downwards. First is the organizational process that defines process for organizational test policies, which dictate policy, strategy, process, procedure, and “other assets”. Test Management Processes are defined at the project level, Dynamic Test Processes are said to control a phase or particular type of testing.

This seems tailored for adoption by the mid-level executive who wants to put their stamp on an organization’s entire testing practice. Over and over again, the standard lays out separate process nodes for each possible step of testing. This exhaustive documentation of the steps involved in one view of testing is way too much for an experienced tester, who would rather provide useful information to stakeholders. It’s still not enough to arm someone with no testing experience to plan and supervise good testing. So who is it for?

When Fear Drives Testing


Software testing is frequently perceived as a high-risk, low-reward activity by people who aren’t testers. It’s thought of as a cost center (“there is no ROI in testing”) and if anything goes wrong, someone’s in trouble. Over and over again, testing is blamed for poor quality, despite the fact that most people who work in software engineering know “you can’t test quality into the product”. Testing is often thought of as less intellectually rigorous than other parts of software engineering, frequently is not a prestigious area to work in, sometimes is led by people without real training, experience, and/or skill in testing, and is often a convenient scapegoat for quality issues – particularly by people who should know better.

Many people that work in testing (rightfully) fear the buck stopping on their desk after a quality failure, and for good reason. If you are likely to have blame imposed for a bug escape, the most rational response by a skilled person might be to interrogate the context and demand the tools and latitude to gather the most comprehensive and useful set of information about the system under test.

If you are controlled by fear, you might shy away from the responsibility, and look for some cover under best practices. After all, if you faithfully observed and obeyed someone else’s plan, you can’t be blamed if the plan fails, right? It wasn’t you, it was the plan!

If you don’t know what you are doing, you might be even more likely to seek the comfort of an externally defined standard that removes your responsibility to decide what to do. If you don’t trust your team (and yourself), you hand off control to someone or something else. Like a prescriptive standard, full of “shall statements” to replace “you thinking”.

The standard is still not detailed enough to be very useful to someone who doesn’t already understand a fair amount about testing, yet an experienced tester could waste a lot of time and effort trying to comply with it. Any discussion of actual techniques seems to be waiting for 29119-4 – at one point promised for late 2014, currently late in the approval process.

You Shall…

There are 138 instances of “shall” in this document. Some of them are highly specific. Some, by reference, contain multitudes. Some are simply nonsense. Some of them are too vague to be useful, though that may make them more applicable in multiple contexts. Some real wisdom can be found in here.

I spent some time pulling apart the various processes, sub-processes, dependencies, and circular references. Rather than try to further sketch out the overall shape of process (and documentation) requirements, I present my 10 most entertaining/concerning/Kafkaesque “Shall Statements” in ISO 29119-2:

  1. The person responsible for organizational test specifications shall implement the following activities and tasks in accordance with applicable organization policies and procedures with respect to the Organizational Test Process.
  2. The organizational test specification requirements shall be used to create the organizational test specification.
  3. Appropriate actions shall be taken to encourage alignment of stakeholders to the organizational test specification.
  4. The traceability between the test basis, feature sets, test conditions, test coverage items, test cases and test sets shall be recorded.
  5. The testing of the feature sets shall be prioritized using the risk exposure levels documented in the Identify and Analyze Risks activity (TP3).
  6. Any risks that have been previously identified shall be reviewed to identify those that relate to and/or can be treated by software testing.
  7. Each required test activity in the Test Strategy shall be scheduled based on the estimates, dependencies and staff availability.
  8. Those actions necessary to implement control directives received from higher level management processes shall be performed.
  9. Readiness for commencing any assigned test activity shall be established before commencing that activity, if not already done.
  10. The test coverage items to be exercised by the testing shall be derived by applying test design techniques to the test conditions to achieve the test completion coverage criteria specified in the Test Plan…
    NOTE 2 Where a test completion criterion for the test item is specified as less than 100% of a test coverage measure, a subset of the test coverage items required to achieve 100 % coverage needs to be selected to be exercised by the testing.

It’s not all baffling. Here’s a richly meaningful shall statement that demonstrates something about the depth necessary to understand context:

A Test Strategy (comprising choices including test phases, test types, features to be tested, test design techniques, test completion criteria, and suspension and resumption criteria) shall be designed that considers test basis, risks, and organizational, project and product constraints…

NOTE 3 This takes into consideration the level of risk exposure to prioritise the test activities, the initial test estimates, the resources needed to perform actions (e.g. skills, tool support and environment needs), and organizational, project and product constraints, such as:
a) regulatory standards; b) the requirements of the Organizational Test Policy, Organizational Test Strategy and the Project Test Plan (if designing a test strategy for a lower level of testing); c) contractual requirements; d) project time and cost constraints; e) availability of appropriately-skilled testers; f) availability of tools and environments; g) technical, system or product limitations.


The last third of 29119-2 is an Annex mapping clauses of other standards (ISO 12207, ISO 15288, ISO 17025, ISO 25051, BS 7925, and IEEE 1008) to 29119-2. Rather than critique these other standards, I will simply question the value and purpose of this exercise. Is it to justify the standard, or to prove that it equals or even supersedes the others?


We still have parts 3 (and 4? soon?) of 29119 to go. Having processes defined before considering what we want to accomplish will guarantee we end at our desired results (whatever that might be), right?

Debates, Rent-Seeking, Personal Attacks, and Apologies

Yesterday, XBOSoft hosted a webinar entitled “The ISO 29119 Software Testing Standard – Friend or Foe?”. Philip Lew hosted a debate, somewhat formally constructed. The panelists were Jon Duncan Hagar, Rex Black, Jean Ann Harrison, and Griffin Jones.

As a close follower of this discussion, I’d say that not a lot of new ground was tread, but the significant issues were discussed, and it was the right size twitter event to participate in. Claire Moss provided detailed play-by-play. Iain McCowatt had some pointed comments. Lalitkumar Bhamare, Perze Ababa, Tim WesternKate Falanaga, and @dsynadinos also contributed. The hashtag was #ISO29119, but some tweets are under #ISO29119Debate. I don’t intend to describe the whole event, but to talk about something specific.

One of the specific things that ignited discussion was the term “rent-seeking“. When Griffin mentioned the term during the debate, it definitely got a strong reaction. Even Phil as the moderator objected. Here is the resulting discussion:

Rex’s response today led to me writing this blog post, because I need more space than Twitter to properly respond.

About Rent-Seeking

I first heard this term in James Christie’s talk at CAST 2015. James borrowed this term from his degree in economics, using it in his talk about standards, certification, and regulation. His hypothesis was that the term “economic rent” was useful in discussing some our community’s concerns, by providing language for discussing how some of the actors/vendors in the testing market can exert control on how business is done in that market. I recommend watching his talk, because he certainly says it better.

The term “rent-seeking” resonates for me, in both professional and political contexts. Like many really delicious words, it is rich in meaning. It describes a complex process that takes place in many industries, in many countries, in many contexts. There are degrees; pointing to a financial benefit from an activity doesn’t mean that it is inherently unethical. It does mean that the interests of people who profit from these arrangements are judged differently.

Some vendors have used standards for financial gain. This should not really be in question.

The definition on the mobile Wikipedia site – the one Rex referenced – is even harsher in making an ethical and value judgment. It says “rent-seeking is expending resources on political activity to increase one’s share of existing wealth without creating wealth.”

All this makes Rent-Seeking a loaded term to use in a discussion. That doesn’t mean it is inaccurate or not useful. Still, I should find a less pejorative term here that describes how a market changes with requirements for entry are added.

The Personal and the Political

I respect and admire Rex Black, who has had a long and successful career in software testing and training. He has worked hard and achieved a lot. He’s smart and funny, and I’ve enjoyed our brief in-person interactions. I think that Rex can point to hundreds (thousands?) of people he’s trained in software testing, and say that he’s has helped many people’s careers. Rex has put a great deal of time and effort into ISTQB, ASTQB, and other testing organizations, and can say that he has given a lot back to testing. He’s spoken and consulted all over the place. It’s more than a little presumptuous for me to call him a colleague, given his accomplishments.

That being said, Rex and I have fundamental disagreements on some issues in software testing. He’s taken a lot of fire from all over the CDT community. I once joked about him trolling CDT, and he suggested the opposite is true. I think that is exactly correct: as the personification of Big Quality to many people in our community, he’s taken some pretty rough criticism, and some of it has been personal.

I am friends with and respect some of the people that Rex has blocked on Twitter, but I can also understand why he’s blocked them. Many professional disagreements end up as personal ones; you don’t have to look too far in our community to see formerly close friends/mentors/partners/collaborators who won’t speak with each other today. In some of these cases, I think the person choosing to disengage has made the right choice. I don’t think any of these people are terrible, just that we’re still basically monkeys that wear clothes and sometimes throw shit at each other.

I won’t adjudicate Twitter behavior – but no one comes out ahead when someone feels attacked. We have professional issues about the current and future states of software testing to debate, and it is unfortunate that we find ourselves here: many people disagreeing with Rex, and him expected to debate dozens of people by himself. I don’t know who to nominate to help him, but I empathize that it must be exhausting. I feel that his willingness to articulate another point of view and engage in debate makes him a resource for our community that we should value.

I think it’s problematic to tell people what they ought to be or are allowed to be offended by. It’s hard not to internalize criticism of work that’s important to you, so that criticism should be done with care. You should be very, very careful about assigning motives to someone, and be even more careful in questioning their ethics. You can make a contribution by being slow to take offense, and patient and forgiving with those who make mistakes, but you also need to take care of yourself.

My Role, and My Apology

I’ve made some pointed comments in debating Rex, trying to walk the line of criticizing specific things without being personal. He has objected to some of the things I’ve said that have been too harsh. When I have agreed, I have apologized. He has accepted these apologies, and has not blocked me. I greatly respect and appreciate that.

Yesterday, I made the implication that Rex offering ISTQB certification classes after helping define the ISTQB standards was a form of Rent-Seeking. I put a sharp edge on it, and threw it out in a tweet with about 5 seconds of thought. I was wrong, and I am sorry I said it.

The reason I was wrong was because the point could have been made without implying that Rex was trying to rig a market, or that he intends to shut down competition, was seeking to purchase favorable regulation like a giant agribusiness, or was otherwise behaving unethically. The “unethical” charge that should not be made casually – and in this case, it’s just not supportable. I should have been more respectful, and been more precise in my criticism.

What I Should Have Said

I believe that all ISO standards are proposed as a standard for governance, whether that is corporate or legally (If it was in person, I might have added a sly comment referencing conservative political philosophy being opposed to needless regulation). I think that these governance standards add overhead and barrier to entry for testing companies, particularly small ones, because they require study and compliance overhead. When a standard is proposed, its very nature is that it wants to become as widely observed as possible.

The claims that ISO29119 makes about correctness and applicability are very broad, and from what I’ve read so far, it does not include qualifications on when, where, or how it should NOT be applied. When the lead author was asked about this, he said you can forego the standard “If you work in your garage (and are) not working with any clients.” I disagree with this, to say the least.

Certifications make similar claims in applicability, and have similar effects in creating barrier for entry for individuals to the profession of testing. When companies require a certification for hiring – that is the goal for every certification, to become the standard measure of competence, right? – they are making it harder for testers to get and change jobs without investing time and study into achieving these certifications. I think that by participating in writing these certifications, and then training people for the certification tests, there is a financial benefit realized while simultaneously imposing a cost for entry, reducing the efficiency of that market.

That is the extent of what I meant to say, and it’s damn close to the definition of Rent-Seeking. It’s still wrong to attack someone’s motives and ethics, and it’s toxic when you are trying to debate important issues.

Again, I am sorry.

Analysis of ISO 29119-1

The first sentence of the ISO 29119 Introduction states:

The purpose of the ISO/IEC/IEEE 29119 series of software testing standards is to define an internationally agreed set of standards for software testing that can be used by any organization when performing any form of software testing.

Don’t think I need to emphasize the “any”s, but go ahead and bold those in your mind if it helps.


Like my other blog posts, this one far exceeds widely-held standards for blog post length, using the best practice metric of word count. So, to summarize:

  • I have some experience with creating ISO Standards in another field. My experience was that it is a difficult and highly political process.
  • Of course our community objects to standards, because they cause crappy testing, here defined as not providing a meaningful, accurate account of quality risks of a project. ISO 29119-1 says it “is informative and no conformance with it is required”.
  • To ISO 29119-1, “Testing” is a “set of activities conducted to facilitate discovery and/or evaluation of properties of one or more test items”, noted to include “planning, preparation, execution, reporting, and management activities, insofar as they are directed towards testing.” It seems that the term “test item” (a system, a software item, a requirements document, a design specification, a user guide) is deliberately constructed to suggest these items can all be tested.
  • This particular document (ISO 29119-1) has a lot of definitions and Naming of Documents. The SDLC model referenced for context is Waterfall-y (Not Iterative).
  • Exploratory Testing, or as it has recently come to be called, “Testing”, is found buried under “Experience-Based Testing”. Further discussion is promised in 29119-2.
  • “Risk-based” is referenced multiple times, with strong language around claims of wide adoption. Given that choosing what to test should always(?) involve evaluating risk, this is hard to argue with: “…truisms…to those new to software testing they are a useful starting point“, indeed.
  • Overall? Several years behind the state of the art, overly focused on formality and control, and barely concerned with technique.

Context and Perspective

I spent a little more than two years in a Working Group like the one that produced ISO 29119. I worked on trusted archiving of digital data, specifically document and records management. Most of the stuff I wrote concerned storage requirements for permanent, reliable, auditable, and discoverable electronic storage of documents. Others worked on PDF standards – one great example of how and where standards are helpful. Of course, a certain vendor associated with PDF tools drove a lot of very specific format standards, so vendor capture is a real concern as well. The last point: standards get written by people who keep showing up.

It was occasionally interesting work to research and write about the reliability and trustworthiness of the different types of proprietary electronic WORM storage at a time when write-once optical media was generally being replaced by software. It was also very deliberative, highly political, entirely subject to who got to edit last, and all progress on publishing anything was blocked by one person’s two-handed, white-knuckled grip on every work item. They were very busy in their consulting business – having successfully marketed their committee position. It was with some relief that I resigned from the group, though there was some useful work being done there. Eventually, there could be guidance for how to properly store electronic documents and records so that data will be permanently preserved – a real problem that lacks clear guidance.

I believe standards can be very useful in some circumstances. A standard for what constitutes proper archival of scanned documents makes a lot of sense. Railways, communications protocols, food labeling, and a thousand more things are all good examples of areas where standards can be very valuable, even essential. I just don’t agree that software testing is one of them.

I strongly identify with the Context-Driven School of Testing (CDT), so it should be no surprise that my default position on any testing standard is against. I’ve said previously that my community could object to a standard without knowing its content, because a standard means that context follows process, whatever its prescriptions. By existing, it conflicts with the idea that context should inform method. As Fiona Charles says: “Where is the standard for writing software?” I’d add “Where are the standards for writing, editing, cooking, or painting?”

I came to my understanding of why making software is not manufacturing on my own, well before I found my way to CDT. To refine my problem statement for Sick Sigma, bug density metrics, standards, and anywhere else where someone is trying to force industrial quality processes and thinking on software: making software is not producing widgets. All software projects are one-offs created by developers, testers, and project staff of various skill levels and engagement, to solve different problems, using different tools and often shared components, based on variably incomplete and conflicting understandings of requirements. In a given context, at a given time, a team of (hopefully) smart and (always) flawed people work together to create something that barely works, and follow-up with as much bug-fixing as they are allotted time, money, and energy for. Requirements, conditions, and the skill and engagement of the people on the team are all moving targets.

Criticism aside, there is a large contingent of people who worked on these standards for many years. I respect their effort, and once someone decided there needs to be a software testing standard, a lot of work went into making these happen. There is some good work in here. There are other things I disagree with. I consider the people that worked on this standard colleagues, and there is no personal animus. We disagree about software testing, and this is reasonable professional debate.

I know that there are people who want predictability and certainty in planning projects. Sometimes the stakes are high enough (or feared to be high enough) that a detailed structure such as this process definition will comfort a stakeholder just by virtue of its specificity and number of control metrics.  I also understand that some of the context in any situation is externally imposed, and not always available for debate.

I think that employing skilled and experienced testers who study context, asking them lots of questions and listening closely to the answers, and leaving broad latitude in implementation is a better strategy. I think that the approach to software testing I espouse can lead to better results than following a cookbook, but it is possible to screw up either (or any!) approach. A meal prepared by a skilled cook from the ingredients on hand is delicious, nutritious, and sustainable. I don’t eat McDonalds, and I don’t recommend anyone else does, either.

If cheap and fast are the objectives, you might make a different choice. There is enough room in the world for people to use whatever approach makes sense for them. The problem is that when a specific approach is published as an international standard, it is asserted to be the best way to test software, using prescription to trump skill. When we join projects that already have engineering managers (or even worse, test managers) invested in following standards, they are A). Almost certainly in dire need of our help, and B). Going to be hard to separate from the security blanket of Bestandardices. Our stakeholders and the craft of testing suffer as a result.

I will report on the standard from my perspective as a Context-Driven Tester. I intend to review the pieces of ISO 29119 separately at first, as they are a lot to digest. Also, these get long enough. Let’s start with Part 1, or ISO/IEC/IEEE 29119-1:2013 as it is formally known. I’ll try to extract points of interest, but when other parts (2-5) of the standard are explicitly referenced, I’ll save the discussion of those issues until we get to the appropriate part of the standard.

This is a lot of stuff to read, friends. I’ve been chipping away at this for weeks. I encourage you to do your own review if you have an interest.

Standard’s Introduction

As noted, the standard starts with a statement of purpose. Next, there is an acknowledgement of many contexts (domains, organizations, methodologies), but says that the standard is applicable in many contexts.

In the next part, words like “conformance” and “required” appear. There is no conformance possible with this section; 29119 sections 2-4 are standards “where conformance can be claimed”. There is something important about safety and liability to think about there.


In the next section, almost a hundred terms are defined. Someday, our community should consider creating a dictionary of terms we can debate from.

“Test case(s)” appears 30 times. The definition of test case includes preconditions (state?), input, and expected results. “Test item” is referenced here as something to be executed, while defined elsewhere as an object of testing (system, piece of software, requirements, document, user documentation). Test cases are noted as the lowest level of test input (cannot be nested) for a “test sub-process”.

Exploratory testing uses “spontaneously” where we would typically use “simultaneously” in front of designs and executes; perhaps this is a simple error. “Unscripting testing” is dynamic testing, or when the tester’s actions are not prescribed by written instructions in a test case. So this implies that they usually are?

“Test coverage” is a percentage to which test coverage items have been exercised by a test case or test cases. This seems difficult to calculate. Test coverage item is an attribute(s) derived from a test condition(s) with an unspecified test design technique.

Testing itself is a “set of activities conducted to facilitate discovery and/or evaluation of properties of one or more test items”, noted to include “planning, preparation, execution, reporting, and management activities, insofar as they are directed towards testing.” It seems that the term “test item” (which as defined here includes both software and documentation) is deliberately constructed to suggest these items can all be tested, and it is explicitly stated that it is not necessary to execute software to test. For example, a specification could be “tested” for correctness against requirements, and this would be considered a testing activity by this standard. It apparently would also be testing to discuss which team member will be responsible for reviewing the standard.

Other notes:

  • “Oracle” is missing altogether; pass/fail criteria is the substituted term.
  • Three separate terms are used to describe parts of equivalence partitioning. This is the only real testing skill described in this section. Later, fuzz testing and a few sampling techniques (for choosing test cases) are referenced, but not described.
  • Black box testing is folded into “specification-based testing”.
  • Documents are Named for Several Things, including Describing the Status of each Test Data Requirement and a Separate Document describing Test Environment Requirements (caps party inspired by ISO 29199-1)

Testing Concepts

Next is background about what testing is and what it is intended to accomplish. This is definitely a target-rich environment, with several points worthy of extended discussion. I’ve pulled out a couple that seem worth deconstructing. My comments in italics.

– Every defect is believed to be introduced by a human’s error or mistake, either in understanding requirements specification or creating software code.

I disagree with this point, as an incomplete understanding of requirements is the only state of discovery I’ve ever seen, meaning, the specification is never “complete”. This is natural and expected when one considers the limitations of communication between any two people. One of the strengths of iterative development is exposing these gaps and eventually fixing them (usually). Thankfully, the standard includes room for “undocumented understanding of the required behaviour”.

– The primary goals of testing are to provide information about the quality of the test item and residual risk. This risk is said to be related to how much the test item has been tested.

If all test items (by the standard’s definition) were equal, there might be something to that, but my experience is that different pieces of software/documents/etc vary greatly in complexity, risk, and in any other meaningful attribute that must be ignored in order to say one piece of software is equivalent to another. What does “how much” mean? Relative to what?

ISO 25010 is referenced for its eight quality characteristics, but the standard later details slightly different characteristics: Functional Suitability, Performance Efficiency, Compatibility, Usability, Reliability, Security, Maintainability, and Portability.

There are many mentions of “Dynamic testing”, but this is said to include executable test items, preparation, and follow-up. More detail is promised in 29119-2.

Testing is said to be a subset of verification and validation. Other standards are referenced: ISO/IEC 12207: software life cycle processes, and 1012-2012 – IEEE: Standard for System and Software Verification and Validation.

At some future date, I would like to figure out what the distinction being made here is; since this standard includes evaluation of specifications as “testing” along with a broad definition of software quality characteristics, it’s hard to imagine what’s left.

There is some discussion of testing in an Organizational and Project context. It is said that an organization might supplement the standard, though it “would adopt” the standard. It is then said that conformity with the organization’s processes is more typical than conforming directly to the standard. It’s still said that if an organization does not have “an appropriate set of processes”, they should apply this standard directly. The case is being made for standardization more than strict adoption of the standard. A welcome bit of guidance, though: “The experience of the industry is that no single test strategy, plan, method or process will work in all situations. Hence organizations and projects should tailor and refine the details of testing with reference to standards such as this.”

Many artifacts are Named and Capitalized, from Organizational Test Policies and Strategies to Project Test Plan, down to”Test Sub-process Plan”, with two examples given as System Test Plan and Performance Test Plan.

Some of these documents may be appropriate in some contexts, but I find heavy documentation to be one of the most undesirable characteristics of industrial testing, and a major contributor to why the Agile community seems to think most testing (besides automated checks) is a waste of time. Time spent writing documentation no one reads or updates is time better spent on gathering information.

There is a very complicated diagram describing the relationship between standards, test strategy, test processes, test policy, etc.

The most important point I see are that standards are put in the same box as Regulations and Laws – an especially toxic outcome of pushing adoption of standards such as these. I’ve written about this before, but it is hard enough getting people to use modern testing techniques without having to give the straw man of liability the illusion of a brain.

“Dynamic Test Processes” appears again – in a diagram saying that test design, test specification, and the “Test Environment Readiness Report” are all necessary to get to Test Execution. The term “Issue (s) Notice” is used post Test-Result, as a decision loop for whether or not to report issues; another pointer to 29119-2.

Software Life Cycle and Testing’s Place

The next part talks about the life cycle of software projects between conception and retirement. ISO/IEC 15288 is referenced as a source for life cycles, and ISO/IEC 25051 is mentioned for testing software written by another company. A Requirements…Design…Coding…Acceptance example is given, and then it is said that defining a development model is out of scope. Still, probably would have been better to use an iterative one.

Quality Assurance is described as a support process required to aid the SDLC:

a set of planned and systematic supporting processes and activities required to provide adequate confidence that a process or work product will fulfill established technical or quality requirements. This is achieved by the imposition of methods, standards, tools, and skills that are recognised as the appropriate practice for the context.

Actually, not that bad!

Measures should be collected during testing, as they can provide information about the quality of test processes and/or test items and the effectiveness of their application on each project.

Oh, there we go.

Testing information is to be provided to Project Management with completion reports and test measures. There is some talk about process improvement, and pushing process improvements across an organization.

“Risk-based Testing”

This is introduced as “Testing is a sampling activity”.  The need to identify product, project, and organizational risks, and then to tailor the test processes to address these risks is described in some detail. “Risk-based” is referenced multiple times, with strong language around claims of wide adaptation. Given that all test choices involves evaluating risk, this is hard to argue with.

There is a distinction made between choosing test cases for requirements coverage, and for risk, which is good. There is some talk about polling a wide group of stakeholders to develop a risk model, and most welcome is an introduction of context, using the Medical Device example of compliance.


Five annexes are included. Some quick tastes:

Annex 1 is about testing as a part of “Verification and Validation”, and presents the following model. The idea of Metrics outside of Testing is worth more contemplation.


Annex 2, Metrics: “In order to monitor and control testing and provide timely information to stakeholders it is necessary to effectively measure the test process…Thus all test efforts need to define and use metrics and provide measures in respect of both products and processes.”


Annex 3 has our first discussion of Iterative Development processes, comparing them to “Sequential” and something called “Evolutionary” that seems to be trying to split the difference.

In Annex 4, there is a discussion of “test sub-processes”, defined earlier in the standard as “test management and dynamic (and static) test processes used to perform a specific test level (e.g. system testing, acceptance testing) or test type (e.g. usability testing, performance testing) normally within the context
of an overall test process for a test project”.

This new term is used constantly throughout the standard, but it doesn’t seem to add a lot of value beyond discussing control of the overall test process.  Examples given here include: Acceptance testing, Detailed design testing, Integration testing, Performance testing, Regression testing, Retesting, System testing, Component testing. There are some tables of each type, listing objectives, claims of detail processes, and technique (usually followed by “As Appropriate”.

My last piece of criticism here is that this supposes these are all distinct and separate activities,as opposed to overlapping and concurrent ones. Perhaps I don’t yet fully understand the usage here.

Annex 5 is about Testing Roles. It names a Strategist who establishes process and ensures conformance, Test Manager who manages to process, and a Tester who executes the processes. There is an interesting discussion about the independence between who writes the code and who tests it that unfortunately concludes with “The intention is to achieve as much independence between those who design the tests and those who produced the test item as possible, within the project’s constraints of time, budget, quality and risk.” Sounds like designing as large a communication gap as possible.

A bibliography lists many ISO and IEE standard documents, plus a ISTQB glossary of terms. Agile Testing (Crispin and Gregory) is listed, but no other books on testing that we would be familiar with are included, nor any contemporary sources.

In Conclusion…

We made it! It looks like we have the result of a committee, several years behind the state of the art, overly focused on formality and control, barely concerned with technique. So about as expected.

The other parts of the standard will be reviewed here soon, and by soon, I mean this year. I have not had any real progress on my attempt to formalize this work with AST, but I will continue to work on that, too.

Speaking Easy, Part 4: Diversity and the Bigger Picture

This is Part 4 of a series of blog posts on why I am volunteering to mentor speakers at testing (and other) conferences for Speak Easy. Part 1 is herePart 2  here, and Part 3 here.

We’ve discussed some very practical points about conferences and presentations. I see connections to larger issues, and I would like to talk about those a bit.

It would be presumptuous of me to attempt to discuss these issues with any authority. Most of the time, what white dudes need to do when the issue of diversity is being discussed is to shut up and listen, since they do most of the talking all the rest of the time anyways. We should also be aware that appropriation is not just about headdresses at Coachella, either. Go listen to other people – preferably people with first-hand experience, please.

My experience is second-hand at best, and I continue to discover more aspects of my privilege. My perspective is less interesting because of these blind spots. I hope that readers will point out anything I am missing, have misstated, or should have mentioned. I welcome the opportunity to be educated, publicly or privately, if you feel like it…with one exception.

Discussion of diversity issues makes many white dudes uncomfortable. Hopefully, this makes you want to learn more and explore that feeling. If you (as a white dude) are feeling the need to express that discomfort by disagreeing with what’s been said in this post, it’s OK if you *don’t* tell us why and how you disagree. If you are frustrated that you haven’t been heard and had your opinion dismissed out of hand, that you’ve been lumped in with other people based on attributes you can’t control and shouldn’t define you…do I need to continue? Also, if you are a “reverse-racism” or “intolerance” guy, read this, this, or watch this.

Conference speaking is a window into our industry, and our society. As a society and as a species, we are still very immature. Most of us have agreed by now that every person of every background should have the same opportunities to succeed, and understand that overt (and non-overt) judgments of people based on demographic information is not only deeply unjust and incompatible with the future most of us say we want, it is harmful to people on the wrong end of this bias, and it is highly inefficient. Only to the far right wing have I said anything remotely controversial in this paragraph; we have a broadly-held, nearly universally proclaimed consensus on these goals.

Where we disagree starts at assessing how close we are to achieving these goals. Even in places where laws purport to guarantee equality and opportunity for all, and are amply reinforced by policies and training, the reality still falls far short, in a thousand small ways that add up to real handicaps. From what I can tell, it’s tougher every place and always to be a woman, a person of color, a recent immigrant, to have a sexual orientation besides hetero, or to have a sexual identity that is not strictly cisgendered. At intersections of these, it gets even harder. These facets of an identity do not define who a person is, their potential, their skills, or their abilities. Unfortunately, they strongly affect how people treat them.

In the workplace, credibility and influence are both earned and given. <- Seriously, read that. I know I throw a ton of in-line links, and I can see from site stats that most people skip them, but this particular one is important. A taste:

Male executives who spoke more often than their peers were rewarded with 10 percent higher ratings of competence. When female executives spoke more than their peers, both men and women punished them with 14 percent lower ratings. As this and other research shows, women who worry that talking “too much” will cause them to be disliked are not paranoid; they are often right.

A typical "leadership gathering"
A typical “leadership gathering”

Leadership dynamics are complicated, but everyone notices, at least subconciously, who talks, who gets interrupted, who is credible, who is challenged, who gets credit, and so forth. Patterns get reinforced all the time, and people are constantly pressured back towards the external expectations others have. This is an effective way to squash dissent and turn collaborative processes into echo chambers for the people that call the meeting and set the agenda…er…I mean, show leadership. When people are pressured to be quiet, they eventually run out of energy. Why should they continue to struggle against the undercurrent of dismissal, when they will punished for speaking up?

Scale-emphasized, still significant
Scale-emphasized, still significant

Tech believes itself to be at least as diverse and a more merit-based industry than others, but it simply isn’t. Tech is sort of like education, marketing, civil service, or other industries where the rank and file are diverse and full of stars from all sorts of backgrounds, but then gets all pasty and lumpy towards the top. At least in the US, we’re losing women year over year in one of our fastest growing industries.

If your response to the discussion of diversity in conference speakers is to point at under-representation in submissions, please think it through one step farther. Why are there fewer submissions from every group besides white guys? Don’t the demographics of our industry suggest there should be a more diverse set of applicants? What are the forces working against that?

I was chagrined the day I finally understood how frequently I was talking over women to restate their ideas, co-opting them and dragging them away from their owners. I’d be even more embarrassed if I told you how recently that was. Dudes, I am paying a lot of attention to this lately, and what I’ve learned is that even with generally good intentions, I’ve been part of the problem. While I will never really understand what it’s like to not have the credibility, influence, and confidence I’ll be listened to just for being me, I will be a lot less of a jerk if I think about it at all. That’s where the bar for being decent on diversity is – pretty low, but still higher than most seem to be willing to reach. Any real awareness is so much better than continuing to coast.

Speaking of coasting, continued acceptance of the advantages we enjoy leads to our world being more crap, because of mediocre white guys taking opportunities they don’t actually deserve from smarter, more qualified people in other demographic groups. Yes, I am willing to live with the potentially painful personal consequences of that, and you should too. The mythical true meritocracy would be a much better place for all of us to live.

Our friends, our spouses, our brothers and sisters – they need us to step forward and help them fix this. Being listened to and taken seriously is a good first step. We can help people be heard, and help them be more confident that they should speak up. As a society, we are doing better lately, but we have a long, long, LOOOONG way to go. Helping people become speakers helps them become leaders, and will make our future better.

Part 5 is out there somewhere, and it’s about my experience learning to speak. It might be a bit before it comes out.

Speaking Easy, Part 3: Developing Your Presentation

This is Part 3 of a series of blog posts on mentoring speakers at testing (and other technical) conferences for Speak Easy. Part 1 is here. Part 2 is here.

In yesterday’s installment, I talked briefly about preparing an abstract. Today, I want to talk more about that, and developing your presentation. We’ll use a theoretical subject that I am not qualified to present on, as you’ll see shortly.

Getting Started

You need an idea to start – what is your presentation going to be about. When you can articulate it in 3-5 compelling paragraphs that include learning takeaways, you’ll be ready to submit it.

Let’s say you spend a good amount of your work on testing web forms. Let’s also say you’ve done some reading about effective GUI design, you follow @mralancooper and other designers, and you discuss interfaces with the people you work with. You know something about this, even if you don’t consider yourself an expert or work as a designer. You might be an expert as a tester, though.

Think about the techniques you use. What heuristics or shortcuts have you developed to do your job? How do you evaluate an interface? Do you have a method to help you remember to apply them? If not, just make a list.

How do you report and contextualize these issues to make your information actionable? What do you think about when you need to summarize what you’ve found to managers/developers/designers/other stakeholders?


You now have takeaways – describing how you do what you do, why you do it, and how you describe what you find. These are things you are an experienced practitioner in, and practical advice in one or both of these areas could be helpful to many people.

Borrowing from yesterday’s post: Look at conference programs/schedules. Here’s the “STARry” larger test conference programs to look at, here is STPCon, and here are some CAST conferencesall linked in a row. Maybe Let’s Test? If you are interested in writing a paper too, you could look at PNSQC. TestBash, Nordic Testing Days…there are dozens more conferences, all around the world, and I apologize to the volunteers that keep all of these conferences running for not including every one. There are people who maintain links for the various software testing conferences, and you can use those to see more. There are many hundreds of presentations every year, and most will need to be replaced next year.

After reading these session descriptions, you now have examples. Which ones are compelling to you? Which ones are not?

Start writing! Your ideas are the payload, so get them laid out and don’t worry about whether it is pretty at first.

Also from yesterday: Work in a local document – you don’t want to compose in someone’s web form and then lose it, or have to remember what you sent in later. Plus, if you are not accepted, you may want to propose again to a different conference.

Select a Target

Do you have a conference in mind that’s coming up? Make sure you understand the deadlines for submission. Make sure you can commit to the dates of the conference if you are selected, and make sure you know how you will get there. You probably won’t get paid by the conference for your travel, lodging, meals, etc, so make sure your employer is on board for the expenses, and your time.

Putting speaking goals into your review is a good hook to have prepared for this conversation. Your employer should be happy to support your personal growth, and to realize the recruiting benefits of having employees speaking. If not, maybe you have friends/family you can stay with near the conference, lowering travel expenses. Still try getting paid for the conference as regular time (not vacation) if you can, as you can point to the “free” conference admission. If you can’t even get that, you may need to make tough choices, including perhaps one about continuing your current employment if you are not in the right environment to grow. I’ve spoken at a conference on vacation, but speaking is work, and you should be spending vacation days on not-work. 

Back to your abstract in progress: Get feedback from multiple sources, such as your Speak Easy mentor. Take some feedback, ignore some feedback, wonder why you asked that person for their opinion, and polish. When you are happy with it (or more likely, when you are nearing the deadline), copy your abstract into the submission form, cross your fingers, and push submit.

There’s a trend of announcing submissions on Twitter. I don’t because I’m an insecure introvert, but that’s up to you. Then, let it go, and wait to hear back.

Outlining Your Ideas

Some weeks after your proposal is sent in, it will be enthusiastically accepted. Uh-oh. Now you have to really work at it! Graciously accept, block off your days, sign up for any hotel deal, and make a calendar reminder to book your flights several weeks before you go.

Find your abstract, and build an outline from it. It would look like this at first if I did it in a few minutes while writing a blog post:

  1. Introduction: You, where you work, what you do. Describe your context.
  2. Issues in testing GUIs
    1. Field Data types and validation
    2. Localization/Date/Currency issues
    3. Usability/Screen Resolution
    4. Ask room for suggestions (limit to 5 minutes)
  3. Techniques
    1. Checking data validation
    2. Duplicate data, other interesting interactions with backend
    3. Incomplete submissions/returning to complete
    4. Browser nav (forward/back/reload)
    5. Some of your collection of weird strings to paste into fields (supplemented by link to your blog, where you have them all collected?)
    6. Your checklist of these techniques and others you didn’t delve into (Other browsers, bad wireless networks, accessibility, etc)
    7. Tools you use for screenshots, http inspection, etc
    8. Ask room for suggestions (5 mins)
  4. Demo on a public-facing website of some of these techniques (10 minutes). Pick a target where you can illuminate an issue or two that you’ve previously described.
  5. Reporting GUI issues – who, what, where, how, and the why that matters for which stakeholder(s)
  6. Takeaways and Questions

45-60 minutes is not so long once you lay it out, particularly when you allocate time for questions and audience interaction. Now you can start putting together your presentation.

Your Presentation

Some people say PowerPoint sucks. It works for me, and for many other people. Some people have learned Keynote. Prezi works for some people. Very few or no slides works for some people. If you can get to the point where you are not leading slide read-along time, you’re good enough for now.

A detailed discussion of what you project while you talk or how to talk is out of scope for this post. What is in scope is the how to prepare: hammer out a rough copy of what you want to show, and start practicing talking in front of it. Fill in the gaps you’ll find once you start discussing the issues, and iterate. Don’t worry about “pretty” yet (unless you need a procrastination technique), worry about completing your content. Your talk is the time to watch, not slides. Slides are only there to support you.

Practice, practice, practice, especially before your first presentation. Audio record yourself. Time yourself. Use the right words instead of the biggest ones, and choose words that are natural for you to say. Make sure you are not talking too fast, that you are speaking clearly, that you are explaining any jargon, and that you are ready for transitions.

You want to identify and knock out “um”, “ah”, “as you know”, “obviously”, and other verbal tics. You need to pause for a couple of moments between slides to signal transitions, and it’s fine to be quiet for longer and let people read/absorb. You might not guess it from my blog posts, but I am a big fan of cutting the fat to make what’s left look better.

If you have not been in front of people yet, this might be a time to get a couple friends together and do karaoke, however well or badly you might do, to experience the queasy stomach feeling in a safer place.

Before the Conference

Aim to have your slides done two weeks before the conference. You will continue to tweak them afterwards, but you probably need to send them in for posting, to preserve the time for fiddling and a couple more practice runs, and reduce last minute-induced crapness. It’s also a good idea to make sure you know how to make your laptop project, have any necessary adapters, etc. You might want to check your slide resolution and make sure it is a common projector resolution.

An aside on slides – some people horde them, many people ask for them, very few actually every look at them. Some conferences want attendees to get them from their site. Some people protect their slides. I put my CAST 2014 presentation on Slideshare, even though I have given other versions of it since and will again. AST also posted a video of it.

Your job is to be generous to everyone that wants to learn from you, responsive to the conference that gave you the opportunity to be heard, and confident and forthright about what’s yours and what isn’t.

Your Session

You will be nervous as your session approaches. Trust your preparation, and channel your energy into practice/last minute slide tweaks if you must, but you are better off not making decisions under duress.

Conferences are valuable learning and networking opportunities. I’ve met many of my favorite people at them. Go to sessions and support other speakers. Volunteer for the conference if they need any help.

The wrong way to prep for your session
The wrong way to prepare for a session

Spend as much time as you can with your fellow conference goers, and see how much you learn from people talking about what works in other contexts. Just don’t stay out too late the night before your session, and don’t get too “dehydrated”.

The session before yours, you will not be paying much attention, so avoid being a distraction by preparing somewhere else. Make sure you have enough water, make sure you don’t have too much, that your laptop is charged, your phone is silent and won’t vibrate, you have a dozen business cards handy for your new fans, and any demos work.

Go to your session’s room as soon as the previous session is over. You need the length of the break to set up your laptop, test A/V and slide clicker, find a clock you can see while you talk, put your phone in your bag, get one glass of water, feel really nervous, and then breathe. I still get butterflies, but it’s a lot better than it used to be. The 4-7-8 technique might help.

Make eye contact, smile, and say hello with each person as they come in, if you can. Accept that you’ve done what you can to prepare, and that you’ll do the best you can. Force yourself to start slow. FADE OUT.

FADE IN: Your room is clapping, and then filling out enthusiastic evaluations. Be gracious to everyone who comes and talks to you after.

Wrapping Up

Jot down any notes you have for yourself. Ask any friends in the room for their feedback.

Turn your phone back on. Star any tweets you like about your session and #followback your new twitter followers.

Congratulate yourself for making it to the other side. Allow yourself more “enjoyment of the conference” that evening. Thank the conference organizers. Add the talk to your LinkedIn profile.

Tell your boss how grateful you are for the support to attend and speak. Look for any local Tester Meetups that you could bring the conference session to. Look at what other conferences you might want to try next; now that you’ve given this talk once, you can improve it.

A few weeks after the conference, watch for speaker evaluations, and take any feedback. Remember that not everyone scores the same way; some will score 3 out of 5 as perfectly respectable. There will be some empty glowing reviews, and almost always, one negative review that after closer reading, you are not sure they were actually in your session.

Whew! It was a long journey, but we got here. Also, it takes a while to prepare for a conference.

In Part 4, I’ll talk about why the subject of diversity matters to me personally.

Speaking Easy, Part 2: You Can Do It!

This is Part 2 of a series of blog posts on volunteering to mentor speakers at testing (and other) conferences for Speak Easy. Part 1 is here.

In case you didn’t get it in the title…

You Can Do It!

You, yes you!

Ryan wants you to speak
Ryan wants you to speak

Really, you can. Hopefully not much more than about 220 days a year, you go to work and do a job. While you are doing your work, you are collecting experiences, researching and implementing methods, getting practice, and breathing in the how and why of what you do. You are accumulating hands-on knowledge every day, and improving at your craft. You know a thing or two about a thing or two, and the world might benefit from you sharing it.

You might look at who speaks at a conference, and think that you don’t match up for some reason. In terms of presentation skills, that might or might not be true at first, but you should never assume that you have less expertise in your subject matter. Many of these speakers have less experience than you at the things you work on. They may have seen more contexts, but they don’t know yours as well as you do. We need fewer blowhards describing the wildly successful implementation of processes they’ve spent talking to managers on the periphery of for a few days, and more sharing by practitioners of techniques and reporting on experiences.

You might not have an obvious subject in mind to talk about when you decide that you want to try speaking. That’s OK. Read conference programs, attend conferences, talk with peers, do some research – you can find something that will spark you. You only need a rough draft of a couple paragraphs of abstract to get started with research and asking for feedback. You don’t have to go all the way through with submitting – you can wait until next year or try a different conference.

Plus, you have these factors working for you in how conferences select speakers:

  1. There are usually not that many submissions. Often, conferences are scrambling to fill the number of speaking slots they have. When you see experienced speakers doubling and tripling up on track sessions, or you see a conference repeatedly publicizing approaching deadlines, you have evidence of this.
  2. There are submissions that are well, not as good as others. The obvious corollary to there not being many submissions is that there are not many abstracts being turned away. Near the end, what gets accepted is helped by the need to pad out a schedule.
  3. Submissions are just summaries. Professional authors will tell you that the best way to sell a book is “on spec”: a sample chapter, and some description of what the book is about. They don’t have to write the whole book until they’ve sold it. Conference presentations work the same way. More on conference submissions later.

hey-you-can-do-itNet-Net: Your proposal doesn’t have to compete with the most compelling submissions from the most experienced speakers. It only has to compete with the last set of submissions that are accepted, or an experienced speaker’s third session.

This all varies, of course, usually down to the track level for conferences that try to maintain subject lines over their conference days. I’ve been turned down for conferences I really wanted to speak at. I’ve also been asked to submit for conferences I had not heard of before I was asked, presumably because they saw my name on other conference programs.

Reviewers and conference staff want to program the best conference they can for attendees, and want great content. New blood is necessary to keep conferences vital and interesting. The old boys network is sort of a thing, but not as prevalent as you might think. Conferences want to have high energy, and that tends to come from people who are still doing the work, not just talking about the work. Killer ideas matter much more than good slides and patter.

So, study what other conference session descriptions looks like, then write three to five compelling paragraphs about what you want to talk about, tailoring it to the format of what you see for previous conferences. Here’s the “STARry” larger test conference programs to look at, here is STPCon, and here are some CAST conferencesall linked in a row. Maybe Let’s Test? If you are interested in writing a paper too, you could look at PNSQC. There are dozens more conferences, all around the world, and I apologize to the volunteers that keep all of these conferences running for not including every one. There are people who maintain links for the various software testing conferences, and you can use those to see more. There are many hundreds of presentations every year, and most will need to be replaced next year. Why not you?

Work in a local document – you don’t want to compose in someone’s web form and then lose it, or have to remember what you sent in later. Plus, if you are not accepted, you may want to propose again to a different conference. Casually embedded idea: you might be able to find an experienced speaker and ask if you can co-propose with them.

Get feedback, polish it, make sure you’re happy before you submit, but remember that you only need to summarize your main points at this stage. You’ll have months between an accepted abstract and a completed talk, and you almost always can revise your abstract for the program, before the conference.

In terms of asking for help directly from the conference, I think it can be a good idea, with some qualifiers. As a reviewer, I don’t mind giving feedback on drafts when it asked for well ahead of deadlines, and I don’t mind coming up with a couple of provocative questions to help sharpen an abstract. I have also been in situations where I felt like I was being politicked for special consideration more than any genuine desire for feedback, and didn’t care for that. People will respond positively to honest seekers.

Maybe you get accepted, maybe you don’t. It hurts a little to get rejected in any context, but it’s going to be OK. Save your abstract, look for opportunities to do more with it (Meetups, etc), and dust it off when the next conference comes along. There are any number of reasons why your talk might be fine, but just not fit for that specific conference. It could be overlapping with someone else’s proposal, or there might be too many talks for that one track. You might not have missed it by much. Sometimes you can get feedback on why, but I’d recommend giving it a few days to cool off before you ask. Just don’t give up!

The responsibility of the reviewers to the conference and its attendees is to select the best content possible, but that is multi-dimensional. A diverse set of subjects, approaches, and experiences creates the richest program. Programs drive attendance, and momentum between conferences depends on successful conferences that people enjoy attending.

schneideryou-can-do-itIt’s hard work and it takes practice to become a good speaker. I couldn’t tell you anything about becoming a great one. But the opportunities to try are not so hard to come by, and you probably have something to talk about.

If you don’t want to do it, that’s fine. It’s not for everyone.

Just don’t decide there isn’t room for you – there is plenty, please come on in. We need you, and we’re really hoping to hear from you soon. How can I help?

In the next post, I will talk about how I develop conference presentations.

Speaking Easy, Part 1: Why it Matters

I recently volunteered to be a mentor for the Speak Easy program that Fiona Charles and Anne-Marie Charrett have started. Being accepted as a mentor is one of the proudest moments of my career. I’d like to talk about why this program matters, why I volunteered, and why this honor means so much to me.

First, the program:

Speak Easy came about when Fiona Charles and Anne-Marie Charrett decided to walk the talk about creating diversity at technical conferences. As conference organisers, they’ve seen the challenges in sourcing diverse speakers for their programs. As experienced speakers they’ve understood and seen the difficulty in getting experience and confidence to speak at a professional level.

This initiative is designed to bridge that gap.

This is a very real thing:

My experience as a conference organizer bears this out. Women are not the only underrepresented group as speakers at most technical conferences; it’s essentially every demographic besides straight white dudes that has a less-than-proportional ratio of conference attendee to conference speaker at most conferences I’ve attended. Try that experiment – assess proportions of the audience relative to the proportions of the speakers – at the next conference you attend, or when looking at a board, executive team, government body, etc. You know, in case constantly being told this is an issue wasn’t enough, and you needed some independent verification.

Why Does It Matter?

Because speaking gigs improve careers. A lot. And these opportunities should be accessible to everyone who deserves them.

Many conferences don’t pay anything beyond a conference admission for the time spent preparing and practicing a track session. Some will help a little with hotel, meals, or other travel expenses. Keynote speakers are more likely to get a small honorarium. None of this compensation is significant compared to the raise a promotion can bring, or the revenue of a consulting gig.

Some people are lucky enough to speak at conferences on an expense account, but consultants give up billable time to attend and speak at conferences. The time spent preparing material is also significant, and is more non-billable time.

So why do we do it? We often call it marketing, because getting your name and your company’s name out there can be of real value, and it’s good to have Google results. Plus, that lets you use marketing budget for travel expenses. But that’s not the only reason – and for most people, it’s not the biggest value they personally get.

Not only are speaking gigs exposure to the world at large for you and your ideas, they are practice for stepping in front of a group of peers and confidently declaring a point of view. Only half of the phrase “thought leadership” is about ideas. That is an important part – study, research, and articulating what you know is a learning exercise that will help you grow. The other half is about confidence, persuading, connecting with an audience, and other skills that are necessary to lead effectively, both formally and technically. The experience of stepping out on front of a room that you think is staring at you, getting over your terror, fielding questions/challenges, and staking out your claim as an expert will increase your effectiveness in a variety of situations. I’ve known a very few people self-possessed enough to confidently speak in front of a group. Everyone else has to get over their impostor syndrome and learn how to do this in order to advance their careers.

Speaking at a conference is one of the best ways to get this experience, and will help your career. It’s a valuable opportunity, and if you hope to lead, either formally or with your ideas, it is something you should strongly consider pursuing. If you would like a gentler on-ramp, consider addressing a Meet-up, or lightning talks/ “speed-geeking” at a conference.

Check back tomorrow for Part 2: You Can Do It!