WOPR24 Experience Report

This blog post relays the findings of a peer workshop on performance and reliability testing and monitoring. I will not provide the usual TL; DR summary this time – if you want this gold, you will have to sift it yourself.

On 22-24 Oct, I attended and facilitated WOPR24, a LAWST-inspired workshop for performance engineers. I have been involved in organizing WOPRs for about 6 years now, and have attended 18(!) WOPRs over the last 11 years.

The attendees of WOPR24 were Ajay Davuluri, Oliver Erlewein, James Davis, Andy Hohenner, Yury Makedonov, Eric Proegler, Mais Tawfik Ashkar, Andreas Grabner, Doug Hoffman, John Meza, Michael Pearl, Ben Simo, and Alon Girmonsky. We were hosted by BlazeMeter in their Mountain View, California office, which was a fantastic venue. Adi BenNun of BlazeMeter took great care of us, making us very comfortable, feeding us very well, and making sure we had everything we could want.

WOPR24Group

I could talk for a long while about WOPR, WOPR’s history, our mission of advancing the practice and community building, how WOPRs are put together, and how Ross Collard changed the trajectory of my career and life by inviting me to WOPR3. At some future date, I will. But for now, so that we can talk about the great content of WOPR24, I will just drop a link about WOPR and let you surf that site.

The Workshop Theme

Each WOPR has a Theme, to help focus the experience reports and lay out what we want to explore. From http://www.performance-workshop.org/wopr24/:

Production is where performance matters most, as it directly impacts our end users and ultimately decides whether our software will be successful or not. Efforts to create test conditions and environments exactly like Production will always fall short; nothing compares to production!

Modern Application Performance Management (APM) solutions are capturing every transaction, all the time. Detailed monitoring has become a standard operations practice – but is it making an impact in the product development cycle? How can we find actionable information with these tools, and communicate our findings to development and testing? How might they improve our testing?

Content – Experience Reports

The primary format of WOPR is Experience Reports (ERs), which are narratives supported by charts, graphs, results, and other relevant data. Each ER is followed by facilitated discussion triggered by the ER. 7 attendees presented ERs over 3 days. Systems we discussed included the following:

  1. An online advertising auction system to algorithmically price and purchase ads for specific user profiles in < 40ms. Presenter used Splunk to model and characterize log events, web hits, application instrumentation, business metrics, and other content.
  2. An automotive parts sourcing SaaS application. Presenter discussed supplementing regular human-conducted load tests with CI automated tests. Lively discussion about thresholds and test environment control/resetting ensued.
  3. A mapping application company’s efforts to test with virtualizing high end video cards (https://www.cdw.com/shop/products/NVIDIA-GRID-K2-graphics-card-2-GPUs-GRID-K2-8-GB/3126398.aspx).
  4. An application that collects, rolls up to big data sets, and displays back to the system’s owner detailed operational metrics from a very large number of embedded systems, distributed around the world.
  5. A order-taking website for a large retailer. Discussion was about launch of a site, and the difficulty of enrolling reluctant development stakeholders in performance testing projects.
  6. A non-profit transportation industry service that publishes rate tables multiple times per day to and from all the vendors in that sector. Discussion of concurrency bugs and how they were reproduced.
  7. A large financial services SaaS provider shared some of the issues around testing mobile in-app video and chat. Generating traffic and evaluating results were some of the technical issues we discussed. 

Content – Exercises

Between ERs, we conducted several guided discussions to explore specific areas of interest. Findings from those are relayed here:

What should we alert on?

In this exercise, we just started calling out thresholds and notes. We definitely didn’t finish.

Monitor What Threshold Notes
CPU > 75% Web/App, > 50-60% DB User + System CPU: For Vus with 4 or less and hyperthreaded, Warning 50 alarm 75, critical 95. With > 4/metal with HT disable warning 75 critical 99 Context-dependent Physical and virtual CPUs Per core and overall 1 minute monitor – number of sequential observations to trigger (two at which level? 3 alarm? 1 critical)? One minute? 5 Minute?
Order Rate Dynamic Baseline (Time/Day/Etc) Oliver: Tues is Max, Friday is Min revenue, order rates, conversion rate, bounce rate Business spend: Advertising money out
Failure Rate http error rate against yesterday Need to have a baseline. Beware of bots, synthetic measures, etc
Queue Length: Threads Web/App, detectable at App? Should have threads available, 2threads/core, beware of thread contention/concurrency events Alert on contention/concurrency?
Connection Pool Utilization JMX: DB Connections, Outgoing Requests DB Connections, external web service calls, depends on app server. Is app waiting for a connection?
TCP Inbound Queue Anything increasing in subsequent samples
Message Q Messages creates/sec, size (no growth), message age, expiring messages present?
CPU Q Load Average/CPU Queue Length > 2 per core Load Average: 4x number of cores
MIPS/Second 3500/second, threshold 2900
Response Time Several times higher than SLA TCP/Response Time can indicate app server overload distribution and median/thresholds vs baseline
Errors (Particular) Class/Types, Rates, and severity Error clusters
CPU Ready Time Any significant percentage
GC Time Percentage of time – what’s impact? Over 10% Full collection? Partial? Age of objects leaving generation
Averaging http status Compare to baseline average literal numbers By request (type), or by page. Watch for dramatic changes Watch for bots > 50% Big difference, prod vs. test. Remove synthetics in test. Load balancer, keep-alive, etc
Throughput Cloud $ rate, watch for bottleneck Network throughput + context switch over Cpu = ratio for sanity check, first bottleneck
GPU utilization in virtualization 75-85% utilization = efficient virtualization
Redirects Count/rate. Redirects on keep-alives Endless loops? Redirects per user, check for max
SAN I/O count
I/O latency Write latency Queue lengths let me see problem faster
Thread context switches
Wait time (DB) Latches, locks, just look for increases Also wait states
Log Volume (and by type)
Cache hit ratio Page life expectancy
DB Connections
Virtual active mem vs real memory Check for Disk swapping
Physical memory free 66% Managed memory Leave small amount free
Network I/O Based on network link rating
Network errors/packet discards
objects generated
Induced GCs
Live Sesions
Disk utilization – space and I/Os
Average SQL Statements/Request Data driven pattern detection problem n+1 query results
Disk space rate consumption
Starting/stopping monitoring agents
Connection Pool: Available
Monitoring Boxes
# restarts
Rate of change in connection pool count
recycling of worker process/app pool
Log messages by severity
thread status – deadlocks?
Business transaction rates – whatever those are
Revenue/sec
Availability
Cloud node thrashing Spin up spin down
Functional audit log/application log Specific events
Transaction span lifecycles
Scale of code change on updates This jar file is x times previous verision
Page size (html) Delta vs previous
User/system ratio
Web Server Request Queue: IIS
Batches/sec 200
%c 1/2/3 VMs in power-save mode? Detected measuring against baseline
Response time total for all requests (SUM) Saw things that were not obvious Correlate against load

What Would an Ideal Dashboard Look Like?

For this activity, we broke up into groups to talk about different contexts. There are some metacomments in each section, reflecting when they came out.

SaaS Company Dashboard

SaaSCompanyDash

This dashboard has horizontal bars to show different metrics for different consumers. The audience is the whole company. The Checkbox/Xs on the left are to provide a two state indicator to easily, rapidly, and broadly detect when something is wrong, by functional area. Here is what each had:

  • CEO – For each metric, the current time period against a previous time period. Demos, Signups, subscriptions, Retain/Drop Rates, and some support metric such as call count
  • CFO – Cost per lead, cost per user, and a sparkline for Revenue
  • CMO – Social media engagement, Mailing opens/click rates, ad conversion
  • CIO – Availability, error rate, Response time, throughput
  • CTO – Deployments, key resource metrics

Each one of these bars is designed for drill-down. Remember, this is the top level.

E-Commerce

For E-Commerce, they decided not to draw a single dashboard. They defined five different functional areas they care about, and drew a Venn diagram to show that there are overlaps of interesting information, but very little that everyone wanted, needed, or would benefit from knowing.

Consider this continuum, generic to specific, from “easy for everyone to understand” to “doesn’t have to be understood except by the individual who uses the dashboard”. You could also think of these as spokes from the center outwards.

Company -> Department -> Team -> Individual

As the questions were posed by them:

  • Who is looking at the dashboard?
  • What is their role?
  • What do they want to see?
  • What actions would they take if they did see something?

The four areas they defined, and the notes on each:

  1. Performance Testing
  2. Business – Adoption Rate, Active/passive campaigns, usage by feature, dollars per aggregate
  3. Development – Separate dashboards for each scrum team
  4. OPS – Active support ticket counts

BigCorp’s Dashboard

BigCorpDash

Here is a dashboard imagined for a very large company. It is intentionally light on detail to avoid having visitors/contractors get information they shouldn’t have. Some of the resulting discussion was about dysfunctions, but these are real safety issues in some organizations.

Consider the following characteristics in deciding what to have in the dashboard:

  • Tooling – what can you show
  • Time – both for measuring effort, and for displaying data in context
  • Security – Don’t show something you shouldn’t – review content with Security officer?
  • Liability – Knowing specific things – or being responsible for exposing specific things – might put a person in a position they don’t want to be
  • Cost – Screens, software, etc.
  • Political dimensions of having potentially out-of-context data be embarrassing to specific executives or individuals. Is this public shaming? Are people stuck with metrics that will never go green? Is that deliberate? What if we move the goalposts so that someone can see green – are they still worth tracking? Should we turn off our sales dashboard when we are clearly going to miss goals, so that we don’t depress morale?
  • NO STOCK PRICES. Or other metrics that are not actually connected to current, actionable data.

Some specifics about this dashboard:

The “Christmas Tree” or dragstrip pattern is for a number of areas to indicate green/yellow/red. This is designed only to communicate whether something interesting is happening.

Other visualization methods – use vertical meters, or speedometers.

The other primary visual is a graph comparing a current time period against a previous, such as this day’s revenue against last year’s/quarter’s/week’s. The interesting suggestion here was to change the background color as an indicator.

Final Metacomments

Focus in the information needs: What is actionable? What is relevant? What is helpful?

Go to people consuming data – what are they needing? Answer that question, instead of starting with what is easy to provide.

Revenue metrics are key

Iterate!

What metrics do we talk about? What metrics do we present? Those metrics already have a life, so we should try to reuse them.

Test for information completeness – establish that the dashboards are truthful and accurate. Make sure the team knows what the data means and where it comes from.

Dashboards provide transparency into current states – but require interpretation. They are starting place for discussions across teams.

Leave space for temporary/rotating items

Bonus Screenshot of an Ops Dashboard Someone Built

Unclear what the news presenter sees on the dashboard that concerns her ;-p

NewsPresenter

What Would We Want to Monitor for Mobile Users?

In this exercise, we listed the attributes that would be interesting to capture for mobile monitoring.

On the device:

  1. Network conditions: latency, bandwidth, jitter, packet loss
  2. Geographic location
  3. Screen size/device
  4. OS version
  5. Battery State
  6. Memory – free/available
  7. Number (list?) of apps running
  8. Powersave mode
  9. Chaperoned?
  10. Carrier
  11. Is the screen cracked or broken?
  12. Signal strength (wireless/cellular)
  13. Accessories
  14. Recent movement/direction/accelerometer data
  15. Contention? Drops/retransmits
  16. Carrier Plan status?
  17. Time: last restart/power cycle
  18. Rooted/Jailbroken?
  19. Date in Service
  20. Accessibility Options
  21. Geographically similar to other devices? People travelling together in a train, for example

In the app:

  1. Screen load RT
  2. Memory footprint
  3. Data transfer
  4. CPU cycles
  5. Gestures captured
  6. Round trips/waterfall
  7. Ad activity
  8. Freemium status?
  9. Known customer
  10. Time of day
  11. Demographics
  12. Date of Install
  13. Version
  14. Date last used
  15. Other app requests
  16. Phone connection right now?

Speaking Easy, Part 3: Developing Your Presentation

This is Part 3 of a series of blog posts on mentoring speakers at testing (and other technical) conferences for Speak Easy. Part 1 is here. Part 2 is here.

In yesterday’s installment, I talked briefly about preparing an abstract. Today, I want to talk more about that, and developing your presentation. We’ll use a theoretical subject that I am not qualified to present on, as you’ll see shortly.

Getting Started

You need an idea to start – what is your presentation going to be about. When you can articulate it in 3-5 compelling paragraphs that include learning takeaways, you’ll be ready to submit it.

Let’s say you spend a good amount of your work on testing web forms. Let’s also say you’ve done some reading about effective GUI design, you follow @mralancooper and other designers, and you discuss interfaces with the people you work with. You know something about this, even if you don’t consider yourself an expert or work as a designer. You might be an expert as a tester, though.

Think about the techniques you use. What heuristics or shortcuts have you developed to do your job? How do you evaluate an interface? Do you have a method to help you remember to apply them? If not, just make a list.

How do you report and contextualize these issues to make your information actionable? What do you think about when you need to summarize what you’ve found to managers/developers/designers/other stakeholders?

Abstract

You now have takeaways – describing how you do what you do, why you do it, and how you describe what you find. These are things you are an experienced practitioner in, and practical advice in one or both of these areas could be helpful to many people.

Borrowing from yesterday’s post: Look at conference programs/schedules. Here’s the “STARry” larger test conference programs to look at, here is STPCon, and here are some CAST conferencesall linked in a row. Maybe Let’s Test? If you are interested in writing a paper too, you could look at PNSQC. TestBash, Nordic Testing Days…there are dozens more conferences, all around the world, and I apologize to the volunteers that keep all of these conferences running for not including every one. There are people who maintain links for the various software testing conferences, and you can use those to see more. There are many hundreds of presentations every year, and most will need to be replaced next year.

After reading these session descriptions, you now have examples. Which ones are compelling to you? Which ones are not?

Start writing! Your ideas are the payload, so get them laid out and don’t worry about whether it is pretty at first.

Also from yesterday: Work in a local document – you don’t want to compose in someone’s web form and then lose it, or have to remember what you sent in later. Plus, if you are not accepted, you may want to propose again to a different conference.

Select a Target

Do you have a conference in mind that’s coming up? Make sure you understand the deadlines for submission. Make sure you can commit to the dates of the conference if you are selected, and make sure you know how you will get there. You probably won’t get paid by the conference for your travel, lodging, meals, etc, so make sure your employer is on board for the expenses, and your time.

Putting speaking goals into your review is a good hook to have prepared for this conversation. Your employer should be happy to support your personal growth, and to realize the recruiting benefits of having employees speaking. If not, maybe you have friends/family you can stay with near the conference, lowering travel expenses. Still try getting paid for the conference as regular time (not vacation) if you can, as you can point to the “free” conference admission. If you can’t even get that, you may need to make tough choices, including perhaps one about continuing your current employment if you are not in the right environment to grow. I’ve spoken at a conference on vacation, but speaking is work, and you should be spending vacation days on not-work. 

Back to your abstract in progress: Get feedback from multiple sources, such as your Speak Easy mentor. Take some feedback, ignore some feedback, wonder why you asked that person for their opinion, and polish. When you are happy with it (or more likely, when you are nearing the deadline), copy your abstract into the submission form, cross your fingers, and push submit.

There’s a trend of announcing submissions on Twitter. I don’t because I’m an insecure introvert, but that’s up to you. Then, let it go, and wait to hear back.

Outlining Your Ideas

Some weeks after your proposal is sent in, it will be enthusiastically accepted. Uh-oh. Now you have to really work at it! Graciously accept, block off your days, sign up for any hotel deal, and make a calendar reminder to book your flights several weeks before you go.

Find your abstract, and build an outline from it. It would look like this at first if I did it in a few minutes while writing a blog post:

  1. Introduction: You, where you work, what you do. Describe your context.
  2. Issues in testing GUIs
    1. Field Data types and validation
    2. Localization/Date/Currency issues
    3. Usability/Screen Resolution
    4. Ask room for suggestions (limit to 5 minutes)
  3. Techniques
    1. Checking data validation
    2. Duplicate data, other interesting interactions with backend
    3. Incomplete submissions/returning to complete
    4. Browser nav (forward/back/reload)
    5. Some of your collection of weird strings to paste into fields (supplemented by link to your blog, where you have them all collected?)
    6. Your checklist of these techniques and others you didn’t delve into (Other browsers, bad wireless networks, accessibility, etc)
    7. Tools you use for screenshots, http inspection, etc
    8. Ask room for suggestions (5 mins)
  4. Demo on a public-facing website of some of these techniques (10 minutes). Pick a target where you can illuminate an issue or two that you’ve previously described.
  5. Reporting GUI issues – who, what, where, how, and the why that matters for which stakeholder(s)
  6. Takeaways and Questions

45-60 minutes is not so long once you lay it out, particularly when you allocate time for questions and audience interaction. Now you can start putting together your presentation.

Your Presentation

Some people say PowerPoint sucks. It works for me, and for many other people. Some people have learned Keynote. Prezi works for some people. Very few or no slides works for some people. If you can get to the point where you are not leading slide read-along time, you’re good enough for now.

A detailed discussion of what you project while you talk or how to talk is out of scope for this post. What is in scope is the how to prepare: hammer out a rough copy of what you want to show, and start practicing talking in front of it. Fill in the gaps you’ll find once you start discussing the issues, and iterate. Don’t worry about “pretty” yet (unless you need a procrastination technique), worry about completing your content. Your talk is the time to watch, not slides. Slides are only there to support you.

Practice, practice, practice, especially before your first presentation. Audio record yourself. Time yourself. Use the right words instead of the biggest ones, and choose words that are natural for you to say. Make sure you are not talking too fast, that you are speaking clearly, that you are explaining any jargon, and that you are ready for transitions.

You want to identify and knock out “um”, “ah”, “as you know”, “obviously”, and other verbal tics. You need to pause for a couple of moments between slides to signal transitions, and it’s fine to be quiet for longer and let people read/absorb. You might not guess it from my blog posts, but I am a big fan of cutting the fat to make what’s left look better.

If you have not been in front of people yet, this might be a time to get a couple friends together and do karaoke, however well or badly you might do, to experience the queasy stomach feeling in a safer place.

Before the Conference

Aim to have your slides done two weeks before the conference. You will continue to tweak them afterwards, but you probably need to send them in for posting, to preserve the time for fiddling and a couple more practice runs, and reduce last minute-induced crapness. It’s also a good idea to make sure you know how to make your laptop project, have any necessary adapters, etc. You might want to check your slide resolution and make sure it is a common projector resolution.

An aside on slides – some people horde them, many people ask for them, very few actually every look at them. Some conferences want attendees to get them from their site. Some people protect their slides. I put my CAST 2014 presentation on Slideshare, even though I have given other versions of it since and will again. AST also posted a video of it.

Your job is to be generous to everyone that wants to learn from you, responsive to the conference that gave you the opportunity to be heard, and confident and forthright about what’s yours and what isn’t.

Your Session

You will be nervous as your session approaches. Trust your preparation, and channel your energy into practice/last minute slide tweaks if you must, but you are better off not making decisions under duress.

Conferences are valuable learning and networking opportunities. I’ve met many of my favorite people at them. Go to sessions and support other speakers. Volunteer for the conference if they need any help.

The wrong way to prep for your session
The wrong way to prepare for a session

Spend as much time as you can with your fellow conference goers, and see how much you learn from people talking about what works in other contexts. Just don’t stay out too late the night before your session, and don’t get too “dehydrated”.

The session before yours, you will not be paying much attention, so avoid being a distraction by preparing somewhere else. Make sure you have enough water, make sure you don’t have too much, that your laptop is charged, your phone is silent and won’t vibrate, you have a dozen business cards handy for your new fans, and any demos work.

Go to your session’s room as soon as the previous session is over. You need the length of the break to set up your laptop, test A/V and slide clicker, find a clock you can see while you talk, put your phone in your bag, get one glass of water, feel really nervous, and then breathe. I still get butterflies, but it’s a lot better than it used to be. The 4-7-8 technique might help.

Make eye contact, smile, and say hello with each person as they come in, if you can. Accept that you’ve done what you can to prepare, and that you’ll do the best you can. Force yourself to start slow. FADE OUT.

FADE IN: Your room is clapping, and then filling out enthusiastic evaluations. Be gracious to everyone who comes and talks to you after.

Wrapping Up

Jot down any notes you have for yourself. Ask any friends in the room for their feedback.

Turn your phone back on. Star any tweets you like about your session and #followback your new twitter followers.

Congratulate yourself for making it to the other side. Allow yourself more “enjoyment of the conference” that evening. Thank the conference organizers. Add the talk to your LinkedIn profile.

Tell your boss how grateful you are for the support to attend and speak. Look for any local Tester Meetups that you could bring the conference session to. Look at what other conferences you might want to try next; now that you’ve given this talk once, you can improve it.

A few weeks after the conference, watch for speaker evaluations, and take any feedback. Remember that not everyone scores the same way; some will score 3 out of 5 as perfectly respectable. There will be some empty glowing reviews, and almost always, one negative review that after closer reading, you are not sure they were actually in your session.

Whew! It was a long journey, but we got here. Also, it takes a while to prepare for a conference.

In Part 4, I’ll talk about why the subject of diversity matters to me personally.

Speaking Easy, Part 2: You Can Do It!

This is Part 2 of a series of blog posts on volunteering to mentor speakers at testing (and other) conferences for Speak Easy. Part 1 is here.

In case you didn’t get it in the title…

You Can Do It!

You, yes you!

Ryan wants you to speak
Ryan wants you to speak

Really, you can. Hopefully not much more than about 220 days a year, you go to work and do a job. While you are doing your work, you are collecting experiences, researching and implementing methods, getting practice, and breathing in the how and why of what you do. You are accumulating hands-on knowledge every day, and improving at your craft. You know a thing or two about a thing or two, and the world might benefit from you sharing it.

You might look at who speaks at a conference, and think that you don’t match up for some reason. In terms of presentation skills, that might or might not be true at first, but you should never assume that you have less expertise in your subject matter. Many of these speakers have less experience than you at the things you work on. They may have seen more contexts, but they don’t know yours as well as you do. We need fewer blowhards describing the wildly successful implementation of processes they’ve spent talking to managers on the periphery of for a few days, and more sharing by practitioners of techniques and reporting on experiences.

You might not have an obvious subject in mind to talk about when you decide that you want to try speaking. That’s OK. Read conference programs, attend conferences, talk with peers, do some research – you can find something that will spark you. You only need a rough draft of a couple paragraphs of abstract to get started with research and asking for feedback. You don’t have to go all the way through with submitting – you can wait until next year or try a different conference.

Plus, you have these factors working for you in how conferences select speakers:

  1. There are usually not that many submissions. Often, conferences are scrambling to fill the number of speaking slots they have. When you see experienced speakers doubling and tripling up on track sessions, or you see a conference repeatedly publicizing approaching deadlines, you have evidence of this.
  2. There are submissions that are well, not as good as others. The obvious corollary to there not being many submissions is that there are not many abstracts being turned away. Near the end, what gets accepted is helped by the need to pad out a schedule.
  3. Submissions are just summaries. Professional authors will tell you that the best way to sell a book is “on spec”: a sample chapter, and some description of what the book is about. They don’t have to write the whole book until they’ve sold it. Conference presentations work the same way. More on conference submissions later.

hey-you-can-do-itNet-Net: Your proposal doesn’t have to compete with the most compelling submissions from the most experienced speakers. It only has to compete with the last set of submissions that are accepted, or an experienced speaker’s third session.

This all varies, of course, usually down to the track level for conferences that try to maintain subject lines over their conference days. I’ve been turned down for conferences I really wanted to speak at. I’ve also been asked to submit for conferences I had not heard of before I was asked, presumably because they saw my name on other conference programs.

Reviewers and conference staff want to program the best conference they can for attendees, and want great content. New blood is necessary to keep conferences vital and interesting. The old boys network is sort of a thing, but not as prevalent as you might think. Conferences want to have high energy, and that tends to come from people who are still doing the work, not just talking about the work. Killer ideas matter much more than good slides and patter.

So, study what other conference session descriptions looks like, then write three to five compelling paragraphs about what you want to talk about, tailoring it to the format of what you see for previous conferences. Here’s the “STARry” larger test conference programs to look at, here is STPCon, and here are some CAST conferencesall linked in a row. Maybe Let’s Test? If you are interested in writing a paper too, you could look at PNSQC. There are dozens more conferences, all around the world, and I apologize to the volunteers that keep all of these conferences running for not including every one. There are people who maintain links for the various software testing conferences, and you can use those to see more. There are many hundreds of presentations every year, and most will need to be replaced next year. Why not you?

Work in a local document – you don’t want to compose in someone’s web form and then lose it, or have to remember what you sent in later. Plus, if you are not accepted, you may want to propose again to a different conference. Casually embedded idea: you might be able to find an experienced speaker and ask if you can co-propose with them.

Get feedback, polish it, make sure you’re happy before you submit, but remember that you only need to summarize your main points at this stage. You’ll have months between an accepted abstract and a completed talk, and you almost always can revise your abstract for the program, before the conference.

In terms of asking for help directly from the conference, I think it can be a good idea, with some qualifiers. As a reviewer, I don’t mind giving feedback on drafts when it asked for well ahead of deadlines, and I don’t mind coming up with a couple of provocative questions to help sharpen an abstract. I have also been in situations where I felt like I was being politicked for special consideration more than any genuine desire for feedback, and didn’t care for that. People will respond positively to honest seekers.

Maybe you get accepted, maybe you don’t. It hurts a little to get rejected in any context, but it’s going to be OK. Save your abstract, look for opportunities to do more with it (Meetups, etc), and dust it off when the next conference comes along. There are any number of reasons why your talk might be fine, but just not fit for that specific conference. It could be overlapping with someone else’s proposal, or there might be too many talks for that one track. You might not have missed it by much. Sometimes you can get feedback on why, but I’d recommend giving it a few days to cool off before you ask. Just don’t give up!

The responsibility of the reviewers to the conference and its attendees is to select the best content possible, but that is multi-dimensional. A diverse set of subjects, approaches, and experiences creates the richest program. Programs drive attendance, and momentum between conferences depends on successful conferences that people enjoy attending.

schneideryou-can-do-itIt’s hard work and it takes practice to become a good speaker. I couldn’t tell you anything about becoming a great one. But the opportunities to try are not so hard to come by, and you probably have something to talk about.

If you don’t want to do it, that’s fine. It’s not for everyone.

Just don’t decide there isn’t room for you – there is plenty, please come on in. We need you, and we’re really hoping to hear from you soon. How can I help?

In the next post, I will talk about how I develop conference presentations.

Managing Test Organization Transformation

Recently, I attended the 9th Workshop on Software Testing in Financial Services (STiFS9) in New York, hosted by Liquidnet. The Theme of the workshop was “Organizational Structure Models for Test Groups at Financial Firms.” One of the discussions that struck a rich vein of ideas was what is needed for successful transitions between types of organizational models in quality organizations.

While the experiences that the participants drew on tended to be with larger organizations in financial services industries, these ideas could be useful for any testing organization’s transformation – or maybe even outside of testing.  Specifically, we were talking about organizations that were moving from a centralized to decentralized testing organizational model – or from decentralized to centralized in organizations with dozens or even hundreds of testers.

STiFS is a LAWST-inspired peer workshop, meaning that the value of the workshop is the product of all of the participants in the workshop, which included Bernie Berger, Kaveri Biswas, Margaret Boisvert, Ross Collard, Joe Lopez, Mike Pearl, Don Pierce, Anna Royzman, Ben Weber, and myself. We captured pages and pages of ideas from these very smart and experienced testers and test managers. Bernie Berger, the CO of STiFS9 (and the driving force behind STiFS) has provided a report on the workshop here.

This post is not an official finding of STiFS9. It doesn’t include all of the discussion or notes and only represents my interpretation and reflection on what I heard. Other participants may have a different take on what was discussed and what it meant. This post is what it meant to me.

A decentralized testing organization, as I mean it here, has testers reporting into specific projects as part of the staff. This approach might be marketed as “(More) Agile“, though it is certainly not necessary for agile, Agile, Scrum, or Certified Pokemon Master processes.

Arguing Over User Stories
What I Think of When I Hear Scrum

A centralized testing organization usually has a reporting structure that includes most or all testers in an organization in one department, which allocates testers for providing testing services to specific projects. Testers report upwards in the testing department and may work on several different projects, depending on the needs and desires of the whole organization. One marketing term associated with this approach is “Testing Center of Excellence.”

2,857 Career Story Points
A True Centre of Excellence

For the testing organization’s transformation project to start successfully, there needs to be top-down support, from the executive levels of the company. This support can be demonstrated and described by clear communication of the organization’s commitment to the transformation and the goals of the transformation. These goals may be to respond to changing market conditions, updated regulations, concerns about the quality and speed of delivery in the organization, or some combination of these and other factors. They should be articulated as the mission that the transformation is designed to accomplish with clear objectives and sufficient justification.

This communication of goals is usually understood as an essential step for aligning the organization, but less obvious is the need for selling the transition to the organization without jargon or “business speak.” This transparency builds trust with employees, who need to feel safe during transitional periods, need to feel included in the process and planning, and to be optimistic about their jobs and the organization’s future after the transformation. Many people are uncomfortable with and even resistant to change; laying the groundwork will help them process the changes to come. Buy-in enlists people in the organization to support the transformation and its goals, and helps retain key employees through periods of uncertainty. This was called “bottom-up trust.”

This leads to another core requirement: an organization’s leadership may draw up a new org chart (and the new org chart should be clear to everyone), but that doesn’t explain *how* the organization changes successfully, can’t describe all of the details and requirements, or fully define the new roles. In fact, almost all of these details will not be understood and defined at the start of an organizational transformation, making broad buy-in even more essential.

The real hard work of solving problems and helping define new structures is likely to be undertaken by “Champions” – key contributors throughout the company who are influential and respected by their co-workers. Many of these champions should *not* be managers, so that their motivation and contributions are clear. Some influence could come from outside consultants, but that will vary by situation and consultant.

The champions should be known to and perhaps recruited by the person that owns the transformation – the transformation owner. This person must be able to solve technical and political problems, make decisions that stick, and manage an ever-mushrooming list of details. The organization should expect that they will get clear, credible, and authoritative communication on progress from this person – and receive it regularly. A trusted source of information helps keep the organization aligned, and may help counter-program the rumor mill that uncertain people will be paying extra attention to.

The transformation owner should drive collaboration and keep people aligned. One way to support this collaboration is impact analysis on stakeholders, to identify problems, and to help create patience during periods of lower efficiency and missed deadlines that will surely occur as the organization is changing direction. This allows the stakeholders to plan and adjust accordingly, by making it explicit that expectations should be renegotiated.

Communication is only one requirement of the role, though. The owner must be able to advocate upwards and across the organization for accountability and buy-in from everyone to the transformation mission – and get it. Short-term revenue and budget pressures will contend with the work needed to create new processes.

The transformation owner can sell leadership on the need for patience and tolerance for initial failures, and run pilot projects to help make the transformation smoother for the rest of the organization. Teams should be able to experiment – to test the transformation – and provide feedback. The willingness to adjust the plan based on this feedback increases the chance of success.

One thing that was noted as frequently missing in transformation plans was engagement with and consideration of risks. There are lots of different kinds of risks; by engaging and enumerating risks, mitigations can be found. Not only should success criteria be understood – failure criteria may be helpful, too! Frequently, rollback is not an option, but contingency planning may be very helpful in getting through rough patches, particularly if the “worst-case” scenario has been understood.

As the details of the plan are being built, sufficient effort should be allocated to workflow analysis. How does work get done today? How will it be done in the future? Some things will fall between the cracks, and the people who would have noticed previously may be doing different things at that point.

An important part of the transformation plan is knowledge transfer. A concept of an “Organizational Level of Knowledge” was described; the cumulative knowledge of the organization is an important asset, and its value should be recognized, and hopefully preserved. Process documentation is easy to identify, but the unwritten rules are harder. Who can really get something done in a given area? How do you put an announcement out on the grapevine? These are not issues that can get resolved with an increased training budget; they took years to develop and be discovered.

Enter beer.

Wine is fine, liquor might indicate a problem...
The Secret Sauce of Transformation

Whether it is used for team building, to provide opportunities for horizontal communication, or as a coping mechanism, there were broad and strong recommendations for using beer to create social situations for team building, casual/informal communication, and reducing stress levels. This touched off a series of comments about the critical effects on employees going through and adjusting to transformation.

Perhaps the most critical issue is having well-defined new responsibilities. People need to know what is expected of them, and they need to trust that they have clear guidelines. This helps them feel secure, helps makes reporting relationships clear, helps set expectations, and helps them adjust to their new position.

It was suggested that there should be additional focus given by HR (and overtime, if necessary) to understand and hellp mitigate impacts on specific people who are dealing with change. People working with new managers are more likely to take complaints or issues to HR instead of the manager; they need to feel that they are heard and their concerns are addressed. This also gives HR a chance to focus attention and effort on keeping key people such as star performers happy and productive.

One of the strong signifiers (or moment where it becomes real) is when people have to move their desks. Movers, phones, networking, etc. all have their own issues and schedules and will need to be coordinated. Offices and workspaces may need to be redesigned.

Sitting with the new team is important, but there are real impacts beyond a change of scenery. If the geographic location changes and people have to move, there should be some care in how to help them understand it. Even changes like an impact to the commute are going to hit harder when surrounded by uncertainty.

People will need training and coaching for new roles and processes. Agile was cited as a commonly experienced part of a transformation, but even if methodology isn’t changing, there are certainly other changes in workflows, and/or required domain/technical knowledge. Testers should feel supported and that there is patience and tolerance for them to skill up to their new roles.

Some of technical issues around testing were collected by the group. Maintenance of automation and preservation of code and test assets was reported as often overlooked. There is a need to resolve the ownership of shared test code and data – or a decision to abandon it. Test labs and equipment will have to find new owners, and it should be clear how to schedule shared resources. Some cleanup should take place after the project completes.

More subtle impacts include ownership and responsibility for compliance concerns and go-forward updates/forwards/changes/retirements of email aliases, message groups, and distribution lists. Managing configuration changes may need significant rethinking, and permissioning was mentioned as a specific problem that had been encountered.

Even when testing is broadly distributed, some testing functions such as Automation or Performance Testing may still be best supported horizontally. This should be considered as part of the plan, including how to request support, and who will lead these functions.

In addition to all of this input about planning, the group had suggestions for conducting the transformation project. Once the transformation project is underway, continuing updates against frequently occurring milestones will help maintain cohesion and focus. Phases of the project are helpful for keeping things on track.

The plan and progress should be highly visible, and some advocated for a metrics regime as part of understanding where the organization was before the transformation, how the organization is managing during the transformation, and how successful the transformation project was in improving the organization’s testing efficiency. There was a lot of energy around engaging with the measurements that would be used by the organization to evaluate the project. Some of the proposed uses of metrics included the awareness of strengths and weaknesses, and current efficiencies and costs. Skepticism about metrics led to a discussion about the need to avoid inherited bias and future gaming of the system by using clearly documented and tangible metrics.

For project management, phases were suggested to keep the steps small enough to manage. There should be frequent check-ins to discuss how things are going, with different groups of people. Milestones should occur regularly, so that efforts remain bite-sized. This also helps build in flexibility by providing opportunities for assessment and adjustment.

It is also important to celebrate reaching these milestones. Celebration of successes should not wait until the entire project is complete, but occur throughout to help build positive momentum and maintain visibility.

Even with frequent phase events, patience will be necessary, both with schedule and results. Significant changes are being implemented, and everyone involved has to give them a chance to succeed. A couple of knowing comments came up here: “Projects unravel in unexpected ways.” and “Don’t panic because things are going perfectly.”

This patience has to include some tolerance for failure, and patience with understanding and correcting failure. There will be some risk for people to speak up and identify failures and their causes; transformation is frequently a top-down exercise. There is likely to be some disagreement about whether something has failed, or about the reasons for failure when the failure is acknowledged. When the organization is stuck, those responsible for planning are likely to blame the execution.

Failures must be engaged with thoughtfully, so that the organization can learn from it and not repeat it. Managing failure is important for the safety of the organization, and everyone involved. There are first order costs on others such as stakeholders, but the testers are going be frustrated and potentially frightened by failures. The organization has to know when to re-evaluate the situation, when to put parts of the plan on hold, when problems are fixable or not, and – perhaps most importantly – to understand and expect that these will be tactical decisions made during the project, when more information is known.

Discussion about completing the transformation (or declaring it done) came up again and again. As one participant put it, “Sooner or later, you have to land the plane”. People need to understand that the new reality is here, and that they can exhale and focus on doing their new jobs well.