How did we miss THAT?

“Oh goodness. How did I miss THAT bug?”

Over the years, I’ve asked myself that question numerous times.

I asked that question when another tester found a blazingly obvious, critical bug that I completely missed. (The answer: I spent too much time tinkering with an ineffective automated script I’d written, and too little time observing the actual behavior of the system. That’s the project where I learned a lot about how NOT to do test automation.)

I asked myself that question, repeatedly, when I participated on a project some years ago now where we shipped software that crashed left and right. (I’m still sorting out the answers to that last one. Catastrophic failures are almost never the result of a single, simple error. And this particular catastrophic failure represented failures at all levels in an organization that had, um, issues. But I digress.)

I asked the question again when I learned that a web site I tested had back-button problems. After all, I was sure I’d tested for that. And I had. But I hadn’t re-tested for it after a particular set of code changes that changed some operations from HTTP GETs to HTTP POSTs. Oops.

And I asked myself that question more recently when I learned that a system I worked on earlier this year failed to save a change, and also failed to report an error, when it encountered data misformatted in a particular way in one specific field. Badly formatted data is one of my specialties, and I couldn’t believe I forgot to test the particular case that resulted in the problem. But it turned out that I did, indeed, fail to test what would happen if you entered “” into a URL field instead of “”. In hindsight, it’s an obvious test. Another lesson learned.

I reflected on those missed bugs when a colleague, Sandeep, a test manager, recently wrote to say that he’s been asking himself “how did my testing team miss that?”

He decided to seek out patterns of testing problems by categorizing escaped bugs according to the hole(s) in testing that allowed the bug to slip through. The idea is to improve the test effort by figuring out the common causes behind escaped bugs.

My initial reaction was, “that makes sense.” If you can identify the top 20% of testing holes that let 80% of the bugs through, and you can make some serious improvements to the test effort.

And my next reaction was, “but be careful.” Sandeep’s intent is good: use lessons learned from escaped bugs to improve testing. However, asking “How did we miss that?” is perilously close to heading down the slippery slope to “How did JoeBob miss that?” to “It’s JoeBob’s fault.” Having talked to Sandeep, I know he’s not trying to play “pin the blame on the tester.”

So I suggested a small reframe.

Instead of categorizing escaped bugs by asking the question, “How did testing miss that?”, categorize them by asking the question, “How can we improve the probability that testing will find bugs like that in the future?”

It’s a subtle difference.

But the result of reframing the question is that instead of identifying categories as noun phrases like “insufficient test data,” we end up with imperative statements like, “add test data.” Those two categories may look almost identical, but only one is actionable. I can “add test data.” The statement prompts me to do something different next time. But “insufficient test data” only gives me something to regret. And regret won’t help me ship better software.

So how can you categorize escaped bugs to improve the test effort without falling into the blaming trap? Try an Affinity Exercise with the question, “What could we do differently next time to increase the probability that if we have another bug like this we’ll catch it in test?”

To prepare:

  1. Choose a team to participate in the activity. Affinity exercises can work with any number of people, but for this particular activity, I find a smaller group – say 3 to 5 people – works best. It’s a good idea to include people with diverse roles and skill sets.
  2. Set up a meeting time and place. Plan for the whole activity to take 2 hours. And arrange to meet in a place with plenty of table and/or wall space.
  3. Gather (or shop for) office supplies. You’ll need:

    • Index cards or sticky notes. Bigger is better. I like 5×8 cards or the SuperSticky 5×8 Post Its.
    • Felt-tip markers. I like Sharpies because they make consistently dark, readable marks. (Beware: Sharpies are permanent. Do NOT confuse your Sharpies and your White Board markers in the conference rooms. Facilities people get tetchy about such mix-ups.)
  4. Gather a list of escaped bugs you want to analyze. If you have a lot of escaped bugs, prioritize them and time box the exercise. (You probably won’t be able to analyze more than 50 in an hour, possibly less, so don’t print out a list of 500.)

In the Meeting:

  1. Review each bug with the team, asking: “What could we do differently next time to increase the probability that if we have another bug like this we’ll catch it in test?”
  2. Have participants write their suggestions on the cards/stickies, one idea per card, in the form of an actionable statement. The suggestions should complete the sentence, “In the next release/iteration/sprint, we can ______.” Tips:
    • Also ask the participants to make their suggestions as concrete and specific as possible. For example, instead of writing “add test data,” write “add titles with ampersands (&) to the test data.”
    • And ask the participants to stick to test-related actions, and avoid blaming individuals. “Revoke NancySue’s checkin priviledges” is not an acceptable suggestion.
  3. When you’ve reviewed all the bugs, or when an hour has passed, stop reviewing bugs. (If you still have lots of bugs to go and want to continue analyzing after an hour, stop anyway. Finish the rest of the exercise – the grouping. When you’ve worked through the whole process, if you still think more analysis would help, you can always do the exercise again.)
  4. Gather all the cards/stickies, and lay them out on a large work surface: a table, the walls, or even the floor can all work well.
  5. Sort through the cards/stickies as a team. The cards are now owned by the team, and everyone should take a hand in organizing them. Encourage participants to move cards that seem alike together so they are stacked together. Continue until the team agrees that it’s satisfied with the stacks of cards/stickies.
  6. Ask the team to give meaningful names to the stacks of cards/stickies. This is the part of the activity where will generate the more abstract categories like “add test data.”

The result of this exercise is a list of categories for improving the testing effort that we can then use to determine which kinds of improvements will have the biggest bang for the buck. And the best part is that the list emerged from the actual problems your software has had in the field rather than being some arbitrary list of theoretical “improvements” based on someone else’s unrelated experience.

But wait; you’re not done. Now that you’ve created a first draft list of categories, test it. (Did I mention I’m Test Obsessed?) Choose a different set of escaped bugs, and assign each to one or more categories from the list. Notice how easy or hard it seems to find a category for each bug. This will give you a lot of feedback about how well the category list will work in practice. You may find the team needs to spend some additional time iterating on the list.

Once you’re satisfied with your list of categories, you can run the numbers to see how many bugs are in each category. Then you can create a Pareto diagram of the results to see what 20% of the improvement opportunities on your list will result in an 80% improvement. Now you can truly leverage escaped bug information into concrete actions that will make the test effort more effective.

Over time, as you try to use that original list to categorize reports of new bugs in the field, you will probably find that the list becomes less and less relevant. I hope so, anyway. It indicates that the improvement efforts are working, that the team has improved the test effort.

That’s when you know it’s time to do the process all over again to classify the next generation of escaped bugs.

It's Like Virtually Being There…

Earlier this year, I worked with an XP team where we had remote team members using a combination of Skype, webcams, and virtual desktop to give them a virtual presence in the team room. I was surprised how well it worked.

Sure, it was a little weird the first time I remote-paired and my pair did the driving: code appeared on the screen as if by magic. Fortunately, my pair was good at talking about his thought process, and his voice filled in the void left by the lack of visible physical cues like a hand hovering over a mouse or restless shifting in the chair.

Prior to that project, I thought it would be extremely difficult to integrate remote employees on an Agile team. But with an always-on Skype connection and a liberal scattering of speakers and microphones, the remote team members were truly present, albeit virtually. They were even there for the hallway chatter. You’d be chatting about something casually with another team member, like ways to structure validations in Ruby on Rails, and a disembodied voice would chime in – just like any other coworker who happened by and who had an opinion.

I began to think of our remote colleagues as coworkers with the curious disability of not having a corporeal body. Those of us in the team room had to make reasonable accommodations like ensuring we kept Skype running on certain boxes, and kept microphones and speakers plugged in. But the remote team members’ ability to contribute was not hampered by the lack of a body. They participated like anyone else.

That’s when I realized that the technology to support remote collaboration has become cheap enough, reliable enough, and ubiquitous enough that people in disparate locations actually can work together remotely as effectively as when they’re in the same room, at least under certain circumstances.

And that makes me wonder if the technology is now advanced enough to support a virtual version of the kind of training I like to do: experiential, interactive, hands-on training.

I should back up a step and explain that I’m not new to virtual training. I’ve been a webinar speaker. And I’ve been a participant in online classes. I know there are all kinds of solutions out there for presentation-based training. But most of the technologies I’ve seen assume there’s a slide deck. The tools do support interactivity, but mostly as Q&A instead of deeper interactive discussions or exercises.

I want something more. I want to be able to do simulations and experiential exercises and real debriefs in a virtual online classroom and have it feel as interactive and collaborative and spontaneous as in a conference room in real-space.

And I think, perhaps, the technology is getting to the point where we can.

Others think so too. James Bach has been experimenting with online virtual training. He recognized that the technology was ready before I did. And it’s going so well he’s planning to do more of it.

So…all this is a long lead in to say that I’ve started experimenting with online training. So far my experiments have been promising but limited. And I’m ready to do more.

If you’re interested in participating in any of my experiments, drop me an email or a note in the comments. I’m looking for volunteers who want to subject themselves to a small sampling of my training and exercises – free – in exchange for feedback. (You will need a broadband connection, webcam, microphone, speakers, and up-to-date browser. You may need a flexible schedule since I want to experiment with 4 – 6 participants, and finding a time that works for everyone can be a challenge. And you’ll need some patience. I’m still experimenting.)