A participant on the agile-testing news group recently asked, “How can you plan for the unexpected?” He was asking about how you design negative tests, and his question reminded me of an exercise I use in my testing classes for doing just that.
This exercise uses some basic Risk Management techniques to come up with a set of tests that will expose potential failures related to a risk that the group decides is important to manage.
It is particularly useful as a kickoff activity prior to a round of exploratory testing with the whole team. It helps folks who usually think in terms of creating (developers, business analysts, product managers) shift their mental focus to thinking about what could go wrong and how to detect vulnerabilities.
The Nightmare Headline News Game
Assemble a group of folks who care about risks on the project. The group could include testers, developers, marketers, the project manager, tech support folks, and anyone else who might have insight into risk. A diverse group is best for prompting lots of ideas.
You will need flipcharts stocked with plenty of paper as well as an assortment of flipchart pens. (Index cards or sticky notes with sharpies can also work well for this.)
Step 1: Brainstorm a List of Serious Failures
Ask the group:
“Imagine you wake up one morning shortly after we’ve launched our software. You fetch the morning paper from your driveway and start scanning the headlines as you walk back into your house. There’s a huge, screaming headline on the front page, above the fold. It’s about our software, and it’s a disaster. What does the headline say?”
Write down responses on a flip chart.
For example, let’s imagine we’re working on an ecommerce site that has a catalog of products and a shopping cart. Some nightmare headlines might be:
- Shopping Cart Software Adds 58% Premium to Bill: Holiday Shoppers Outraged at the Overcharges
- Man Surprised by Shipment of 827 Garden Gnomes, says “But I Only Ordered One!”
- Billed, not Delivered: Software Glitch Causes Consumers to be Billed for Merchandise that Never Ships
- Security Hole in Shopping Cart Software Enables Hackers to Steal Credit Card Info
Encourage the group to have fun with this and embellish their headlines with details.
Step 2: Choose a Big Risk to Work on
Post the list of nightmare headlines where everyone can see them.
Ask the group to scan the list looking for a nightmare that stands out as:
(You want to focus your efforts on the risks that are likely and that software testing can help with.)
Once the group has had a chance to look through the list, ask for nominations. Choose one to start.
You could get fancy with multi-voting or an affinity exercise to distill the list down to the most important risks to manage at this point, but I don’t recommend it. Taking time to do so is likely to bring down the energy level in the room, and is unlikely to add significant value to the end result. It’s better to use any time available to analyze more risks using steps 3 and 4 than to winnow the list of risks down until it is perfect.
Step 3: Brainstorm Contributing Causes
Write the chosen failure at the top of a sheet of flip chart paper. Then ask the group: “What could possibly cause this problem?”
Write down the responses on the flipchart.
For example, let’s imagine we decide to work on, ‘Man Surprised by Shipment of 827 Garden Gnomes, says “But I Only Ordered One!”‘ Contributing factors might include:
- The shopper corrupted the data in the quantity number field (entered a letter, etc.)
- The shopper hit Refresh in the browser multiple times
- The front end of the web app corrupts the data
- The back end software has a bug that results in random quantity numbers being recorded in the database or reported on the order/picklist sent to the warehouse
Step 4: Refine Causes into Test Cases
Post the list of contributing factors where everyone can see them.
Ask participants to choose a contributing factor, then identify test cases that would reveal the problem if it were going to happen.
For example, “shopper corrupted the data” immediately suggests attempting to change the quantity to illegal values including negative numbers, decimal numbers, numbers in scientific notation, extremely large numbers, letters, punctuation, etc.
“Shopper hit Refresh” suggests a series of browser navigation-related tests including refresh, back, forward, history, and bookmarks.
“Front end corrupts data” suggests testing to verify that the web app sends the right data (the key/value pairs in the POST request are correct and the keys are unique) and that the server does the right thing with it (the values that make it into the database are correct).
Lather, Rinse, Repeat
As time and interest allows, choose another big risk from the list generated in Step 2, and analyze it using Steps 3 and 4. The more practice the team has at coming up with test cases based on plausible risks, the more they’ll do it even when they aren’t in a brainstorming meeting.
Note that if you don’t already have checklists of generic tests, these brainstorms are a great way to create them. You’re likely to find that the same kinds of tests come up over and over and over again.