There’s a discussion going on over on the software-testing discussion group about a customer’s delivery requirement that the software be handed over with an acceptance test script.
I want to illustrate my perspective on this topic with a short story. If stories aren’t your thing, skip to the end of the post. I make my real point there.
A Fictional Digression with Little Green Men, a Magic Transporter Stick, and a Jar of Mustard
It’s just after dusk. You’re hanging out on your back porch enjoying the evening.
Suddenly with a whir and a whine, an alien space ship lands in your back yard. Two little green guys pop out. At first, you hear a bizarre series of clicks and whistles, but soon you hear a mechanical voice. You guess it must be a translator device. The voice says.
“Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”
You aren’t sure you heard them right. This is too surreal. Mustard?
You stare at the green guys. They’re kind of roly-poly. Each one has five eyes on stalks on the top of its head, and five squat arms with a mass of tentacles coming out of the middle of their bodies. And they are green. Decidedly, definitely green.
“Little green men,” you mutter under your breath.
They repeat their message: “Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”
You shake your head to clear it.
One half of your brain is still boggling at the fact that there are aliens hanging out by your rhododendrons.
The other half of your brain is remembering the huge economy-sized jar of mustard you got the last time you were at MegaStuff. You weren’t ever going to finish it anyway.
“Yeah, hang on,” you say. You go to fetch the jar of mustard. After all, you are the neighborly, helpful sort.
You are also the careful sort. As you are rummaging in the pantry it occurs to you that you don’t know what this “advanced technology” is. What does it do? How does it work? Is it safe? What if this is a plot to get a human to press the Go button on a Planet Death Device?
You can’t be too careful, after all. Time to ask some questions.
You come back outside, holding up the jar of mustard. The aliens nearly dislocate their eye stalks staring at it. All ten eyes are gleaming at you. It’s disconcerting.
Undaunted, you start your questions. “Before I give this to you,” you say, “what is this advanced technology? What does it do?”
The aliens each swivel one eye stalk up toward your face. The other four eyestalks remain firmly rooted on the mustard jar.
“It’s a portable personal surface transporter,” says the mechanical voice. “It will take you wherever on Earth you want to go.”
You consider this for a moment. A portable personal surface transporter. That could be handy. You consider your 45 minute commute each way every day, the long drives to see family, and the last time you paid an arm and a leg for a cramped economy seat in the back of the plane because you promised yourself you’d make it off your home continent while you were still young.
You look at the jar of mustard in your hand. You consider the possibilities.
Seems like a more than fair trade. A jar of mustard for instant transportation anywhere? That’s even worth some risks.
“OK,” you say. You walk over to the aliens and thrust the jar of mustard toward them. You have to bend down to get close enough. One of the aliens seizes the jar from your grasp with three of its hands. You’re surprised at how easily it hefts the weight of the jar given its size. All five of its eyestalks are on the jar, waving around the lid. Its hands are a sea of motion as its tentacles move over the large plastic container.
You look at the other alien. It’s riveted on the jar too. “What is it about these guys and mustard?” you wonder.
Then you notice that the other alien is holding a stick with a ball on the end in one of his hands. On him, it’s the right size for a walking stick. For you, it’s a little larger than a magic wand. A magic wand with a bulbous end.
“Magic wand indeed,” you mutter under your breath.
The alien hands it to you. It’s heavier than it appeared. Smooth, polished. “How does it work?” you ask.
A short series of clicks and whistles emits from one of them, you can’t tell which. Then the mechanical voice says: “Tap the bottom to turn it on.” Both aliens still absorbed by the mustard. One is holding the bottom of the jar while the other unscrews the lid.
You tap the bottom of the stick part of the device. The ball on the top starts glowing.
“Now what?” you ask.
One alien eyestalk swivels in your direction. “Touch the top wherever you want to go.”
You look at the ball again. It’s now a globe. A blue-green globe. A model of the Earth.
“So I touch New Zealand and I’m there? Really?” you turn it trying to find New Zealand. Once found, your hand hovers over the country.
“Wait!” one of the aliens swivels all five eyestalks over to you. He has a hand poised over the mustard jar. “WAIT!”
He moves toward you, eyestalks waving wildly. You decide that’s probably an indication of panic. You stop moving your hand toward the globe.
“What?” you ask.
The alien shuffles over and peers at the globe then your hand. He reaches up with three of his arms and grasps your hand in his tentacles.
“Oh dear,” the mechanical voice now seems to be coming from the vicinity of your feet. “Your hands are too big. This won’t work. No. If you touch the globe you could end up anywhere. Anywhere. You might not end up in New Zealand at all. Your hand is so big you could even end up in Tasmania. More likely you’ll drown in the middle of the ocean. Can’t have that. I’d feel terrible. You’ll have to enter the geocode instead of touching the globe.”
The mechanical voice is flat, emotionless. But you sense concern and perhaps fear in the clicks and whistles that the alien is making.
“Enter the geocode?”
“Yes,” the alien nods his eyestalks. “Let me show you.”
The alien takes the stick and manipulates it with his tentacles. He disappears.
Then, from over the fence, you hear: “Over here!” You look. The alien is waving at you from across the neighbor’s yard. Then a moment later he’s back by your side.
“See,” says the alien, pointing to the stick. “You need to press the sequence of buttons here with the geocode of where you want to go.” He hands you the stick. You look at it. The stick appears smooth and featureless.
The alien is acting like this is the most natural thing in the world. You’re baffled.
“Enter the geocode?” you ask again, hoping he’ll explain in more detail.
In the meantime you look over at the other alien. He appears to be enjoying the mustard. A lot. Noticing your attention, the second alien nods an eyestalk in your direction. “This is good stuff,” he says. Two eyestalks swivel to his compatriot. “Dude, you have to try this.” He holds out the jar.
The alien who has been helpfully explaining the portable personal surface transporter to you shuffles over to his friend. He is now lost in the mustard. An arm reaches into the jar. His eyestalks relax, drooping slightly.
“Oh, that is good. It’s been too long. Glad we stopped by Earth!” Both the aliens turn back toward their ship.
“WAIT!” you shout. “We’re not done. I don’t know how to use this thing! I don’t know how to tell if it’s working right! How do I enter the coordinates? How can I tell if I entered them correctly? HOW DO I KEEP MYSELF FROM LANDING IN THE MIDDLE OF THE OCEAN?”
The aliens seem impatient, but the helpful one turns back. He shuffles back to you and grasps your hand again. Manipulating your fingers with his tentacles he touches your fingers to the stick. “Here,” he says. “We’ll go over to there together.”
You can feel slight, ever so slight, impressions on the stick. Under his guidance, you tap on them. As you tap each one, it pulses back, apparently indicating that it received the tap. Partway through the sequence, you see a light appear on the side of the stick. A few more taps and another light appears. One last tap, and the globe flashes briefly. The alien tugs at your finger, pulling it to the other side of the stick and presses it down firmly. The next thing you know you’re on the other side of the rhododendrons.
“And back,” he says. He guides you to repeat the actions, but this time with a different sequence of taps. Different lights appear on the stick. The globe flashes again. Suddenly you’re back in front of the alien ship.
You have the hang of manipulating the buttons on the device now. But how will you know what the sequences are for different areas? What do the lights mean? How will you know if something has gone wrong?
You realize the aliens have turned back to their ship again.
“WAIT!” you shout. “I STILL DON’T KNOW HOW TO OPERATE THIS THING!”
You watch ten eyes all roll at the same time on only two bodies. The effect is disconcerting. But you need answers. You hold your ground, fearful and angry. You do not want to try randomly entering sequences. Who knows where you could end up?
“Don’t you have some documentation or something?” you plead.
One of the aliens shuffles up the ramp into the ship, then returns faster than you thought possible with a stack of paper.
“Here,” he says. “It’s a translation. It won’t be perfect. But it tells you what you should expect.”
You scan the document. It contains instructions for tapping, but even more importantly, it contains expected results. You realize that the first light you saw on the stick indicated that the stick accepted the latitude. The second indicated it accepted the longitude. The flash on the globe, had you looked at it carefully, would have given you some idea of where the coordinates would be taking you. The final tap on the other side of the stick confirmed the destination and activated the device. You notice that there is a cancel button next to the confirm button.
The next time you look up, you realize that the aliens are already in their ship. A soft whine from their engines and they’ve taken off. A scrap of paper floats down to you. “So long and thanks for the mustard!”
You go inside to look up the geocode for New Zealand.
Back to the Point
Now, let’s imagine for a moment that you are the hero in our story.
Unless you have a death wish, you’re probably not going to just start testing that thing to see what it does. Not really. You’re not going to push any buttons unless you have a very clear idea of what the expected result is.
Instead, you’re going to follow the instructions in the documentation exactly. And you want to know what you should expect to happen each step of the way. If something is going wrong at any point, you want to know right away so you can minimize the damage.
That’s not really testing: you only take actions where you know exactly what to expect. Testing involves experimenting with unknowns.
To frame your expectations, you need information. But the information you need is different from what you find in a typical user guide.
It is, however, very much like what you would find in a scripted acceptance test:
Touch buttons – 3 6 . 8 6 1 9 7
See the 3rd blue light.
Touch buttons 1 7 4 . 7 7 4 1 6 6
See the 4th green light.
Touch the Destination button.
See a flash on the Globe at Auckland.
Touch the Confirm button.
Arrive in Auckland Domain, Auckland New Zealand.
See greenery.
Now let’s imagine that you work for a huge company that has bought a software package. It’s not exactly alien technology. It may not even be particularly advanced technology. But it’s important to the business or the business would not have purchased it.
Let’s say that you’re not the end user for this software. You are in IT. This is a system that the sales team has purchased. Since you’re part of central IT, and you drew the short straw this month, you are responsible for the successful roll out of this new software.
You don’t actually understand what it does. In fact, you don’t really understand your sales people or what they do. You think they might be aliens. But you know that it’s gonna be on your head if you screw this up. So you want to make sure that everything is operating as expected.
You also know that there are finicky details about how the software is deployed and configured that make our imaginary portable personal surface transporter stick look like an etch-a-sketch.
How are you supposed to know if you set up the software right? If it’s configured correctly? If it’s doing what it’s expected to do?
Note that you don’t want to test the software. Even if you had time to test the software (you don’t), it’s not your job to do so. Someone else in your company made the decision to buy the software, so you’re not evaluating it. And the vendor presumably already tested that the software does what they intended it to do, at least some of the time. You’re not trying to find problems. You just want to get it to work and present evidence to the sales team that their precious new sales automation solution is ready to go.
In short, you need to conduct a final acceptance test after you have everything set up and configured.
But you are not qualified to design an acceptance test for this package. You do not know the software. You do not understand what it does. Your job is to install it and make sure that it’s wired up correctly.
The sales team is not qualified to conduct that test. They’re not even trained on the new system yet. The committee that decided to buy the software isn’t qualified to conduct the test. Some of them can barely retrieve their email unassisted.
So you need the people who made the software to tell you what to do and what you should expect.
Back to the software-testing group discussion: this is why customers ask for an acceptance test script to be delivered with software.
It is a perfectly reasonable request. The customer is not stupid or lazy or incompetent. The customer is asking for a deliverable that they need in order to ensure that the product that they purchased is operating as expected in their environment: with their permissions and authentication and security schemes, connecting to their customized databases, on their network with packet filters and firewalls. They just need to know it’s working. And reading the user guide won’t help them figure that out. Not efficiently. So they want an acceptance test. Think of it as a diagnostic checklist.
Oh, and if your organization is doing ATDD, you’d already have an acceptance test script: you could just give them a prettied-up version of the natural language expectations without the automation.