Acceptance Tests as a Customer Deliverable

There’s a discussion going on over on the software-testing discussion group about a customer’s delivery requirement that the software be handed over with an acceptance test script.

I want to illustrate my perspective on this topic with a short story. If stories aren’t your thing, skip to the end of the post. I make my real point there.

A Fictional Digression with Little Green Men, a Magic Transporter Stick, and a Jar of Mustard

It’s just after dusk. You’re hanging out on your back porch enjoying the evening.

Suddenly with a whir and a whine, an alien space ship lands in your back yard. Two little green guys pop out. At first, you hear a bizarre series of clicks and whistles, but soon you hear a mechanical voice. You guess it must be a translator device. The voice says.

“Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You aren’t sure you heard them right. This is too surreal. Mustard?

You stare at the green guys. They’re kind of roly-poly. Each one has five eyes on stalks on the top of its head, and five squat arms with a mass of tentacles coming out of the middle of their bodies. And they are green. Decidedly, definitely green.

“Little green men,” you mutter under your breath.

They repeat their message: “Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You shake your head to clear it.

One half of your brain is still boggling at the fact that there are aliens hanging out by your rhododendrons.

The other half of your brain is remembering the huge economy-sized jar of mustard you got the last time you were at MegaStuff. You weren’t ever going to finish it anyway.

“Yeah, hang on,” you say. You go to fetch the jar of mustard. After all, you are the neighborly, helpful sort.

You are also the careful sort. As you are rummaging in the pantry it occurs to you that you don’t know what this “advanced technology” is. What does it do? How does it work? Is it safe? What if this is a plot to get a human to press the Go button on a Planet Death Device?

You can’t be too careful, after all. Time to ask some questions.

You come back outside, holding up the jar of mustard. The aliens nearly dislocate their eye stalks staring at it. All ten eyes are gleaming at you. It’s disconcerting.

Undaunted, you start your questions. “Before I give this to you,” you say, “what is this advanced technology? What does it do?”

The aliens each swivel one eye stalk up toward your face. The other four eyestalks remain firmly rooted on the mustard jar.

“It’s a portable personal surface transporter,” says the mechanical voice. “It will take you wherever on Earth you want to go.”

You consider this for a moment. A portable personal surface transporter. That could be handy. You consider your 45 minute commute each way every day, the long drives to see family, and the last time you paid an arm and a leg for a cramped economy seat in the back of the plane because you promised yourself you’d make it off your home continent while you were still young.

You look at the jar of mustard in your hand. You consider the possibilities.

Seems like a more than fair trade. A jar of mustard for instant transportation anywhere? That’s even worth some risks.

“OK,” you say. You walk over to the aliens and thrust the jar of mustard toward them. You have to bend down to get close enough. One of the aliens seizes the jar from your grasp with three of its hands. You’re surprised at how easily it hefts the weight of the jar given its size. All five of its eyestalks are on the jar, waving around the lid. Its hands are a sea of motion as its tentacles move over the large plastic container.

You look at the other alien. It’s riveted on the jar too. “What is it about these guys and mustard?” you wonder.

Then you notice that the other alien is holding a stick with a ball on the end in one of his hands. On him, it’s the right size for a walking stick. For you, it’s a little larger than a magic wand. A magic wand with a bulbous end.

“Magic wand indeed,” you mutter under your breath.

The alien hands it to you. It’s heavier than it appeared. Smooth, polished. “How does it work?” you ask.

A short series of clicks and whistles emits from one of them, you can’t tell which. Then the mechanical voice says: “Tap the bottom to turn it on.” Both aliens still absorbed by the mustard. One is holding the bottom of the jar while the other unscrews the lid.

You tap the bottom of the stick part of the device. The ball on the top starts glowing.

“Now what?” you ask.

One alien eyestalk swivels in your direction. “Touch the top wherever you want to go.”

You look at the ball again. It’s now a globe. A blue-green globe. A model of the Earth.

“So I touch New Zealand and I’m there? Really?” you turn it trying to find New Zealand. Once found, your hand hovers over the country.

“Wait!” one of the aliens swivels all five eyestalks over to you. He has a hand poised over the mustard jar. “WAIT!”

He moves toward you, eyestalks waving wildly. You decide that’s probably an indication of panic. You stop moving your hand toward the globe.

“What?” you ask.

The alien shuffles over and peers at the globe then your hand. He reaches up with three of his arms and grasps your hand in his tentacles.

“Oh dear,” the mechanical voice now seems to be coming from the vicinity of your feet. “Your hands are too big. This won’t work. No. If you touch the globe you could end up anywhere. Anywhere. You might not end up in New Zealand at all. Your hand is so big you could even end up in Tasmania. More likely you’ll drown in the middle of the ocean. Can’t have that. I’d feel terrible. You’ll have to enter the geocode instead of touching the globe.”

The mechanical voice is flat, emotionless. But you sense concern and perhaps fear in the clicks and whistles that the alien is making.

“Enter the geocode?”

“Yes,” the alien nods his eyestalks. “Let me show you.”

The alien takes the stick and manipulates it with his tentacles. He disappears.

Then, from over the fence, you hear: “Over here!” You look. The alien is waving at you from across the neighbor’s yard. Then a moment later he’s back by your side.

“See,” says the alien, pointing to the stick. “You need to press the sequence of buttons here with the geocode of where you want to go.” He hands you the stick. You look at it. The stick appears smooth and featureless.

The alien is acting like this is the most natural thing in the world. You’re baffled.

“Enter the geocode?” you ask again, hoping he’ll explain in more detail.

In the meantime you look over at the other alien. He appears to be enjoying the mustard. A lot. Noticing your attention, the second alien nods an eyestalk in your direction. “This is good stuff,” he says. Two eyestalks swivel to his compatriot. “Dude, you have to try this.” He holds out the jar.

The alien who has been helpfully explaining the portable personal surface transporter to you shuffles over to his friend. He is now lost in the mustard. An arm reaches into the jar. His eyestalks relax, drooping slightly.

“Oh, that is good. It’s been too long. Glad we stopped by Earth!” Both the aliens turn back toward their ship.

“WAIT!” you shout. “We’re not done. I don’t know how to use this thing! I don’t know how to tell if it’s working right! How do I enter the coordinates? How can I tell if I entered them correctly? HOW DO I KEEP MYSELF FROM LANDING IN THE MIDDLE OF THE OCEAN?”

The aliens seem impatient, but the helpful one turns back. He shuffles back to you and grasps your hand again. Manipulating your fingers with his tentacles he touches your fingers to the stick. “Here,” he says. “We’ll go over to there together.”

You can feel slight, ever so slight, impressions on the stick. Under his guidance, you tap on them. As you tap each one, it pulses back, apparently indicating that it received the tap. Partway through the sequence, you see a light appear on the side of the stick. A few more taps and another light appears. One last tap, and the globe flashes briefly. The alien tugs at your finger, pulling it to the other side of the stick and presses it down firmly. The next thing you know you’re on the other side of the rhododendrons.

“And back,” he says. He guides you to repeat the actions, but this time with a different sequence of taps. Different lights appear on the stick. The globe flashes again. Suddenly you’re back in front of the alien ship.

You have the hang of manipulating the buttons on the device now. But how will you know what the sequences are for different areas? What do the lights mean? How will you know if something has gone wrong?

You realize the aliens have turned back to their ship again.

“WAIT!” you shout. “I STILL DON’T KNOW HOW TO OPERATE THIS THING!”

You watch ten eyes all roll at the same time on only two bodies. The effect is disconcerting. But you need answers. You hold your ground, fearful and angry. You do not want to try randomly entering sequences. Who knows where you could end up?

“Don’t you have some documentation or something?” you plead.

One of the aliens shuffles up the ramp into the ship, then returns faster than you thought possible with a stack of paper.

“Here,” he says. “It’s a translation. It won’t be perfect. But it tells you what you should expect.”

You scan the document. It contains instructions for tapping, but even more importantly, it contains expected results. You realize that the first light you saw on the stick indicated that the stick accepted the latitude. The second indicated it accepted the longitude. The flash on the globe, had you looked at it carefully, would have given you some idea of where the coordinates would be taking you. The final tap on the other side of the stick confirmed the destination and activated the device. You notice that there is a cancel button next to the confirm button.

The next time you look up, you realize that the aliens are already in their ship. A soft whine from their engines and they’ve taken off. A scrap of paper floats down to you. “So long and thanks for the mustard!”

You go inside to look up the geocode for New Zealand.

Back to the Point

Now, let’s imagine for a moment that you are the hero in our story.

Unless you have a death wish, you’re probably not going to just start testing that thing to see what it does. Not really. You’re not going to push any buttons unless you have a very clear idea of what the expected result is.

Instead, you’re going to follow the instructions in the documentation exactly. And you want to know what you should expect to happen each step of the way. If something is going wrong at any point, you want to know right away so you can minimize the damage.

That’s not really testing: you only take actions where you know exactly what to expect. Testing involves experimenting with unknowns.

To frame your expectations, you need information. But the information you need is different from what you find in a typical user guide.

It is, however, very much like what you would find in a scripted acceptance test:

Touch buttons – 3 6 . 8 6 1 9 7
See the 3rd blue light.
Touch buttons 1 7 4 . 7 7 4 1 6 6
See the 4th green light.
Touch the Destination button.
See a flash on the Globe at Auckland.
Touch the Confirm button.
Arrive in Auckland Domain, Auckland New Zealand.
See greenery.

Now let’s imagine that you work for a huge company that has bought a software package. It’s not exactly alien technology. It may not even be particularly advanced technology. But it’s important to the business or the business would not have purchased it.

Let’s say that you’re not the end user for this software. You are in IT. This is a system that the sales team has purchased. Since you’re part of central IT, and you drew the short straw this month, you are responsible for the successful roll out of this new software.

You don’t actually understand what it does. In fact, you don’t really understand your sales people or what they do. You think they might be aliens. But you know that it’s gonna be on your head if you screw this up. So you want to make sure that everything is operating as expected.

You also know that there are finicky details about how the software is deployed and configured that make our imaginary portable personal surface transporter stick look like an etch-a-sketch.

How are you supposed to know if you set up the software right? If it’s configured correctly? If it’s doing what it’s expected to do?

Note that you don’t want to test the software. Even if you had time to test the software (you don’t), it’s not your job to do so. Someone else in your company made the decision to buy the software, so you’re not evaluating it. And the vendor presumably already tested that the software does what they intended it to do, at least some of the time. You’re not trying to find problems. You just want to get it to work and present evidence to the sales team that their precious new sales automation solution is ready to go.

In short, you need to conduct a final acceptance test after you have everything set up and configured.

But you are not qualified to design an acceptance test for this package. You do not know the software. You do not understand what it does. Your job is to install it and make sure that it’s wired up correctly.

The sales team is not qualified to conduct that test. They’re not even trained on the new system yet. The committee that decided to buy the software isn’t qualified to conduct the test. Some of them can barely retrieve their email unassisted.

So you need the people who made the software to tell you what to do and what you should expect.

Back to the software-testing group discussion: this is why customers ask for an acceptance test script to be delivered with software.

It is a perfectly reasonable request. The customer is not stupid or lazy or incompetent. The customer is asking for a deliverable that they need in order to ensure that the product that they purchased is operating as expected in their environment: with their permissions and authentication and security schemes, connecting to their customized databases, on their network with packet filters and firewalls. They just need to know it’s working. And reading the user guide won’t help them figure that out. Not efficiently. So they want an acceptance test. Think of it as a diagnostic checklist.

Oh, and if your organization is doing ATDD, you’d already have an acceptance test script: you could just give them a prettied-up version of the natural language expectations without the automation.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

13 Responses to Acceptance Tests as a Customer Deliverable

  1. Chris McMahon July 22, 2010 at 4:56 pm #

    FWIW, we did exactly this when releasing location engine software to big telecom companies. This was stuff ran that ran deep inside proprietary networks. And since the same engine that allows you to be connected to the nearest Domino’s Pizza also drives wireless 911 calls, it was critical that it be installed and configured correctly. We put a lot of effort into shipping good acceptance tests for that code.

  2. Karol Magda July 22, 2010 at 8:55 pm #

    But then sending guy that knows something about the software alongside with the software may be just better way of doing things.

  3. Tola Anjorin Jnr July 22, 2010 at 10:24 pm #

    Please what does an acceptance test script look like and how will customers use it, if they cant even work with the software?

  4. ehendrickson July 22, 2010 at 10:32 pm #

    It’s a step by step script that describes the actions to take and the expected results. (See the fictional example in the blog post.) And it’s not the only documentation: I would expect that in this kind of context there would also be an installation/setup guide and a user manual. Armed with all that documentation, a technically savvy person should be able to install the software and follow the script to verify that the software is doing what it is supposed to do.

  5. James Bach July 22, 2010 at 11:35 pm #

    “It is a perfectly reasonable request. The customer is not stupid or lazy or incompetent.”

    It’s only a reasonable request in the expanded sense of “there are no stupid questions.” People can ask for whatever they think they want, and perhaps they should. But it’s not a reasonable *expectation* on their part that a shallow and poorly understood “acceptance test” should provide any significant confidence that we know anything about how the product “works.”

    Such acceptance tests are primarily a ceremonial activity, akin to praying. I’m not knocking prayer, but prayer is about managing your own feelings about life. It’s not a reliable way to get things done.

    It is not a responsible act to perform a test you don’t understand and then use that as a warrant for saying “the product works.” Even if you do understand the test, if it’s a poor test, it’s still no warrant.

    There are good alternatives to such practice and we ought to know what they are and advise our clients accordingly.

    To bring it back to the aliens… No, I’m not going to operate this device at all. In fact, I’m not going to take the mustard deal. It’s a business arrangement that I don’t understand, and the alien’s craving for mustard and apparent inability to get it from other places makes me deeply suspicious.

    I realize that many people feel powerless at work. But at least, as an aspiration, we can move toward personal responsibility and technical ethics.

    — James

  6. ehendrickson July 23, 2010 at 12:19 am #

    Stakeholders have questions; testers have answers.

    In this context, we have an IT department performing an acceptance test of a package that they installed and configured at the behest of someone else who made the purchasing decision.

    The question at hand is not whether or not the package “works” in the broad sense of the word. In this situation, the stakeholder is not asking the tester to second-guess their decision to buy the product. And that means the stakeholder probably will not appreciate a comprehensive report of everything the tester thinks is wrong with the product. Indeed, the stakeholder is likely to view such a report as a waste of time at best. In a politically charged organization the stakeholder may see a comprehensive test report on the product as an attack on their authority to make the purchasing decision.

    So I’ll suggest that the real question here is whether or not the package has been installed and configured correctly. For that type of test, I’ll suggest that the product “works” if, for each of a given, pre-defined set of stimuli, the software offers the response that the software vendor expects.

    That’s what an acceptance test script should specify: a pre-defined set of conditions and actions along with the corresponding expected responses.

    As you say, such a document is not a sufficient guide for assessing the product’s fitness for use.

    But that’s not the goal of the kind of acceptance testing I just described. “Is this thing installed right?” is the software equivalent to “<tap, tap, tap> Is this thing on? Can you hear me?” And it’s a perfectly valid question in the right context. Thus answering the question is not merely ceremonial. Nor is it irresponsible or unethical to create a script to help someone answer that question.

  7. Rick Scott July 23, 2010 at 4:13 am #

    Yes! I think we’re running up against the fact that “acceptance test” is a bit of a misnomer, especially here. It’s more of a “rejection test”. You don’t know that the system perfectly suits your needs if the test passes, but you do know that something is not working correctly if the test fails.

  8. Markus Gaertner July 23, 2010 at 6:04 am #

    You’re disagreeing about the purpose. While Elisabeth raises the point of documentation purpose for the customer on how to operate the product and what to expect, James and Rick are raising the point, that it’s not a real test. You both are right, but you disagree about the purpose of the Acceptance Test in first place.

  9. ehendrickson July 23, 2010 at 2:34 pm #

    Hi Markus,

    You’re right, of course.

    Though I think part of the problem is that like every other term in testing, the phrase “Acceptance Testing” is overloaded.

    With Acceptance Test Driven Development, the Acceptance Tests are checks that verify the system meets expectations.

    But when a Product Owner does Acceptance Testing, it’s to determine whether or not they believe that the feature as implemented conforms to both the letter and spirit of the expectations of the story. So that kind of Acceptance Testing absolutely should be a more thorough exercise of the system and would necessarily involve exploratory testing. When I have seen Product Owners who did not have the time or inclination to do a more thorough job of exploring the implemented stories before accepting, they usually regretted having accepted the story without enough information. (A little like buying a house without inspecting it.)

    So just within Agile the phrase Acceptance Testing has multiple meanings. Add in all the other meanings of Acceptance Testing from over the years and we have quite a mess.

    That’s why I’ve been adding in the rest of the context in the discussion. It’s important to be clear about which kind of Acceptance Testing we’re talking about. And back on the original topic from software-testing: my suspicion is that the request was for an Acceptance Test as in “does it work as expected *here*?” than “do we find this product to be acceptable?”

    Elisabeth

  10. ehendrickson July 23, 2010 at 2:37 pm #

    Love the term “rejection test.” 🙂

  11. Michael Bolton July 27, 2010 at 7:19 pm #

    Disconnected reactions:

    1) When I read about the aliens and their cries of “MUSTARD”, the first thing that came to my mind was a huge two-or-three-letter consultancy and their cries of “MONEY! We will trade you our advanced technology for MONEY”. Somehow in this story the tester decides to fetch a pile of [ money | mustard ]. That’s very different from the situation that you propose later, in which you are dealing with someone else’s (probable) mess. The confusion between who is the stakeholder, who is the tester, and what constitutes acceptability leads to much distress when we talk about “acceptance tests”.

    “You don’t actually understand what it does. In fact, you don’t really understand your sales people or what they do. You think they might be aliens. But you know that it’s gonna be on your head if you screw this up. So you want to make sure that everything is operating as expected.”

    If that’s the case (as, sadly, it is for so many testers in this world), you need a new job. It’s unreasonable for anyone to hold a job in which you’re expected to know “everything is operating as expected” despite a profound ignorance of what “everything”, “operating”, and “expected” means. After you’re been given the “acceptance test”, you still don’t understand how the product works (see below).

    Absolutely everyone up the chain has abdicated his or her responsibility when, in a context like this, they put the blame for product failure on the tester. In this case, you don’t need an acceptance test. You need lessons in self-defense. You probably also need the services of a good recruiter to help you find a testing job.

    2) I’ve been pointing out since at least 2006 that automated, ATDD-style acceptance tests have been incorrectly named. I published a paper on acceptance testing at the PNSQC, and “rejection test” was exactly the term that I used in both presentations. One enormous problem that I identified in that paper is that “acceptance” describes a relationship. A product that is “acceptable” to the vendor may be in no way acceptable to the customer, even in cases where the vendor and the customer have agreed on representative examples of how the product should work. Such examples are explications of specifications. They don’t demonstrate “acceptability” any more than an ISTQB exam and certification demonstrate that a person is an “acceptable” tester.

    3) “Stakeholders have questions; testers have answers.”

    Checkers have answers. Automated acceptance tests have answers. Testers have answers too, but testers have something else: the capacity to ask questions about possible threats to stakeholders’ value. Let me give you an example:

    Touch buttons – 3 6 . 8 6 1 9 7
    See the 3rd blue light.
    Touch buttons 1 7 4 . 7 7 4 1 6 6
    See the 4th green light.
    Touch the Destination button.
    See a flash on the Globe at Auckland.
    Touch the Confirm button.
    Arrive in Auckland Domain, Auckland New Zealand.
    See greenery.

    What happens when I make a mistake on data entry? Can I correct it?

    After I enter the first set of numbers, the third blue light does come on. But so does the second, and so does the first. Is that a problem, or something that I should expect?

    I realize that there’s ambiguity here between south latitude (given the New Zealand reference) and north latitude; east longitude and west longitude Yet it’s not at all clear to me when I should expect to arrive in New Zealand and when I should expect be dumped in the North (or in one case, South) Pacific.

    When I see greenery, that’s nice; not only did the aliens get the latitude and longitude right, but they also got the altitude right. What happens when I put in co-ordinates for Katmandu, Nepal? Will I materialize at the same altitude, inside a mountain?

    Come to think of it, can this thing take me anywhere but New Zealand?

    I could go on, but I won’t. My big question is this: After someone dies by being hit by a car upon materializing, is the stakeholder going to be okay with the fact that I didn’t raise these questions, especially when he thinks that I’m accepting his mustard for a job we both call “testing”?

    4) It’s not that providing a setup script and some automated checks to detect specific problems with the setup is a bad practice. It’s a wonderful convenience for everyone concerned, especially when the scripts include comprehensible diagnostics (note: comprehensible; I didn’t mean to say comprehensive, because that’s not available). But why call that an “acceptance test”? Why not call it what it is: a setup and configuration checker?

    —Michael B.

  12. ehendrickson July 28, 2010 at 10:56 pm #

    Hi Michael,

    Wow! What a great meaty comment.

    I’m totally with you about calling things “checkers” instead of tests. I want to do away with the word “test” altogether and instead adopt terminology check and explore.

  13. Jon Bach July 28, 2010 at 11:10 pm #

    Do you think that Acceptance Tests imply the product will always work the same way when it was tested at Acceptance time?

    At a different time, on a different configuration, under different circumstances, the same code that passed “Acceptance Testing” might fail. Will the customer care that a programmer said “well, you saw that it worked on my machine at Acceptance Time”?

    What do you think of the notion of configuration testing and risk — to advertise the specifics of the platforms that were tested and to what degree problems were either seen or not seen. Is that overkill?

    Maybe if we called it “Acceptable Testing” for when the customer says, “yeah, we know you couldn’t test everything on all platforms, but we see it’s working on your stuff and so it’s likely to work on ours — it’s probably acceptable.” (risk)

    As for me, something passes Acceptance when it meets my needs or solves a problem but also when I think the risk of acceptance is worth the investment I made.

Leave a Reply