Some years ago I had lunch with a QA specialist who invited me to visit him at work. He wanted to show off how he had used a macro recorder to automate his testing. Over lunch I offered the opinion that test automation is a programming activity. The QA specialist vehemently disagreed with me.

“I don’t need to program to automate my tests!” he said, waving his fork. “And I don’t want to program!” His fork punctuated his words. “All those FOR loops. Never could get the hang of FOR loops. And what’s the point anyway!” The fork was becoming dangerous. I considered ducking.

I couldn’t help but notice that he seemed angry that FOR loops were too complicated, and yet he was trying to automate tests. I haven’t visited that company again and I have no idea how they’re doing. But that guy? He scared me. No, it wasn’t the fork that scared me. It was the attitude about programming and automated tests.

Not Everyone Has to Code

I am often asked whether QA/Test people need to know how to program.

I always answer the same: QA/Test people do not need to know how to program, even on Agile projects. Everyone on a team brings something to the table. QA/Test people my not bring programming skills, but that’s OK, they don’t need to. The programmers already have that covered. Rather, QA/Test people bring testing skills and analysis skills and critical thinking skills and communication skills. Some bring other non-programming technical skills like database admin or network admin or system admin skills. And some do bring programming skills, and that’s great too. Whatever their skills, everyone brings something to the table.

And non-programming testers can collaborate very effectively with programmers to create awesome test automation (just ask Lisa Crispin).

But someone who is scared of FOR loops doing the coding part of test automation in isolation by recording and playing back stuff? Seems to me like that’s a good way for everyone on the project to waste a huge amount of time and become very frustrated.

Is Record-and-Playback a Solution to the Wrong Problem?

I’ve been thinking about that guy today as I’ve been thinking about record-and-playback test automation. I have had several conversations about record-and-playback test automation over the last few months with a wide variety of people.

Some people have said, “Yes, we discovered that what you are saying is true. So while we may start by recording a script, we modify it so much that it is not recognizable as a recorded script when we’re done.”

Others have said, “I only use the record-and-playback for quick and dirty test-assistance scripts. I know it can’t create real, robust test automation.”

Still others – usually vendors – have said, “Yes, I know you don’t like record-and-playback. But others do, and they want that capability. They want to automate their tests without programming.”

So individual contributors generally recognize that record-and-playback can be a helpful stepping stone but it is not a strategy. Yet at least some vendors still are very attached to record-and-playback, and customers apparently still demand it.

I wonder if record-and-playback is an attempt to solve the problem that QA/Test groups think they have rather than the real, underlying problem? It seems to me that the reasoning usually goes something like this:

1. We need automated tests to provide fast feedback.
2. Someone needs to write those tests.
3. It has the word “test” in it, so it must belong to the QA/Test group.
4. The QA/Test group doesn’t have much in the way of programming skills.

Therefore, the reasoning concludes, we must need something that will allow QA/Test groups to automate tests without knowing how to program. Or we must make all QA/Test people learn to code. That second thing isn’t gonna happen, so we need a tool to take care of the first solution.

The problem is that item #3 in the list is a total fallacy. Just because something has the word “test” in it does not automatically mean that it should be assigned to a designated, independent tester. If we get rid of the original constraint #3, we simplify the problem and open a world of alternative solutions. So let’s state the situation instead as:

1. We need automated tests to provide fast feedback.
2. Someone needs to write those tests.
3. The QA/Test group may not have much in the way of programming skills.

So the people who are really good at coding can do the programming part of automating tests, and the non-programming testers can collaborate with them to make sure the automated tests are useful and effective. And I do mean collaborate. Not “hand off a specification to…”, not “make a request of…”, and certainly not “supervise.” I mean collaborate as in work together on, at the same time.

Worse, is Record-and-Playback a Substitute for the Real Solution?

So if the reason customers demand record-and-playback capability in test automation tools is that it enables people who don’t know how to code to automate tests, it makes me wonder why they’re making non-programmers do programming work.

The most common reason I hear from QA/Test people is that the programmers won’t automate tests, so the testers have to do it. The most common reason I hear from the programmers is that they don’t have time, and besides, there is a whole QA/Test group assigned to that kind of work.

But it seems to me like the real issue here is that we’re trying to use a tool to as a substitute for something else. We’re using it as an alternative to real collaboration.

So now when I hear someone tell me that they’re using record-and-playback a lot in their test automation strategy, it suggests that perhaps the test effort and development effort aren’t integrated, that the organization is still operating in silos, and that we need to work on breaking down barriers to collaboration before we start talking seriously about tools.

There’s a discussion going on over on the software-testing discussion group about a customer’s delivery requirement that the software be handed over with an acceptance test script.

I want to illustrate my perspective on this topic with a short story. If stories aren’t your thing, skip to the end of the post. I make my real point there.

A Fictional Digression with Little Green Men, a Magic Transporter Stick, and a Jar of Mustard

It’s just after dusk. You’re hanging out on your back porch enjoying the evening.

Suddenly with a whir and a whine, an alien space ship lands in your back yard. Two little green guys pop out. At first, you hear a bizarre series of clicks and whistles, but soon you hear a mechanical voice. You guess it must be a translator device. The voice says.

“Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You aren’t sure you heard them right. This is too surreal. Mustard?

You stare at the green guys. They’re kind of roly-poly. Each one has five eyes on stalks on the top of its head, and five squat arms with a mass of tentacles coming out of the middle of their bodies. And they are green. Decidedly, definitely green.

“Little green men,” you mutter under your breath.

They repeat their message: “Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You shake your head to clear it.

One half of your brain is still boggling at the fact that there are aliens hanging out by your rhododendrons.

The other half of your brain is remembering the huge economy-sized jar of mustard you got the last time you were at MegaStuff. You weren’t ever going to finish it anyway.

“Yeah, hang on,” you say. You go to fetch the jar of mustard. After all, you are the neighborly, helpful sort.

You are also the careful sort. As you are rummaging in the pantry it occurs to you that you don’t know what this “advanced technology” is. What does it do? How does it work? Is it safe? What if this is a plot to get a human to press the Go button on a Planet Death Device?

You can’t be too careful, after all. Time to ask some questions.

You come back outside, holding up the jar of mustard. The aliens nearly dislocate their eye stalks staring at it. All ten eyes are gleaming at you. It’s disconcerting.

Undaunted, you start your questions. “Before I give this to you,” you say, “what is this advanced technology? What does it do?”

The aliens each swivel one eye stalk up toward your face. The other four eyestalks remain firmly rooted on the mustard jar.

“It’s a portable personal surface transporter,” says the mechanical voice. “It will take you wherever on Earth you want to go.”

You consider this for a moment. A portable personal surface transporter. That could be handy. You consider your 45 minute commute each way every day, the long drives to see family, and the last time you paid an arm and a leg for a cramped economy seat in the back of the plane because you promised yourself you’d make it off your home continent while you were still young.

You look at the jar of mustard in your hand. You consider the possibilities.

Seems like a more than fair trade. A jar of mustard for instant transportation anywhere? That’s even worth some risks.

“OK,” you say. You walk over to the aliens and thrust the jar of mustard toward them. You have to bend down to get close enough. One of the aliens seizes the jar from your grasp with three of its hands. You’re surprised at how easily it hefts the weight of the jar given its size. All five of its eyestalks are on the jar, waving around the lid. Its hands are a sea of motion as its tentacles move over the large plastic container.

You look at the other alien. It’s riveted on the jar too. “What is it about these guys and mustard?” you wonder.

Then you notice that the other alien is holding a stick with a ball on the end in one of his hands. On him, it’s the right size for a walking stick. For you, it’s a little larger than a magic wand. A magic wand with a bulbous end.

“Magic wand indeed,” you mutter under your breath.

The alien hands it to you. It’s heavier than it appeared. Smooth, polished. “How does it work?” you ask.

A short series of clicks and whistles emits from one of them, you can’t tell which. Then the mechanical voice says: “Tap the bottom to turn it on.” Both aliens still absorbed by the mustard. One is holding the bottom of the jar while the other unscrews the lid.

You tap the bottom of the stick part of the device. The ball on the top starts glowing.

“Now what?” you ask.

One alien eyestalk swivels in your direction. “Touch the top wherever you want to go.”

You look at the ball again. It’s now a globe. A blue-green globe. A model of the Earth.

“So I touch New Zealand and I’m there? Really?” you turn it trying to find New Zealand. Once found, your hand hovers over the country.

“Wait!” one of the aliens swivels all five eyestalks over to you. He has a hand poised over the mustard jar. “WAIT!”

He moves toward you, eyestalks waving wildly. You decide that’s probably an indication of panic. You stop moving your hand toward the globe.

“What?” you ask.

The alien shuffles over and peers at the globe then your hand. He reaches up with three of his arms and grasps your hand in his tentacles.

“Oh dear,” the mechanical voice now seems to be coming from the vicinity of your feet. “Your hands are too big. This won’t work. No. If you touch the globe you could end up anywhere. Anywhere. You might not end up in New Zealand at all. Your hand is so big you could even end up in Tasmania. More likely you’ll drown in the middle of the ocean. Can’t have that. I’d feel terrible. You’ll have to enter the geocode instead of touching the globe.”

The mechanical voice is flat, emotionless. But you sense concern and perhaps fear in the clicks and whistles that the alien is making.

“Enter the geocode?”

“Yes,” the alien nods his eyestalks. “Let me show you.”

The alien takes the stick and manipulates it with his tentacles. He disappears.

Then, from over the fence, you hear: “Over here!” You look. The alien is waving at you from across the neighbor’s yard. Then a moment later he’s back by your side.

“See,” says the alien, pointing to the stick. “You need to press the sequence of buttons here with the geocode of where you want to go.” He hands you the stick. You look at it. The stick appears smooth and featureless.

The alien is acting like this is the most natural thing in the world. You’re baffled.

“Enter the geocode?” you ask again, hoping he’ll explain in more detail.

In the meantime you look over at the other alien. He appears to be enjoying the mustard. A lot. Noticing your attention, the second alien nods an eyestalk in your direction. “This is good stuff,” he says. Two eyestalks swivel to his compatriot. “Dude, you have to try this.” He holds out the jar.

The alien who has been helpfully explaining the portable personal surface transporter to you shuffles over to his friend. He is now lost in the mustard. An arm reaches into the jar. His eyestalks relax, drooping slightly.

“Oh, that is good. It’s been too long. Glad we stopped by Earth!” Both the aliens turn back toward their ship.

“WAIT!” you shout. “We’re not done. I don’t know how to use this thing! I don’t know how to tell if it’s working right! How do I enter the coordinates? How can I tell if I entered them correctly? HOW DO I KEEP MYSELF FROM LANDING IN THE MIDDLE OF THE OCEAN?”

The aliens seem impatient, but the helpful one turns back. He shuffles back to you and grasps your hand again. Manipulating your fingers with his tentacles he touches your fingers to the stick. “Here,” he says. “We’ll go over to there together.”

You can feel slight, ever so slight, impressions on the stick. Under his guidance, you tap on them. As you tap each one, it pulses back, apparently indicating that it received the tap. Partway through the sequence, you see a light appear on the side of the stick. A few more taps and another light appears. One last tap, and the globe flashes briefly. The alien tugs at your finger, pulling it to the other side of the stick and presses it down firmly. The next thing you know you’re on the other side of the rhododendrons.

“And back,” he says. He guides you to repeat the actions, but this time with a different sequence of taps. Different lights appear on the stick. The globe flashes again. Suddenly you’re back in front of the alien ship.

You have the hang of manipulating the buttons on the device now. But how will you know what the sequences are for different areas? What do the lights mean? How will you know if something has gone wrong?

You realize the aliens have turned back to their ship again.

“WAIT!” you shout. “I STILL DON’T KNOW HOW TO OPERATE THIS THING!”

You watch ten eyes all roll at the same time on only two bodies. The effect is disconcerting. But you need answers. You hold your ground, fearful and angry. You do not want to try randomly entering sequences. Who knows where you could end up?

“Don’t you have some documentation or something?” you plead.

One of the aliens shuffles up the ramp into the ship, then returns faster than you thought possible with a stack of paper.

“Here,” he says. “It’s a translation. It won’t be perfect. But it tells you what you should expect.”

You scan the document. It contains instructions for tapping, but even more importantly, it contains expected results. You realize that the first light you saw on the stick indicated that the stick accepted the latitude. The second indicated it accepted the longitude. The flash on the globe, had you looked at it carefully, would have given you some idea of where the coordinates would be taking you. The final tap on the other side of the stick confirmed the destination and activated the device. You notice that there is a cancel button next to the confirm button.

The next time you look up, you realize that the aliens are already in their ship. A soft whine from their engines and they’ve taken off. A scrap of paper floats down to you. “So long and thanks for the mustard!”

You go inside to look up the geocode for New Zealand.

Back to the Point

Now, let’s imagine for a moment that you are the hero in our story.

Unless you have a death wish, you’re probably not going to just start testing that thing to see what it does. Not really. You’re not going to push any buttons unless you have a very clear idea of what the expected result is.

Instead, you’re going to follow the instructions in the documentation exactly. And you want to know what you should expect to happen each step of the way. If something is going wrong at any point, you want to know right away so you can minimize the damage.

That’s not really testing: you only take actions where you know exactly what to expect. Testing involves experimenting with unknowns.

To frame your expectations, you need information. But the information you need is different from what you find in a typical user guide.

It is, however, very much like what you would find in a scripted acceptance test:

Touch buttons – 3 6 . 8 6 1 9 7
See the 3rd blue light.
Touch buttons 1 7 4 . 7 7 4 1 6 6
See the 4th green light.
Touch the Destination button.
See a flash on the Globe at Auckland.
Touch the Confirm button.
Arrive in Auckland Domain, Auckland New Zealand.
See greenery.

Now let’s imagine that you work for a huge company that has bought a software package. It’s not exactly alien technology. It may not even be particularly advanced technology. But it’s important to the business or the business would not have purchased it.

Let’s say that you’re not the end user for this software. You are in IT. This is a system that the sales team has purchased. Since you’re part of central IT, and you drew the short straw this month, you are responsible for the successful roll out of this new software.

You don’t actually understand what it does. In fact, you don’t really understand your sales people or what they do. You think they might be aliens. But you know that it’s gonna be on your head if you screw this up. So you want to make sure that everything is operating as expected.

You also know that there are finicky details about how the software is deployed and configured that make our imaginary portable personal surface transporter stick look like an etch-a-sketch.

How are you supposed to know if you set up the software right? If it’s configured correctly? If it’s doing what it’s expected to do?

Note that you don’t want to test the software. Even if you had time to test the software (you don’t), it’s not your job to do so. Someone else in your company made the decision to buy the software, so you’re not evaluating it. And the vendor presumably already tested that the software does what they intended it to do, at least some of the time. You’re not trying to find problems. You just want to get it to work and present evidence to the sales team that their precious new sales automation solution is ready to go.

In short, you need to conduct a final acceptance test after you have everything set up and configured.

But you are not qualified to design an acceptance test for this package. You do not know the software. You do not understand what it does. Your job is to install it and make sure that it’s wired up correctly.

The sales team is not qualified to conduct that test. They’re not even trained on the new system yet. The committee that decided to buy the software isn’t qualified to conduct the test. Some of them can barely retrieve their email unassisted.

So you need the people who made the software to tell you what to do and what you should expect.

Back to the software-testing group discussion: this is why customers ask for an acceptance test script to be delivered with software.

It is a perfectly reasonable request. The customer is not stupid or lazy or incompetent. The customer is asking for a deliverable that they need in order to ensure that the product that they purchased is operating as expected in their environment: with their permissions and authentication and security schemes, connecting to their customized databases, on their network with packet filters and firewalls. They just need to know it’s working. And reading the user guide won’t help them figure that out. Not efficiently. So they want an acceptance test. Think of it as a diagnostic checklist.

Oh, and if your organization is doing ATDD, you’d already have an acceptance test script: you could just give them a prettied-up version of the natural language expectations without the automation.

“We can’t automate everything. It costs too much.”

I’ve heard that concern—that test automation costs too much—multiple times when talking with folks in the middle of an Agile transition.

They know that automated regression testing is important in an Agile context. But they’re concerned about the costs. And they can’t even imagine getting to a point where all the regression tests are automated.

They’re right to be concerned. Their organizations have typically struggled just to reach a bare minimum of automated test coverage.

So why is that? Why have these organizations historically struggled with test automation?

Usually the organization’s test automation strategy involves a dedicated team creating automated tests after the code is written. At least that was probably their approach before the Agile transition. It’s the strategy promoted by most of the test automation tool vendors back in the 90’s.

(Some of those vendors even persist in promoting that kind of test-last automation strategy today. Unbelievable but true. Don’t even get me started. I’ll just rant. It won’t be pretty.)

But until the organization adopts a new test automation approach, they’re stuck with what they have. And the result of the traditional approach is that:

1. The test automation is being written in a language or tool that is only accessible (or known) to a small number of specialists. That group of specialists is a bottleneck.
    
2. To get anything done, that group of specialists has to reverse engineer the developed software to figure out how to shoehorn automation onto an untestable interface, typically a GUI. (Ow. Pain. Ow.) And they have to go through the GUI: the system quite probably has business logic intertwingled with the GUI layer. Even if it doesn’t, the test automation specialists probably don’t know how to reach into the code base to bypass the GUI.
    
3. The specialists may be test automation experts, but they are usually not professional programmers. That means that while they can make test automation tools sing, dance, and jump through hoops, they usually have not studied program design principles (SOLID, patterns). Most test automation code I see that’s written by specialists is kinda smelly. (I don’t blame them. My code was pretty dang smelly too when I was a test automation specialist. Pairing with real professional developers did a world of good for my programming skills.)
    
4. The previous generation of commercial specialized test automation tools enforce their own architecture making it dang near impossible to apply software design principles even if you do understand them.
    
5. The specialized commercial test automation tools usually cost an arm and a leg. (Need another license to execute the tests on the CI server? Fork over another 5 figures worth of licensing and maintenance fees.)

Bottom line: the reason test automation costs so much is that it’s done in a silo far removed from the development effort.

Buffered from the consequences of design decisions that decrease testability, the developers continue to create software that’s nigh onto impossible to automate.

And isolated from the technical expertise of how the software was constructed, the test automation specialists are in a situation where they cannot help but be both inefficient and ineffective.

Highly functioning Agile teams break down those silos. With the most effective Agile teams I’ve seen, everyone on the team is involved in automating the tests. And the automated tests go into the same shared source repository where the developers check in the code.

When we integrate the test automation effort with the development effort, we reduce the cost of test automation drastically. In doing so, we fundamentally change the cost/benefit tradeoff equation so that fully automated regression tests start looking like an achievable goal instead of an impossible pipe dream.

It’s time to stop imagining that test automation and programming are two separate activities done by different people with different job titles working in different departments.

That approach didn’t work particularly well with the V-Model, but it’s a flat out fail in an Agile context where programming and automating are part-and-parcel of the overall software development effort.

So what have I been up to for the last 7 months? There’s the usual stuff: a fair bit of client work, some conferences, and a whole lot of travel: Finland, Germany, Japan, and various locations in the US.

But the most exciting news is that Agilistry Studio is open! We’ve held a bunch of events there including an Open Source Test Automation Love In (OSTATLI), a week-long immersion dubbed “Agile Up to Here” (see writeups by Alan Cooper and Jon Bach), and a bunch of meetups. We have classes coming up there including an immersive Agile transformation simulation (WordCount). Please sign up!