I wasn’t even aware that I had been nominated, so when David Hussman called me at home shortly after 7:30AM on Tuesday August 10 to tell me that I had won, I was beyond surprised. Gobsmacked? Flummoxed? Yes, . . . → Read More: On Winning the Gordon Pask Award at Agile2010]]> On Friday August 13, I accepted the Agile Alliance’s Gordon Pask award at the Agile 2010 conference in Orlando.

I wasn’t even aware that I had been nominated, so when David Hussman called me at home shortly after 7:30AM on Tuesday August 10 to tell me that I had won, I was beyond surprised. Gobsmacked? Flummoxed? Yes, those words fit. Also grateful, honored, and delighted. I immediately made arrangements to go to Orlando to accept the award in person.

Even now I find it difficult to articulate what the award means to me. I am amazed to have been nominated. To have won? I am enormously pleased to have my work in the Agile community validated by such an honor. I think back to the prior winners and am ecstatic to be in their company. And I feel incredibly flattered to have been chosen alongside Liz Keogh who I respect and admire tremendously.

My inability to articulate my feelings led to near-paralysis in the days leading up to the official (short) ceremony. I had most of Tuesday and all of Wednesday and Thursday to organize my thoughts, but I made very little progress.

I thought about what to say during the long flight to Orlando on Thursday. When we landed in Orlando I had no more idea what I would say than when we left San Francisco.

Confessing my uncertainty about what to say to Matthew Barcomb during the banquet on Thursday night, I joked that I could use my time on stage to help him find his missing VGA adapter. “Thanks for the award!” I said, “Now, where is Matthew Barcomb. Yes, there you are. Stand up. OK, now who borrowed Matthew’s VGA adapter? Could you go over there and return it to him please?”

Matthew laughed. Brian Marick chimed in. “You could thank me,” he joked.

“OK,” I said. “What shall I thank you for?”

“For resigning from the Pask committee so that I’m no longer there to blackball women or testers.”

We laughed.

Then we drank some more. As a group we found our way to “Mexico” in Epcot Center where we hung with the Version One folks. Then we meandered over to “England” where Shane Hastie put some delicious but deadly concoction of a cocktail into my hands. We made our way to the Dance Hall on the boardwalk where we danced to pounding music and I lost track of my head.

Keenly aware that I had to be onstage between 9AM and 9:30, I switched to water at midnight and made my way back to my hotel at 1AM. Before going to sleep, I set two alarms, a primary for 7AM and a backup for 7:30AM. Then I fell asleep, blissfully unencumbered by any thoughts whatsoever.

Friday morning dawned.

The “strum” sound from my iPhone woke me with a start. It was my backup alarm. That meant it was 7:30AM. I shook away the cobwebs of a surreal dream involving Craftsman architecture and escaping cookie dough. Bleary-eyed, turned off the alarm. No time to hit Snooze.

I wondered why my primary alarm on my iPad, set for 7:00AM, hadn’t gone off. I looked over at the iPad. The display showed the alarm clock app prompting me to dismiss the alarm. Apparently it had gone off, but silently. Whoopsie. Good thing I always set two alarms. I turned off the alarm clock on the iPad and shuffled toward the coffee maker.

I took a mental inventory. I felt better than I had any right to given that I’d been in California less than 24 hours prior, had flown across country, and had been out partying until 1AM. I congratulated myself for not overindulging the night before, at least not too much, and for going to bed early enough to get 6 hours of sleep.

My mind turned to preparing for the day. 7:30AM. That left me an hour in which to make myself presentable, have some coffee, and reflect on what I would say on stage before I had to walk over to the Dolphin. I still needed to iron my shirt, but I would have plenty of time for that. And I could organize my acceptance speech in my head while I ironed.

Then my glance fell on the hotel room clock.

As a general rule, I don’t use hotel clocks. I find the alarms on my iPhone and iPad tend to be more reliable, and their time is usually more accurate.

But this time I wished that I had consulted the hotel room clock sooner.

It read 9:17AM.

If it really was 9:17… then I was due on stage RIGHT NOW. I could picture JB or Jim Newkirk calling my name, looking for me.

My mind raced. Which clock was right? I looked outside. Bright. Sunny. Of course. It’s Florida. Bright and sunny are normal. That didn’t mean much. I checked my own internal sense of time. Because I’d slept in 3 different time zones in the space of a week, my internal sense of time was completely broken.

My iPhone now read 7:33. I recalled my stop-over in Denver. I have had my iPhone get “stuck” on the wrong time zone before. And given that my iPad was another Apple product, it was possible that it could have the same issue.

Nagging doubts remained. If it was really that late why hadn’t someone called me? Surely someone would have called me. But maybe they couldn’t find me. I was staying at the Yacht Club because the Dolphin was full. And maybe they didn’t have my cell number. Another doubt surfaced. If the problem was that my iPhone and iPad were “stuck” on Denver time, why was it not an exact 2-hour difference? Why did my iPhone now say 7:34 and the hotel clock said 9:18?

But I had to admit to myself that it was entirely plausible that the hotel room clock was closer to correct than my own devices.

My stomach plummeted.

I quickly searched my address book for cell phone numbers of everyone I could think of. Phil Brock. Jessica Ambrose. JB Rainsberger. David Hussman. No luck. All my contact information was on my computer back in California. In an attempt to travel light I only had my iPad with me and it didn’t have all my contact info. So I had no way to contact anyone at the conference.

Full panic mode set in. I was seized with the notion that it was after 9AM. Despair and disbelief washed over me. I had flown all the way across the country just to accept the award. I felt so honored to have been selected. Then I stayed out too late and slept through my chance to say thank you. In doing so, I threw the honor back in the face of the committee. I’ve made mistakes before. I have disappointed people. But this would be among the worst of my screwups.

I threw on some clothes. Maybe, just maybe, if I got out the door fast enough and ran full tilt to the Dolphin, I could make it before 9:30. I would look like a disaster, but I would be there.

Grabbing my phone and room key I glanced at the mirror on my way out. It showed me an unforgiving image. I was a total mess. “What will people think?” I wondered. “How will they interpret my appearance?” Drunk, I decided. People would look at my hair and mismatched clothes and think I’m drunk from the night before. That’s the only reason someone would show up looking like this. My stomach sank lower.

I decided that showing up looking like a complete mess was preferable to not showing up at all. I raced out the door.

The nagging doubts resurfaced. The first thing to do, I decided, was to find out exactly what time it really is.

A hapless tourist wandered into my field of view. I raced up to him.

“WHAT TIME IS IT??!?” I demanded.

His gave me a look that plainly said he thought I was nuts. I privately agreed with his assessment. He checked his watch.

“About 7:48,” he reported.

The adrenaline that had fueled my scrambled exit from my hotel room abated. I shook with relief. My mind stopped racing. It would all be OK after all. I felt a sense of joyous reprieve. The sinking feeling reversed, roller-coasted into elation.

“THANK GOD! THANK YOU! I FEEL LIKE SCROOGE!” I shouted at the tourist with the watch. The bit about Scrooge made perfect sense in my head, but I suspect it served to confirm the tourist’s diagnosis of CRAZY.

I ran back into my room and spent the next 5 minutes just remembering how to breathe.

The remainder of my morning went as originally planned. In the next 40 minutes, I made myself presentable, had some coffee, and reflected on what to say on stage. And I laughed a little at my foolishness over the time confusion.

At 8:30AM (10:17AM HCT – Hotel Clock Time) I made my way from the Yacht Club to the Dolphin. I contemplated possible acceptance speeches as I walked, my heels thunking on the wood of the boardwalk.

I considered telling the story of my morning and the clock mixup, but decided it was too off-topic.

I considered doing an Oscars-style “Thank you to…” in which I would thank everyone I’d learned from in the Agile community, but decided that it was a massively long list, it would take too long, and no matter how careful I tried to be I would forget someone important.

I considered saying something about the past controversy around the award and the fact that two women had won this year when none had one before, but decided that was too divisive a message.

I considered gushing appreciations about the award and what it meant to me, but decided I’d probably end up babbling “Thank you I’m so honored!” over and over.

My mind was still churning even as the morning session began.

Mercifully, Liz was called up to the stage first. She spoke eloquently of community. I appreciated and agreed with her sentiments. But I knew that I could not get away with uttering, “Yeah. What Liz said!” I had to say something of my own.

Then it was my turn. As I walked up the steps I still did not really know what I would say.

But I had the germ of an idea. Brian had said I should thank him.

He had been joking. But he was right. I did need to thank him. Just not for the reason he suggested.

Brian is responsible for my starting down the path to learn about Agile, and he ushered me into the Agile community.

Brian started telling me about Extreme Programming sometime around 1999 or 2000. I ignored him for a couple years. Then at his urging, I went to see Kent Beck speak in 2001 and finally understood what Brian had been talking about. Set on the path to learning about Agile, I sought more sources of learning. I participated in one of Josh Kerievsky’s XP immersion classes. I finagled my way onto a Pivotal Labs project. I attended both XP Universe and the Agile Developer Conference (the forerunners to the Agile20XX conferences). Then Brian suggested that we do a session together at ADC2003 on Exploratory Testing in Agile. And so I started presenting at Agile conferences.

In short, if Brian had not introduced me to Agile concepts, principles, values, and the surrounding community, I wouldn’t be doing what I do today.

So, in my acceptance speech I said thank you to Brian. My words met with a few laughs, and I am still half expecting Brian to shoot me an email saying “WTF??!?”

But while I singled out Brian, I also want to thank members of the broader community. Fellow consultants and coaches. People I’ve worked with at my clients. Members of the AA-FTT community. Members of the BayXP and BayAPLN communities. Members of the local user groups that I’ve spoken to. The fine folks who work at Pivotal Labs, Atomic Object, CodeCentric, and Reaktor Innovations. Everyone who has come to Agilistry for events. Agile Alliance members.

All of you have been part of this journey. And I am immensely grateful to each and every one of you.

Communities take on a life of their own and deserve to be recognized and celebrated in their own right.

And I also wish to celebrate the individuals that make up the community.

So many, many thanks to all of you for being part of my Agile journey. I am honored. And grateful. And flattered. And extremely appreciative to be surrounded by a circle of such incredible individuals, part of an amazing community.

I want to illustrate my perspective on this topic with a short story. If stories aren’t your thing, skip to the end of the post. I make my real point . . . → Read More: Acceptance Tests as a Customer Deliverable]]> There’s a discussion going on over on the software-testing discussion group about a customer’s delivery requirement that the software be handed over with an acceptance test script.

I want to illustrate my perspective on this topic with a short story. If stories aren’t your thing, skip to the end of the post. I make my real point there.

A Fictional Digression with Little Green Men, a Magic Transporter Stick, and a Jar of Mustard

It’s just after dusk. You’re hanging out on your back porch enjoying the evening.

Suddenly with a whir and a whine, an alien space ship lands in your back yard. Two little green guys pop out. At first, you hear a bizarre series of clicks and whistles, but soon you hear a mechanical voice. You guess it must be a translator device. The voice says.

“Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You aren’t sure you heard them right. This is too surreal. Mustard?

You stare at the green guys. They’re kind of roly-poly. Each one has five eyes on stalks on the top of its head, and five squat arms with a mass of tentacles coming out of the middle of their bodies. And they are green. Decidedly, definitely green.

“Little green men,” you mutter under your breath.

They repeat their message: “Greetings! Do you have a jar of mustard? We will trade you our advanced technology for mustard.”

You shake your head to clear it.

One half of your brain is still boggling at the fact that there are aliens hanging out by your rhododendrons.

The other half of your brain is remembering the huge economy-sized jar of mustard you got the last time you were at MegaStuff. You weren’t ever going to finish it anyway.

“Yeah, hang on,” you say. You go to fetch the jar of mustard. After all, you are the neighborly, helpful sort.

You are also the careful sort. As you are rummaging in the pantry it occurs to you that you don’t know what this “advanced technology” is. What does it do? How does it work? Is it safe? What if this is a plot to get a human to press the Go button on a Planet Death Device?

You can’t be too careful, after all. Time to ask some questions.

You come back outside, holding up the jar of mustard. The aliens nearly dislocate their eye stalks staring at it. All ten eyes are gleaming at you. It’s disconcerting.

Undaunted, you start your questions. “Before I give this to you,” you say, “what is this advanced technology? What does it do?”

The aliens each swivel one eye stalk up toward your face. The other four eyestalks remain firmly rooted on the mustard jar.

“It’s a portable personal surface transporter,” says the mechanical voice. “It will take you wherever on Earth you want to go.”

You consider this for a moment. A portable personal surface transporter. That could be handy. You consider your 45 minute commute each way every day, the long drives to see family, and the last time you paid an arm and a leg for a cramped economy seat in the back of the plane because you promised yourself you’d make it off your home continent while you were still young.

You look at the jar of mustard in your hand. You consider the possibilities.

Seems like a more than fair trade. A jar of mustard for instant transportation anywhere? That’s even worth some risks.

“OK,” you say. You walk over to the aliens and thrust the jar of mustard toward them. You have to bend down to get close enough. One of the aliens seizes the jar from your grasp with three of its hands. You’re surprised at how easily it hefts the weight of the jar given its size. All five of its eyestalks are on the jar, waving around the lid. Its hands are a sea of motion as its tentacles move over the large plastic container.

You look at the other alien. It’s riveted on the jar too. “What is it about these guys and mustard?” you wonder.

Then you notice that the other alien is holding a stick with a ball on the end in one of his hands. On him, it’s the right size for a walking stick. For you, it’s a little larger than a magic wand. A magic wand with a bulbous end.

“Magic wand indeed,” you mutter under your breath.

The alien hands it to you. It’s heavier than it appeared. Smooth, polished. “How does it work?” you ask.

A short series of clicks and whistles emits from one of them, you can’t tell which. Then the mechanical voice says: “Tap the bottom to turn it on.” Both aliens still absorbed by the mustard. One is holding the bottom of the jar while the other unscrews the lid.

You tap the bottom of the stick part of the device. The ball on the top starts glowing.

“Now what?” you ask.

One alien eyestalk swivels in your direction. “Touch the top wherever you want to go.”

You look at the ball again. It’s now a globe. A blue-green globe. A model of the Earth.

“So I touch New Zealand and I’m there? Really?” you turn it trying to find New Zealand. Once found, your hand hovers over the country.

“Wait!” one of the aliens swivels all five eyestalks over to you. He has a hand poised over the mustard jar. “WAIT!”

He moves toward you, eyestalks waving wildly. You decide that’s probably an indication of panic. You stop moving your hand toward the globe.

“What?” you ask.

The alien shuffles over and peers at the globe then your hand. He reaches up with three of his arms and grasps your hand in his tentacles.

“Oh dear,” the mechanical voice now seems to be coming from the vicinity of your feet. “Your hands are too big. This won’t work. No. If you touch the globe you could end up anywhere. Anywhere. You might not end up in New Zealand at all. Your hand is so big you could even end up in Tasmania. More likely you’ll drown in the middle of the ocean. Can’t have that. I’d feel terrible. You’ll have to enter the geocode instead of touching the globe.”

The mechanical voice is flat, emotionless. But you sense concern and perhaps fear in the clicks and whistles that the alien is making.

“Enter the geocode?”

“Yes,” the alien nods his eyestalks. “Let me show you.”

The alien takes the stick and manipulates it with his tentacles. He disappears.

Then, from over the fence, you hear: “Over here!” You look. The alien is waving at you from across the neighbor’s yard. Then a moment later he’s back by your side.

“See,” says the alien, pointing to the stick. “You need to press the sequence of buttons here with the geocode of where you want to go.” He hands you the stick. You look at it. The stick appears smooth and featureless.

The alien is acting like this is the most natural thing in the world. You’re baffled.

“Enter the geocode?” you ask again, hoping he’ll explain in more detail.

In the meantime you look over at the other alien. He appears to be enjoying the mustard. A lot. Noticing your attention, the second alien nods an eyestalk in your direction. “This is good stuff,” he says. Two eyestalks swivel to his compatriot. “Dude, you have to try this.” He holds out the jar.

The alien who has been helpfully explaining the portable personal surface transporter to you shuffles over to his friend. He is now lost in the mustard. An arm reaches into the jar. His eyestalks relax, drooping slightly.

“Oh, that is good. It’s been too long. Glad we stopped by Earth!” Both the aliens turn back toward their ship.

“WAIT!” you shout. “We’re not done. I don’t know how to use this thing! I don’t know how to tell if it’s working right! How do I enter the coordinates? How can I tell if I entered them correctly? HOW DO I KEEP MYSELF FROM LANDING IN THE MIDDLE OF THE OCEAN?”

The aliens seem impatient, but the helpful one turns back. He shuffles back to you and grasps your hand again. Manipulating your fingers with his tentacles he touches your fingers to the stick. “Here,” he says. “We’ll go over to there together.”

You can feel slight, ever so slight, impressions on the stick. Under his guidance, you tap on them. As you tap each one, it pulses back, apparently indicating that it received the tap. Partway through the sequence, you see a light appear on the side of the stick. A few more taps and another light appears. One last tap, and the globe flashes briefly. The alien tugs at your finger, pulling it to the other side of the stick and presses it down firmly. The next thing you know you’re on the other side of the rhododendrons.

“And back,” he says. He guides you to repeat the actions, but this time with a different sequence of taps. Different lights appear on the stick. The globe flashes again. Suddenly you’re back in front of the alien ship.

You have the hang of manipulating the buttons on the device now. But how will you know what the sequences are for different areas? What do the lights mean? How will you know if something has gone wrong?

You realize the aliens have turned back to their ship again.

“WAIT!” you shout. “I STILL DON’T KNOW HOW TO OPERATE THIS THING!”

You watch ten eyes all roll at the same time on only two bodies. The effect is disconcerting. But you need answers. You hold your ground, fearful and angry. You do not want to try randomly entering sequences. Who knows where you could end up?

“Don’t you have some documentation or something?” you plead.

One of the aliens shuffles up the ramp into the ship, then returns faster than you thought possible with a stack of paper.

“Here,” he says. “It’s a translation. It won’t be perfect. But it tells you what you should expect.”

You scan the document. It contains instructions for tapping, but even more importantly, it contains expected results. You realize that the first light you saw on the stick indicated that the stick accepted the latitude. The second indicated it accepted the longitude. The flash on the globe, had you looked at it carefully, would have given you some idea of where the coordinates would be taking you. The final tap on the other side of the stick confirmed the destination and activated the device. You notice that there is a cancel button next to the confirm button.

The next time you look up, you realize that the aliens are already in their ship. A soft whine from their engines and they’ve taken off. A scrap of paper floats down to you. “So long and thanks for the mustard!”

You go inside to look up the geocode for New Zealand.

Back to the Point

Now, let’s imagine for a moment that you are the hero in our story.

Unless you have a death wish, you’re probably not going to just start testing that thing to see what it does. Not really. You’re not going to push any buttons unless you have a very clear idea of what the expected result is.

Instead, you’re going to follow the instructions in the documentation exactly. And you want to know what you should expect to happen each step of the way. If something is going wrong at any point, you want to know right away so you can minimize the damage.

That’s not really testing: you only take actions where you know exactly what to expect. Testing involves experimenting with unknowns.

To frame your expectations, you need information. But the information you need is different from what you find in a typical user guide.

It is, however, very much like what you would find in a scripted acceptance test:

Touch buttons – 3 6 . 8 6 1 9 7
See the 3rd blue light.
Touch buttons 1 7 4 . 7 7 4 1 6 6
See the 4th green light.
Touch the Destination button.
See a flash on the Globe at Auckland.
Touch the Confirm button.
Arrive in Auckland Domain, Auckland New Zealand.
See greenery.

Now let’s imagine that you work for a huge company that has bought a software package. It’s not exactly alien technology. It may not even be particularly advanced technology. But it’s important to the business or the business would not have purchased it.

Let’s say that you’re not the end user for this software. You are in IT. This is a system that the sales team has purchased. Since you’re part of central IT, and you drew the short straw this month, you are responsible for the successful roll out of this new software.

You don’t actually understand what it does. In fact, you don’t really understand your sales people or what they do. You think they might be aliens. But you know that it’s gonna be on your head if you screw this up. So you want to make sure that everything is operating as expected.

You also know that there are finicky details about how the software is deployed and configured that make our imaginary portable personal surface transporter stick look like an etch-a-sketch.

How are you supposed to know if you set up the software right? If it’s configured correctly? If it’s doing what it’s expected to do?

Note that you don’t want to test the software. Even if you had time to test the software (you don’t), it’s not your job to do so. Someone else in your company made the decision to buy the software, so you’re not evaluating it. And the vendor presumably already tested that the software does what they intended it to do, at least some of the time. You’re not trying to find problems. You just want to get it to work and present evidence to the sales team that their precious new sales automation solution is ready to go.

In short, you need to conduct a final acceptance test after you have everything set up and configured.

But you are not qualified to design an acceptance test for this package. You do not know the software. You do not understand what it does. Your job is to install it and make sure that it’s wired up correctly.

The sales team is not qualified to conduct that test. They’re not even trained on the new system yet. The committee that decided to buy the software isn’t qualified to conduct the test. Some of them can barely retrieve their email unassisted.

So you need the people who made the software to tell you what to do and what you should expect.

Back to the software-testing group discussion: this is why customers ask for an acceptance test script to be delivered with software.

It is a perfectly reasonable request. The customer is not stupid or lazy or incompetent. The customer is asking for a deliverable that they need in order to ensure that the product that they purchased is operating as expected in their environment: with their permissions and authentication and security schemes, connecting to their customized databases, on their network with packet filters and firewalls. They just need to know it’s working. And reading the user guide won’t help them figure that out. Not efficiently. So they want an acceptance test. Think of it as a diagnostic checklist.

Oh, and if your organization is doing ATDD, you’d already have an acceptance test script: you could just give them a prettied-up version of the natural language expectations without the automation.

I’ve heard that concern—that test automation costs too much—multiple times when talking with folks in the middle of an Agile transition.

They know that automated regression testing is important in an Agile context. But they’re concerned about the costs. And they can’t even imagine getting to a point where all . . . → Read More: Why Test Automation Costs Too Much]]> “We can’t automate everything. It costs too much.”

I’ve heard that concern—that test automation costs too much—multiple times when talking with folks in the middle of an Agile transition.

They know that automated regression testing is important in an Agile context. But they’re concerned about the costs. And they can’t even imagine getting to a point where all the regression tests are automated.

They’re right to be concerned. Their organizations have typically struggled just to reach a bare minimum of automated test coverage.

So why is that? Why have these organizations historically struggled with test automation?

Usually the organization’s test automation strategy involves a dedicated team creating automated tests after the code is written. At least that was probably their approach before the Agile transition. It’s the strategy promoted by most of the test automation tool vendors back in the 90′s.

(Some of those vendors even persist in promoting that kind of test-last automation strategy today. Unbelievable but true. Don’t even get me started. I’ll just rant. It won’t be pretty.)

But until the organization adopts a new test automation approach, they’re stuck with what they have. And the result of the traditional approach is that:

1. The test automation is being written in a language or tool that is only accessible (or known) to a small number of specialists. That group of specialists is a bottleneck.
    
2. To get anything done, that group of specialists has to reverse engineer the developed software to figure out how to shoehorn automation onto an untestable interface, typically a GUI. (Ow. Pain. Ow.) And they have to go through the GUI: the system quite probably has business logic intertwingled with the GUI layer. Even if it doesn’t, the test automation specialists probably don’t know how to reach into the code base to bypass the GUI.
    
3. The specialists may be test automation experts, but they are usually not professional programmers. That means that while they can make test automation tools sing, dance, and jump through hoops, they usually have not studied program design principles (SOLID, patterns). Most test automation code I see that’s written by specialists is kinda smelly. (I don’t blame them. My code was pretty dang smelly too when I was a test automation specialist. Pairing with real professional developers did a world of good for my programming skills.)
    
4. The previous generation of commercial specialized test automation tools enforce their own architecture making it dang near impossible to apply software design principles even if you do understand them.
    
5. The specialized commercial test automation tools usually cost an arm and a leg. (Need another license to execute the tests on the CI server? Fork over another 5 figures worth of licensing and maintenance fees.)

Bottom line: the reason test automation costs so much is that it’s done in a silo far removed from the development effort.

Buffered from the consequences of design decisions that decrease testability, the developers continue to create software that’s nigh onto impossible to automate.

And isolated from the technical expertise of how the software was constructed, the test automation specialists are in a situation where they cannot help but be both inefficient and ineffective.

Highly functioning Agile teams break down those silos. With the most effective Agile teams I’ve seen, everyone on the team is involved in automating the tests. And the automated tests go into the same shared source repository where the developers check in the code.

When we integrate the test automation effort with the development effort, we reduce the cost of test automation drastically. In doing so, we fundamentally change the cost/benefit tradeoff equation so that fully automated regression tests start looking like an achievable goal instead of an impossible pipe dream.

It’s time to stop imagining that test automation and programming are two separate activities done by different people with different job titles working in different departments.

That approach didn’t work particularly well with the V-Model, but it’s a flat out fail in an Agile context where programming and automating are part-and-parcel of the overall software development effort.

But the most exciting news is that Agilistry Studio is open! We’ve held a bunch of events there . . . → Read More: Look! An Update!]]> So what have I been up to for the last 7 months? There’s the usual stuff: a fair bit of client work, some conferences, and a whole lot of travel: Finland, Germany, Japan, and various locations in the US.

But the most exciting news is that Agilistry Studio is open! We’ve held a bunch of events there including an Open Source Test Automation Love In (OSTATLI), a week-long immersion dubbed “Agile Up to Here” (see writeups by Alan Cooper and Jon Bach), and a bunch of meetups. We have classes coming up there including an immersive Agile transformation simulation (WordCount). Please sign up!

Dave Nicolette commented that he thinks my definition actually describes Lean. He defines Agile in terms of the Agile Manifesto.

I replied to Dave elsewhere, but wanted to post my response here too since this is a topic that comes up frequently.

I have trouble defining Agile solely in terms of . . . → Read More: Why I Define Agile in Terms of Results]]> Not everyone agrees with my definition of Agile.

Dave Nicolette commented that he thinks my definition actually describes Lean. He defines Agile in terms of the Agile Manifesto.

I replied to Dave elsewhere, but wanted to post my response here too since this is a topic that comes up frequently.

I have trouble defining Agile solely in terms of the Agile Manifesto.

Mind you, I believe the Agile Manifesto is a great document. With it, the original signatories gave the industry a focal point, a fulcrum on which to turn. They forever changed our industry by distilling the difference, in terms of their guiding values and principles, between the lightweight approaches they used and the then-generally accepted formal processes & industry “Best Practices.”

And turn we have. “Agile,” at least as a buzzword, is mainstream. Increasingly organizations are adopting Agile methods to remain competitive.

However, I see the manifesto as a beginning, not a definitive end. The community has learned much in intervening years.

Part of what I’ve personally learned is that defining Agile in terms of results short circuits two of the bigger problems I see plaguing the community: 1) religious debates and 2) superficial but ultimately hollow attempts at transitions that result in frAgile processes.

Where people define Agile in terms of practices, I see more instances of Cargo Cult adoption (“We’re Agile because we stand up every morning!”) and religious dogmatism (“You don’t TDD?!? You can’t possibly be Agile!”).

Where people define Agile in terms of values, I see more instances of Agile-as-an-excuse (“Documentation? No. We don’t document anything. We’re Agile!”).

But where people define Agile in terms of results, I see greater focus on the ultimate goal: value to the business.

As it happens, the best way we’ve found to achieve that goal so far—or at least the best way I’ve seen so far—involves embracing the values and principles of the manifesto.

But I believe that remembering why those values and principles are important, remembering the result we’re trying to achieve, is essential. And so I define Agile in terms of results.

The request was for the keynote talk that I just presented at PNSQC. I’ve had several requests for the timeline that resulted, so I figure the easiest (and therefore . . . → Read More: My PNSQC Keynote with Agile Timeline]]> A few days ago, I tweeted that I was looking for nominations for events for an Agile timeline and am extremely grateful for all the responses I received.

The request was for the keynote talk that I just presented at PNSQC. I’ve had several requests for the timeline that resulted, so I figure the easiest (and therefore fastest) way to share the resulting timeline would be to share my slides.

Here they are (pdf, ~1Mb). Enjoy! (As always, comments/questions/critiques welcome.)

Stop. Seriously. Just stop. It’s getting in the way.

If you are accustomed to heavyweight test management solutions, you might not realize the extent to which a test management tool is more of . . . → Read More: Specialized Test Management Systems are an Agile Impediment]]> If you work in an Agile organization and are using a heavy weight specialized tool for test management, I have an important message for you:

Stop. Seriously. Just stop. It’s getting in the way.

If you are accustomed to heavyweight test management solutions, you might not realize the extent to which a test management tool is more of an impediment than an aid to agility. But for Agile teams, it is. Always. Without exception.

I don’t make such claims lightly and I don’t expect you to accept my claims at face value. So let me explain.

The Agile Alternative to Test Management

The things you need need to manage the test effort in an Agile context are whatever you are already using for the: Backlog; Source Control Management (SCM) System; Continuous Integration (CI) System; and Automated Regression Tests.

That’s it. You don’t need any other tools or tracking mechanisms.

Any test-specific repository will increase duplication and add unnecessary overhead to keep the duplicate data in sync across multiple repositories. It will also probably necessitate creating and managing cumbersome meta data, like traceability matrices, to tie all the repositories together.

All that overhead comes at a high cost and adds absolutely no value beyond what SCM, CI, & the Backlog already provide.

But, But, But…

I’ve heard any number objections to the notion that Agile teams don’t need specialized test management systems. I’ll tackle the objections I hear most often here:

But Where Do the Tests Live?
Persistent test-related artifacts go in one of two places:

• High-level acceptance criteria, test ideas, and Exploratory Testing charters belong in the Backlog with the associated Story.
• Technical artifacts including test automation and manual regression test scripts (if any) belong in the Source Control System versioned with the associated code.

And Where Do We Capture the Testing Estimates?
In Agile, we ultimately care about Done Stories. Coded but not Tested means Not Done. Thus the test effort has to be estimated as part of the overall Story implementation effort if we are to have anything even remotely approaching accurate estimates. So we don’t estimate the test effort separately, and that means we don’t need a separate place to put test estimates.

How Do I Prioritize Tests?
Agile teams work from a prioritized backlog. Instead of prioritizing tests, they prioritize Stories. And Stories are either Done or not. Given that context, it does not make sense to talk about prioritizing the tests in isolation.

Hello, I Live in the Real World. There is Never Enough Time to Test. How Do I Prioritize Tests Given Time Pressure?
If the Story is important enough to code, it’s important enough to test. Period. If you’re working in an Agile context it is absolutely critical that everyone on the team understands this.

But Testing is Never Done. Seriously, How Do I Prioritize What To Test?
This isn’t really a test management problem. This is a requirements, quality, and testing problem that test management solutions offer the illusion of addressing.

The answer isn’t to waste time mucking about in a test management tool attempting to manage the effort, control the process, or prioritize tests. Every minute we spend mucking about in a test management tool is a minute we’re not spending on understanding the real state of the emerging system in development.

The answer instead is to invest the time in activities that contribute directly to moving the project forward: understanding the Product Owner’s expectations; capturing those expectations in automated acceptance tests; and using time-boxed Exploratory Testing sessions to reveal risks and vulnerabilities.

What about the Test Reports?
Traditional test management systems provide all kinds of reports: pass/fail statistics, execution time actuals v. estimated, planned v. executed tests, etc. Much of this information is irrelevant in an Agile context.

The CI system provides the information that remains relevant: the automated test execution results. And those results should be 100% Green (passed) most of the time.

What about Historical Test Results Data?
Most teams find that the current CI reports are more interesting than the historic results. If the CI build goes Red for any reason, Agile teams stop and fix it. Thus Agile teams don’t have the same kind of progression of pass/fail ratios that traditional teams see during a synch and stabilize phase. And that means historic trends usually are not all that interesting.

However, if the team really wants to keep historic test execution results (or are compelled to do so as a matter of regulatory compliance), the test results can be stored in the source control system with the code.

Speaking of Regulatory Compliance, How Can We Be in Compliance without a Test Management System?
If your context involves FDA, SOX, ISO, or just internal audit compliance, then you probably live in a world where:

• If it wasn’t documented, it didn’t happen
• We say what we do and do what we say
• Test repeatability is essential

In that context, specialized test management solutions may be the defacto standard, but they’re not the best answer. If I’m working on a system where we have to be clear, concrete, and explicit about requirements, tests, and execution results, then I would much rather do Acceptance Test Driven Development. ATDD provides the added value of executable requirements. Instead of the tests and requirements just saying what the system should do, they can be executed to demonstrate that it does.

Certainly, doing ATDD requires effort. But so does maintaining a separate test management system and all the corresponding traceability matrices and overhead documentation.

Our Management Requires Us to Use a Specialized Test Management System. Now What?
Send them the URL to this post. Ask them to read it. Then ask them what additional value they’re getting out a test management system that they wouldn’t get from leveraging SCM, CI, the Backlog, and the automated regression tests.

So, have I convinced you? If not, please tell me why in the comments…

So now this post . . . → Read More: Adventures with Auto-Generated Tests and RSpec]]> This post started out as a quick little entry about a cool parlor trick you can do with RSpec to make it work for auto-generated test data. But in the middle of writing what was supposed to be a simple post, my tests found a subtle bug with bad consequences. (Yeah for tests!)

So now this post is about auto-generated tests with RSpec, and what I learned hunting down my bug.

Meet RSpec

In case you haven’t encountered RSpec before, it’s one of the Behavior Driven Development developer test frameworks along with JBehave, EasyB, and others.

Each RSpec test looks something like this:

  it "should be able to greet the world" do
      greet.should equal("Hello, World!")
  end

I used RSpec to TDD a solution to a slider puzzle code challenge posted on the DailyWTF.

Auto-Generating LOTS of Tests with RSpec

So let’s imagine that you’re testing something where it would be really handy to auto-generate a bunch of test cases.

In my particular case, I wanted to test my slider-puzzle example against a wide range of starting puzzle configurations.

My code takes an array representing the starting values in a 3×3 slider puzzle and, following the rules of the slider puzzle, attempts to solve it. I knew that my code would solve the puzzle sometimes, but not always. I wanted to see how often my little algorithm would work. And to test that, I wanted to pump it through a bunch of tests and give me pass/fail statistics.

I could write individual solution tests like this:

  it "should be able to solve a board" do
      @puzzle.load([1, 2, 3, 4, 5, 6, 8, 7, nil])
      @puzzle.solve
      @puzzle.solved?.should be_true
  end

But with 362,880 possible permutations of the starting board, I most certainly was NOT going to hand code all those tests. I hand coded a few in my developer tests. But I wanted more tests. Lots more.

I knew that I could generate all the board permutations. But then what? Out of the box, RSpec isn’t designed to do data driven testing.

It occurred to me that I should try putting the “it” into a loop. So I tried a tiny experiment:

  require 'rubygems'
  require 'spec'

  describe "data driven testing with rspec" do

      10.times { | count |
          it "should work on try #{count}" do
              # purposely fail to see test names
              true.should be_false
          end
      }

  end

Lo and behold, it worked!

I was able then to write a little “permute” function that took an array and generated all the permutations of the elements in the array. And then I instantiated a new test for each:

  describe "puzzle solve algorithm" do
      permutations = permute([1,2,3,4,5,6,7,8,nil])

      before(:each) do
        @puzzle = Puzzle.new
      end

      permutations.each{ |board|
          it "should be able to solve [#{board}]" do
              @puzzle.load(board)
              @puzzle.solve
              @puzzle.solved?.should be_true
          end
      }
  end

Sampling

Coming to my senses, I quickly realized that it would take a long, long time to run through all 362,880 permutations. So I adjusted, changing the loop to just take 1000 of the permutations:

  permutations[0..999].each{ |board|
      it "should be able to solve [#{board}]" do
          @puzzle.load(board)
          @puzzle.solve
          @puzzle.solved?.should be_true
      end
  }

That returned in about 20 seconds. Encouraged, I tried it with 5000 permutations. That took about 90 seconds. I decided to push my luck with 10,000 permutations. That stalled out. I backed it down to 5200 permutations. That returned in a little over 90 seconds. I cranked it up to 6000 permutations. Stalled again.

I thought it might be some kind of limitation with rspec and I was content to keep my test runs to a sample of about 5000. But I decided that sampling the first 5000 generated boards every time wasn’t that interesting. So I wrote a little more code to randomly pick the sample.

My tests started hanging again.

My Tests Found a Bug! (But I Didn’t Believe It at First.)

Curious about why my tests would be hanging, I decided to pick a sample out of the middle of the generated boards by calling:

  permutations[90000..90999]

The tests hung. I chose a different sample:

  permutations[10000..10999]

No hang.

I experimented with a variety of values and found that there was a correlation: the higher the starting number for my sample, the longer the tests seemed to take.

“That’s just nuts,” I thought. “It makes no sense. But…maybe…”

In desperation, I texted my friend Glen.

I was hoping that Glen would say, “Yeah, that makes sense because [some deep arcane thing].” (Glen knows lots of deep arcane things.) Alas, he gently (but relentlessly) pushed me to try a variety of other experiments to eliminate RSpec as a cause. Sure enough, after a few experiments I figured out that my code was falling into an infinite loop.

Once I recognized that it was my code at fault, it didn’t take long to isolate the bug to a specific condition that I had not previously checked. I added the missing low-level test and discovered the root cause of the infinite loop.

It turns out that my code had two similarly-named variables, and I’d used one when I meant the other. The result was diabolically subtle: in most situations, the puzzle solving code arrived at the same outcome it would have otherwise, just in a more roundabout way. But in a few specific situations the code ended up in an infinite loop. (And in fixing the bug, I eliminated one of the two confusing variables to make sure I wouldn’t make the same mistake again.)

I never would have found that bug if I hadn’t been running my code through its paces with a large sample of the various input permutations. So I think it’s appropriate to have discovered the bug, thus demonstrating the value of high-volume auto-generated tests, while writing about the mechanics of auto-generating tests with RSpec.

In the meantime, if you would like to play with my slider puzzle sample code and tests, I’ve released it under Creative Commons license and posted it on github. Enjoy! (I’m not planning to do much more with the sample code myself, and can’t promise to provide support on it. But I’ll do my best to answer questions. Oh, and yes, it really could use some refactoring. Seriously. A bazillion methods all on one class. Ick. But I’m publishing it anyway because I think it’s a handy example.)