I recently had the opportunity to visit AboutUs in Portland, OR. In case you’re not familiar with AboutUs, they’ve created a community-driven wikified guide to websites.

(So how appropriate that I’m blogging this on the 14th birthday of the Wiki, as Ward Cunningham reminded us over Twitter, another catalytic technology.)

Anyway, AboutUs is doing very cool stuff. For example, here’s the Quality Tree Software, Inc. profile. AboutUs automatically generated that profile the first time I searched on qualitytree.com. Then I was able to modify the automatically generated profile, and add relevant tags. And as a result of tagging, here’s what happens when I search AboutUs for “Agile Testing.”

I really enjoyed the open and inviting atmosphere in the AboutUs offices. Huge windows run along the outer walls of the offices, letting in a ton of natural light. And the office has an open floor plan featuring 100% mobile workstations. The result is a highly functional, adaptable space that feels light and airy. (This picture that the AboutUs folks posted on Flikr captures the feel of the space well.) Add in the openness of the people there, and you have a groovy, fun, and productive work environment. I was sorry I could only stay for a few short hours.

While I was there, Ward interviewed me for his series of lightning interviews.

Ward’s gentle interview style made the whole process feel very natural, like a normal conversation. And I really enjoy conversations with Ward. So if it looks like I’m having a blast in the interview, it’s because I am!

I hope they let me come back and visit again soon!

Certification of software professionals has been a hot topic for quite a while. At least 15 years. Maybe longer.

I keep hoping that the whole thing will blow over.

But it hasn’t. And it’s not going to. Too many people have too much of a financial stake in the success of certifications. Certification customers, including individuals and their employers, want certifications to have value. Certification providers want to continue making money.

But while I’ve been able to ignore most development and test certification initiatives up until now, I don’t think I am going to be able to ignore Agile certifications for much longer.

So I guess it’s time for me to talk about this publicly. I’ll start with tester certifications.

General certification programs, like the ISTQB tester certification, focus on knowledge of “best practices” and definitions. I have nothing against learning the material in the ISTQB Syllabus. There’s good stuff in there (even if the most recent books in the Foundations bibliography were published in 2004). However, I do have a problem with charging a whopping huge amount of money for test preparation classes, and testing people on their ability to memorize the contents on a Body of Knowledge, then slapping a Certified Tester stamp on their forehead.

The classes surely have some value. The trainers I know who teach certification prep classes certainly have much to offer. I see no harm in learning what these people have to teach.

But the cost of these classes is high. Certification preparation class can cost hundreds of dollars more than comparable non-certification classes, and the ISTQB test fee is another couple of hundred dollars.

And for what?

It is not clear to me that there is any evidence demonstrating a positive correlation between competence at software testing and possession of an ISTQB certification. (Some wags have argued that there is a negative correlation. I’m not going there.)

Rather, I suspect there is no correlation. I do not believe that certified software testers are any better at testing, on average, than uncertified testers.

And because I do not think there is a correlation between tester certification and competence, I see no value in software testing certifications. I think they’re a marketing scheme concocted to increase training revenues.

But people buy into this stuff, and classes leading to certification outsell classes that don’t lead to certification.

It’s important for me to note that I don’t have any problem with certifications in a specific technology. When Microsoft certifies someone as an MCSE, it means that Microsoft, the creator of a technology, is certifying that the candidate has met minimum competency requirements in that technology. Microsoft is not pretending to certify someone as a developer; they’re certifying that the candidate knows some specific fiddly details about a specific technology related to development.

It may seem like I’m splitting hairs. But it’s an important difference. There is a right and a wrong answer for specific technical questions like how to change a Windows Registry setting (hint: it’s not the form you submit to Microsoft to register your copy of Windows). General topics, like Software Testing, are not so clear cut. What’s right in one context could be dead wrong in another.

So, enough on tester certifications. I’ve successfully ignored them up until now, and plan to continue ignoring them in the future.

What I’m really concerned about are general Agile certifications.

I started hearing the rumblings around Agile certification some years back. In response, the Agile Alliance published a statement about certifications. It’s a good statement. I’m delighted the AA published it. I was in the room when Brian Marick and a small group decided to write the statement and I think they did a fabulous job on it. That can’t have been an easy thing to write.

And I was delighted when Laurent Bossavit and Brian Marick started WeVouchFor, a different kind of certification involving endorsements of competence rather than tests of knowledge.

Sure, there’s always a risk with endorsements that it becomes a kind of you-scratch-my-back-I-scratch-yours mutual love-fest. But I think that even such reciprocal endorsement arrangements say far more than commercial exam-based certifications that have a pass rate in the high 90%s. I’m not alone in that perspective, but endorsements alone are not enough for a lot of people. They want the certification.

I do understand the desire to have Agile certifications.

Agile is relatively new. There are a lot of people, and companies, sporting big ol’ “Now with Agile!” stickers slapped on top of their old RUP/CMM/CMMi/current-hot-thing stickers. So it can be difficult to tell those with deep Agile understanding from those who think they’ll make more money by adopting the hot buzzword-of-the-month.

And so employers look for objective evidence, like certifications, that someone who claims to know Agile actually knows what they’re talking about. And individuals want those certifications as a form of evidence.

The only real Agile certifications that I am aware of right now are the various Scrum certifications. Since Scrum is now the most popular Agile process, it’s no surprise that the Certified Scrum Master (CSM) is the most commonly-held Agile-related certification available today.

As an aside, it’s my understanding that the CSM designation started as a kind of an in-joke. I got my CSM by taking the CSM class. In it, Ken Schwaber said that the certification meant that we probably knew a little more at the end of class than we did at the beginning. But he wasn’t guaranteeing it. And then he taught us all the “secret handshake” (woof) so we could prove to other CSMs that we were in the club. (For the record, I took the CSM class so I could meet Ken Schwaber and learn about Scrum from one of the originators. The certification was a side effect of taking the class. The AHA! moments and resulting deep learnings are far more valuable to me than the certification.)

Then the Scrum Alliance decided to take the CSM, and other Scrum certifications, seriously. They put teeth in the certification.

And that’s fine. It seems to me that the Scrum certifications are like technology certifications. The Scrum Alliance is certifying knowledge of Scrum, a specific process. They’re not trying to certify general knowledge across all things Agile. They’re not saying that being a CSM means you’re generally competent. They’re just saying that being a CSM means you know what a Scrum Master does within the Scrum process. You know the mechanics.

Further, the CSM class is fabulous with or without certification. It’s experience-based, participative, interactive and leads to deep learning. I recommend it.

But this week, I am again hearing the rumblings for general Agile certifications, not just Scrum certifications.

People are asking me how to become Certified Agile Testers. The very thought makes me queasy. Agile Testing isn’t a process or a technology. It’s testing in an Agile context. And that’s not something I know how to certify someone in.

And just today I ran across a site run by the self-proclaimed World Agile Qualifications Board. And that made me angry. Really angry. Angry enough to write this post and not to link to their site. You can search them out if you want to, but I won’t drive traffic their way.

Alan Page suggested on Twitter that perhaps it’s an April Fool’s Joke, like Waterfall2006.com. I’m hoping he’s right. I’d rather feel stupid for falling for the joke than outraged at the reality.

However, even if it turns out to be a practical joke, the strength of my anger surprised me.

On reflection, and being brutally honest, I realized that it’s an anger borne of fear.

I fear that the quest for certification and availability of general Agile certifications, no matter how dodgy, will lead people away from the non-certification classes and services that I, and others like me, offer.

In this economy, that could be disastrous for my business.

I would like to believe that people in this industry are sufficiently discerning that they will come to my Agile Testing classes because my classes are valuable. The kinds of things I teach are not the kinds of things that are certifiable.

How do you certify someone on the realization that they’ve been playing the hero all too often? Or on the bone-deep visceral understanding around the effects of changes in feedback latency. These are the kinds of lessons that I believe participants in my Agile Transformation simulation learn. And they are not things that translate to a certification exam.

And so I am afraid, even though I know that fear is a lousy compass.

Being afraid makes me feel pressure to offer something certification related. But doing so goes against my principles for all the reasons I’ve already explained.

Maybe the best thing I could do would be to offer a self-certification. Want a certification? Declare yourself certifiably Test Obsessed. Here’s the certificate:

I was honored to be included on the lunch and learn panel at the Software Quality Association of Denver (SQuAD) conference this week. One of the questions that came up had to do with triaging bugs in an Agile context. Here’s my answer, in a bit more detail than I could give at the panel.

The short answer is that there should be so few bugs that triaging them doesn’t make sense. After all, if you only have 2 bugs, how much time do you need to spend discussing whether or not to fix them?

When I say that, people usually shake their head. “Yeah right,” they say. “You obviously don’t live in the real world.” I do live in the real world. Truly, I do. The problem, I suspect, is one of definition. When is a bug counted as a bug?

In an Agile context, I define a bug as behavior in a “Done” story that violates valid expectations of the Product Owner.

There’s plenty of ambiguity in that statement, of course. So let me elaborate a little further.

Let’s start with the Product Owner. Not all Agile teams use this term. So where my definition says “Product Owner,” substitute in the title or name of the person who, in your organization, is responsible for defining what the software should do. This person might be a Business Analyst, a Product Manager, or some other Business Stakeholder.

This person is not anyone on the implementation team. Yes, the testers or programmers may have opinions about what’s a bug and what’s not. The implementation team can advise the Product Owner. But the Product Owner decides.

This person is also not the end user or customer. When end users or customers encounter problems in the field, we listen to them. The Product Owner takes their opinions and preferences and needs into account. But the Product Owner is the person who ultimately decides if the customer has found something that violates valid expectations of the behavior of the system.

Yes, that does put a lot of responsibility on the shoulders of the Product Owner, but that’s where the responsibility belongs. Defining what the software should and should not do is a business decision, not a technical decision.

Speaking of expectations, let’s talk about that a little more.

When the Product Owner defines stories, they have expectations about what the story will look like when it’s done. The implementation team collaborates with the Product Owner on articulating those expectations in the form of Acceptance Criteria or Acceptance Tests.

It’s easy to tell if the software violates those explicit expectations. However, implicit expectations are a little more difficult. And the Product Owner will have implicit expectations that are perfectly valid. There is no way to capture every nuance of every expectation in an Acceptance Test.

Further, there are some expectations that cannot be captured completely. “It should never corrupt data or lose the user’s work,” the Product Owner may say, or “It should never jeopardize the safety of the user.” We cannot possibly create a comprehensive enough set of Acceptance Tests to cover every possibility. So we attend to both the letter of the Acceptance Tests and the spirit, and we use Exploratory Testing to look for unforeseen conditions in which the system misbehaves.

Finally, let’s talk about “Done.” Done means implemented, tested, integrated, explored, and ready to ship or deploy. Done doesn’t just mean coded, Done means finished, complete, ready, polished.

Before we declare a story “Done,” if we find something that would violate the Product Owner’s expectations, we fix it. We don’t argue about it, we don’t debate or triage, we just fix it. This is what it means to have a zero tolerance for bugs. This is how we keep the code base clean and malleable and maintainable. That’s how we avoid accumulating technical debt. We do not tolerate broken windows in our code. And we make sure that there are one or more automated tests that would cover that same case so the problem won’t creep back in. Ever.

And since we just fix them as we find them, we don’t need a name for these things. We don’t need to prioritize them. We don’t need to track them in a bug tracking system. We just take care of them right away.

At this point someone inevitably asks, “But don’t we need to track the history of the things we fix? Don’t we want to collect metrics about them?” To that I answer “Whatever for? We’ve caught it, fixed it, and added a test for it. What possible business value would it have to keep a record of it? Our process obviously worked, so analyzing the data would yield no actionable improvements.”

If we are ever unsure whether something violates the Product Owner’s expectations we ask. We don’t guess. We show the Product Owner. The Product Owner will say one of three things: “Wow, that’s a problem,” or “That’s outside the scope of this story, I’ll add it to the backlog,” Or “Cool! It’s working exactly as I want it to!” If the Product Owner says it’s a problem, we fix it.

If the Product Owner says “Technically, that’s a bug, but I would rather have more features than have you fix that bug, so make a note of it but leave it alone for now” then we tell the Product Owner that it belongs on the backlog. And we explain to the Product Owner that it is not a bug because it does not violate their current expectations of the behavior of the software.

Someone else usually says at this point, “But even if the Product Owner says it’s not a problem, shouldn’t we keep a record of it?” Usually the motivation for wanting to keep a record of things we won’t fix is to cover our backsides so that when the Product Owner comes back and says “Hey! Why didn’t you catch this?” we can point to the bug database and say “We did too catch it and you said not to fix it. Neener neener neener.” If an Agile team needs to keep CYA records, they have problems that bug tracking won’t fix.

Further, there is a high cost to such record keeping.

Many of the traditional teams I worked with (back before I started working with Agile teams) had bug databases that were overflowing with bugs that would never be fixed. Usually these were things that had been reported by people on the team, generally testers, and prioritized as “cosmetic” or “low priority.”

Such collections of low priority issues never added value: we never did anything with all that information. And yet we lugged that data forward from release to release in the mistaken belief that there was value in tracking every single time someone reported some nit picky thing that the business just didn’t care about.

The database became more like a security blanket than a project asset. We spent hours and hours in meetings discussing the issues, making lists of issues to fix, and tweaking the severity and priority settings, only to have all that decision making undone when the next critical feature request or bug came in. If that sounds familiar, it’s time to admit it: that information is not helping move the project forward. So stop carrying it around. It’s costing you more than it’s gaining you.

So when do we report bugs in an Agile context?

After the story is Done and Accepted, we may learn about circumstances in which the completed stories don’t live up to the Product Owner’s expectations. That’s when we have a bug.

If we’re doing things right, there should not be very many of those things. Triaging and tracking bugs in a fancy bug database does not make sense if there are something like 5 open issues at any given time. The Product Owner will prioritize fixing those bugs against other items in the product backlog and the team will move on.

And if we’re not doing things right, we may find out that there are an overwhelming number of the little critters escaping. That’s when we know that we have a real problem with our process. Rather than wasting all that time trying to manage the escaping bugs, we need to step back and figure out what’s causing the infestation. Stop the bugs at the source instead of trying to corral and manage the little critters.

Shameless plug alert!

Dale Emery and I are offering the 3-day Agile Testing Series of classes in Pleasanton, CA on April 22 – 24, 2009, and in Portland, OR on April 28 – 30, 2009.

We’ve structured the three days as a la carte offerings. You can take one, two, or all three days.

• Day 1 is “Adapting to Agile,” also known as The WordCount Simulation.
• Day 2 is “Acceptance Test Driven Development (ATDD) in Practice”
• Day 3 is “Exploratory Testing in an Agile Context”

We have discounted pricing right now: early bird pricing is just $499/day instead of the usual $599. In addition, if you register 2 or more people for the same class simultaneously, you can take advantage of our 20% group discount. And we’re offering a “Buy 3 get 1 free” Guest Pass offer where if you buy the whole series, you’ll get a Guest Pass to give to a friend or colleague so they can join in the fun (some restrictions apply, see terms and conditions for details).

And if that’s not enough, I’ve created a limited coupon offer for my blog readers. The first 5 people who register for each class using the coupon code “testobsessed” will receive an extra 10% discount on the class fee.

It had already been a frustrating morning. Too little sleep, and too little coffee. Squabbling kids. Traffic. Parents who didn’t know how to use the carpool lane at school and just stopped in the middle of the road, holding up traffic for 5 minutes while Junior fetched his lunchbox from the depths of the giant SUV. By the time I got to work, I had my cranky pants on.

And so when I read through a series of messages in some online forum or another in which some poor newbie was asking questions about Agile that indicated that they Just Didn’t Get It, I had to bite my tongue. I was not going to take out my frustrations with my morning commute on some poor unsuspecting person asking a perfectly legitimate, if naive, question.

But I couldn’t just ignore it. The wrongness of the assumptions in the message nagged at me, distracted me, kept me from focusing on the stack of work I intended to do.

So I wrote the following on Twitter:

If the org has 1-1 map btwn old vocab & new (e.g. WBS==sprint backlog, PM==Scrum Mastr, status meeting==Daily Scrum) they’re doin’ it wrong.

Turns out that was just the beginning. It seems there’s no end of ideas about how NOT to do Agile. I soon found myself writing about misinterpretations of TDD:

If the org thinks TDD means the testers write test documents before coding starts, they’re doin’ it wrong.

If the devs think TDD means writing *all* the unit tests in advance, and only then writing the production code, they’re doin’ it wrong.

@qualityfrog picked up on that with his contribution:

If those without title ‘developer’ aren’t included as team members, they’re doing it wrong.

Then @woodybrood suggested I should tag them. Thus, the #notagile tag was born. My most recent #notagile entries:

If an Agile team’s typical cycle time for critically important PO-requested fixes is measured in months, they’re doin’ it wrong. #notagile

If the Agile process has a reqs sprint and a design sprint and a code sprint and a test sprint, they’re doin’ it wrong. #notagile

I’m finding that having a place to put How Not to do Agile ideas is helping me. It’s cathartic. I’m not finding myself distracted the urge to write long diatribes on my blog anymore. And I’m pretty sure it’s lowering my blood pressure.

Follow along or feel free to contribute your own to the Twitter stream (tag your tweet with #notagile).

I’m planning all the little logistical details for my upcoming trip to Portland, Oregon to talk with the XPDX group about ATDD. Details like how I’ll get from point A to point B.

The TriMet site for the fabulous lightrail (that I am hoping will make it possible for me to cancel that car reservation after all) provides an interactive trip planner. There’s just one small problem:

Apparently TriMet thinks I can walk on water. Alas, that would be incorrect. Methinks I’d better plan on taking the bridge.

On February 5, a group of us gathered in my office on First Street in Pleasanton for the first ever “Open Source Test Automation Tool Love In” (OSTATLI). Joining in were: Dale Emery, Jeffrey Fredrick, Kevin Lawrence, Dave Liebreich, Ken Pier, and Chris Sims.

Several folks have asked me to post the results of the meeting, and others have asked me how to host one in their area.

I’ve been pleasantly surprised by the strong response to OSTATLI. At its core, the gathering was just an excuse for like-minded folks to spend a day geeking out together. Perhaps the interest was because of the name?

In any case, I must confess that I find it a little hard to post “the” results for the meeting. It’s just not the type of meeting that leads to just one set of results.

However, here’s my best attempt to provide a glimpse into the day from my perspective.

After our initial chitchat, and a brief moment in which I panicked while Chris convinced the wifi that it it wanted to allow us access to the Internet, we set some intentions for the day.

Some were about exploring tools, like:

• Play with Cucumber
• Get feedback on FIT v. NUnit
• Understand Selenium
• A taste of RSpec
• The cool tool that I’ve never heard of but should

Others were specific tasks, such as:

• Get SafariWatir testing my local WordPress instance “the right way”
• Use Robot Framework to create an initial set of Acceptance Tests for “VDL”

And we captured a list of the tools represented in the room, either tools about which someone was curious, or tools around which someone had expertise:

• RSpec
• Selenium/SeleniumRC
• DBUnit
• Abbot
• Marathon
• UISpec4J
• Robot Framework
• FIT
• Fitnesse/Slim
• Diff (built into the operating system)
• Firebug
• Cubic Test
• Cucumber
• Concordion

We spent the rest of the morning on projects related to our stated intentions, working in small groups. After lunch, we gathered around the projector for a demo by Ken of a custom test harness that SocialText has written around Selenium. Then Kevin did an impromptu demo of Cubic Test, and we all oohed and aahed over the graphical test representation.

A few folks had to leave early, and the rest of us talked for a while in the office, until we remembered that there’s a pub around the corner that would allow us to talk about geek stuff with a beer in our hands.

Overall, I think it was a fabulous success, tons of fun, and I am most grateful to all the participants and to everyone who has expressed interest.

For another perspective on OSTATLI, see Dave Liebreich’s account.

Also, I’m delighted that the idea is catching on. Al Snow is already planning a similar kind of meeting in the Atlanta area.

I’m in the process of writing marketing copy for a series of public classes that Dale Emery and I will be offering on April 22 – 24 in Pleasanton, CA and April 28 – 30 in Portland, OR.

(I am also still working on getting the registration system to bend to my will, so you can’t register just yet. I’ll post something here when the registration system finally goes live…soon, soon…)

Anyway…

I was wondering if I could impose on the collective wisdom of the general community to help me hone my message?

First, some background…

The classes are a 3-day series of Agile Testing related classes.

• Day 1 is “Adapting to Agile,” also known as The WordCount Simulation.
• Day 2 is “Acceptance Test Driven Development (ATDD) in Practice”
• Day 3 is “Exploratory Testing in an Agile Context”

The classes can be taken individually, or in combination.

From a marketing perspective, I want to include a little Question and Answer style blurb about what days someone should plan to take. For example:

I’m in a QA/QE/Testing group, and my organization is adopting Agile. This is all new to me. Where should I start?

We strongly recommend that you take all three classes in this series.

The first day, “Adapting to Agile,” will give you an opportunity to experience an Agile transformation and see how the whole team (not just testers) adapts testing-related activities as the context changes. Along the way, you’ll learn how test activities can support Agility by increasing visibility and feedback. And you’ll learn how to spot waste and focus on customer value in testing.

The second day, “Acceptance Test Driven Development (ATDD) in Practice,” will give you an understanding of how test specialists can contribute throughout the development cycle. ATDD might also address the concerns you might have around how you’ll be able to derive tests without having requirements documents.

The third day, “Exploratory Testing in an Agile Context,” will teach you about applying Session-Based Exploratory Testing as part of a sprint or iteration. It will also answer concerns you might have around how an Agile team tests for the risks and vulnerabilities that are not covered by the automated tests.

And as long as I’m doing that, I wanted to extend the idea to include questions and answers aimed at convincing team members that Agile Testing classes aren’t just for testers. To that end, I wrote the following:

I’m a Developer. Are these classes just for testers?

Nope! We want you to participate! Please join us!

As a Developer on an Agile team, you contribute a great deal to the testing-related activities. These classes will help you learn how to collaborate with testers and business stakeholders on various testing-related activities to ensure that the whole team is getting the feedback they need to keep the project moving forward. These classes also might help you convince other people in your organization that testing activities are a shared responsibility on an Agile team.

I’m a Business Analyst/Product Owner/XP Customer. Should I come to these classes? If so, which one?

If you are responsible for defining what the software should do on an Agile project, then you are also ultimately responsible for accepting the software. And yet you don’t have time to test it thoroughly by yourself. You need the help and support of the technical team to be sure when you accept software that it meets your expectations. The practice of Acceptance Test Driven Development is particularly important for that. So if you can come to only one day of these classes, we recommend you come for Day 2, the ATDD class.

If you can come to two days, we recommend that you also take the “Adapting to Agile” class because it will allow you to explore the connection between stories and acceptance tests in a microcosm.

Of course, if you can come for all three days we think you’ll find it very worthwhile. The third day on Exploratory Testing will give you ideas for ways to explore the emerging system to ensure that it really does meet your needs.

Really? I should sign up? But I’m not in QA, and I’m not a Tester. I’m a …

Please excuse us for interrupting.

We hear this a lot: “Great! You have an Agile Testing class! I’ll send my QA department!” There is an unfortunate implicit assumption that the only people who have to worry about testing are the designated Testers or QA people.

In an Agile context, everyone tests.

So no matter what role you play, if you have a role on an Agile team, you have some testing-related responsibilities. Whether you are an Architect, Developer, Database Designer, UI Designer, Technical Writer, Product Owner, Scrum Master, XP Customer, Tester, QA Manager, Quality Engineer, SDET, Build Meister, Configuration Manager, Team Lead, Test Lead, Development Manager, etc., if you’re working on an Agile team that delivers software, you need to know how testing activities in Agile help move the project forward. And you need to be prepared to play your part in ensuring that the software is adequately tested before calling it “Done.”

And by the way, if you are thinking, “I’m not a tester, so I don’t need these classes,” then you REALLY need these classes to find out what you’re missing.

Before I publish all this prose as part of the marketing materials, I would really like some community feedback. What do you think? Is it worth publishing this kind of thing? Is that last little bit too harsh? Too annoying? Is it convincing? What would make it more convincing?

Thanks for any feedback you care to give me.

The subject of how much to automate, and the related topic of how to calculate the ROI for test automation, comes up on a regular basis. In fact, it popped up on a couple of the mail lists I read recently.

Usually there’s at least one person arguing that test automation is expensive and that there are situations in which it just doesn’t make sense, so we should automate selectively. We should pick and choose, they say. We should automate wisely. We should automate only those tests where the investment is justified. The rest will stay manual, and that’s only sensible, they say.

I understand their concern.

In some contexts, particularly where there is a legacy code base that was created without automated tests, the cost to create and maintain each automated test is extraordinarily high.

Further, the value of those tests is often less than it could be.

The value in any test is in the information that it provides. But when many of the test failures are because the tests, and not the code, are wrong, the information provided by the whole suite of tests is deemed unreliable and untrustworthy. Information only has value to the extent that we can trust it.

Thus, the automated tests in that kind of context are both insanely expensive and low in value. Some years ago this was the norm. In many organizations, sadly, this is still the norm.

But it doesn’t have to be that way.

Successful Agile teams typically follow at least a subset of the XP development practices like TDD and Continuous Integration. Oh, sure, you can be Agile without doing TDD.

But teams that do practice TDD and ATDD wind up with large suites of automated tests as a side effect. (Yes, it’s a side effect. Contrary to what some people think, TDD is a design technique, not a testing technique. We do TDD because it leads to clean, well-factored, malleable, testable code. The automated tests are just a nice fringe benefit.)

Moreover, the resulting set of automated tests is so much more valuable because the test results are typically reliable and trustworthy. When the tests pass, we know it means that the code still meets our expectations. And when the tests fail, we’re genuinely surprised. We know it means there’s something that broke recently and we need to stop and fix it.

And we get that information frequently. Developers writing code execute the unit tests every few minutes, and are thus able to tell almost immediately when they’ve broken something that used to work. Similarly, the Continuous Integration system executes the unit and acceptance regression tests with every code check in. All those automated tests result in incredibly fast feedback.

Anyone who has taken an economics or business class is probably aware of the time-value of money. Net Present Value says that money in your hand today is worth more than that same amount of money in your hand tomorrow.

The same is true of information. Information sooner is worth more than the same information later. Automated tests executed through a continuous integration system give us a lot of information fast, and that has enormous value.

So Agile teams that have solid engineering practices in place typically find it that each incremental test costs very little to create and maintain, and the value of those tests are huge because the information is reliable and it’s delivered so quickly. In such a context it no longer makes sense to debate the ROI for a single given test. In the time we have the debate, we could just write the test.

People who are accustomed to living in the first context, where automation is hard to create, painful and costly to maintain, and doesn’t offer all that much value, often find it hard to imagine such a context. From their perspective, ROI is so very uncertain because the price is high and the value is low.

But instead of refining our ROI arguments, I suggest changing the equation. Adopt development practices that lower the cost of automation and increase the value so much that we just don’t ever have to argue about whether or not a given test is worth automating.

Are you an open source test automation tool aficionado? I am. That’s why I’m organizing OSTATLI, a small, non-commercial, invitation-only gathering next Thursday in my office in Pleasanton to provide an opportunity for us to express our mutual love of open source test automation tools.

Participants will bring their laptops, loaded with their favorite tools, and we’ll spend a day messing around with various tools to see what each one can do. I’ll provide beverages and wifi.

Want to come play? My office is small, so participation is limited. But getting an invite is easy: just email me.