The Case for Open Source Testing Tools

In case I haven’t said this often or loudly enough, I think open source test automation tools are our future.

The tool vendors simply cannot keep pace with changing technology. Each technology transition (Win32 to .NET; forms-based web apps to AJAX; etc.) prompts another round of “Oh crud, how do I get the tool to handle this?” Worse, specialized testing tools tend to encourage us v. them rifts rather than facilitating collaboration. Finally, the cost for commercial tools will always be high: they have a tiny niche market and a high cost structure. One example of that cost structure: a large sales force dedicated to visiting customers to do sales pitches, and corresponding high customer acquisition costs. Oh, and have you ever noticed that you can’t get a straight answer out of a Big Tool Vendor about price, and always have to go through a quoting process? I won’t name names here, but if you’ve been through a test automation tool evaluation process you know what I’m talking about.

Of course, this has worked in the past because vendors are adept at positioning their tools and coming up with ROI figures. But in the last few years, open source tools and frameworks have emerged that give you similar capabilities (or better depending on your requirements) than the commercial tools, and for free.

And yet there are still some managers who cling to commercial tools. I could say they’re acting out of fear, and fear is a lousy compass. I could say that, but I won’t. The truth is that these managers have valid concerns. They don’t want to invest time, and thus money, in a dead end solution. A free dead end solution is still a dead end. So, here’s my attempt to address the most common concerns I hear.

Lots of open source projects are abandoned. How do I know the tool I choose won’t be one of the abandoned ones?

Alas, life is uncertain. You don’t know; the tool you choose may end up abandoned in a few months. But the future of any given commercial tool may be uncertain as well. At least with an open source solution you have the source, so you can keep it running if needed. And if the tool is really that good there are probably others, like you, who use it and who would be willing to work with you to keep the project alive.

To put your mind at ease, however, here are the ages of some of the more popular open source test tools/frameworks:

  • JUnit released 1999 (or perhaps earlier)
  • FIT released 2002 (or perhaps earlier)
  • Fitnesse released 2003
  • Watir released 2003
  • Selenium released 2004

Even more telling than the age of the tools is the impact they’re having in the commercial world. Consider JUnit. JUnit was the inspiration for NUnit. And NUnit was the model for the Unit Testing Framework in Visual Studio 2005 Team System. The xUnit family of unit test harnesses are now the defacto standard for unit testing. I don’t think there’s been a single commercial automated testing tool that has had that kind of impact.

“Free” can be really expensive. Who is accountable if I need help?

If something goes wrong with the tool, if you have a question, or if you’re just having trouble even getting started with it, post a message to the discussion group for the tool.

If the tool has an active community, you’ll usually have a response within a matter of hours, and, if you’ve encountered a bug, a fix or workaround within a few days. The people who respond are passionate about the tool. They are dedicated to the community. And they will do their utmost to help others have the same success they’ve had.

Want some idea of the level of support you can expect? Check out these mailing list archives:

Commercial tools are easier to use. Won’t it become more difficult to hire qualified people if we go with open source tools?

Commercial tools only look easier to use. In practice, all test automation involves programming. The sales people will happily talk about how the ease-of-use features make the tool accessible to even completely non-technical testers, but every organization that I am aware of that succeeded with test automation ended up with specialists doing the heavy lifting.

These specialists are people who have invested an enormous amount of time to learn the ins and outs of a particular tool. They’ve had to learn all the quirks of the proprietary test development environment. Such people are rare. If you adopt a tool where the automators use a standard language (Ruby, Perl, JavaScript, VB, etc.) and general purpose development environment, the field of potential automation candidates widens rather than narrows.

Further, you already have a great resource on hand: your programmers. Chances are your programmers know a few scripting languages already. They can help. Getting the programmers more involved in the test automation effort is likely have another benefit: improved testability. When the programmers find themselves reaping the pain of poor testability, they tend to make changes (like giving all page elements an id attribute) to make automating tests easier.

What about record and playback? Every commercial tool has record and playback.

First, my obligatory rant: record and playback is overrated. It’s fine for creating throwaway scripts, and for creating a first draft script that you then refactor without mercy until it doesn’t even resemble the original recording. But record and playback is not a test automation strategy. It is at best a tool and at worst a crutch. Be wary of record and playback.

That said, there are now recorders for some of the open source tools. In particular, see Selenium IDE (with record and playback). Lots of folks have asked for recorders, and I have no doubt we’ll see more in the future.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

5 Responses to The Case for Open Source Testing Tools

  1. Chris McMahon November 10, 2006 at 5:47 pm #

    Nice to see you writing so much.

    I’ve had success putting this discussion in terms of investment. For a proprietary tool, you invest your cash with a company in whose tool you believe. In return, when you need help, you get the right to call an 800 number where there is presumably an expert on the other end.

    For an open-source tool, you invest your time with a community in whose work you believe. This investment mostly takes the form of asking and answering questions, with occasionally the opportunity to contribute something material to the project, like code or documentation.

    I mentioned this on a little local list and had a very knowledgeable person with lots of enterprise experience with open-source point out times when the proprietary tool is preferable: http://tech.groups.yahoo.com/group/four-corners-open-source-focos/message/14 (Note the display doesn’t show the whole conversation. It’s worth it to read more of the msgs in the thread.)

  2. Frederic Torres December 15, 2006 at 6:46 pm #

    Elisabeth,
    Is this mean that there is no more room for commercial test tools if they use standard languages and IDE, do the job and are reasonably priced.

    ;>

    Frederic Torres
    http://www.InCisif.net
    Web Testing with C# or VB.NET (and IronPython soon)

  3. Elisabeth Hendrickson December 15, 2006 at 9:56 pm #

    Hi Frederic,

    I think that the old commercial model of specialized tools with hamstrung proprietary languages doesn’t work well given the open source alternatives. But I do think there is room for commercial test tools.

    For a commercial tool to be compelling to me, it would have to give me a level of sophistication beyond what the open source frameworks offer, while still supporting good coding practices in test automation. That last point is why I’m not a big fan of record and playback: generated code is usually bloated, difficult to read, full of hard coded data, and awful to maintain.

    So, what would be compelling? One example of something the open source frameworks don’t do (as far as I know, and I welcome corrections): test generation. Check out http://www.agitar.com/solutions/products/software_agitation.html

    Elisabeth

  4. Irk April 8, 2007 at 6:50 pm #

    It also depends on the way how the test code is generated and organized. For example, SWEA (http://webuissoft.com) separates binding to page elements from the script code. It helps create resilient to UI changes scripts. The script recording can save a lot of time on script development. Without script recording we will spend more time on writing tests than on writing code we are paid for.

Trackbacks/Pingbacks

  1. Do Testers Have to Write Code? « Test Obsessed - October 20, 2010

    […] of the number of test automation tool mentions, more than half are for free or open source tools. I’ve been saying for a while that the commercial test automation tool vendors ought to be nervous. I believe that this data backs me up. The revolution I predicted in 2006 is well under way and […]