In case I haven’t said this often or loudly enough, I think open source test automation tools are our future.
The tool vendors simply cannot keep pace with changing technology. Each technology transition (Win32 to .NET; forms-based web apps to AJAX; etc.) prompts another round of “Oh crud, how do I get the tool to handle this?” Worse, specialized testing tools tend to encourage us v. them rifts rather than facilitating collaboration. Finally, the cost for commercial tools will always be high: they have a tiny niche market and a high cost structure. One example of that cost structure: a large sales force dedicated to visiting customers to do sales pitches, and corresponding high customer acquisition costs. Oh, and have you ever noticed that you can’t get a straight answer out of a Big Tool Vendor about price, and always have to go through a quoting process? I won’t name names here, but if you’ve been through a test automation tool evaluation process you know what I’m talking about.
Of course, this has worked in the past because vendors are adept at positioning their tools and coming up with ROI figures. But in the last few years, open source tools and frameworks have emerged that give you similar capabilities (or better depending on your requirements) than the commercial tools, and for free.
And yet there are still some managers who cling to commercial tools. I could say they’re acting out of fear, and fear is a lousy compass. I could say that, but I won’t. The truth is that these managers have valid concerns. They don’t want to invest time, and thus money, in a dead end solution. A free dead end solution is still a dead end. So, here’s my attempt to address the most common concerns I hear.
Lots of open source projects are abandoned. How do I know the tool I choose won’t be one of the abandoned ones?
Alas, life is uncertain. You don’t know; the tool you choose may end up abandoned in a few months. But the future of any given commercial tool may be uncertain as well. At least with an open source solution you have the source, so you can keep it running if needed. And if the tool is really that good there are probably others, like you, who use it and who would be willing to work with you to keep the project alive.
To put your mind at ease, however, here are the ages of some of the more popular open source test tools/frameworks:
- JUnit released 1999 (or perhaps earlier)
- FIT released 2002 (or perhaps earlier)
- Fitnesse released 2003
- Watir released 2003
- Selenium released 2004
Even more telling than the age of the tools is the impact they’re having in the commercial world. Consider JUnit. JUnit was the inspiration for NUnit. And NUnit was the model for the Unit Testing Framework in Visual Studio 2005 Team System. The xUnit family of unit test harnesses are now the defacto standard for unit testing. I don’t think there’s been a single commercial automated testing tool that has had that kind of impact.
“Free” can be really expensive. Who is accountable if I need help?
If something goes wrong with the tool, if you have a question, or if you’re just having trouble even getting started with it, post a message to the discussion group for the tool.
If the tool has an active community, you’ll usually have a response within a matter of hours, and, if you’ve encountered a bug, a fix or workaround within a few days. The people who respond are passionate about the tool. They are dedicated to the community. And they will do their utmost to help others have the same success they’ve had.
Want some idea of the level of support you can expect? Check out these mailing list archives:
Commercial tools are easier to use. Won’t it become more difficult to hire qualified people if we go with open source tools?
Commercial tools only look easier to use. In practice, all test automation involves programming. The sales people will happily talk about how the ease-of-use features make the tool accessible to even completely non-technical testers, but every organization that I am aware of that succeeded with test automation ended up with specialists doing the heavy lifting.
These specialists are people who have invested an enormous amount of time to learn the ins and outs of a particular tool. They’ve had to learn all the quirks of the proprietary test development environment. Such people are rare. If you adopt a tool where the automators use a standard language (Ruby, Perl, JavaScript, VB, etc.) and general purpose development environment, the field of potential automation candidates widens rather than narrows.
Further, you already have a great resource on hand: your programmers. Chances are your programmers know a few scripting languages already. They can help. Getting the programmers more involved in the test automation effort is likely have another benefit: improved testability. When the programmers find themselves reaping the pain of poor testability, they tend to make changes (like giving all page elements an id attribute) to make automating tests easier.
What about record and playback? Every commercial tool has record and playback.
First, my obligatory rant: record and playback is overrated. It’s fine for creating throwaway scripts, and for creating a first draft script that you then refactor without mercy until it doesn’t even resemble the original recording. But record and playback is not a test automation strategy. It is at best a tool and at worst a crutch. Be wary of record and playback.
That said, there are now recorders for some of the open source tools. In particular, see Selenium IDE (with record and playback). Lots of folks have asked for recorders, and I have no doubt we’ll see more in the future.