The Changing Landscape of Test Automation

In case you haven’t noticed, there’s been a big shift in the world of system-level test automation over the last few years. These free/open source tools are rapidly gaining traction:

What’s happening? Why are open source frameworks gaining such popularity?

Perhaps this trend started as a result of dissatisfaction with the big commercial test automation tools. In the late 1990’s, the heavyweight specialized test automation vendors positioned their Record-and-Playback tools as a way of enabling non-technical testers to leverage testing expertise into automation.

There was only one problem: Record-and-Playback didn’t work.

Wait. That’s not fair. The tools “worked” in that it was possible to record a script and play it back. Usually. Unless the recorder failed to capture a mouse gesture or two.

But even when the tools “worked,” they were painful and produced totally unmaintainable tests. This was not the silver bullet the vendors had promised. Many organizations discovered that the cure of test automation was worse than the disease of manual testing. The ones that succeeded did so because they realized that no matter what a vendor promises, test automation is a form of programming and test scripts are code.

But the commercial tools didn’t just cause testers pain: they divided organizations. People who considered themselves Real Programmers shunned the specialized commercial tools, leaving the tools to the ghetto backwater of disenfranchised QA teams. The result was predictably tragic: the teams that adopted commercial test automation solutions had to fight everyone and everything in their organization to keep their scripts alive. They had to fight for improved testability in the software under test; they had to fight against changes that would break the automated scripts; they had to fight their management to get budget for sufficient licenses and training; they had to fight for time and resources to maintain the existing scripts; and they had to fight the tools to get useful results. Too much fighting, not enough productive working.

Some individuals began creating free alternatives to commercial solutions. One such alternative that remains popular is Win32::GUI, a Perl module that allows programmers to drive Win32 GUI interfaces. Another is a Perl module for driving Web applications: WWW::Mechanize.

But just having free drivers available didn’t result in a fundamental shift in the industry. That’s because a comprehensive test automation solution requires more than just a way to drive the application under test (as I explain in my post “Do We Need Specialized Test Automation Tools?”).

I think the current trend in open source tools is due to:

  • Key people in the industry who have generously contributed their time, effort, and insight, including Kent Beck, Ward Cunningham, Rick Mugridge, Bob Martin, Micah Martin, Bret Pettichord, lots of Thoughtworks people, and a bunch of folks I’m forgetting to mention.
  • The combined effect of open source testing and continuous integration tools.
  • The increased adoption of Agile practices encouraging automated testing at all levels in a project.

The result is that the latest crop of open source test automation frameworks are better than ever. Teams are discovering that there are viable alternatives to buying a high-priced specialized tool that divides the team. Instead, they’re discovering that the open source frameworks foster collaboration between programmers and testers and subject matter experts. And that collaboration is helping to make teams more adaptable.

So, what does this all mean to the commercial world of test automation tool vendors? Be afraid. Be very afraid.


Subscribe to our e-mail newsletter to receive updates.

12 Responses to The Changing Landscape of Test Automation

  1. Ed Gibbs November 4, 2006 at 9:20 pm #

    I think your observations are spot on. We’ve been moving to Scrum for about 18 months now and we haven’t had a lot of of success with the commercial record and playback tools. In our case that would be Mercury Quick Test Pro and Winrunner.

    The scenario plays out again and again that the testers don’t want to automate the scripts until the interface has really settled down, and the customer is pretty much allowed to change it every 30 days. The final result is not much gets automated until late in the project many Sprints in when the tester feels like the risk of change is low. So all you really get is a regression test suite eventually.

    We’re diving into Fitnesse finally to see if this helps aleviate things.

    Glad to see you’re blogging again.

  2. esh November 7, 2006 at 6:08 pm #

    Hi Ed,

    Thanks for your comment! I hope Fitnesse works better for you than the commercial tools have. No matter how you slice it, automating acceptance tests is one of the most difficult Agile practices to achieve. But I truly believe that the available open source tools support Agile much better than their commercial counterparts.


  3. Binil Thomas November 25, 2006 at 6:35 am #


    One tool you forgot to mention is Sahi –


  4. Titus Brown February 17, 2007 at 7:38 pm #


    a friend just forwarded on this entry after I posted my figleaf sectioning code,

    which is at least a step towards some of the cool stuff you talk about above.

    You might also consider twill,, for simple programmable Web browsing. It’s based on the Python port of www::mechanize.


  5. John Lockhart March 24, 2007 at 2:41 pm #

    And of course, Canoo WebTest

  6. Mark Aberdour June 3, 2007 at 3:09 pm #

    Just came across your blog today from a link on the WebLOAD site, good post. I’ve been putting together an ever-expanding list of these tools over at – there is LOTS of activity going on – 4 years ago my list stood at 50 tools, today it’s over 340 with new submissions on a weekly basis. Looking forward to exploring your blog some more, looks like great content.

  7. manjunath June 24, 2007 at 11:31 pm #


    can u send me the DAR analysis on SAHI tool.

    check the URL below

  8. Frank Cohen July 16, 2007 at 12:08 pm #

    A shamless plug for my own open-source service governance and test automation tool: PushToTest TestMaker. Screencasts and details are found at

    -Frank Cohen

  9. Dattatraya July 16, 2008 at 4:37 am #

    hi can anyone tell me how we can automate the system test manual procedure to automate without using paid tools

  10. Shira March 15, 2009 at 8:25 am #

    Have a look at JSystem –
    it’s also a good Java automation tool for complete System testing


  1. Functional Test Tools: the Next Generation (part 1 of 2) « Test Obsessed - December 14, 2010

    […] open source tools are gaining traction. And it is true that a new generation of testing tools emerged in the last several years based […]

  2. RadView and Open Source « Test Obsessed - December 14, 2010

    […] and Open Source May 24th, 2007 I’ve been saying for a long time that Open Source testing tools are our future. It seems at least one test tool vendor agrees with […]