What Problem Would Next-Generation Functional Testing Tools Solve?

On the toolsmith-guild discussion forum, Russ DeWolfe asked me what problem I was trying to solve with all this next generation tools talk.

Russ brings up a good point. I talked a lot about what I want, but not about the core problem that I’d like to see solved. So here’s what I hope a next generation of functional tools could do.

JUnit revolutionized unit testing by creating an incredibly simple yet powerful model: setup/test/teardown. Further, it provided a bare bones framework for implementing and executing your tests. In the past, a developer wanting to automate unit testing would have to write some kind of harness, would have to think about results reporting, and would have to think through how to structure the tests. JUnit simplified all that and allowed the developer to focus on just writing one test at a time. The result: a developer can write a unit tests within minutes. Thus JUnit significantly lowered the barrier to automated unit tests.

Fully automated functional testing still has numerous barriers. And Agile teams that want to integrate automated acceptance testing with their continuous integration process need fully automated tests.

Watir is an excellent example of something that helps break through barriers…it’s possible to create a lot of tests very quickly with Watir.

But after a while it becomes difficult to manage those tests. And reporting is still a challenge. And the business people who must agree that the automated tests reflect their acceptance criteria have a hard time (sometimes) understanding what all that Ruby code is doing.

Various people have tried various structures including table driven things (like FIT/FITnesse) and Domain Specific Languages. But none of these approaches has quite succeeded in transforming the state of the practice the way JUnit did for unit testing.

So I’d like a thing, or more likely, a collection of things, that break down the barriers to effective, fully automated end-to-end tests in a lightweight way, like what JUnit accomplished. I’d like it to be general enough that it could be used anywhere. But like JUnit, I’d like to see that general applicability achieved through simplicity rather than through gold plating, tack-another-feature-on-the-beast.


Subscribe to our e-mail newsletter to receive updates.

4 Responses to What Problem Would Next-Generation Functional Testing Tools Solve?

  1. Paul Rogers February 22, 2007 at 2:18 pm #

    I completely agree with this, and also your earlier posts on the next generation of test tools.

    In many places that Ive done automation, Ive been asked “So what does your automation test?” My usual answer of “check the code” isnt usually the answer people want.
    When there are subtle differences, this becomes more important ( eg, 2 links to get to a help page, does the automation need to test both, how do we easily tell people which link we use)

    Ive attempted to do this by tying the watir tests to an app like test director. It required alot of work, and still really didnt acheive what we wanted.

  2. Patrick Lightbody February 27, 2007 at 9:25 pm #

    I really enjoyed your “next gen” posts, as well as this. As the founder of OpenQA (home of Watir and Selenium), and a develop on Selenium, I have a lot of thoughts on this. In fact, I even created a product called HostedQA that tries to address these very issues (the company/product was recently sold and will be announced soon). I’m not sure why I didn’t have you in my RSS reader sooner, but you’re there now 🙂

    I’ve written up an initial response to some of your thoughts here in my blog entry, with the plan to follow up with two more posts in the coming days on my take on how next gen tools (like HostedQA, I hope) can help. You can find the entry here: http://blogs.opensymphony.com/plightbo/2007/02/next_gen_testing_tools.html

  3. Jerrad Anderson February 28, 2007 at 7:40 am #

    I started a long comment that ended up being to long for a comment so I posted it here:


  4. John Lockhart March 23, 2007 at 11:49 pm #

    I have just completed an initial automation project for the web booking engine for Air New Zealand, using Canoo Webtest. While challenging, I believe it sucessfully answered many of the issues above. Using an Ant script gave the benefits of Ant and XML and the (mixed blessing) of being able to run without driving a “real” browser or having to modify anything in the web app or server stack. By careful nesting of clearly named xml entities for duplicated function (similar to action words but simpler and perhaps more crude) and the use of ant/java properties for mapping to the web pages and for test data values as well as setting and following clear guidelines we achieved a very useful result very quickly (compared to traditional automation projects, which I’ve been involved in) and have left a legacy that can be maintained (although not by completely non-technical staff – but is that ever the case with automation? Despite what some companies like Worksoft say, I doubt it).

    I am not a developer, though I was one in a previous life/decade (but not a java one – boy it’s a different world new!), and while challenging both techically and as a paradigm shift, I found the tool useable.

    John (P.S. I met you before this all started at the Wellington conference in Aug 06 Elisabeth).