Do We Need Specialized Test Automation Tools?

In 2001, Bret Pettichord wrote an essay calling test tool vendors onto the carpet for giving us testers hamstrung scripting languages.  For me, it was a mind-blowing piece of writing.  By questioning the value of specialized test tool scripting languages, Bret’s essay made me rethink aspects of test automation that I’d taken for granted.I realized that at their core, commercial GUI testing tools like Mercury WinRunner or QuickTestPro, Segue Silk, Compuware QARun, et al give us six basic capabilities:

  1. Manipulate and inspect elements of a graphical user interface
  2. Compare expected results to actual results
  3. Group and/or select tests to execute
  4. Report results
  5. Recover from unexpected events
  6. Hide the complexity of all of the above so it’s incredibly easy to use

In order to get these capabilities, there are high up front licensing costs as well as less readily quantifiable costs such as learning the tool and designing, implementing, debugging, deploying, and maintaining the automated scripts.

I started to wonder: what if we could get 80% of the benefit for 20% of the cost by using a standard development environment instead of a specialized commercial tool?

As far as I’m concerned, items 1 – 5 are essential for GUI-based test automation.  So I set about trying to find a way to get the benefit of items 1 – 5 using what I called “shoestring automation.”

It’s worth noting here that only item 1 is specific to GUI testing.  Items 2 – 5 would apply just as well to non-UI testing (like for APIs or servers).  So there are a variety of ways to get items 2 – 5.  But how to manipulate the user interface using a standard language without doing anything special to the software under test?

Initially the answers I found involved non-maintainable, fragile approaches such as sending keystrokes to the application.  These approaches can work, but maintaining a suite of tests created this way would be a nighmare.

Now I think I have some answers, at least for a limited set of applications.  I’m experimenting with automating .NET applications through the GUI using C# and .NET Reflection from within nUnit.

Credit goes where credit is due: I was inspired by James McCaffrey’s article on MSDN on how to “Build Quick and Easy UI Test Automation Suites with Visual Studio .NET.”  And while I’d originally planned to do my experimenting using Ward Cunningham’s FIT (Framework for Integrated Test), it was Richard Hale Shaw who convinced me to try it with nUnit first.

As I said, I’m still experimenting.  But I’ll have some results to post here soon.  And I’ll be presenting what I have to date at SD in March.

And for the record, item 6 on the list of commercial GUI testing tool capabilities is a special brand of snake oil that deserves its own rumination.  More on that another day.