Development tools have become orders of magnitudes more powerful in the last several years with intellisense; keyword coloring; automated refactoring across entire code bases; tight integration with pick-your-favorite-flavor of xUnit; and tight integration with source control. Software development today is nothing like software development when I started learning to code in the 1980s.
But while we’ve seen huge leaps in unit testing, tools to support functional testing haven’t changed all that much in recent years.
Yes, open source tools are gaining traction. And it is true that a new generation of testing tools emerged in the last several years based around the idea of driving applications using keywords or actions rather than click-by-click scripting. And some years ago the commercial tools finally figured out that separating test data and test actions with data-driven testing was a pretty neat idea. But that’s old news now.
We’re overdue for a major step forward in functional testing tools. So a bunch of us have been talking about “what’s next in test automation?” And at the same time, several folks have taken steps to answer that question.
Ward Cunningham (father of Wikis, FIT, and a host of other innovations) recently walked me through something he’s calling Process Explorer. To explore Process Explorer for yourself, check out these tests.
Process Explorer allows you to explore the automated functional tests through a variety of representations. You can see the tests in a traditional way, as a series of commands like “check(‘data here’).” You can see the executed tests with results embedded. You can also see a table with the number of times any given action is executed in current batch of use cases. You can explore each action and see which use cases exercise it. And even more groovy, you can see a swim-lane representation of all the actions in a use case organized by actor.
In the swim lane view, as you mouse over each step in the use case, a little bubble pops up showing you the result of the step. That isn’t a static picture that has to be maintained. It’s live HTML generated on the fly by the chunk of code responsible for returning a partial page from the server after an AJAX request.
Holy cats. Fully automated, live, on-the-fly results generation. What an incredible way to explore an AJAX app. And the swim lane representation is pure genius for workflow oriented testing.
And Ward isn’t the only one working on innovative ways of expressing tests.
At AWTA, Brian Marick demonstrated a prototype of some recent work he’s doing with test-driving GUIs with wireframes. The thing that has me really excited about his approach is the idea of annotating a GUI with expectations.
And Jennitta Andrea has been talking about a next generation of tools that know how to refactor tests, and that connect tests and code in a way that’s completely transparent to the tester.
The problem she’s highlighted is that if you have functional tests written in FIT or Fitnesse, there’s a connection between your tests expressed in HTML and your fixtures written in C#/Java/etc. But the IDE doesn’t understand that connection. So if you change the name of a method in your fixture, your FIT table isn’t automatically updated. Changing the code breaks the tests.
Further, if you have bunches of repetitive tests, you’d probably want to do simple refactorings like Extract Method. But unless you’re writing your tests in a language with an IDE that supports automated refactoring, you get to do all that refactoring by hand. How 1990’s.
Jennitta envisions a world in which your IDE would understand the connection between your FIT tables and your fixtures. She also imagines tools that fully support test refactorings like extracting commonly used data or sequences.
To get more people talking about such things, Jennitta hosted a packed session at Agile Open NW a couple weeks ago. We had a great discussion, ranging across topics ranging from limitations of current solutions to roadblocks to possible futures.
Now that Ward and Jennitta and Brian have helped me imagine what’s possible, I want even more.
Just one example: Ward’s Process Explorer only supports creating/editing tests in the command form. The graphical representations are read-only. The commands are all in text files, and thus aren’t refactorable. They hook into the code in development using a FIT-like keyword mechanism. I want to be able to manipulate the tests in the swim-lane representation as well as the code view. I want to be able to drag and drop actions around to change sequences. And I want to borrow Brian’s idea of annotating GUIs with expectations, and annotate the swim lane view with expectations about results that will then automatically be translated into “check” commands in the script view.
The more I pondered these ideas, the more I realized that they are key elements in the next generation of functional testing tools. And that inspired me to synthesize these ideas into a bold vision. Thus, I humbly present my vision of Test Tools: the Next Generation…
…on Monday, in part 2 of 2. (Everybody loves a cliffhanger.)
Happy weekend!