From the mailbox: selecting test automation tools

A long time ago, all the way back in 1999, I wrote an article on selecting GUI test automation tools. Someone recently found it and wrote me an email to ask about getting help with evaluating tools. I decided my response might be useful for other people trying to choose tools, so I turned it into a blog post.

By the way, so much has changed since my article on GUI testing tools was published back in 1999 that my approach is a little different these days. There are so many options available now that weren’t 12 years ago, and there are new options appearing nearly every day it seems.

Back in 1999 I advocated a heavy-weight evaluation process. I helped companies evaluate commercial tools, and at the time it made sense to spend lots of time and money on the evaluation process. The cost of making a mistake in tool selection was too high.

After all, once we chose a tool we would have to pay for it, and that licensing fee became a sunk cost. Further, the cost of switching between tools was exorbitant. Tests were tool-specific and could not move from one tool to another. Thus we’d have to throw away anything we created in Tool A if we later decided to adopt Tool B. Further, any new tool would cost even more money in licensing fees. So spending a month evaluating tools before making a 6-figure investment made sense.

But now the market has changed. Open source tools are surpassing commercial tools, so the license fee is less of an issue. There are still commercial tools, but I always recommend looking at the open source tools first to see if there’s anything that fits before diving into commercial tool evaluations.

So here’s my quick and dirty guide to test tool selection.

If you want a tool to do functional test automation (as opposed to unit testing), you will probably need both a framework and a driver.

  • The framework is responsible for defining the format of the tests, making the connection between the tests and test automation code, executing the tests, and reporting results.
  • The driver is responsible for manipulating the interface.

So, for example, on my side project entaggle.com, I use Cucumber (framework) with Capybara (driver).

To decide what combination of framework(s) and driver(s) are right for your context…

Step 1. Identify possible frameworks…

Consideration #1: Test Format

The first thing to consider is if you need a framework that supports expressing tests in a natural language (e.g. English), or in code.

This is a question for the whole team, not just the testers or programmers. Everyone on the project must be able to at least read the functional tests. Done well, the tests can become executable requirements. So the functional testing framework needs to support test formats that work for collaboration across the whole team.

Instead of assuming what the various stakeholders want to see, ask them.

In particular, if you are contemplating expressing tests in code, make very sure to ask the business stakeholders how they feel about that. And I don’t mean ask them like, “Hey, you don’t mind the occasional semi-colon, right? It’s no big deal, right? I mean, you’re SMART ENOUGH to read CODE, right?” That kind of questioning backs the business stakeholders into a corner. They might say, “OK,” but it’s only because they’ve been bullied.

I mean mock up some samples and ask like this: “Hey, here’s an example of some tests for our system written in a framework we’re considering using. Can you read this? What do you think it’s testing?” If they are comfortable with the tests, the format is probably going to work. If not, consider other frameworks.

Note that the reason that it’s useful to express expectations in English isn’t to dumb down the tests. This isn’t about making it possible for non-technical people to do all the automation.

Even with frameworks that express tests in natural language, There is still programming involved. Test automation is still inherently about programming.

But by separating the essence of the tests from the test support code, we’re able to separate the concerns in a way that makes it easier to collaborate on the tests, and further the tests become more maintainable and reusable.

When I explain all that, people sometimes ask me, “OK, that’s fine, but what’s the EASIEST test automation tool to learn?” Usually they’re thinking that “easy” is synonymous with “record and playback.”

Such kinds of easy paths may look inviting, but it’s a trap leads into a deep dark swamp from which there may be no escape. None of the tools I’ve talked about do record and playback. Yes, there is a Selenium recorder. I do not recommend using it except as a way to learn.

So natural language tests facilitate collaboration. But I’ve seen organizations write acceptance tests in Java with JUnit using Selenium as the driver and still get a high degree of collaboration. The important thing is the collaboration, not the test format.

In fact, there are advantages to expressing tests in code.

Using the same unit testing framework for the functional tests and the code-facing tests removes one layer of abstraction. That can reduce the complexity of the tests and make it easier for the technical folks to create and update the tests.

But the times I have seen this work well for the organization is when the business people were all technology savvy so they were able to read the tests just fine even when expressed in Java rather than English.

Consideration #2: Programming Language

The next consideration is the production code language.

If your production code is written in… And you want to express expectations in natural language, consider… Or you want to express expectations in code, consider…
Java Robot Framework, JBehave, Fitnesse, Concordion JUnit, TestNG
Ruby Cucumber Test::Unit, RSpec
.NET Specflow NUnit

 

By the way, the tools I’ve mentioned so far are not even remotely close to a comprehensive list. There are lots more tools listed on the AA-FTT spreadsheet. (The AA-FTT is the Agile Alliance Functional Testing Tools group. It’s a program of the Agile Alliance. The spreadsheet came out of work that the AA-FTT community did. If you need help interpreting the spreadsheet, you can ask questions about it on the AA-FTT mail list.)

So, why consider the language that the production code is written in? I advocate choosing a tool that will allow you to write the test automation code in the same language (or at least one of the same languages if there are several) as the production code for a number of reasons:

  1. The programmers will already know the language. This is a huge boon for getting the programmers to collaborate on functional test automation.
  2. It’s probably a real programming language with a real IDE that supports automated refactoring and other kinds of good programming groovy-ness. It’s critical to treat test automation code with the same level of care as production code. Test automation code should be well factored to increase maintainability, remove duplication, and exhibit SOLID principles.
  3. It increases the probability that you’ll be able to bypass the GUI for setting up conditions and data. You may even be able to leverage test helper code from the unit tests. For example, on entaggle.com, I have some data generation code that is shared between the unit tests and the acceptance tests. Such reuse drastically cuts down on the cost of creating and maintaining automated tests.

Consideration #3: The Ecosystem

Finally, as you are considering frameworks, consider also the ecosystem in which that framework will live. I personally dismiss any test framework that does not play nicely with both the source control system and the automated build process or continuous integration server. That means at a bare minimum:

  • All assets must be flat files, no binaries. So no assets stored in databases, and no XLS spreadsheets (though comma separated values or .CSV files can be OK). In short, if you can’t read all the assets in a plain old text editor like Notepad, you’re going to run into problems with versioning.
  • It can execute from a command line and return an exit code of 0 if everything passes or some other number if there’s a failure. (You may need more than this to kick off the tests from the automated build and report results, but the exit code criteria is absolutely critical.)

 

Step 2. Choose your driver(s)…

A driver is just a library that knows how to manipulate the interface you’re testing against. You may actually need more than one driver depending on the interfaces in the system you’re testing. You might need one driver to handle web stuff while another driver can manipulate Windows apps.

Note that the awesome thing about the way test tools work these days is that you can use multiple drivers with any given functional testing framework. In fact, you can use multiple drivers all in a single test. Or you can have a test that executes against multiple interfaces. Not a copy of the test, but actually the same test. By separating concerns, separating the framework from the driver, we make it possible for tests to be completely driver agnostic.

Choosing drivers is often a matter of just finding the most popular driver for your particular technical context. It’s hard for me to offer advice on which drivers are good because there are so many more drivers available than I know about. Most of the work I do these days is web-based. So I use Selenium / WebDriver.

To find a specific driver for a specific kind of interface, look at the tools spreadsheet or ask on the AA-FTT mail list.

Step 3. Experiment

Don’t worry about choosing The One Right tool. Choose something that fits your basic criteria and see how it works in practice. These days it’s so much less costly to experiment and see how things go working with the tool on real stuff than to do an extensive tool evaluation.

How can this possibly be? First, lots of organizations are figuring out that the licensing costs are no longer an issue. Open source tools rule. Better yet, if you go with a tool that lets you express tests in natural language it’s really not that hard to convert tests from one framework to another. I converted a small set of Robot Framework tests to Cucumber and it took me almost no time to convert the tests themselves. The formats were remarkably similar. The test automation code took a little longer, but there was less of it.

Given that the cost of making a mistake on tool choice is so low, I recommend experimenting freely. Try a tool for a couple weeks on real tests for your real project. If it works well for the team, awesome. If not, try a different one.

But whatever you do, don’t spend a month (or more) in meetings speculating about what tools will work. Just pick something to start with so you can try and see right away. (As you all know, empirical evidence trumps speculation. :-))

Eventually, if you are in a larger organization, you might find that a proliferation of testing frameworks becomes a problem. It may be necessary to reduce the number of technologies that have to be supported and make reporting consistent across teams.

But beware premature standardization. Back in 1999, choosing a single tool gave large organizations an economy of scale. They could negotiate better deals on licenses and run everyone through the same training classes. Such economies of scale are evaporating in the open source world where license deals are irrelevant and training is much more likely to be informal and community-based.

So even in a large organization I advocate experimenting extensively before standardizing.

Also, it’s worth noting that while I can see a need to standardize on a testing framework, I see much less need to standardize on drivers. So be careful about what aspects of the test automation ecosystem you standardize on.

Good luck and happy automating…

14 Responses to From the mailbox: selecting test automation tools

  1. Jim Hazen December 1, 2011 at 4:49 pm #

    Elisabeth,

    “Test automation is still inherently about programming”… How true, how true. Plus, everything else you said about treating automation as a development type project (design, tools, people/talent, versioning, ease of maintenance consideration and code reuse) is spot on target.

    Have you been reading my posts on other threads?? Hmm… strange how we sometimes think alike. Great post my friend.

    Regards,

    Jim Hazen

  2. James December 2, 2011 at 4:06 pm #

    Hi Elisabeth,

    This is great stuff. I wrote about technology choice (on the back of GUI automation, as it happens) a couple of weeks ago and I think we’re nicely complementary: http://qahiccupps.blogspot.com/2011/11/shall-we-ask-magic-8-ball_23.html

    James

  3. Mark Levison December 2, 2011 at 9:43 pm #

    Excellent post – small details. RobotFramework speaks natively to python and is getting better at speaking to .NET. FitNesse supports .NET.

    I mention this only because your audience may dismiss these as options too soon.

    Cheers
    Mark Levison

  4. Lisa Crispin December 3, 2011 at 3:51 pm #

    It’s so smart to start with test format. I’ve always given the advice you have for points 2 and 3. But it wasn’t until I recently read Markus Gaertner’s new _ATDD By Example_ manuscript that it hit home to me that first, we must decide the test format that works best for testers, programmers, business experts, BAs, everyone who will need to use the tests. Then go looking for the tool that supports that format. Of course, some of that is because 10 years ago, we didn’t have so many different options about format.

    I hope readers understand the point that this is something for the whole team to do together. I love automating tests, but it was only this year after hearing Gojko Adzic’s StarEast keynote “Sleeping with the Enemy” that I realized the programmers on my team ought to be writing ALL the automation code. I’m not terrible at coding, but I only spend a small percentage of my time writing code. Finding the most maintainable approach is not natural to me the way it is for my programmer teammates.

    Now we just have to get everyone to read your post! :->

  5. George Dinwiddie December 3, 2011 at 6:27 pm #

    Great stuff, Elisabeth, as usual.

    For working with Java code, I’ve had great success using Cucumber running under JRuby. This allows accessing the Java code with ease, but also allows using the brevity and convenience of Ruby in the step definitions.

    Aslak is currently working on Cucumber-JVM as a native Java implementation (replacing Cuke4Duke).

  6. Suraj Sundarrajan December 6, 2011 at 7:50 pm #

    Excellent article that will serve as a reference material for testers while evaluating automation tools for their specific testing needs. As a test automation enthusiast myself, I found the tips & recommendations greatly valuable and I look forward to more such enlightening articles from you.

  7. yuval December 19, 2011 at 2:16 pm #

    This is a great article
    visit me at: http://www.qatestingtools.com and see more tools, more specification, more article
    If you would like I can publish this as well

  8. testobsessed December 19, 2011 at 5:21 pm #

    Hi Yuval,

    I’m glad you like the article. I don’t want you to republish it on your site, but you are (of course) welcome to link to it. Thanks!

    Elisabeth

  9. Chris L December 22, 2011 at 7:24 pm #

    Great post, Elisabeth! The challenge we see that is becoming exponentially greater is the number of languages being used. We also now have to deal with Objective-C/Cocoa/etc. as the number of platforms (web, iphone, android, etc.) we have to support increases. Any thoughts?

  10. Bryan Oakley January 9, 2012 at 3:22 pm #

    In your table of tools to consider you suggest robot framework for java but not for .net. I’m not sure if you excluded robot for .net because you think it won’t work, or if you just think it’s not ideal, but my team is having great success using it for our .net application.We use it to drive both API and Selenium tests, which is great since we use one testing language for both types of tests. It took remarkably little glue to be able to write robot framework keywords in a .net language in addition to its built in support for python.

    The thing I like best about robot — and why we chose it over specflow — is that we have the freedom and ability to write tests in a BDD style, a procedural style, or a mix of the two. We’re not locked into one specific type of test.

    I wish more people knew about the robot framework — it’s surprisingly powerful.

  11. Logan January 13, 2012 at 4:31 am #

    Just started following this blog. My second read this morning and I’m delighted to have stumbled upon such a good blog.

    This post just summarizes what every automation test lead should be aware of. Having done this tool selection a couple of times for my organisation, I must say every point mentioned in this post is a must do for tool selection. Start your experiment as soon as possible and take your own time (I mean don’t rush through) before coming to a conclusion. I’ve seen projects struggle by choosing tools through more meetings and less experiment.

    recommended article.

Trackbacks/Pingbacks

  1. Рубрика «Полезное чтиво». Выпуск 13 « XP Injection - December 6, 2011

    [...] From the mailbox: selecting test automation tools – отличная инструкция по выбору инструментов для автоматизации тестирования [...]

  2. The First Links Post of 2012 | Learning Agile and Lean - January 1, 2012

    [...] Elizabeth Hendrickson: From the mailbox: selecting test automation tools [...]

  3. Weekly Article Links #3 « Practical Tips on Software Testing - January 4, 2012

    [...] Load Testing with JMeter: Part 1 – Getting Started From the mailbox: selecting test automation tools [...]

Leave a Reply