Why Test Automation Costs Too Much

“We can’t automate everything. It costs too much.”

I’ve heard that concern—that test automation costs too much—multiple times when talking with folks in the middle of an Agile transition.

They know that automated regression testing is important in an Agile context. But they’re concerned about the costs. And they can’t even imagine getting to a point where all the regression tests are automated.

They’re right to be concerned. Their organizations have typically struggled just to reach a bare minimum of automated test coverage.

So why is that? Why have these organizations historically struggled with test automation?

Usually the organization’s test automation strategy involves a dedicated team creating automated tests after the code is written. At least that was probably their approach before the Agile transition. It’s the strategy promoted by most of the test automation tool vendors back in the 90’s.

(Some of those vendors even persist in promoting that kind of test-last automation strategy today. Unbelievable but true. Don’t even get me started. I’ll just rant. It won’t be pretty.)

But until the organization adopts a new test automation approach, they’re stuck with what they have. And the result of the traditional approach is that:

  1. The test automation is being written in a language or tool that is only accessible (or known) to a small number of specialists. That group of specialists is a bottleneck.
  2. To get anything done, that group of specialists has to reverse engineer the developed software to figure out how to shoehorn automation onto an untestable interface, typically a GUI. (Ow. Pain. Ow.) And they have to go through the GUI: the system quite probably has business logic intertwingled with the GUI layer. Even if it doesn’t, the test automation specialists probably don’t know how to reach into the code base to bypass the GUI.
  3. The specialists may be test automation experts, but they are usually not professional programmers. That means that while they can make test automation tools sing, dance, and jump through hoops, they usually have not studied program design principles (SOLID, patterns). Most test automation code I see that’s written by specialists is kinda smelly. (I don’t blame them. My code was pretty dang smelly too when I was a test automation specialist. Pairing with real professional developers did a world of good for my programming skills.)
  4. The previous generation of commercial specialized test automation tools enforce their own architecture making it dang near impossible to apply software design principles even if you do understand them.
  5. The specialized commercial test automation tools usually cost an arm and a leg. (Need another license to execute the tests on the CI server? Fork over another 5 figures worth of licensing and maintenance fees.)

Bottom line: the reason test automation costs so much is that it’s done in a silo far removed from the development effort.

Buffered from the consequences of design decisions that decrease testability, the developers continue to create software that’s nigh onto impossible to automate.

And isolated from the technical expertise of how the software was constructed, the test automation specialists are in a situation where they cannot help but be both inefficient and ineffective.

Highly functioning Agile teams break down those silos. With the most effective Agile teams I’ve seen, everyone on the team is involved in automating the tests. And the automated tests go into the same shared source repository where the developers check in the code.

When we integrate the test automation effort with the development effort, we reduce the cost of test automation drastically. In doing so, we fundamentally change the cost/benefit tradeoff equation so that fully automated regression tests start looking like an achievable goal instead of an impossible pipe dream.

It’s time to stop imagining that test automation and programming are two separate activities done by different people with different job titles working in different departments.

That approach didn’t work particularly well with the V-Model, but it’s a flat out fail in an Agile context where programming and automating are part-and-parcel of the overall software development effort.

19 thoughts on “Why Test Automation Costs Too Much

  1. I tend to agree. Non-programmers – and I’m one of them – make lousy test automators. The best people I’ve seen write automated code were already skilled developers in their own right.

    I think I’m pretty much screwed when it comes to automation – my programming skills are poor to non-existent. Trying to find the time to learn has been rather fruitless. Given that a new role I’m moving into will no doubt require automation skills at some point, I’m looking at being up the proverbial river of excrement without adequate means of locomotion.

  2. Have you ever come across teams where test automation is a silo within the development team? I work in a team where we have test automation specialists who do not interface with the rest of the test team.

    Those of us on the other side of the test team are not plugged into the automation at all and it seems to be creating a rift where the testers are so busy with the exploratory testing that there’s never a chance to touch automation. I’d love to say that this is a simple case of prioritizing, but it’s hard to know where to begin.

    So in this situation, how can testers approach plugging in to what’s going on with the test automation?

  3. Nice thoughts and my sentiments exactly. Coming from a programming background I know how hard it is for a tester without programming design experience to build automated tests. You need to work with the developer to effective perform your function as an automated tester. The other myth that testers need not know the coding language and its structure is wrong. It always helped me pinpoint the bug to the developer and even fix it for him/her to maintain the flow. All these factors help keep the project on track and reduce the automation cost no matter which development model you follow V or Agile. I think it requires a degree of tuning to the testing profession and we need to open the lid on time boxing to a realistic approach. Open minded approach to automation works better than the legacy idea of wait till the application is matured. No harm in keeping half cooked automated script ready before the code is delivered, it keeps you focussed.

  4. Ah, silos. I’ve dealt with them on many an occasion. And I’ve found that the most effective way to address the issue is to make friends. Build the relationship; break down the silo. First steps are simple: ask the friendliest-seeming of the folks on the other side of the silo wall out to lunch or out for an appropriate beverage after work. You can also invite them to your weekly meetings or to present a brown-bag session to show off what they’re doing. Nearly everyone likes to show off their work for an appreciative audience. (Just make sure that everyone understands it’s show-and-tell, not an opportunity to complain. Handle complaints separately and delicately; a strong negative reaction could jeopardize the relationship you’re building.) When the relationship is strong enough, then you can propose working together on stuff. Just keep building the relationship one step at a time.

  5. Fully agree. This also goes along well with my slogan when hiring people for QA: “The best programmer is only a good QA guys.” and “The best QA guy has to be a good programmer.”

    The smaller the company the less likely is the silo approach. The bigger the company grows the more fences you will see, more decoupling will happen, and finally the entire development of software slows down to a halt or everything starts to suck so much that people will just jump of the wagon.

  6. > When we integrate the test automation effort with the development effort, we reduce the cost of test automation drastically.

    Hm. I’ve seen the cost reduced considerably, but never “drastically”, if drastically means an order of magnitude. It’s still expensive. One reason costs don’t go down as “drastically” as they might, is that tasks tend to stick in development until they are done-done. High-functioning teams make this work, once they’ve hit their stride, but going from not-testing to testing doesn’t happen overnight.

    IMO, the cost/benefit ratio tips upward both because the cost comes down, AND because the benefit (usually negligible without programmer involvement) increases drastically.

    In other words, programmers should be involved in testing not because it’s tolerably cheap (it isn’t) but because it’s cheaper than deferring the tests, and because you finally get decent ROI on their time.

  7. Actually I agree: test automation now is not as it should be. Test automation should be treated as any other development process with all risks. In other case we will end up with those 5 statements and evenmore.
    So, interpolating:
    1. Languge should be wide spread, fully object oriented.
    2. Automation developer should have experience and access to initial application(it is often problem to get it).
    3. TA developer require to know all technologies(at least at basic level) used in AUT and additionaly know problematics of test automation.
    4. Choose library that access needed interface rather than tool with its verdorscript and IDE, it will make your TA project architecture more maintainable.
    5. Vendors spend lots of money on advertizing their tools, they tell: buy a tool, give them to any person and you will got automation. Vendors are interested in selling tools, not makeing autoamtion, so you will not hear from thyem that automation is as much as hard or even more complicated(recall GUI automation) as AUT development.

  8. Completely agree with all the above mentioned points and from personal experience yes, pair programming with professional developers can help better code

  9. Automating testing can have positive impacts in many areas and there are many success stories to provide hope that test automation will save money and solve
    some testing problems. It is important to really understand the potential costs and benefits before undertaking the kind of change automation implies if only to plan well to make the most of them.

  10. So are you saying developers should also take responsibility for test automation? With all due respect to developers, when the pressure is on, the first thing sacrificed is unit testing. So now test automation will be sacrificed first, and then unit testing. I still see some “automation silos” adding huge value to the final product especially in older software where the GUI is the safest place to automate against. Depending on the context, I say, let developers focus on development, and let testers with programming skills focus on automation.

  11. Very well said and sounds exciting developers doing regression automation. However the ground reality is different. As rightly said by SuperMario first thing developer sacrifice is UNIT testing and that’s what happens. It’s not that they can’t do regression automation(may be some developer would find it less as a skilled job and would show some attitude for not doing it), it’s probably would be more costly affair than putting a dedicated automation team to that job. Regression automation is just not automation, there is a maintenance as well. Once development is done they get busy with either defect fixes or implementing next release new feature. How would they find time to maintain the automation? If the automation is not well maintain then it’s of no use.
    In short in today’s world also we need those dedicated team to make automation a “success story” and if we work out a proper plan for ROI then it really works out well.

  12. Some of the drawbacks of Automation Testing are,

    1) If the application / product do not have many release cycles, it is not worth the time and money automating for one/two cycles.
    2) The automation process is expensive.
    3) All areas of the application/product cannot be automated. There would be certain limitations in certain areas.

  13. The Emperor wears no clothes!

    Traditional automated tested whereby a script runs and effectively pretends to be a user (moving mouse, pressing buttons and typing) is fantastic. On paper. Reality often sets in, after spending a couple of million dollars, that the shiney tool the marketing person from Wonderful Test Tools Ltd has a few basic issues that were not communicated early;

    1. Record and Playback is great for demoing the tool and showing basic simple tests like logging in. But in a commercial real-world scenario it is little more than a pre-scripting function to get basic information.

    2. Anyone using the Automation tool to create or maintain tests has to be a good programmer. Tool vendors call them ‘scripts’ but they are actually real computer code; with all the issues code has. To make maintainable the ‘script’ code must be very well written and laid out with a good understanding of programming subjects such as function calling, variables & types, parsing, loops, decision constructs etc etc… Otherwise it is a very basic test at best and a rats nest of unmaintainable and unreliable liability at worst.

    3. Automation is not flexible. The test envirnonment and test data must be absolutely solid with access and control of all endpoints by the automation framework. Any difference, even slight (IE. A login password expiring), could cause a whole suite of tests to fail and sometimes be costly to rework every script.

    However, there are some tools that genuinely help to get Automation working by addressing the fundamental problem; good testers are rarely good developers (and vice versa!). My favourite uses nothing more than Excel spreadsheets to allow testers to write tests consisting of explicit steps (It also allows the test suites and object maps – if fact everything! – to be in Excel spreadsheets). It takes the action/data steps and generates – in realtime – test scripts for the underlying tool (IE. QTP/Selenium/CodedUI/Etc) and the executes the scripts building decent results documentation.

    When done well, Automation is your friend. It can be used for creating frameworks that can be used by other depts (IE. Integration, deployment, staging etc). The metrics can be used to monitot progress, predict actual release-dates, get test-coverage and acurately measure risk. But when done badly, Automation can do more damage to a project budget than a lightning strike and little more than a toy for developer wannabies.

  14. Hello,

    I can only partly agree on this topic.
    For the design of embedded systems you cannot involve developer through all validation steps, because they can’t have the knowledge to apply testing on all different stage (integration testing, Hardware in the loop testing and system testing requires a different experience).

    Supposing we reduce the scope of their validation to component testing, I don’t think to the developers should validate their own work. Because they will lack of objectivity, and they will tend (in my opinion) to validate their code and not what the customer is expecting, which are two different things.

    Nevertheless, i agree on the fact that testers should be trained on development to automate efficiently. I myself struggle to have a clever testing design which allows re-usability and easy maintenance.

    Sincerely yours.

Comments are closed.