How Much to Automate? Agile Changes the Equation

The subject of how much to automate, and the related topic of how to calculate the ROI for test automation, comes up on a regular basis. In fact, it popped up on a couple of the mail lists I read recently.

Usually there’s at least one person arguing that test automation is expensive and that there are situations in which it just doesn’t make sense, so we should automate selectively. We should pick and choose, they say. We should automate wisely. We should automate only those tests where the investment is justified. The rest will stay manual, and that’s only sensible, they say.

I understand their concern.

In some contexts, particularly where there is a legacy code base that was created without automated tests, the cost to create and maintain each automated test is extraordinarily high.

Further, the value of those tests is often less than it could be.

The value in any test is in the information that it provides. But when many of the test failures are because the tests, and not the code, are wrong, the information provided by the whole suite of tests is deemed unreliable and untrustworthy. Information only has value to the extent that we can trust it.

Thus, the automated tests in that kind of context are both insanely expensive and low in value. Some years ago this was the norm. In many organizations, sadly, this is still the norm.

But it doesn’t have to be that way.

Successful Agile teams typically follow at least a subset of the XP development practices like TDD and Continuous Integration.

Oh, sure, you can be Agile without doing TDD. But by and large effective Agile teams do at least some of the XP practices.

And teams that do practice TDD and ATDD wind up with large suites of automated tests as a side effect.

(Yes, it’s a side effect. Contrary to what some people think, TDD is a design technique, not a testing technique. We do TDD because it leads to clean, well-factored, malleable, testable code. The automated tests are just a nice fringe benefit. Similarly ATDD is more about requirements than about testing. ATDD drives out ambiguity early, helps us understand the full scope of a given story, and ensures that we have a shared understanding of what “Done” looks like. But I digress.)

Moreover, the resulting set of automated tests is so much more valuable because the test results are typically reliable and trustworthy. When the tests pass, we know it means that the code still meets our expectations. And when the tests fail, we’re genuinely surprised. We know it means there’s something that broke recently and we need to stop and fix it.

And we get that information frequently. Developers writing code execute the unit tests every few minutes, and are thus able to tell almost immediately when they’ve broken something that used to work. Similarly, the Continuous Integration system executes the unit and acceptance regression tests with every code check in. All those automated tests result in incredibly fast feedback.

Anyone who has taken an economics or business class is probably aware of the time-value of money. Net Present Value says that money in your hand today is worth more than that same amount of money in your hand tomorrow.

The same is true of information. Information sooner is worth more than the same information later. Automated tests executed through a continuous integration system give us a lot of information fast, and that has enormous value.

So Agile teams that have solid engineering practices in place typically find it that each incremental test costs very little to create and maintain, and the value of those tests are huge because the information is reliable and it’s delivered so quickly. In such a context it no longer makes sense to debate the ROI for a single given test. In the time we have the debate, we could just write the test.

People who are accustomed to living in the first context, where automation is hard to create, painful and costly to maintain, and doesn’t offer all that much value, often find it hard to imagine such a context. From their perspective, ROI is so very uncertain because the price is high and the value is low.

But instead of refining our ROI arguments, I suggest changing the equation. Adopt development practices that lower the cost of automation and increase the value so much that we just don’t ever have to argue about whether or not a given test is worth automating.

12 thoughts on “How Much to Automate? Agile Changes the Equation

  1. Wow, I had never thought in terms of information having a higher PV, like money does. What a great way to look at it. I am seeing so many teams, not only my own, do what you are saying, automation becomes just a natural component of development. This economic angle helps explain why it is so valuable.

  2. This makes a lot of sense, I hadn’t thought of it like this either. Just curious, what would you suggest for those companies who aren’t doing TDD/ATDD but are looking to automate their tests? A lot of the people I speak with seem to be searching for a silver bullet that will let them automate everything without developers being involved. What’s the best way to manage their expectations without turning them off of automation entirely?

    Elisabeth replies:

    That’s a huge question. The short answer is that there is no one single conversation that is going to adjust all their expectations. It takes a series of conversations. And demonstrations. And proof-of-concept spikes. And references to outside experience reports. It takes persistence and step-by-step changes, making automation a little less expensive incrementally, while simultaneously making it a little more valuable. It’s more of a culture change than a technical challenge (though technical challenges abound). You’ll have to convince developers to participate, and you’ll have to get QA to let them. And you’ll have to convince management that the cost of these changes are worth it by helping them understand how different things can be.

  3. I completely agree with you. However, change is never easy. How would you advice teams to actually do this “Adopt development practices that lower the cost of automation and increase the value so much that we just don’t ever have to argue about whether or not a given test is worth automating.”?

    Obviously, if you don’t have a descent test bed, you have a lot of bugs, so you have little time to invest in creating hard-to-write tests. It’s an uphill struggle. What can one do?

    Elisabeth responds:

    Take it one small step at a time. Fortunately there are numerous small (or relatively small) steps that you can take to begin the journey. Get the tests versioned with the code. Get the tests running against every build using a continuous integration system. Take half a day to refactor the tests to extract helper methods/functions for common sequences. Take half a day to resolve intermittent failures. Take half a day to clean up as many “known failures” as you can so the expectation is that the tests will run clean. Declutter the test suite: throw out or rewrite crufty tests that are so convoluted no one actually knows what they do. Take half a day to pair with a developer to fix a particularly annoying test. Take one step. Then take another one. And another. The next thing you know, the automated tests are in decent shape.

  4. Hi Elisabeth,

    You do a pretty good job summarizing the arguments that “sapientists” like me make against massive automation. But you haven’t mentioned what I consider the most important one: an high-automation attitude systematically discourages testing of areas that are hard to automate (long and interesting scenario tests, with deep oracles, for instance) and tends to train people to think of testing as whatever it is that the test tools do, instead of what humans do.

    TDD testing does not lead to a rich and interesting set of tests. In fact, developers discourage each other from creating interesting unit tests, as I discovered during the Agile Fusion conference, some years back. Chris Sepulveda kept saying NO NO NO when I tried to write a really biting unit test. Why? Well, I was slowing down the coding!

    It’s an ongoing struggle for those of us who have dedicated ourselves to the testing craft, as such, to remind developers that programming can’t solve every problem. It won’t save your marriage. It won’t solve the financial crisis. And it doesn’t replace testing skill.

    — James

  5. Totally agree. One thing I definitely learned during my work managing software projects for use in risk management and trading is that there are environments where part of the picture is equivalent to no picture. In order to hedge a portfolio, you must have complete information about its composition.

    Tests are your code’s risk management tool so it is important to have a complete picture and to be able to trust the information you are getting. “No broken windows” must apply to tests at least as much as it applies to code.

    Taking the time-value approach is also very important. I am motivated to give this flavor to future conversations with my clients in finance. However your approach is even better if it does to testing what Thomas Edison did to candles:

    “We will make electricity so cheap that only the rich will burn candles.”


  6. The first part I couldn’t have said in a better fashion. In the second part you raise a very interesting point as Lisa already pointed out about informational time-value.

    The point James missed in his comment from my point of view is, that Agile practices demand to combine test automation with exploratory techniques. Just the combination of test automation on repeatable tasks and the human mind on tricky parts of the system built a ground for sucess. The automation is there to free one from repeating tedious tests over and over again while having the safety not to break any available functionality.

  7. Great article! One other distinction we’ve found important to make … automated unit testing frameworks (running unit tests written by the developers) is a Quality Control (QC) practice. Functional testing, whether automated or manual is a Quality Assurance practice … its goal being to ensure fit for purpose. QC is binary … the code either passes QC or it doesn’t … i.e the automated unit tests must run 100% or production stops until they do. QA is not binary … each QA decision is a business value judgment based on the business goals at the time and the risks associated with any quality issues that may exist. The wonderful evolution that has occurred where TDD and automated unit testing frameworks are in place is just as you describe … writing tests is now an act of design, not testing and QA is elevated to the role of creating the greatest business value.

  8. Great Article. Reply from Markus is spot on and is what I have been practicing for long in Agile projects. Test Automation and exploratory testing is not mutually exclusive, they live together and are more effective that way. IMO, value addition by doing both, test automation and exploratory testing is much greater than doing _either_ exploratory or automated testing.

    With new releases after every Sprint, it is very difficult to imagine successful releases without combination of automated tests for confidence and exploratory testing to identify critical defects. Both are useful and have proven their value many times in the past.

  9. Hi Elisabeth,

    I completely agree. Automation, testing and agility are very good bedfellows.

    In my experience testing isn’t given the priority it needs. Perhaps if the best developers started pairing up more to raise each others game we would see some really smart automated testing.

    I’m really interested to hear from anyone who feels their team is ahead of the game in continuous integration and automated testing (better still if based in London UK).



  10. On large bases of legacy code write tests for new features, refactored sections, and for any defects that are introduced or arise during testing. This builds a strong base of tests, and it will turn out that some sections of the legacy app are more fragile than others. The fragile section of the legacy system will end up with a robust framework rapidly.

    Automated test suites don’t replace usability or exploratory testing. In fact, this type of testing can be very valuable and the automated testing frees up time for this higher value effort.

  11. Starting from the bottom of your post …

    >>But instead of refining our ROI arguments, I suggest changing the equation.

    In IT services, outsourced environments the story is completely diffrent. First of all automation mostly (95%) GUI centric and is done by testers having specific COTS tools ( QTP, WinRunner etc) knowledge (alone). ROI is always the bone of contention. Automation is always deployed as an “add” on with Sole aim of reducing and cutting down the cycle time/testing cost. Executives want “business case” for funding automation and services provides provide all sorts of magical figures – 80% reduction in cycle, 150% increase in test coverage and so on.

    Here the notion of ROI hinges on how many times you repeat a specific test and the investment you do in buying tool and creating/maintaining the solution.

    >>> In some contexts, particularly where there is a legacy code base that was created without automated tests …

    There is another angle to it. There are those “black box” testers that write manual test cases that need to be automated in the future. This is a typical IT services scenario. In such case, one manager crudely mentioned – “let us *convert* all these manual test cases to automation scripts” (no mention of any code at all)

    >>> Usually there’s at least one person arguing that test automation is expensive.

    In fact the whole discussion starts with this point. Interestingly enough, in IT domain, that is not the problem at all as the worry is more about when will I recover my investment back and how much testing will I cut?

    >> there are situations in which it just doesn’t make sense, so we should automate selectively.

    Again interestingly enough, this question is rarely asked in IT scenario. The mandate automation all you can and save me testing time.


  12. What do you suggest for teams which refrain from adopting TDD as tests add to development cost? Many a times project management are more comfortable with traditional style of having QC team do the testing, and would not like to invest in cost of TDD.

    Elisabeth responds:

    It’s not really a question of comfort, but of effectiveness, right? If the team wants the benefits of highly testable software with a comprehensive and maintainable set of automated tests that provide feedback, the team will have to adopt practices that lead to that outcome. The team can succeed without adopting TDD, but I have no idea how to succeed at changing the automation equation without involving the developers in automating the tests.

    Secondly, I have observed that sometimes we are working on a project which is based on technology which we have not used earlier. In such projects, team has to ramp up skills to develop functional code, but they may not be able to acquire skills to do a TDD with that technology, or the technology itself may not be TDD-friendly. For instance, for one of our projects, we used Spring WebFlow as MVC technology for front end. We had a team which were new to Webflow and more over, we did not have much idea on how to go about automating testing of every piece of web gui functionality. Hence, even though I knew that TDD should be done, but we could not do due to lack of skills. So what do you suggest to teams who are faced with technology choices which itself has learning curve and team gets lost in getting functionality developed with no time to focus on testing aspect?

    Elisabeth responds:

    Yes, I ran into this problem recently in my own work when using a new library. The recipe for TDDing something in a new technology where you’re still trying to figure out how the technology works is to spike, learn, shred, and then TDD it right. That means you start by hacking together something that works well enough so you can learn how the technology works. Then you shred that initial solution, ripping (or commenting) out all that hacked together code. And start over test-first. Does it take longer? Sure. But the result is more maintainable, testable software with fewer bugs, and also much deeper learning.

Comments are closed.