“We can’t automate everything. It costs too much.”
I’ve heard that concern—that test automation costs too much—multiple times when talking with folks in the middle of an Agile transition.
They know that automated regression testing is important in an Agile context. But they’re concerned about the costs. And they can’t even imagine getting to a point where all the regression tests are automated.
They’re right to be concerned. Their organizations have typically struggled just to reach a bare minimum of automated test coverage.
So why is that? Why have these organizations historically struggled with test automation?
Usually the organization’s test automation strategy involves a dedicated team creating automated tests after the code is written. At least that was probably their approach before the Agile transition. It’s the strategy promoted by most of the test automation tool vendors back in the 90’s.
(Some of those vendors even persist in promoting that kind of test-last automation strategy today. Unbelievable but true. Don’t even get me started. I’ll just rant. It won’t be pretty.)
But until the organization adopts a new test automation approach, they’re stuck with what they have. And the result of the traditional approach is that:
- The test automation is being written in a language or tool that is only accessible (or known) to a small number of specialists. That group of specialists is a bottleneck.
- To get anything done, that group of specialists has to reverse engineer the developed software to figure out how to shoehorn automation onto an untestable interface, typically a GUI. (Ow. Pain. Ow.) And they have to go through the GUI: the system quite probably has business logic intertwingled with the GUI layer. Even if it doesn’t, the test automation specialists probably don’t know how to reach into the code base to bypass the GUI.
- The specialists may be test automation experts, but they are usually not professional programmers. That means that while they can make test automation tools sing, dance, and jump through hoops, they usually have not studied program design principles (SOLID, patterns). Most test automation code I see that’s written by specialists is kinda smelly. (I don’t blame them. My code was pretty dang smelly too when I was a test automation specialist. Pairing with real professional developers did a world of good for my programming skills.)
- The previous generation of commercial specialized test automation tools enforce their own architecture making it dang near impossible to apply software design principles even if you do understand them.
- The specialized commercial test automation tools usually cost an arm and a leg. (Need another license to execute the tests on the CI server? Fork over another 5 figures worth of licensing and maintenance fees.)
Bottom line: the reason test automation costs so much is that it’s done in a silo far removed from the development effort.
Buffered from the consequences of design decisions that decrease testability, the developers continue to create software that’s nigh onto impossible to automate.
And isolated from the technical expertise of how the software was constructed, the test automation specialists are in a situation where they cannot help but be both inefficient and ineffective.
Highly functioning Agile teams break down those silos. With the most effective Agile teams I’ve seen, everyone on the team is involved in automating the tests. And the automated tests go into the same shared source repository where the developers check in the code.
When we integrate the test automation effort with the development effort, we reduce the cost of test automation drastically. In doing so, we fundamentally change the cost/benefit tradeoff equation so that fully automated regression tests start looking like an achievable goal instead of an impossible pipe dream.
It’s time to stop imagining that test automation and programming are two separate activities done by different people with different job titles working in different departments.
That approach didn’t work particularly well with the V-Model, but it’s a flat out fail in an Agile context where programming and automating are part-and-parcel of the overall software development effort.