I talk to a lot of people in organizations that use some flavor of Agile. Almost all of them, even the teams that are succeeding wildly with Agile, struggle with testing. It’s easy to say that we test throughout the cycle. It’s harder to do it.
Some teams are really struggling with testing, and it’s affecting their ability to get stories all the way done in a timely manner and with a high enough level of quality. I hear comments like these frequently:
“We’re implementing stories up to the last minute, so we can never finish testing within the sprint.”
“We are trying to automate the acceptance tests, but we’re about to give up. The tests get out of sync with the code too easily. Nine times out of ten, when the build is ‘red’ it’s because the tests are wrong.”
“I’m afraid we’re missing bugs because we never have time to explore the system. But we’re too busy running the regression tests to take time out to explore.”
Using a variation on the 5 Why’s, I dig into the issue with these folks. What I’ve found is that there is one common unifying root cause at the heart of all these challenges:
There is an (unfortunate) belief that testers test, programmers code, and the separation of the two disciplines is important.
In some cases, people within the organization hold this belief explicitly. They subscribe to the notion that the only valid testing is that which is done by an independent tester. Just in case you happen to be among that group, let me dispel the programmers-can’t-test myth right now.
Programmers most certainly can test. Anyone who can wrap their heads around closures and patterns and good design is perfectly capable of wrapping their heads around risks and corner cases and test heuristics. For that matter, some of the best programmers I’ve worked with also turned out to be some of the best testers.
Perhaps your objection is a little different: “Sure, programmers can test,” you say. “But they can’t be objective about their own code. They could test someone else’s but not their own.”
Well, yes. Blindspots tend to perpetuate.
However, as both a tester and a programmer I can tell you that at least for me, time pressure is much more of an issue than inherent subjectivity.
When I feel time pressure, I rush. When I rush, I forget stuff. Later when I find bugs in production, it’s in the areas that I forgot about, in the places where I rushed. Just testing someone else’s code won’t address the problem that time pressure leads to rushing.
However, pairing can address both problems: subjectivity and rushing the job. Pairing with someone else while testing—say, for example, having a programmer pair with a tester—can both ensure we’re testing from multiple perspectives and also that we’re not unduly rushing through while failing to notice that the installer just erased the hard drive.
In other cases, however, the people I am talking to already buy into the idea that programmers can test.
“We don’t suffer from the belief that testers and programmers should be kept separate,” they object. “We believe programmers should test! And our programmers do test! But we still struggle with finishing the regression testing during a sprint.”
“If everyone on the team believes in programmers testing, why aren’t the programmers pitching in to run the manual regression tests?” I counter.
“Because they don’t have time…”
“…because they’re too busy writing new code that the testers won’t have time to test?”
“Right. You’re telling me testers test and programmers code.”
So, back to our original problem: the team is struggling to complete testing within a sprint.
Throwing more testing bodies at the problem will not solve the issue. It will result in spending time to bring the new testers up to speed and to filter through large swaths of feedback that doesn’t actually help move the project forward.
Throwing a separate team of test automators at the problem might work as a temporary band-aid but it will end up being very inefficient and expensive in the long run. The separate team of test automators won’t be able to change the source code to improve testability so they will spend more time fighting the code than testing it.
The long term sustainable solution is both simple and brutally difficult: recognize that testing and quality are the responsibility of the whole team, not any given individual.
This is so much easier said than done. Sure, we can say “everyone is responsible for testing and quality.” But when it’s the end of the sprint and the product owner is pushing for more features, it takes an enormous amount of strength and courage to say, “We have undone testing tasks stacking up. Coding more features will not help. We need to focus on testing what we already have.”
For that matter, spending programmer time on making automated tests execute faster and more reliably might seem like pure indulgence in the face of project deadlines.
And internal process metrics that measure programmers and testers separately just exacerbate the problem. Any time programmers are measured on lines of code, checkins, or implemented story points, while testers are measured on defect counts and test cases executed, we’re going to have problems getting team members to see testing as a whole team responsibility.
But when we can get the team to see testing as part of developing a working solution, wonderful things can happen.
- Our inventory of coded-but-not-tested stories dissipates as stories no longer languish in the “To Be Tested” column on the task board. We no longer have to deal with the carrying cost of stories that might or might not work as we intended.
- Programmers executing manual regression tests are in a better position to see both opportunities to automate, and also opportunities to pare down duplication.
- Testers and programmers can collaborate on creating test automation. The result will be significantly better automation than either testers or programmers would have written on their own, created much more efficiently.
- As the level of regression test automation increases, testers have more time to do the higher value activity of exploratory testing.
Testing is an activity. Testers happen to be really good at it. We need testers on Agile teams. But if we want real agility, we need to see that completing testing as part of the sprint is the responsibility of the whole team, not just the testers.
And that means we have to do away with the barriers—whether beliefs or metrics or external pressure—that reinforce the “testers test, programmers code” divide.
If you or someone you know is struggling with the question of how to integrate testing throughout the lifecycle in Agile, please check out the Agile Testing Class I’m teaching with Dale Emery at Agilistry Studio on October 11 – 13.