Testing is a Whole Team Activity

I talk to a lot of people in organizations that use some flavor of Agile. Almost all of them, even the teams that are succeeding wildly with Agile, struggle with testing. It’s easy to say that we test throughout the cycle. It’s harder to do it.

Some teams are really struggling with testing, and it’s affecting their ability to get stories all the way done in a timely manner and with a high enough level of quality. I hear comments like these frequently:

“We’re implementing stories up to the last minute, so we can never finish testing within the sprint.”

“We are trying to automate the acceptance tests, but we’re about to give up. The tests get out of sync with the code too easily. Nine times out of ten, when the build is ‘red’ it’s because the tests are wrong.”

“I’m afraid we’re missing bugs because we never have time to explore the system. But we’re too busy running the regression tests to take time out to explore.”

Using a variation on the 5 Why’s, I dig into the issue with these folks. What I’ve found is that there is one common unifying root cause at the heart of all these challenges:

There is an (unfortunate) belief that testers test, programmers code, and the separation of the two disciplines is important.

In some cases, people within the organization hold this belief explicitly. They subscribe to the notion that the only valid testing is that which is done by an independent tester. Just in case you happen to be among that group, let me dispel the programmers-can’t-test myth right now.

Programmers most certainly can test. Anyone who can wrap their heads around closures and patterns and good design is perfectly capable of wrapping their heads around risks and corner cases and test heuristics. For that matter, some of the best programmers I’ve worked with also turned out to be some of the best testers.

Perhaps your objection is a little different: “Sure, programmers can test,” you say. “But they can’t be objective about their own code. They could test someone else’s but not their own.”

Well, yes. Blindspots tend to perpetuate.

However, as both a tester and a programmer I can tell you that at least for me, time pressure is much more of an issue than inherent subjectivity.

When I feel time pressure, I rush. When I rush, I forget stuff. Later when I find bugs in production, it’s in the areas that I forgot about, in the places where I rushed. Just testing someone else’s code won’t address the problem that time pressure leads to rushing.

However, pairing can address both problems: subjectivity and rushing the job. Pairing with someone else while testing—say, for example, having a programmer pair with a tester—can both ensure we’re testing from multiple perspectives and also that we’re not unduly rushing through while failing to notice that the installer just erased the hard drive.

In other cases, however, the people I am talking to already buy into the idea that programmers can test.

“We don’t suffer from the belief that testers and programmers should be kept separate,” they object. “We believe programmers should test! And our programmers do test! But we still struggle with finishing the regression testing during a sprint.”

“If everyone on the team believes in programmers testing, why aren’t the programmers pitching in to run the manual regression tests?” I counter.

“Because they don’t have time…”

“…because they’re too busy writing new code that the testers won’t have time to test?”

“Um, yeah…”

“Right. You’re telling me testers test and programmers code.”

“Oh.”

So, back to our original problem: the team is struggling to complete testing within a sprint.

Throwing more testing bodies at the problem will not solve the issue. It will result in spending time to bring the new testers up to speed and to filter through large swaths of feedback that doesn’t actually help move the project forward.

Throwing a separate team of test automators at the problem might work as a temporary band-aid but it will end up being very inefficient and expensive in the long run. The separate team of test automators won’t be able to change the source code to improve testability so they will spend more time fighting the code than testing it.

The long term sustainable solution is both simple and brutally difficult: recognize that testing and quality are the responsibility of the whole team, not any given individual.

This is so much easier said than done. Sure, we can say “everyone is responsible for testing and quality.” But when it’s the end of the sprint and the product owner is pushing for more features, it takes an enormous amount of strength and courage to say, “We have undone testing tasks stacking up. Coding more features will not help. We need to focus on testing what we already have.”

For that matter, spending programmer time on making automated tests execute faster and more reliably might seem like pure indulgence in the face of project deadlines.

And internal process metrics that measure programmers and testers separately just exacerbate the problem. Any time programmers are measured on lines of code, checkins, or implemented story points, while testers are measured on defect counts and test cases executed, we’re going to have problems getting team members to see testing as a whole team responsibility.

But when we can get the team to see testing as part of developing a working solution, wonderful things can happen.

  • Our inventory of coded-but-not-tested stories dissipates as stories no longer languish in the “To Be Tested” column on the task board. We no longer have to deal with the carrying cost of stories that might or might not work as we intended.
  • Programmers executing manual regression tests are in a better position to see both opportunities to automate, and also opportunities to pare down duplication.
  • Testers and programmers can collaborate on creating test automation. The result will be significantly better automation than either testers or programmers would have written on their own, created much more efficiently.
  • As the level of regression test automation increases, testers have more time to do the higher value activity of exploratory testing.

Testing is an activity. Testers happen to be really good at it. We need testers on Agile teams. But if we want real agility, we need to see that completing testing as part of the sprint is the responsibility of the whole team, not just the testers.

And that means we have to do away with the barriers—whether beliefs or metrics or external pressure—that reinforce the “testers test, programmers code” divide.

If you or someone you know is struggling with the question of how to integrate testing throughout the lifecycle in Agile, please check out the Agile Testing Class I’m teaching with Dale Emery at Agilistry Studio on October 11 – 13.

28 Responses to Testing is a Whole Team Activity

  1. Curt Hibbs September 10, 2011 at 7:19 am #

    There should not be a “To Be Tested” column on the task board at all. The story code and its test should be considered an inseparable unit, and it should not be allowed to check in the code without checking in the corresponding test. The test is part of the definition of done. No test means the story is not done… period.

    That simple rule addresses nearly every issue raised in the post.

  2. testobsessed September 10, 2011 at 12:59 pm #

    I agree with Curt. Too often, however, I see task boards with a test column. (The worst one was when test happened AFTER customer accept. It didn’t work very well. Shocking, I know.)

  3. Laurent Bossavit September 11, 2011 at 2:06 pm #

    I ascribe the persistence of this unfortunate belief largely to Glenford Myers’ 1976 book “Software Reliability – Principles and Practices” which asserted as an “axiom” that “Developers shouldn’t ever test their own code”. This was interpreted way too liberally (for instance, even if narrowly true, developers could test each other’s code) as a reason to separate the activities.

    It isn’t at all clear to me that the separation into two distinct roles “tester” and “developer” is doing the industry more good, overall, than it does harm. But even without questioning this division, it is obvious that many harmful anti-patterns arise from building tall walls between the two.

  4. Halperinko - Kobi Halperin September 12, 2011 at 4:38 am #

    Very nice post, as a tester who has not done agile yet, I wonder how you might cope with larger activities, where regression backlog becomes much bigger than the actual newly added content of that iteration.

    Halperinko – Kobi Halperin

  5. Flora September 12, 2011 at 11:12 am #

    Interesting read

  6. Otávio Macedo September 12, 2011 at 2:34 pm #

    Perfect!

    This kind of artificial separation is the root of many, many problems in software development. And it is not only about “developers vs. testers”. See, for example, the separation “developers vs. operations”, another big source of problems (hence, DevOps). But the good news is that the agile community is starting to pay more attention to the problem and I hope things will get better in the near future.

    Aside from the question of separation, the broad substance of your text comes down to “programmers are doing a poor job at testing.” But it seems that you’ve left out the other side of the coin: most testers I’ve have worked with have no interest or skills in programming. I would expect, at the very least, that they could contribute some ideas to the overall design of the applications they test. Could you write something about it?

  7. Dave Simpson September 12, 2011 at 2:44 pm #

    While I work for a company where developers write and execute tests, these are all automated. When it comes to manual testing dedicated testers are used because the argument is that a Developer costs xx/hour while a tester might be half that. So it is economically more prudent (in management’s eyes) for developers to spend their time writing code not pushing buttons. Again I stress we have a good comprehensive set of automated acceptance/integration tests as well as unit – however we do lack the bit in the middle – component, functional, whatever you want to call them.

  8. Eric Jimmink September 12, 2011 at 2:55 pm #

    Great post. I certainly agree that a team ought to be able to cope without a “to be tested” column.
    If it does exist on a board, then I urge the team to use a simple (Kanban) rule to put a limit on the number of items in that column. Say, for instance, that a team member places a fourth item in the test column, and that number equals the limit. Then, by definition, no new user story may be picked up. Instead, that team member must assist in testing one of the other three stories, and help get those to Done first.

    As a side note: the issue was raised that most people have some ‘blind spot’ in testing their own work. If a team is large enough, then for each story one or more developers may agree during sprint planning not to pick up any coding tasks for that story, as they will likely end up assisting in testing.

  9. Erwin September 12, 2011 at 2:59 pm #

    I agree with Otavio, as a tester you can raise questions on the designs, help clarify the designs and requirements and suggest directions for solving defects. This way you can add value as well as save time and money.

    Furthermore, I don’t see a problem with having a ‘Test’ column on the board. It helps to visualize progress of seperate tasks. What really helps in my team is having demo sessions with the whole team so that everyone, be it tester or programmer, gains a comprehensive view of the application. In these session issues are pinpointed due to mutual insight and every member regains focus on the sprint goals. Although I am a dedicated tester, no one has the illusion that everything can be tested on the final day of the sprint.

  10. Marcus September 12, 2011 at 2:59 pm #

    The root cause of the lack of / quality of testing is in most cases the glaringly obvious bit you mention in the paragraph
    “This is so much easier said than done. Sure, we can say “everyone is responsible for testing and quality.” But when it’s the end of the sprint and the product owner is pushing for more features, it takes an enormous amount of strength and courage to say, “We have undone testing tasks stacking up. Coding more features will not help. We need to focus on testing what we already have.””

    More often than not Sales / Customer Service want to wow their customers with as many features as possible to show how ‘capable’ the company is, but this soon falls apart when the features delivered have bugs-a-plenty and the customer loses confidence in the application as a whole. Better to have one feature less this sprint with what is shipped being well tested, but try telling that to your sales person who will likely dismiss the testing as unnecessary chaff in comparison to adding the extra feature they promised.. (until of course they are sitting in a room demoing the system with the new features only for the customer to spot bugs here and there.. upon their return the sales person is sure to start yelling at lack of proper testing!)

    Developers know they should test, testers know they should test, sales people want to sell the product and company and being sales are often more adept at persuading the higher up bodies where resources should be prioritised.. e.g. in the favour of sales.

  11. Sathish September 12, 2011 at 3:59 pm #

    Valuable post. Any thoughts on what would you recommend if testing and development teams are not co-located?

  12. Mark Wilden September 12, 2011 at 4:11 pm #

    I’m surprised to read an article (and its comments) about agile and testing without seeing TDD mentioned. While TDD isn’t a quality assurance tool – some say TDD isn’t even about testing – it certainly mitigates many of the scenarios described here.

  13. Meinte Boersma September 12, 2011 at 5:58 pm #

    “Anyone who can wrap their heads around closures and patterns and good design is perfectly capable of wrapping their heads around risks and corner cases and test heuristics.”

    Given that this is the precondition for dispelling the programmers-can’t-test myth, apparently you’ve been fortunate enough to only have worked with people satisfying it. Unfortunately, I’ve seen enough “programmers-can’t-program truth” as well ;) On the other hand, I also know testers that can program – not good enough to be developer but good enough to code up automated tests :)

    Nice post!

  14. Santosh September 12, 2011 at 6:25 pm #

    Well I think if the testers start testing very early in the process even when a user story is not fully done, and work closely with the developers with the findings instead of getting busy and waste time tracking the number of bugs he/she found in the bug tracker the quality may improve depending on the team dynamics. I tend to question the testing team members on what they can do to reduce the number of bugs in the QA box instead of focusing on finding out bugs before it is released to the customer for demo.

    A tester should share and walk through the test cases with the programmer while he is developing the application to reduce the number of bugs and focus on finding bugs early in the process by testing not fully developed user stories even. I think it needs skill to test half baked user stories and still add value.

  15. SuperMario September 12, 2011 at 7:15 pm #

    Testing is a Whole Team Activity: I do agree. But putting this into practice is just not that simple. Congratulations to those who made this work, but for the rest of us, we have a mountain to climb. Most testers like breaking things and most developers like creating useful software. This in my opinion is two completely different mindsets. I have seen great success where a developer pairs with a tester. As a team, they have achieved so much more in one sprint. So it can be done!

    How do you enforce this in Agile though? The team decides right? If left to the team to decide, it won’t happen in my opinion. I can just hear the argument: “I did not study computer science for three years or pass the Java certification exam, to test software!”

  16. Marco September 13, 2011 at 6:37 am #

    Hi SuperMario, if someone came to me with that argument I would show them the door. Imagine the quality of his/her unit tests and therefore code…

    Elisabeth, I couldn’t agree more with your assessment. I find that a similar type of problem is on the analysis front, where the perception is that all analysis is done by BAs, as opposed to collaboratively specifying. The result is that often teams pile up ‘inventory’ of user stories which won’t be touched for months, and when the time comes to address them, they are out of date.

    I suppose the underlying principle seems to be that true agility is not possible without applying Lean principles.

    To those interested on these topics, you may wish to read “Bridging the Communication Gap” and “Specification by Example – How successful teams deliver the right software” by Gojko Adzic.

  17. Eb September 13, 2011 at 12:54 pm #

    This is all about the mindset. I think practices fo ATDD, BDD, TDD etc etc help to harmonize the agile teams thoughts surrounding testing as a practice. That being said, there are still different types of tests and if you have testers that represent end users, then there is nothing wrong with a “test” column on the board – as long as you are clear as to why it is there. For example, exploratory testing by testers is done at that point.

    Eb

  18. Fabian September 13, 2011 at 2:14 pm #

    Quick comment on the task board comment:
    You can have this column, but it has to have a sensible meaning.

    In a project we had this column, and it was called “verify”.
    But for a card to reach this column, we already had written unit or acceptance tests.

    The point was to have somebody from the team who did not actively work on the task check the task with a more “outside” view. Perhaps she will run the test differently or expect something different.

    So verify is more a safty net, or four-eyes principle, rather the addition of missing tests.

    Verify ranged from easy “yep, runs on integration server fine” to “the design looks ok, and code is readable, the test description matches the story and demoing the feature gives great wow. well done”

    Fabian

  19. Oaz September 15, 2011 at 6:07 am #

    The ISO 13485, a standard which describe requirements for the design of medical devices states the following :
    “Top management shall establish the interrelation of all personnel who manage, perform and verify work affecting quality, and shall ensure the independence and authority necessary to perform these tasks.”

    I’m pretty sure that there are standards in other areas saying the same thing : the output of a task shall be verified by a person that is independent of the person performing the task.

    How can one be compliant to those standards and do not fall into the “programmers-can’t-test” trap?

  20. BJ September 18, 2011 at 5:54 pm #

    To me, “programmer-can’t-test” has a lot to do with the “testing-at-the-end” anti-pattern. Suppose a story is divided into two task; a) implementation, b) testing, and I put on my programmer hat and take on the implementation task. The time I would state the the implementation task is done, is when I have convinced myself that the implementation works. Then someone might suggest that I take on the testing task. Ridiculous, neither do I need nor do I want to test the story – I am convinced that the story works, if I were not convinced it works, I would still be doing the implementation task.
    The only time I can put on a tester hat and put my mind to find all the corner cases that need to be handled, is before I’ve finished the implementation task. This leads us to to the Specification By Example/ATTD practice on the story level, and the “red – green – refactor”/TDD practice on the code level. First implement, then test – why would I want to do that?
    BJ

  21. Michael Mayer October 2, 2011 at 12:58 pm #

    I agree with this blog post. However: If you have a “semi-agile” approach with very comprehensive specs (that were written upfront by the customer), you need separate testers that know all specs very well in order to confirm that the feature was implemented in the right way. In a large team, not everyone can understand the whole project in all details and has all browsers installed (in case of Web apps). So a “To Be Tested” ticket status makes sense, in many environments. Also, most teams should first work on useful unit tests before they invest time in extensive acceptance tests (especially in the beginning, when the user interface might change a lot).

  22. Oana October 4, 2011 at 12:28 pm #

    Excellent post! While reading I had the impression you’re talking about the company I work for. We have the same issue.. Fast delivery, testing on the last minute…Ups tehre are some things left untested due to the lack of time..and so on…
    I agree that “if we want real agility, we need to see that completing testing as part of the sprint is the responsibility of the whole team, not just the testers. ”
    I also think this is the responsibility of PMs to track the project status. PMs work with estimates, they know how much is assigned to development, testing, etc. A good tracking of project status could avoid functionalities left untested. I agree there could be some exceptions (the case when customers request some Change Requests – new functionality, to be delivered with the original delivery).

  23. Riaz October 10, 2011 at 11:55 am #

    I could not agree more with Santosh.

    The company I recently worked for very much likes to separate tasks and creates exactly the sorts of problems mentioned in the article. I have 3 developers I worked with who were very very keen to work closely with me to produce a much better end product. Constant communication, lots of questions back and forth, consultation with users and management and back we go, round and round again. Using this super quick, open & direct communication the testing times were reduced, alignment of the end product with the business expectations was brilliant and the solution was readily accepted.

    The relationships within teams as with all things in life, need to be open, constructive, supportive and focused on the end goal always.

    It takes a village…..

  24. Logan January 13, 2012 at 5:52 am #

    Tester attitude is much more important. Their job is not to break things. They should work closely with the developer programmers to help them understand the piece of software they are writing. More often developer programmers have issues with technology and how to do things and tend to forget (concentration lapse) on what they are doing.

    From the testing point of view, developer programmers can help test automation by building the framework and utilities (ui controls repository, functions to interact with application and so on), where as the testers can focus on manual testing and easy automation of the system using the utilities created by the developers.

    I don’t buy the idea of asking the developers to execute manual test cases. that simply will not always work in the real world.

    Co-location is imperative to achieve this. Very good article indeed.

  25. Uday January 17, 2013 at 6:27 am #

    Great Post!
    I agree with what Loagn said, in all the teams I have worked so far many of the developers put less effort in understating the complete functionality or business use of the application. They tend to concentrate on implementation of task in hand. Hence co-operation will be the key.

Trackbacks/Pingbacks

  1. Новая рубрика «Полезное чтиво» « XP Injection - September 12, 2011

    [...] Testing is a Whole Team Activity – отличная статья о том, что тестирование является общей командной активностью, без которой не получится выпустить качественный продукт [...]

  2. Verantwortung für Qualität und Tests liegt im Team - Inspect & Adapt - September 15, 2011

    [...] ihrem lesenswerten Blogeintrag “Testing is a Whole Team Activity” geht Elisabeth Hendrickson den Gründen auf den Grund. Vielen Dank an cfisch3r für den [...]

  3. The Embedded Tester « Manage to Test - April 12, 2012

    [...] In Agile, testers aren’t “embedded” on teams any more than programmers are, or analysts, or any other role that is needed on the team. To say that they are suggests that this is an option, or a particular strategy you might employ to help with Agile development. It’s not! It’s an essential part of it! [...]

Leave a Reply