Logo Elisabeth Hendrickson’s Thoughts on Testing, Agile, and Agile Testing

Diluting the Tester Role

January 25th, 2008
Filed under Ruminations, Teamwork

In a comment on my last post, Shrini asked:

“While hitting hard at ‘Jargon’ based software testing experts, you also appear to give the impression that ‘testing’ is ‘everyone’s job’ (as quality) and seem to dilute the importance and role of testing in software world – you might want to clarify. Don’t you think that you can hit at these jargon creators without diluting the role of testing?”

Yes, thank you Shrini. I appreciate your question, and I would like to clarify.

I do believe that testing is so crucial that it is everyone’s job.

And I also believe that software development teams need people who have taken the time to become really good at testing.

I believe that these two ideas are compatible. Let me ’splain.

On more than one occasion I have worked with software development teams that were under such time pressure to write code, that they felt that they had no time available to test. They left the testing to a designated team of independent testers. The result in each case was code that was so bug-riddled that stabilizing it took months. During these long months of playing whack-a-bug, the testers and developers were constantly battling, management was perpetually screaming, and no one was happy. Predictably, even when the software did ship, it was prone to failure. In weekly meetings that followed the release, Tech Support let us all know that the quality was unacceptable and that the customers were cranky. Subsequent new development efforts were hampered by the need to patch critical bugs in the field. We were in technical debt up to our eyeballs.

(At this point, I’m guessing that at least one person I have never met in person is reading this and thinking “Wow! She worked here!” I probably didn’t; it’s an all-too-universal experience. Kinda like Dilbert.)

These experiences led me to write “Better Testing, Worse Quality,” a paper that explores the system effects that lead improvements in testing to yield even more fragile software.

And then I started working with Extreme Programming teams and discovered what I’d been missing. I experienced how Test Driven Development and Continuous Integration and Collective Code Ownership and Paired Programming and continuous Refactoring led to a solid code base. Project after project, I observed that the XP teams I was working with were achieving significantly better results than the code-and-fix teams of my past. Oh, the XP projects weren’t perfect. We still had bugs. But we didn’t play whack-a-bug for months on software that was supposed to be done-except-for-testing. We didn’t need stabilization phases. The software was always stable. It might not do everything yet, but what it did, it did well.

I also witnessed the power of Customer Acceptance Testing. I realized that the people responsible for defining the requirements are in the best position to determine whether or not the implementation matches their vision.

But at the same time, I recognized that I had something to offer the team. Compared to the professional programmers, my programming skills were meager. Compared to the product managers, my understanding of the product vision was superficial. But compared to any of the other team members, my understanding of where the risks and vulnerabilities were likely to hide was superb.

I used my skills to provide feedback to the team, to point out implementation bugs and requirements ambiguities and risks. I explored the implementation and reported what I found back to the team. I thought up tests no one else had thought up before.

But I did all these things in collaboration with the team. We pooled our knowledge. And we shared responsibility for testing activities. Testing became an integral part of the software development process. We could no more separate the development and testing activities than we could have separated our hands from our bodies, leaving our hands to type on the keyboard while the rest of our bodies took a break.

Perhaps paradoxically, spreading responsibility for testing to the whole team increased the overall test effectiveness. The testing mindset became pervasive. The test effort wasn’t diluted; instead it grew and flourished.

(I could write bad prose with corny analogies involving a comparison between diluting a drop of colored dye in water and spreading seeds in the wind. I could point out that inanimate objects become weak or disintegrate when spread, while living things - like ideas and knowledge - grow. But I won’t. Oh dret. I just did. Um, never mind.)

So anyway…bad analogies aside…

I do believe that everyone is responsible for testing, even on teams that don’t claim to be doing Agile or XP. The developers test that the code does what they expect and intend, and that new changes don’t break existing expectations. The business stakeholders test that the implementation is what they had in mind. Testers - those who have studied testing long enough and hard enough to get good at it - bring specialized skills to the table to support the rest of the team.

Testers apply critical thinking skills and analytical abilities, coming up with new questions to ask the software. “What if a user does this after that?” we ask. “What if the system is in this state when that happens? What if the data looks like this? What if we configure it like that?”

But testers are most effective when they do this with, not for, the team.

4 Comments

James Bach
Jan 25, 2008
1:51 pm

Testers serve the project team. Good testers work with the team.

But do you find that there are aspects and activities of testing that non-testers find boring and unnecessary? One of the problems I have with the policy that testing is everyone’s job is that many people don’t like doing it. You can *say* it’s their job, but you can’t compel people to treat that role with care in every detail.

I just had a fireplace insert installed. Obviously *I* am a very important stakeholder. You might say that I set the requirements for this stove. But I’m not interested in testing it. I just want to use it. When the installer finished his meticulous and interminable installation process, he announced that it must first be tested with a paper-only fire. My reaction was “what could go wrong? It’s a metal box! Just pack it wood and go for broke!”

That’s why I have a specialist do these things. I would cut corners. It’s my job to see that it is safely installed, perhaps, but my judgment and interest is not up to it.

I have also seen this in software projects. I saw it in the Mac System 7.0 project, where it was announced that “everyone’s job is to test System 7.0″ while those of us on other projects rolled our eyes and said to ourselves “Yeah, like our bosses are going to forgive us for delaying our other duties to play with System 7.0″

I saw it at the Agile Fusion conference, where a group of testers and developers were working on a demonstration project and Cem argued for the important of configuration testing. The developers (all of whom professed to consider testing important) expressed no interest in doing any of that. Now, maybe configuration testing was a good idea or maybe it wasn’t, but in the argument, I noticed that the tester was raising a standard testing issue, and the developers’ reaction sounded to me like kids being told to brush their teeth and wash behind their ears. Of course, I’m a tester. The need for configuration testing seems obvious to me.

Trained testers are generally biased toward “do more testing.” If I’m a project manager, I think I want someone with that bias on my team. I also want people with a different bias. I worry about a team that doesn’t have a diversity of biases and roles.

And the person who sets the requirements may be lousy at determining when they are met, because they want them met. They want to believe they are met. It may require a great deal of patience and wisdom to say “no, let’s do more testing.” Seeing something and deciding if you like what you see is only a small part of testing. A bigger part of it is deciding how to look, and how much looking to do, and slogging through all that looking.

 
Shrini Kulkarni
Jan 29, 2008
9:37 am

Elisabeth,

I am glad (or feel honored) that you wrote a whole blog post on my comment. However, I still feel that my original question has not been addressed completely.

Let me restate my major comment. The original post was titled as “obfuscating Jargon”. As a reader of that post, I expected to read about “obfuscating Jargons” and their pros, cons etc. The content of the post appear to have diluted by stating “Testing is everyone responsibility”, “Good testing is really very simple” (BTW these two statements are great in their own - fieldstones that can further really expand on and write many posts). I feel that the connection between the title of the post and these statements is no clear to me or it is not very direct. If you exclude these from the original post and just talk about the ill effects of jargons and how software community responds to them - the whole post reads just perfect. I, therefore, feel that you diluted the content of the post, first and then diluted the role of testing.

This post rightly addresses “dilution” aspect of tester role. I have few comments to make.

Testing is every one’s responsibility — OK, even if some of team are skilled or inclined or willing to do it? Please note, Testing requires wide variety and diverse skill sets. Few of us practice to be good at doing good testing. So, I am having trouble in accepting that “everyone can do good testing” - if they are willing to do any testing”. However, tester by collaborating with others in the team can make good testing happen so there has to be a testing leadership in the team.

>>>I do believe that testing is so crucial that it is everyone’s job.

and I believe that “Testing is so crucial that it can not not be left to every one - common responsibility”.

So, the problems I see with your assertion and subsequent explanation of how everyone can do or own testing are -
1. Not all are skilled at doing good testing
2. Not all may be willing to do any testing (good or bad)
3. “everybody-somebody” problem - when something is left to everyone, no one actually takes it seriously and does something good.

>>>And I also believe that software development teams need people who have taken the time to become really good at testing.

I agree … We do need such developers but then it is their secondary responsibility. So we do need people whose sole work is do testing so that they bring relevant skills.

>>>They left the testing to a designated team of independent testers. The result in each case was code that was so bug-riddled that stabilizing it took months.

What do you think was the reason(s) - independent testing? Testing that is completely left to the testers? or management’s inability to handle the bug inflow and factor appropriate development effort?

If the code is riddled with bugs - was it due to testers? I would say they have done good job and exposing as many bugs they can and it was program/project management teams failure to manage corresponding development effort. What do you say?

>>>I experienced how Test Driven Development and Continuous Integration and Collective Code Ownership and Paired Programming and continuous Refactoring led to a solid code base.

Well, I think, you are very well aware that having a solid code base (of the product under test) is one part (small) part of big universe in which software operates. Being an associate of Jerry Weinberg, you would appreciate the idea that software is a system and during its operation, interactions with various other systems - platform (OS etc), Organization and business process, Social system, Cultural system of users and so on.
That is the big picture and it is complex to test such interrelated sets of interactions. That is why we need testing experts who system thinkers and model the systems and their interactions. So code base is one of several systems and impact of having a good or not-so-good code base needs to be viewed in this context.

It is my opinion that agilists (James Bach can help here - not sure if it is “a”gile or “A”gile) and extreeme programming propents so how seem to overlook the complex picture of some many diverse systems interacting while a softare is in use. They seem to simplify that “other” systems are like “constants” in a mathematical equation. It is like the hypothesis that newton made while discovering law of gravitation between two bodies. Newton hypothesised that the force of attraction between two bodies all about those two bodies (mass and distance) and ignored the effect of any third body.

The Agile and Extreme programming world also appear to be traveling the path of Newton. Somehow we need to balance analytical thinking (understanding complex behaviors of a system by studying the individual parts of the system) by systems thinking which focuses on the big picture and the dynamics and dependency interactions of constituent parts.

I believe “systems thinking” is missing largely in agile community. I could be wrong - I am saying this as I have not seen examples of systems thinking in agile world.
What are your views?

 
Bruce Daley
Feb 20, 2008
4:27 pm

It seems to this observer the key sentence in your last post was “I used my skills to provide feedback to the team”. At heart testing is not about finding bugs, at heart it is about providing feedback - balanced, meaningful, actionable feedback that moves the software development process forward. What could be a greater enemy of feedback than obfuscating jargon?

Keep up the good work!

 
Michael Ludgate
Apr 22, 2008
12:28 pm

In response to Shrini mainly, and I felt it needed clarifying (or exploring if not agreed); A Tester as a job position, is a super-set of the activity of testing. I believe Elisabeth is referring to ‘testing’ as an activity. As a tester I perform a range of activities other than a more dictionary style definition of tester: technical manual writer, in-line help authoring, proof-reading, specification interpretation, moral & legal adviser, security analyst, stand-in-customer, customer liaison, support desk training, product guru, researcher, automated scripter, trend analyst.

While I except that my situation (while not unique) in a small company is not mirrored in larger organisations. I’m positive that every tester as some point has been required to perform a task that isn’t even defined by our non test-obsessed colleagues as ‘testing’.

You can’t do a code walk-through by yourself, you can’t get the development team to add testable features or consider writing anything with a test-centric manor - by yourself. Unfortunately I’m a pupil of the old skool, where developers shriek away from testing as some dirty task, performed by aspiring programmers.

The value of the tester as a position (if filled reasonably well) is the ability of that individual to provide concise objective unbiased approach to tasks that require it.

It’s said that familiarity breads contempt, however I’ve found that separation does the same and worse. Working closer with other parts of the company should be seen as an opportunity to innovate processes. Spend time thinking about your job role and those around you, work closer but retain your sense of identity. To continue with Elisabeth’s reference to ‘corny analogies’ (I love my corny analogies, and certainly don’t aspire for good prose) A tester who works 100% in isolation, accepting release candidates and returning bug lists is closely akin to a potted plant. Knowledge of the big picture, contact with the stakeholders, credibility and of course strong roots, can easily be forfeited for isolation.

 

Leave a Comment

Because of the rise in blog-spam, I've turned on comment moderation. If it takes a while before your comment appears, I hope you understand.

Moderation Policy: I approve substantive comments. I reject ads. And if I don't know whether it's substantive or advertising, it sits in my moderation queue until I get sick of looking at it, at which point I reject it, kind of like the questionable meatloaf in the fridge. But please be assured that I think long and hard before clicking that reject link. I really am grateful for every comment any human takes the time to make. (Spambots, not so much. But if you're reading this, you're probably human.) So please contribute to the conversation...