“They never give us enough time to automate our tests, and then they
complain at us that we don’t test fast enough!” J. shook her head.
“And when I want to hire more people to help automate, they tell me I
have too many people already! Management blames me because testing
takes too long, but they won’t support me in fixing the problem.
What’s wrong with them!?!”

J. is a QA manager in an organization that’s adopting Scrum. She’s
frustrated, and understandably so. From her point of view, she’s
being squeezed in all directions. The developers are producing
releasable code every month. But for her team to run a regression
test cycle - mostly manual - takes 6 weeks. That’s too long. Just
one test cycle exceeds the sprint length by 2 weeks. J. feels
tremendous pressure to reduce the time it takes to test the software.
Yet at the same time, she feels like she’s not getting any support to
do the one thing she can see that will help reduce the test cycle
time: automate the regression tests.

I’ve had some visibility into J.’s situation for some time now. J.’s
team has been trying - and failing - to automate the regression suite
for the last two years. They aren’t making any headway because as
soon as they get one script working, another one breaks. The
automation is brittle, error-prone, and incredibly expensive to create
and maintain. That’s in part because they’ve been using a cumbersome
commercial tool that doesn’t support creating maintainable tests.
It’s also because the user interface was not designed with test
automation in mind. Many UI elements don’t have IDs, and the ones
that do use automatically generated IDs that change with each build.
In short, the combination of the tool and the software under test
equals a test automation nightmare. It’s no wonder J.’s team is not
making headway.

Yet J. persists. Doing more of the same kind of test automation
that’s already failing doesn’t make much sense to me, but she
disagrees. “We just need more time!” she says.

The problem is that J. is still thinking in terms of silos. She
thinks all testing tasks must be done by QA people using specialized
QA tools. It simply would not occur to her to suggest that
development help automate tests. Nor does she suggest that developers
and testers collaborate on making the UI more testable. Instead, she
says, “QA can’t go that fast. Slow down.”

J. doesn’t want to acknowledge that test automation created by a
siloed QA team working in isolation to reverse-engineer existing
software and automate tests against an untestable UI using proprietary
tools accessible only to a few select team members is guaranteed to be
incredibly expensive both to create and to maintain, and also
ridiculously fragile. In short, her approach just isn’t going
to work.

Unfortunately, J.’s story is likely to have an unhappy ending - at
least for J. and her team. Her strategy of trying to get development
to slow down, and telling management that they can’t release monthly,
is backfiring. The development team is already bypassing QA for small
changes and getting good results. But J. is undeterred. My past
observations tell me that no matter what the reaction of the people
around her, she will keep doing the same thing and expect different
results.

But maybe, just maybe, by telling J.’s story here, I can help someone,
somewhere.

So allow me to repeat the moral of this story:

When QA works in isolation, creating automated tests after the
software is theoretically “done,” using proprietary tools that are
available only to a select few team members, the results will be a
fragile, unmaintainable mess.

For test automation to work well, it must be created in collaboration
with the whole team and the resulting test automation code must be
treated as code. That means it should be versioned with the source
code, executed with each and every build, and created and maintained
as part of the overall development cycle rather than as an afterthought.

And when a Test/QA group insists on keeping within their silo when the
rest of the organization adopts Agile practices, they will end up
bypassed and irrelevant as the rest of the organization finds ways to
move forward without their help.

In a comment on my last post, Shrini asked:

“While hitting hard at ‘Jargon’ based software testing experts, you also appear to give the impression that ‘testing’ is ‘everyone’s job’ (as quality) and seem to dilute the importance and role of testing in software world – you might want to clarify. Don’t you think that you can hit at these jargon creators without diluting the role of testing?”

Yes, thank you Shrini. I appreciate your question, and I would like to clarify.

I do believe that testing is so crucial that it is everyone’s job.

And I also believe that software development teams need people who have taken the time to become really good at testing.

I believe that these two ideas are compatible. Let me ’splain.

On more than one occasion I have worked with software development teams that were under such time pressure to write code, that they felt that they had no time available to test. They left the testing to a designated team of independent testers. The result in each case was code that was so bug-riddled that stabilizing it took months. During these long months of playing whack-a-bug, the testers and developers were constantly battling, management was perpetually screaming, and no one was happy. Predictably, even when the software did ship, it was prone to failure. In weekly meetings that followed the release, Tech Support let us all know that the quality was unacceptable and that the customers were cranky. Subsequent new development efforts were hampered by the need to patch critical bugs in the field. We were in technical debt up to our eyeballs.

(At this point, I’m guessing that at least one person I have never met in person is reading this and thinking “Wow! She worked here!” I probably didn’t; it’s an all-too-universal experience. Kinda like Dilbert.)

These experiences led me to write “Better Testing, Worse Quality,” a paper that explores the system effects that lead improvements in testing to yield even more fragile software.

And then I started working with Extreme Programming teams and discovered what I’d been missing. I experienced how Test Driven Development and Continuous Integration and Collective Code Ownership and Paired Programming and continuous Refactoring led to a solid code base. Project after project, I observed that the XP teams I was working with were achieving significantly better results than the code-and-fix teams of my past. Oh, the XP projects weren’t perfect. We still had bugs. But we didn’t play whack-a-bug for months on software that was supposed to be done-except-for-testing. We didn’t need stabilization phases. The software was always stable. It might not do everything yet, but what it did, it did well.

I also witnessed the power of Customer Acceptance Testing. I realized that the people responsible for defining the requirements are in the best position to determine whether or not the implementation matches their vision.

But at the same time, I recognized that I had something to offer the team. Compared to the professional programmers, my programming skills were meager. Compared to the product managers, my understanding of the product vision was superficial. But compared to any of the other team members, my understanding of where the risks and vulnerabilities were likely to hide was superb.

I used my skills to provide feedback to the team, to point out implementation bugs and requirements ambiguities and risks. I explored the implementation and reported what I found back to the team. I thought up tests no one else had thought up before.

But I did all these things in collaboration with the team. We pooled our knowledge. And we shared responsibility for testing activities. Testing became an integral part of the software development process. We could no more separate the development and testing activities than we could have separated our hands from our bodies, leaving our hands to type on the keyboard while the rest of our bodies took a break.

Perhaps paradoxically, spreading responsibility for testing to the whole team increased the overall test effectiveness. The testing mindset became pervasive. The test effort wasn’t diluted; instead it grew and flourished.

(I could write bad prose with corny analogies involving a comparison between diluting a drop of colored dye in water and spreading seeds in the wind. I could point out that inanimate objects become weak or disintegrate when spread, while living things - like ideas and knowledge - grow. But I won’t. Oh dret. I just did. Um, never mind.)

So anyway…bad analogies aside…

I do believe that everyone is responsible for testing, even on teams that don’t claim to be doing Agile or XP. The developers test that the code does what they expect and intend, and that new changes don’t break existing expectations. The business stakeholders test that the implementation is what they had in mind. Testers - those who have studied testing long enough and hard enough to get good at it - bring specialized skills to the table to support the rest of the team.

Testers apply critical thinking skills and analytical abilities, coming up with new questions to ask the software. “What if a user does this after that?” we ask. “What if the system is in this state when that happens? What if the data looks like this? What if we configure it like that?”

But testers are most effective when they do this with, not for, the team.

I just got back from CITCON where I met an amazing group of incredibly cool people.

At dinner after the conference, a group of us compared “First Programming Experiences.” Me, I wrote my first lines of code in BASIC on a trash-80 when I was in 8th grade. The guy sitting to my right, Zach, wrote his first code on a Commodore 64. The guy across the table, Matt, wrote his first lines of code in HTML on a Windows laptop when he was something like 9. That should you an idea of our relative ages.

Matt learned to program in an entirely different era than Zach and me. Back when Zach and I were learning to program, we couldn’t just Google for an answer. We couldn’t order a tech book from Amazon. Wikipedia wasn’t an option. We couldn’t even post our question to a news group. (If we’d happened to have tech savvy parents, we might, maybe have had access to a BBS. But neither of us were so lucky.) We could ask the people around us for ideas answers. But if you happened to be the only geek with one of those early personal computers among your circle of friends, the chance of getting an answer was pretty slim.

In telling us the story of how he learned to code, Zach lamented: “The one thing I could never figure out was how to do a raster interrupt.”

It occurred to me that with about 30 people assembled, many of them geeks about our age, surely one of them knew how to do raster interrupts on a Commodore 64. “Let’s try a social experiment,” I suggested. “Write your question down and we’ll send it around the group to see if anyone knows the answer.” So Zach wrote his question on a cocktail napkin (the only thing handy). And we sent it around.

Most people just shook their heads and laughed. But one person hollered back down the table, “Who wants to know how to do a raster interrupt on a Commodore 64?”

“I do,” Zach hollered back.

“Well I don’t remember the exact syntax,” came the reply. “But you access the upper level memory registers. There wasn’t much documentation on them at the time, so figuring out the right numbers was a bear.”

Behold the power of community. Put your questions out there, and someone in the community will have an answer.

(Oh, and for the curious, Google also does a good job of finding the answer: address 53274 ($D01A).)