“They never give us enough time to automate our tests, and then they complain at us that we don’t test fast enough!” J. shook her head. “And when I want to hire more people to help automate, they tell me I have too many people already! Management blames me because testing takes too long, but they won’t support me in fixing the problem. What’s wrong with them!?!”

J. is a QA manager in an organization that’s adopting Scrum. She’s frustrated, and understandably so. From her point of view, she’s being squeezed in all directions. The developers are producing releasable code every month. But for her team to run a regression test cycle – mostly manual – takes 6 weeks. That’s too long. Just one test cycle exceeds the sprint length by 2 weeks. J. feels tremendous pressure to reduce the time it takes to test the software. Yet at the same time, she feels like she’s not getting any support to do the one thing she can see that will help reduce the test cycle time: automate the regression tests.

I’ve had some visibility into J.’s situation for some time now. J.’s team has been trying – and failing – to automate the regression suite for the last two years. They aren’t making any headway because as soon as they get one script working, another one breaks. The automation is brittle, error-prone, and incredibly expensive to create and maintain. That’s in part because they’ve been using a cumbersome commercial tool that doesn’t support creating maintainable tests. It’s also because the user interface was not designed with test automation in mind. Many UI elements don’t have IDs, and the ones that do use automatically generated IDs that change with each build. In short, the combination of the tool and the software under test
equals a test automation nightmare. It’s no wonder J.’s team is not making headway.

Yet J. persists. Doing more of the same kind of test automation that’s already failing doesn’t make much sense to me, but she disagrees. “We just need more time!” she says.

The problem is that J. is still thinking in terms of silos. She thinks all testing tasks must be done by QA people using specialized QA tools. It simply would not occur to her to suggest that development help automate tests. Nor does she suggest that developers and testers collaborate on making the UI more testable. Instead, she says, “QA can’t go that fast. Slow down.”

J. doesn’t want to acknowledge that test automation created by a siloed QA team working in isolation to reverse-engineer existing software and automate tests against an untestable UI using proprietary tools accessible only to a few select team members is guaranteed to be incredibly expensive both to create and to maintain, and also ridiculously fragile. In short, her approach just isn’t going to work.

Unfortunately, J.’s story is likely to have an unhappy ending – at least for J. and her team. Her strategy of trying to get development to slow down, and telling management that they can’t release monthly, is backfiring. The development team is already bypassing QA for small changes and getting good results. But J. is undeterred. My past observations tell me that no matter what the reaction of the people around her, she will keep doing the same thing and expect different results.

But maybe, just maybe, by telling J.’s story here, I can help someone, somewhere.

So allow me to repeat the moral of this story:

When QA works in isolation, creating automated tests after the software is theoretically “done,” using proprietary tools that are available only to a select few team members, the results will be a fragile, unmaintainable mess.

For test automation to work well, it must be created in collaboration with the whole team and the resulting test automation code must be treated as code. That means it should be versioned with the source code, executed with each and every build, and created and maintained as part of the overall development cycle rather than as an afterthought.

And when a Test/QA group insists on keeping within their silo when the rest of the organization adopts Agile practices, they will end up bypassed and irrelevant as the rest of the organization finds ways to move forward without their help.

In a comment on my last post, Shrini asked:

“While hitting hard at ‘Jargon’ based software testing experts, you also appear to give the impression that ‘testing’ is ‘everyone’s job’ (as quality) and seem to dilute the importance and role of testing in software world – you might want to clarify. Don’t you think that you can hit at these jargon creators without diluting the role of testing?”

Yes, thank you Shrini. I appreciate your question, and I would like to clarify.

I do believe that testing is so crucial that it is everyone’s job.

And I also believe that software development teams need people who have taken the time to become really good at testing.

I believe that these two ideas are compatible. Let me ’splain.

On more than one occasion I have worked with software development teams that were under such time pressure to write code, that they felt that they had no time available to test. They left the testing to a designated team of independent testers. The result in each case was code that was so bug-riddled that stabilizing it took months. During these long months of playing whack-a-bug, the testers and developers were constantly battling, management was perpetually screaming, and no one was happy. Predictably, even when the software did ship, it was prone to failure. In weekly meetings that followed the release, Tech Support let us all know that the quality was unacceptable and that the customers were cranky. Subsequent new development efforts were hampered by the need to patch critical bugs in the field. We were in technical debt up to our eyeballs.

(At this point, I’m guessing that at least one person I have never met in person is reading this and thinking “Wow! She worked here!” I probably didn’t; it’s an all-too-universal experience. Kinda like Dilbert.)

These experiences led me to write “Better Testing, Worse Quality,” a paper that explores the system effects that lead improvements in testing to yield even more fragile software.

And then I started working with Extreme Programming teams and discovered what I’d been missing. I experienced how Test Driven Development and Continuous Integration and Collective Code Ownership and Paired Programming and continuous Refactoring led to a solid code base. Project after project, I observed that the XP teams I was working with were achieving significantly better results than the code-and-fix teams of my past. Oh, the XP projects weren’t perfect. We still had bugs. But we didn’t play whack-a-bug for months on software that was supposed to be done-except-for-testing. We didn’t need stabilization phases. The software was always stable. It might not do everything yet, but what it did, it did well.

I also witnessed the power of Customer Acceptance Testing. I realized that the people responsible for defining the requirements are in the best position to determine whether or not the implementation matches their vision.

But at the same time, I recognized that I had something to offer the team. Compared to the professional programmers, my programming skills were meager. Compared to the product managers, my understanding of the product vision was superficial. But compared to any of the other team members, my understanding of where the risks and vulnerabilities were likely to hide was superb.

I used my skills to provide feedback to the team, to point out implementation bugs and requirements ambiguities and risks. I explored the implementation and reported what I found back to the team. I thought up tests no one else had thought up before.

But I did all these things in collaboration with the team. We pooled our knowledge. And we shared responsibility for testing activities. Testing became an integral part of the software development process. We could no more separate the development and testing activities than we could have separated our hands from our bodies, leaving our hands to type on the keyboard while the rest of our bodies took a break.

Perhaps paradoxically, spreading responsibility for testing to the whole team increased the overall test effectiveness. The testing mindset became pervasive. The test effort wasn’t diluted; instead it grew and flourished.

(I could write bad prose with corny analogies involving a comparison between diluting a drop of colored dye in water and spreading seeds in the wind. I could point out that inanimate objects become weak or disintegrate when spread, while living things – like ideas and knowledge – grow. But I won’t. Oh dret. I just did. Um, never mind.)

So anyway…bad analogies aside…

I do believe that everyone is responsible for testing, even on teams that don’t claim to be doing Agile or XP. The developers test that the code does what they expect and intend, and that new changes don’t break existing expectations. The business stakeholders test that the implementation is what they had in mind. Testers – those who have studied testing long enough and hard enough to get good at it – bring specialized skills to the table to support the rest of the team.

Testers apply critical thinking skills and analytical abilities, coming up with new questions to ask the software. “What if a user does this after that?” we ask. “What if the system is in this state when that happens? What if the data looks like this? What if we configure it like that?”

But testers are most effective when they do this with, not for, the team.

I just got back from CITCON where I met an amazing group of incredibly cool people.

At dinner after the conference, a group of us compared “First Programming Experiences.” Me, I wrote my first lines of code in BASIC on a trash-80 when I was in 8th grade. The guy sitting to my right, Zach, wrote his first code on a Commodore 64. The guy across the table, Matt, wrote his first lines of code in HTML on a Windows laptop when he was something like 9. That should you an idea of our relative ages.

Matt learned to program in an entirely different era than Zach and me. Back when Zach and I were learning to program, we couldn’t just Google for an answer. We couldn’t order a tech book from Amazon. Wikipedia wasn’t an option. We couldn’t even post our question to a news group. (If we’d happened to have tech savvy parents, we might, maybe have had access to a BBS. But neither of us were so lucky.) We could ask the people around us for ideas answers. But if you happened to be the only geek with one of those early personal computers among your circle of friends, the chance of getting an answer was pretty slim.

In telling us the story of how he learned to code, Zach lamented: “The one thing I could never figure out was how to do a raster interrupt.”

It occurred to me that with about 30 people assembled, many of them geeks about our age, surely one of them knew how to do raster interrupts on a Commodore 64. “Let’s try a social experiment,” I suggested. “Write your question down and we’ll send it around the group to see if anyone knows the answer.” So Zach wrote his question on a cocktail napkin (the only thing handy). And we sent it around.

Most people just shook their heads and laughed. But one person hollered back down the table, “Who wants to know how to do a raster interrupt on a Commodore 64?”

“I do,” Zach hollered back.

“Well I don’t remember the exact syntax,” came the reply. “But you access the upper level memory registers. There wasn’t much documentation on them at the time, so figuring out the right numbers was a bear.”

Behold the power of community. Put your questions out there, and someone in the community will have an answer.

(Oh, and for the curious, Google also does a good job of finding the answer: address 53274 ($D01A).)

Once upon a time, I worked on a project where the developer protested “SCOPE CREEP!” to every bug report I filed.

Sadly, the two of us built up a lot of animosity arguing over whether or not the bugs I found were bugs or enhancements. I reasoned that I was testing conditions that were likely to occur in the real world, and “not crashing” did not count as an enhancement. The programmer argued that he’d done what he’d been asked to do and that it was too late to add more work to his plate. “No one said anything about the software being able to handle corrupt data!” he snapped.

Much to the programmer’s chagrin, management tended to agree with my assessments. The programmer ended up doing a whole lot of rework, grousing the whole time.

I later realized that the programmer thought I was making up new requirements as I went along. Seriously. He thought my tests were unfair.

Of course, that’s not what I intended.

The way I saw it, my testing was revealing answers to questions no one had thought to ask before: What if this file is locked? What if that connection is broken? What if the data is corrupted? If I’d been involved in the project earlier, I would have asked the questions earlier. But this was a waterfallish project, and testing happened at the very end of the process.

I know I’m not alone in having had a project like that. Some situations are even more dysfunctional. “They yell at me when I find bugs,” lamented one tester, “and they yell at me for not finding bugs if a user finds a problem in the field. I can’t win.”

Can’t win. Can’t break even. Can’t quit. At least not without dire consequences, like unemployment. Sometimes testing feels like Ginsberg’s restated laws of thermodynamics. The only way out of the trap is to change the game.

Fortunately, testers have the opportunity to change the game when we’re part of the team from the beginning, like on Agile projects. We can take those same testing skills we apply at the end of the project that enable us to find good, deep bugs, and use them to elicit specific acceptance criteria with good examples.

In fact, this is one of the most important ways people with testing expertise can help Agile teams.

But one stumbling block for some testers new to Agile his how, exactly, to do this.

First, you have to be able to imagine tests based on only the sketchiest idea of what the software under development is supposed to do.

Since this is the situation most testers work in, we’ve already honed that skill. We know how to take a hand-wavy set of statements like “users belong to groups; groups have permissions; permissions allow for create, read, update, and/or deleting of floozibitzes” and turn them into 193 test cases. So let’s assume that as a tester, you know how to identify potentially interesting conditions, actions, sequences, configurations, and such.

Now let’s explore how to leverage that skill at the beginning of a project, transforming tests into questions that will prompt stakeholders to discuss acceptance criteria in concrete terms.

First, remember the anatomy of a test. Tests generally involve some setup (”Log in as a user with read only permissions”), one or more actions (”Double click the floozibit record”), and one ore more expected results (”Verify the Edit button does not appear”).

Let’s rephrase that as a sentence: “Given this setup, take some action, and verify the expected results.”

With traditional tests, we know the expected results in advance. But when we’re still exploring requirements, we’re trying to determine the expected behavior. We only have our imagined setup and actions to work with. But we can get some pretty detailed information if we ask the question in the form: “If we have this setup and take these actions, what do you expect to happen?”

Let’s take a couple examples. Where you might be tempted to design a test around a condition like corrupted data or NULL values, instead frame it as a question:

“How should the software respond if it encounters corrupted data?”

“What if the data in this field were NULL in the database?”

Of course, sometimes when we ask that question, we get an answer like “Gosh, I dunno.” That doesn’t help much. Fortunately, testers are also good at figuring out what a reasonable expected result might be based on all kinds of things like past versions of the software, comparable products, and past experience. We can suggest some possible expected results in our questions: “If we have this setup and take these actions, should the software do this or that?”

“If the software encounters corrupt data, is it better for it to attempt a repair or discard the data or something else?”

“If the data in this field is NULL in the database, should the software turn that into a zero-length string?”

Of course, just because we can imagine a test doesn’t automatically make it interesting. Just as you might offer potential real world use cases to make a bug report more compelling, you can offer use cases to make a condition more interesting when eliciting requirements:

“That could happen if the database were improperly restored, or with data migrated from a legacy database.”

“What if a user bookmarkes a page for an item that is later deleted?”

Where you might define negative tests, instead ask about allowed/disallowed actions:

“Should a user with Delete Account permissions be able to delete the last Administrative Account?”

Traditional software development processes define the relationship between requirements and tests by insisting that tests be written from requirements. As a result, bad requirements lead to bad tests, or to a whole lot of arguing about what is and is not in scope.

Forget the arguing. Let’s focus on gaining alignment, and gaining it early. And as testers, we’re in a unique position to help the team do just that.
Anyone who is good at designing tests can use that skill to frame a wide variety of specific, thoughtful, and thought-provoking questions around expected behavior. And that means everything you know about testing can contribute to establishing a well-defined, specific, shared understanding of what the team is building.

On every project, we make commitments based on negotiated agreements, even when we don’t think we’re negotiating. We agree to accomplish certain tasks by a given deadline. We agree to follow a particular process. We agree to work late one day so we can leave early another. Or we agree to work over a weekend because we want to do whatever it takes to make the project succeed.

But sometimes we discover that the negotiated agreement no longer fits for us for some reason. We need to revisit the agreement. We need to re-negotiate.

Recently I had the great fun of watching a group of people work through a simulation. The simulation required that the participants work as a team to build a product for which they would get “paid.” Several rounds had passed and the team had not yet “shipped.” The team was divided into several separate working groups, and, just like in real life, there was a little tension between the groups. Some folks were getting frustrated. There was some finger pointing going on. The participants were scattered around the room, each group working independently. Close to the end of the simulation, I was beginning to wonder if the participants would be successful and make their “revenue” goals.