Test Obsessed

A couple weeks ago I tweeted:

I prefer: - Recovery over Perfection - Predictability over Commitment - Safety Nets over Change Control - Collaboration over Handoffs

— ElisabethHendrickson (@testobsessed) May 6, 2015

Apparently it resonated. I think that’s more retweets than anything else original I’ve said on Twitter in my seven years on the platform. (SEVEN years? Holy snack-sized sound bytes! But I digress.)

@jonathandart said, “I would love to read a fleshed out version of that tweet.”

OK, here you go.

First, a little background. Since I worked on Cloud Foundry at Pivotal for a couple years, I’ve been living the DevOps life. My days were filled with zero-downtime deployments, monitoring, configuration as code, and a deep antipathy for snowflakes. We honed our practices around deployment checklists, incident response, and no-blame post mortems.

It is within that context that I came to appreciate these four simple statements.

Recovery over Perfection

Something will go wrong. Software might behave differently with real production data or traffic than you could possibly have imagined. AWS could have an outage. Humans, being fallible, might publish secret credentials in public places. A new security vulnerability may come to light (oh hai, Heartbleed).

If we aim for perfection, we’ll be too afraid to deploy. We’ll delay deploying while we attempt to test all the things (and fail anyway because ‘all the things’ is an infinite set). Lowering the frequency with which we deploy in order to attempt perfection will ironically increase the odds of failure: we’ll have fewer turns of the crank and thus fewer opportunities to learn, so we’ll be even farther from perfect.

Perfect is indeed the enemy of good. Striving for perfection creates brittle systems.

So rather than strive for perfection, I prefer to have a Plan B. What happens if the deployment fails? Make sure we can roll back. What happens if the software exhibits bad behavior? Make sure we can update it quickly.

Predictability over Commitment

Surely you have seen at least one case where estimates were interpreted as a commitment, and a team was then pressured to deliver a fixed scope in fixed time.

Some even think such commitments light a fire under the team. They give everyone something to strive for.

It’s a trap.

Any interesting, innovative, and even slightly complex development effort will encounter unforeseen obstacles. Surprises will crop up that affect our ability to deliver. If those surprises threaten our ability to meet our commitments, we have to make painful tradeoffs: Do we live up to our commitment and sacrifice something else, like quality? Or do we break our commitment? The very notion of commitment means we probably take the tradeoff. We made a commitment, after all. Broken commitments are a sign of failure.

Commitment thus trumps sustainability. It leads to mounting technical debt. Some number of years later find themselves constantly firefighting and unable to make any progress.

The real problem with commitments is that they suggest that achieving a given goal is more important than positioning ourselves for ongoing success. It is not enough to deliver on this one thing. With each delivery, we need to improve our position to deliver in the future.

So rather than committing in the face of the unknown, I prefer to use historical information and systems that create visibility to predict outcomes. That means having a backlog that represents a single stream of work, and using velocity to enable us to predict when a given story will land. When we’re surprised by the need for additional work, we put that work in the backlog and see the implications. If we don’t like the result, we make an explicit decision to tradeoff scope and time instead of cutting corners to make a commitment.

Aiming for predictability instead of commitment allows us to adapt when we discover that our assumptions were not realistic. There is no failure, there is only learning.

Safety Nets over Change Control

If you want to prevent a given set of changes from breaking your system, you can either put in place practices to tightly control the nature of the changes, or you can make it safer to change things.

Controlling the changes typically means having mechanisms to accept or reject proposed changes: change control boards, review cycles, quality gates.

Such systems may be intended to mitigate risk, but they do so by making change more expensive. The people making changes have to navigate through the labyrinth of these control systems to deliver their work. More expensive change means less change means less risk. Unless the real risk to your business is a slogging pace of innovation in a rapidly changing market.

Thus rather than building up control systems that prevent change, I’d rather find ways to make change safe. One way is to ensure recoverability. Recovery over perfection, after all.

Fast feedback cycles make change safe too. So instead of a review board, I’d rather have CI to tell us when the system is violating expectations. And instead of a laborious code review process, I’d rather have a pair work with me in real time.

If you want to keep the status quo, change control is fine. But if you want to go fast, find ways to make change cheap and safe.

Collaboration over Handoffs

In traditional processes there are typically a variety of points where one group hands off work to another. Developers hand off to other developers, to QA for test, to Release Engineering to deliver, or to Ops to deploy. Such handoffs typically involve checklists and documentation.

But the written word cannot convey the richness of a conversation. Things will be missed. And then there will be a back and forth.

“You didn’t document foo.”
“Yes, we did. See section 3.5.1.”
“I read that. It doesn’t give me the information I need.”

The next thing you know it’s been 3 weeks and the project is stalled.

We imagine a proper handoff to be an efficient use of everyone’s time, but they’re risky. Too much can go wrong, and when it does progress stops.

Instead of throwing a set of deliverables at the next team down the line, bring people together. Embed testers in the development team. Have members of the development team rotate through Ops to help with deployment and operation for a period of time. It actually takes less time to work together than it does to create sufficient documentation to achieve a perfect handoff.

True Responsiveness over the Illusion of Control

Ultimately all these statements are about creating responsive systems.

When we design processes that attempt to corral reality into a neat little box, we set ourselves up for failure. Such systems are brittle. We may feel in control, but it’s an illusion. The real world is not constrained by our imagined boundaries. There are surprises just around the corner.

We can’t control the surprises. But we can be ready for them.

Oh, hai internets. It’s been a while. Did you miss me?

Let me tell you what I’ve been up to.

In the fall of 2012 I shut down my consulting practice (Quality Tree Software) and my studio (Agilistry), and took a job with Pivotal. Actually, to be precise, I joined Pivotal Labs; Pivotal did not even exist in 2012.

Pivotal came into existence in April 2013 as a spin out from EMC and VMWare. Labs is part of Pivotal, but we are more of a product company focused on cloud and data than a services company. We work on Cloud Foundry, an open source Platform-as-a-Service (PaaS), and have our own distribution as well as our hosted service. The other side of our business is data. Our Big Data Suite includes GPDB (an MPP database), HAWQ (SQL on Hadoop), and Gemfire (an in-memory data grid).

My role at Pivotal has evolved in the time I’ve been there.

For the first couple years, I was the Director of Quality Engineering on Cloud Foundry. It’s a title I swore I would never take again. But my job was different than you might imagine. I did not direct the efforts of quality engineers. Rather, I paid attention to our feedback cycles. Teams own their tests, their CI pipelines, and ultimately the quality of their deliverables. I just helped connect the dots. By the way, if you want to know more about quality and testing on Cloud Foundry, I did get out of the building long enough to give a talk on it at the Wikimedia Foundation. I also gave a talk on the Care and Feeding of Feedback cycles at DOES2014.

In the last few months I moved over to our Data organization in Palo Alto. This changed my commute substantially, so my family and I are moving this summer. That will be an adventure. We’ve been in the same house for 17 years. So wish me luck with that.

Along with the move to our data org, my title changed. We removed the word “quality” from it since what I do does not look anything like traditional quality engineering. So I’m now a director of engineering. But the work I do on a daily basis with our Data teams looks a lot like what I did with Cloud Foundry: I’m deeply involved in hiring, cross-team coordination, improving our release practices, improving builds and CI to make the developer workflow better, and coordinating with our product organization to make sure teams have a steady stream of high value work.

I’m also doing my best to climb the steep learning curve of MPP databases and Hadoop. It helps that I worked at Sybase once upon a time. But that was 20 years ago. So between the fact that I was doing very different work 20 years ago, I’ve forgotten much of what I learned, and things have changed a bit in 2 decades, my prior database experience is only helping me a little in understanding my new context.

I have to say that I love working at Pivotal. I adore the people, am fascinated by the products, and am passionate about the way we work. Coming back to Pivotal was like coming home. (After all, Pivotal Labs is where I learned Agile over a decade ago.)

Some of you have noted that I don’t get out much anymore. I’m not at conferences and I don’t travel much. Since I’m in an inward facing role it’s difficult for me to carve out time to get out into the community. I’d like to see my industry friends more often and I am always honored to be invited to speak. But I turn down the vast majority of speaking invitations. My job takes up all my available time and brain cells.

So that’s what I’m up to and why I’ve been silent here for so long. I do have things to say though. I’ve learned a lot in the last 30 months. And I’m learning more every day. So I hope to carve out time to share what I’m learning here. But no promises about when, exactly, I’ll post.

I rested my head against the window, drowsy from a long day, full tummy, and steady hum of the train. My husband sat next to me absorbed in a game. It’s a long ride from San Francisco, where we work, to Pleasanton, where we live. I thought I might take a nap.

A disembodied voice crackled, announcing the next station: “West Oakland.”

The train stopped, the doors opened. I watched the passengers. The train car was about half full. A few people got off. Three black youths, a girl and two boys, got on. They were talking animatedly, happy. All three were well-dressed: fashionable clothes that looked new and fit well. Tight black pants. Caps. A little bling. They’d taken the time to look nice, and they were out for a good time.

One of the boys sat while the girl and the other boy stood, balancing by holding onto the straps suspended from the ceiling of the car. They talked and joked together. I felt a little voyeuristic watching them share this private moment of fun, but I couldn’t help myself. I was too taken with their delight and energy. Their faces lit with joy as they laughed. The girl and the boy hung from the straps and swung their weight around, enjoying the interplay of gravity and momentum of the train and their own upper body strength.

The girl clung to the strap then braced her other hand on the side of the train. Walking her feet up the train side, she executed an impressive flip. She did it again, obviously pleased with herself. “You try doing that in heels!” she crowed. She did it again. I noted with admiration that she was wearing boots with 5″ heels.

I guessed the kids to be somewhere between 15 and 20. It’s an in-between age. Young enough to treat a train as a jungle gym but old enough for a night out on their own. I thought about my own kids, about the same age.

The disembodied crackling voice announced the next station: Lake Merritt. The train stopped. The kids stayed on. The train started again. The train abruptly came to a screeching halt. “That’s not good,” the passenger in front of me muttered. I looked around but couldn’t tell why we were stopped. I leaned my head back against the window; my husband stayed immersed in his game. The trains don’t run on time here. Stopping wasn’t unusual.

Two BART police officers stepped onto the car. Another guy, a passenger, trailed behind them. The passenger with the cops said something I couldn’t hear and gestured toward the three youths. He was a mousy white guy in a button-down oxford shirt, corduroy pants, and a fleece vest. He looked like the kind of guy who drives an SUV with Save the Planet bumper stickers.

The first police officer addressed the youths. “There’s been a complaint. I need you to step off the train.”

“What’s the complaint?” the girl asked.

“We didn’t do anything,” the boy said.

“There’s been a complaint,” the police officer insisted. “Please come with me.”

“What’s the complaint?” the girl repeated.

The mousy passenger who had made the accusation faded back. I didn’t see where he went. Now it was just the cops and the kids.

“We got a complaint that you were asking for money.” The cop’s voice carried.

“We didn’t do anything!” the boy repeated, louder.

“Please come with me,” the cop insisted. “This train can’t move until you get off.”

The crowd on the car grew restless, impatient. The two cops flanked the three youths and attempted to escort them off the train. For a minute, it looked like the kids were going to comply. The boy who was seated stood, headed toward the open door, but then changed his mind. “I didn’t do anything. I was just sitting there,” he said. He got back on the train.

The girl’s face had gone from joy to furor. “It’s that racist white guy,” she yelled, loud enough for the whole train car to hear. “We didn’t do anything, but he thinks we did because we’re black.”

A passenger stood up, Asian, “They really didn’t do anything,” he said to the cops. Then to the kids: “Let’s just step off the train so they can make their report.”

Another passenger stood up. A black woman. “Really, officers, these kids didn’t do anything.”

By this time I was on the edge of my seat. It was so unfair. These kids were just out for a good time. They had done nothing besides being full of energy and life. They hadn’t asked for money. They were being harassed by the cops for something they had not done.

I’m still not sure what made me decide to act on my sense of injustice. Maybe I arrogantly thought the police would listen to me, a middle-class, middle-aged, white woman. Maybe I saw the faces of my own kids in the black youths and imagined them being detained on hearsay. Whatever the trigger, I stood up, moving past my husband who looked at me quizzically. I walked up the train aisle to the officer who had been barking orders.

“Sir, they really didn’t do anything. They just got on at the last stop.” My voice was steady. Calm. Quiet. Pitched for the circle of people involved, not for the rest of the train. Internally I was amazed that my adrenaline wasn’t amped up, that my voice was not shaking. I don’t normally confront cops. It is not at all like me to tell a cop how to do his job. I couldn’t believe I was confronting a cop and not even nervous about it.

By this time the crowd on the train was growing antsy. A few were calling for the kids to get off the train so they could get where they were going, but far more were calling for the cops to leave the kids alone. Several more passengers stood up and started shouting at the cops:

“These kids didn’t do anything!”

“Leave them alone!”

“Racist white guy can’t tell blacks apart!”

“It must have been someone else!”

“Let the kids go!”

I felt self-conscious standing in the middle of everything. I’d played my bit part in the unfolding drama. I returned to my seat.

A woman on the other side of the car leaned over toward me, “I could have done that,” she said. She was white like me, graying hair cut shoulder length in a bob. She looked like she could live in the house next door in my white-picket-fence neighborhood.

“I’m sorry?” I was confused by her words. What does that even mean, I could have done that? If you could have why didn’t you? Did she mean that she could not have done what I did?

“I could have done what you did,” she repeated.

I smiled and nodded. I didn’t know what else to say.

My attention turned back to the growing situation. More passengers had stood up to defend the kids. Another BART police officer entered the car. The increasing tension and buzz from the crowd made it hard to hear. I realized that there were several more police officers waiting on the platform. The situation had escalated rapidly. This could become ugly.

“CHECK THE BUCKET,” the girl shouted.

For the first time I noticed that the kids had an amplifier and a bucket with them, and suddenly I saw a missing piece of the puzzle. The kids were street performers. They weren’t out for a night of clubbing. They were dancers.

Sometimes street performers come onto the BART trains. They strut up and down the aisle, put on a show, and ask for money. I personally find being a captive audience to these shows annoying. I never give the performers money because I don’t want to encourage it. But I did not realize that such behavior warranted police intervention on this scale.

Nor did I think that these kids had been dancing for money on this train. I looked around for the accuser but couldn’t see him.

I was incensed by the entire situation. The accuser had disappeared, the situation was escalating, and these innocent kids were now targets.

“This isn’t right,” I told my husband. I stood up, shouldered my backpack in case I decided to leave the train, and pushed past my husband’s knees for the second time. Without even thinking about whether or not my husband would follow me or would be upset with me for getting involved, I stepped into the situation again.

I sidled up to the cop who had originally been barking orders and who was now standing between the kids and the rest of the passengers.

“They really didn’t do anything,” I said into his ear. “They didn’t. This isn’t right. I will be happy to make a statement. What can I do to help?”

The cop turned to me. “They might not have done anything originally,” he said, “but now they are resisting. If they don’t get off the train we will have to arrest them.”

Another police officer entered the train. The mood of the passengers was growing dark, roiling. It became a palpable thing, shifting energies threatening to spill over into violence. My stomach fluttered.

I turned around, realized that half a dozen people had their cell phones out, recording the incident. I idly wondered if I would show up on youtube videos. I decided I didn’t care. This wasn’t right and I wouldn’t stand for it. These kids were being detained, harassed, and possibly arrested all because the mousy guy in the fleece vest didn’t like being bothered by dancers panhandling on BART. These kids might have danced on BART at some point, but not tonight, not on this train. They didn’t deserve this.

Another officer entered the train. It was now five officers to the three kids. The Asian passenger who had originally stepped up to intervene turned to the kids, “Why don’t we get off the train and get this handled. Then we can all get onto the next train.”

The police closed their circle, puffing themselves up, herding the kids to the exit. I followed them to the door. I was about to step off the train. I turned back to see if my husband was paying attention. He was staring at me, his face open, questioning. “This isn’t right,” I mouthed at him. He stood up, following me.

I turned my attention back to the kids. The cops had one of the boys in an arm lock and pressed him up against the wall of the station. It looked like it hurt. They had the girl surrounded. I couldn’t see her, but I could hear her screaming. I heard another voice — the other boy? the Asian passenger who had intervened? a cop? — telling her “This isn’t worth it. Stop struggling.”

I stepped off the train onto the platform.

The cops who had the girl surrounded had her hands behind her back. She was struggling and screaming. They began half pulling, half dragging her down the platform toward the station exit. “LET GO OF ME,” she shouted. “LET GO OF ME LET GO OF ME LET ME GO!”

An officer stood in the middle of the chaos just looking, sizing it all up. He was new on the scene. He was also the only officer not in the middle of the chaos. I walked up to him. “Sir, they didn’t do anything,” I said. I watched his face. Dark skin. Brown eyes. A guy who’d been having a quiet night. He turned to me.

“You’re a witness?” he asked.

“Yes sir. I was on the car. These kids didn’t do anything. This isn’t right.” The girl was still screaming. Each panicked shriek felt like shrapnel in my soul. That could be my daughter, I thought. So very not right. “I will be happy to make a statement,” I said.

“OK,” the cop replied. He took out a notebook, asked me my name. He had to ask me to spell my last name a second time. I handed him my driver’s license to make things easier for him. He started taking notes, then looked up at my husband and me. “I left my recorder in my office,” he said. “Would you be willing to go there to make a statement?”

I agreed immediately. To my surprise, my husband agreed as well. “Of course,” he said. We would probably be an hour late, or more, getting home. This was worth it.

We followed the officer off the platform, past the turnstiles, and through a door marked “authorized personnel only.” He led us down a maze of corridors, past a bank of cubicles, and into his office. As promised, he took out a recorder and interviewed us for a verbal statement. My husband and I each told our version of the events. They lined up. Each of us emphasized that the kids had done nothing wrong.

“Let me just clarify a few points,” the officer said after we finished our statements. “In your opinion, did the officers use excessive force at any time?” I said they had, citing the painful hands-behind-the-back restraining technique and the panicked girl screaming. My husband said they had not, but pointed out that the officers should have just let the kids go when so many passengers testified that the kids were innocent.

The officer asked a few more questions: had the officers used mace, batons, or physical violence on the kids? had they acted unprofessionally?

We finished the interview, the officer escorted us back into the station, said that internal affairs would probably be contacting us in connection with the investigation, then said goodbye.

As we waited for the next train, I played over the evening in my mind.

I realized that the statements we made existed only on a recorder in the cop’s office. The kids wouldn’t know we made the recorded statement. If they got lawyers, the lawyers wouldn’t know. Our testimony might never be heard by anyone involved in the case against the kids.

The more I played over the questions that the officer had asked us, the more I realized that the questions were all about the conduct of the other officers, not about the kids and their innocence.

The internal affairs investigation would, no doubt, turn into a huge deal. The situation had escalated so rapidly, fueled by racial tension. So many people had recorded videos. The train had been halted for 15 minutes or more. Passengers were irate. There would be much attention on the conduct of the cops.

I didn’t care about the cops. I wanted to help the kids.

Somehow, I doubted that I had.

Next time I see dancers on BART, I’m giving them whatever is in my wallet. I’ll think of it as their defense fund.

When I wrote my blog post a few weeks back celebrating 15 years in business, I had no idea that in a few short weeks I would be making sweeping changes. I knew I was restless in my current work, and also tired of being on the road all the time. But I did not realize how ready I was for a major change.

Hoo-boy, was I ready for change though. Rapid, sweeping change. Folks who follow me on Twitter know that in the last couple of weeks, I’ve: delivered my PragProg book Explore It!, to production; started a new job at Pivotal Labs; and closed down both Agilistry Studio and my consulting practice, Quality Tree Software, Inc.

I’ve posted farewell letters on both the Quality Tree and Agilistry Studio websites. You can read those for details about those closings. Eventually, however, I’ll be taking those websites down. This site will be my primary web presence.

I want to take a moment here to talk about what’s next for me.

My new job means that I won’t be training or consulting any more, nor do I intend to participate in many conferences for a while. There’s just so much for me to do and learn with my new job that I don’t have the mental bandwidth for anything else for the foreseeable future.

That said, I do intend to keep pushing the boundaries of software testing and overall development practices. I believe that the traditional software QA model is fundamentally and irretrievably broken. It’s time for new models. I want to figure out how to articulate those new models simply, and I hope that my work at Pivotal Labs will help me do that.

In the coming months, I might be writing here more. Or not. I’m not sure yet. We’ll just have to see how things go.

So that’s what I’m up to. What changes are on the horizon for you?

So, there’s another round of gender/race drama going down on teh intarwebs.

It seems that there was a conference, BritRuby, slated to run next year in Manchester, England. Only there was some criticism on Twitter of their all-white-guy lineup. Suddenly, without warning, the organizers canceled the conference. The organizers published an official statement on the conference website, but Sean Handley, one of the organizers, offered a much better and more detailed explanation.

I was unaware of BritRuby until it was canceled. Even if I’d known about it, I couldn’t have participated. So if you think this is a post about BritRuby itself, it’s not. I don’t have an opinion one way or another about BritRuby or their speaker lineup.

But here’s why I do care: because I think it’s deeply unfortunate that anyone is blaming a Twitter conversation about diversity for the cancellation of the conference. That’s absurd. The issue is not whether the Twitterverse has an opinion. The issue is that sponsors pulled out.

Hang on, let’s back up a step.

The official statement on the BritRuby website said, “Sadly, BritRuby was used as the arena to air these issues on Twitter and this has fundamentally destroyed any chance we had of addressing these issues.” And there’s a bunch of Twittering about how people were dogpiling on BritRuby.

Figuring things must have gotten really nasty, I went searching for the smoking gun on Twitter.

I couldn’t find it.

Where were the activist bullies? Where were the accusations of sexism or misogyny? I found none of that. Instead I found expressions of regret:

Nice speaker lineup for @britruby. Except for the 100% white guys part.

— Josh Susser (@joshsusser) November 17, 2012

And offers of help:

@britruby I’m sure we could hook you up with people who’d be interested in helping be a part of organizing and would help drive diversity.

— ashe dryden (@ashedryden) November 17, 2012

Plus some defensiveness:

@raganwald @anthonycgreen Well said… #BritRuby will keep fighting this issue but stands firm in its views on how speakers are selected.

— British Ruby Conf (@BritRuby) November 17, 2012

There were no flame fests, no diatribes. The more I dug, the more it seemed like a relatively mild discussion on Twitter got reflected back and turned around until people who expressed their opinion were being attacked viciously in tweets with words like “Shame on you” and “Disgusting” and “You ruined everything.”

It appeared that the mere question of diversity had a polarizing effect, dividing people into Us v. Them without anyone actually having thrown the first stone.

This is the underlying tragedy: we cannot have a discussion about the issue of diversity in technology without teh intarwebs hitting resonant frequency until something falls apart.

Ultimately, that’s why I care and why I am wading into the fray myself. If we cannot talk about diversity issues openly, we cannot address them. So attempting to shut down discussion of the issue, to shame people into silence, is the same as actively opposing diversity.

If you personally choose not to address diversity issues, that’s fine. Don’t address them. Keep your own silence. But don’t expect that others will ignore the issue just because you want to ignore it. Don’t ask them to be silent.

If you become fed up enough to wade into the general mayhem and express your opinions, do a little research first. Don’t fly off the handle at accusations that someone said something. Find the original quotes. Respond to actual statements, not to incendiary reactions to over-reactions to original statements.

What is at risk here isn’t just a conference. It’s something much bigger: the ability to discuss very real, very personal issues in an open forum.

Update thanks to Laurent Bossavit:

@testobsessed Here’s a gist of *all* criticism leveled at BritRuby during 24h between “white guys” twt & cancellation: gist.github.com/4108671

— Laurent Bossavit (@Morendil) November 19, 2012

The full list:

Still seems pretty mild to me; not nearly enough to bring down a whole conference all by itself.

But wait, it gets better. Laurent then searched to find the reactions to the original comments. It’s those reactions where folks reframed the original comments using trigger words like “sexism” and “racism.” See Laurent’s link for details.

@testobsessed Now compare it to the reactions (and I’ve just sampled the early ones): bit.ly/Qohbtz

— Laurent Bossavit (@Morendil) November 19, 2012

I was going through some of my old notes yesterday and stumbled across a sketch I made of a diagram of effects showing how test managers become ineffectual. (I re-created it in Illustrator since no one but me could have read my original sketch.)

I’d like to say that back when I was a test manager I never succumbed to this cycle. Alas, I’m sure I did. It’s almost inevitable.

Reading through those old notes was a good reminder about why I don’t manage independent QA/Test groups anymore.

I’d much rather work with organizations to integrate testing activities throughout the cycle and across all roles. The traditional role of a test manager in organizations that practice test-last development is a miserable job. Add in a sprinkle of dysfunctional QA blaming while simultaneously undercutting any efforts the QA manager makes to increase quality? The job becomes a recipe for depression and substance abuse.

Fortunately, I escaped in time.

I have wasted countless hours of my life arguing to fix bugs in bug triage meetings.

Bug advocacy is a core skill for testers in traditional software development organizations that follow code-then-test practices. Over time, I got reasonably good at it. I could explain to both business and technical stakeholders not only the symptoms of the bug and steps to reproduce it, but also the corresponding impact to users. I could help my internal stakeholders see the connection between the risk that the bug represented and the core value of whatever system we were working on. In short, I became adept at making a business case to fix bugs.

It was awful. I hated every minute of it.

Every bug triage meeting was torture. They were long and contentious. I didn’t understand then, nor do I understand now, why an organization would bother spending so much time and money on testing if they weren’t going to do anything with the information that testing revealed.

These days I choose to do hands-on-the-keyboard work only in organizations where they value clean code so highly that I don’t have to draw on those bug advocacy skills.

But even now, there is a mistaken notion in the software development community that bugs are inevitable, that you can’t fix them all, and that it makes sense to make a fix/defer decision on bugs based on some notion of ROI.

This line of thinking is a deathtrap.

It might be a slow, lingering death. It might, in fact, be such a slow death that it’s not apparent how each defer decision is bringing death for the product, or company, a little closer. But once we start down the apologists path of deferring “minor” bugs in mountainous inventories of defect backlogs, we’re inviting a sinister guest into our midst.

If you think I’m being overly dramatic, consider this: two of the companies where I spent the most time in bug triage meetings are now dead. So I think I know a deathtrap when I see it. Let me share their stories.

One of the companies sold software directly to consumers. The other was a paid service with a software component distributed through large OEMs. In both cases we “didn’t have time” to fix all the bugs. There were always higher priorities. We had to get features out. We had to make deadlines. We had to respond to customer demands.

In both cases the executive staff impressed on us that every day we delayed a release incurred a massive cost. They pushed us to be pragmatic. Reasonable. Make good business decisions about tradeoffs.

The executives were not wrong. Both companies were in rapidly changing markets where speed was critical. In the case of the company that sold to OEM partner companies, the deadlines were incredibly real. Miss the drop date by even a single day, and we wouldn’t be on the platform. That meant we wouldn’t see revenue on that platform for another 6 months.

Under such conditions it’s easy to see how we made the decisions we did.

But remember, both companies are dead.

It wasn’t the bugs that killed us directly. Rather, the bugs became a pervasive infection. They hampered us, slowing down our productivity. Eventually we were paralyzed, unable to make even tiny changes safely or quickly.

Whenever we considered the question of whether to defer or fix a bug, we looked at the obvious costs associated with releasing software with known bugs: annoyed customers, tech support, and the overhead associated with releasing patches for the most egregious cases.

We were right about those costs. So we weren’t surprised when the resulting interruptions and escalations forced us to split our efforts between new development and maintenance. We just hired more people to keep up with the increased demands and told ourselves that it was the cost of doing business in the software industry.

However, it was the hidden costs that drained us of life: hours lost arguing about bugs in bug triage meetings; hours lost stumbling over the same known issues again and again; hours lost fighting with a fragile and error-prone code base to make even small changes; hours lost cataloging and categorizing the backlog. It was demoralizing and immensely expensive.

All those hours leached away, drained our capacity. We could see it happening but were powerless to stop it. By the time we realized how much trouble we were in, it was too late. The infection had taken root and nothing short of a total overhaul of the entire code base would make things better.

The consumer software company eventually died because a more nimble competitor came along. We simply could not keep up with the changing market demands in a rapidly evolving field.

The OEM partner died because we just couldn’t generate enough revenue. What little revenue we had disappeared as one-by-one, the large OEMs dropped us. The OEM’s internal staff had no faith in our software so when their customers complained to their tech support, the OEM tech support blamed our software.

There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little “cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again, you will spend hours upon hours on it.

This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill. They do it slowly, painfully, but relentlessly.

Want your software to evade that horrific death?

Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified; attend to your broken windows early.

And whatever you do, don’t accept the apologist’s excuse that bugs in production are inevitable, so the only pragmatic approach is to make a cost/benefit decision on whether or not to fix them. The cost is almost always far higher than anyone might guess.

A reader wrote me to ask:

I know what metrics to use to manage a traditional phased project. But what metrics do you use on Agile projects?

I started drafting my answer in a private email but I decided that it was time to put my answer on the record.

This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue using the same metrics they were using for traditional projects even though the process is fundamentally different.

The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful.

• Requirements coverage is meaningless if the definition of “Done” includes tested. The number will always be 100%. If it’s not 100%, you have a deeper problem that metrics will not help you solve.
• Test pass/fail ratios are not useful. If a test that was passing starts failing, the team stops and fixes the regression right away.
• Counting the defects found internally is not particularly helpful, unless it’s a big number, and then we stop to figure out why there are so many bugs.
• Traditional QA defect ratio-based metrics, especially defect detection efficiency, can be actively harmful. It’s particularly horrific if the testers are being judged based on DDE. In every situation I’ve seen and heard about where that happened, the testers spent more time arguing about what was and was not a bug than actually helping to move the project forward.

So that’s what I don’t do. Before I launch into the metrics that I do use, there are some caveats I have to tell you.

First, I only use metrics to get a 50,000 foot view on what’s going on. They provide an indicator that there’s something to investigate, but they don’t give me enough understanding to make critical decisions. To make important decisions, I have to dig into the details.

Second, I do not use metrics to compare teams or individuals. Ever. This is important. The best way to screw up the usefulness of any process metric is to use it to judge people. This is a basic principle of metrics and has nothing to do with the development process. But apparently it’s something that needs repeating; I still see managers trying to use velocity or defect metrics to compare teams.

I even saw one manager institute the notion of “personal velocity” on an Agile team. He thought it was great. He wanted to know who the solid producers were, and he thought it led to greater attention to individual responsibility. But he was blind to the side effects: people did not help each other because they knew their own personal productivity would decline if they spent their time helping others.

Third, I am much more focused on qualitative than quantitative measures. Counting leads to the illusion that we can understand something because we can quantify it. But numbers can be very misleading. Worse, counting leads to perverse incentives all too often. Even when managers are not assessing people based on process metrics, counting things affects behavior.

Given all that, here’s the list of what I do use:

1. The core measure I use is velocity, or Running Tested Features as Ron Jeffries calls it. Note that the “Tested” part is important: velocity is only meaningful if Tested is part of the definition of Done.
2. I sometimes look at cycle time (the time it takes from when a story moves from To Do into In Progress to the time it’s Done). If the team is having difficulty producing Running Tested Features, measuring the cycle time of the simplest possible change is an enlightening diagnostic. If taking a tiny, self-contained, non-risky change all the way to Done takes more than an hour, then it’s not at all surprising the team can’t get real stuff completed. At that point I start digging to find out what the impediments are.
3. I use the Continuous Integration system to tell me about the health of the build. Sometimes if the build seems to be red a lot I count the time the build stays red, or the time the build has been green. (Think of a big poster: “Days since last build fail: 15”) But any time you count build metrics there is a risk of affecting checkin behavior. I’ve seen a team that is so scared of breaking the build that they stopped checking in altogether. Clearly this is not ideal.
4. I do measure code coverage on the unit tests. I don’t have any illusion that code coverage tools can actually tell me anything about how good the testing was, but if there are large swaths of code with no unit tests, I view that as technical debt and start paying it down.
5. I do pay attention to defects reported from the field. If there are enough of them, counting might be useful. But counting isn’t necessary. One of the more effective things I’ve seen is a support group that created visibility around field problems by posting the big ones on a big visible chart prominently located where everyone (testers, developers, executives, etc.) would see it.