Checking In

Well, so this is awkward. I kind of disappeared. I’ve been focusing all my time, energy, and attention on my day job at Pivotal. It’s been fun! And wild! We went public and I got to be part of the team that stood on the podium at the New York Stock Exchange! A bucket list item I didn’t even know I had! Along the way I’ve gotten to work with some amazing folks on super interesting products.

I’m just popping back in now to update my wordpress theme. Some very kind folks let me know that my site was kinda borked and that made them sad. And here I am at SFO with time on my hands because of a 2 hour flight delay. It was this or try to plow through another hundred messages in my embarrassingly and perpetually overflowing inbox. And oh, the yaks I had to shave to get to this point, with a new and hopefully unborked theme! Three password recovery workflows. The discovery that my old theme is no longer being maintained, apparently. Searching for new themes. Getting overwhelmed. Settling on the default theme that comes with any wordpress installation. It was a whole herd of yaks! My flight will be boarding before I know it.

As long as I’m here, I’ll confess that I don’t quite know what to do with this blog. Apparently it’s useful at least to some folks who take the time to notify me when things are broken. For that matter, it’s been useful to me to have a public repository of things I’ve written: I find myself foraging among my old writings periodically and sending along links to folks instead of writing blog length emails. But I don’t even look at comments anymore. (There are 571 in moderation. I’ll bet 568 of them are spam. So yeah. Not gonna look. Really sorry, other three people!) And I haven’t been writing anything I’d want to post. So it’s a static repository. WordPress is totally overkill, but migrating it to anything else would require more time and energy than I have right now.

So here we are. A weak update just to say “yo, maybe the css will work in chrome now?”

Maybe now that I’ve recovered all the passwords and found my admin page I might post again? We’ll see. In the meantime, please do continue to let me know if you experience problems with the blog. Thanks!

I Prefer This Over That

A couple weeks ago I tweeted:

Apparently it resonated. I think that’s more retweets than anything else original I’ve said on Twitter in my seven years on the platform. (SEVEN years? Holy snack-sized sound bytes! But I digress.)

@jonathandart said, “I would love to read a fleshed out version of that tweet.”

OK, here you go.

First, a little background. Since I worked on Cloud Foundry at Pivotal for a couple years, I’ve been living the DevOps life. My days were filled with zero-downtime deployments, monitoring, configuration as code, and a deep antipathy for snowflakes. We honed our practices around deployment checklists, incident response, and no-blame post mortems.

It is within that context that I came to appreciate these four simple statements.

Recovery over Perfection

Something will go wrong. Software might behave differently with real production data or traffic than you could possibly have imagined. AWS could have an outage. Humans, being fallible, might publish secret credentials in public places. A new security vulnerability may come to light (oh hai, Heartbleed).

If we aim for perfection, we’ll be too afraid to deploy. We’ll delay deploying while we attempt to test all the things (and fail anyway because ‘all the things’ is an infinite set). Lowering the frequency with which we deploy in order to attempt perfection will ironically increase the odds of failure: we’ll have fewer turns of the crank and thus fewer opportunities to learn, so we’ll be even farther from perfect.

Perfect is indeed the enemy of good. Striving for perfection creates brittle systems.

So rather than strive for perfection, I prefer to have a Plan B. What happens if the deployment fails? Make sure we can roll back. What happens if the software exhibits bad behavior? Make sure we can update it quickly.

Predictability over Commitment

Surely you have seen at least one case where estimates were interpreted as a commitment, and a team was then pressured to deliver a fixed scope in fixed time.

Some even think such commitments light a fire under the team. They give everyone something to strive for.

It’s a trap.

Any interesting, innovative, and even slightly complex development effort will encounter unforeseen obstacles. Surprises will crop up that affect our ability to deliver. If those surprises threaten our ability to meet our commitments, we have to make painful tradeoffs: Do we live up to our commitment and sacrifice something else, like quality? Or do we break our commitment? The very notion of commitment means we probably take the tradeoff. We made a commitment, after all. Broken commitments are a sign of failure.

Commitment thus trumps sustainability. It leads to mounting technical debt. Some number of years later find themselves constantly firefighting and unable to make any progress.

The real problem with commitments is that they suggest that achieving a given goal is more important than positioning ourselves for ongoing success. It is not enough to deliver on this one thing. With each delivery, we need to improve our position to deliver in the future.

So rather than committing in the face of the unknown, I prefer to use historical information and systems that create visibility to predict outcomes. That means having a backlog that represents a single stream of work, and using velocity to enable us to predict when a given story will land. When we’re surprised by the need for additional work, we put that work in the backlog and see the implications. If we don’t like the result, we make an explicit decision to tradeoff scope and time instead of cutting corners to make a commitment.

Aiming for predictability instead of commitment allows us to adapt when we discover that our assumptions were not realistic. There is no failure, there is only learning.

Safety Nets over Change Control

If you want to prevent a given set of changes from breaking your system, you can either put in place practices to tightly control the nature of the changes, or you can make it safer to change things.

Controlling the changes typically means having mechanisms to accept or reject proposed changes: change control boards, review cycles, quality gates.

Such systems may be intended to mitigate risk, but they do so by making change more expensive. The people making changes have to navigate through the labyrinth of these control systems to deliver their work. More expensive change means less change means less risk. Unless the real risk to your business is a slogging pace of innovation in a rapidly changing market.

Thus rather than building up control systems that prevent change, I’d rather find ways to make change safe. One way is to ensure recoverability. Recovery over perfection, after all.

Fast feedback cycles make change safe too. So instead of a review board, I’d rather have CI to tell us when the system is violating expectations. And instead of a laborious code review process, I’d rather have a pair work with me in real time.

If you want to keep the status quo, change control is fine. But if you want to go fast, find ways to make change cheap and safe.

Collaboration over Handoffs

In traditional processes there are typically a variety of points where one group hands off work to another. Developers hand off to other developers, to QA for test, to Release Engineering to deliver, or to Ops to deploy. Such handoffs typically involve checklists and documentation.

But the written word cannot convey the richness of a conversation. Things will be missed. And then there will be a back and forth.

“You didn’t document foo.”
“Yes, we did. See section 3.5.1.”
“I read that. It doesn’t give me the information I need.”

The next thing you know it’s been 3 weeks and the project is stalled.

We imagine a proper handoff to be an efficient use of everyone’s time, but they’re risky. Too much can go wrong, and when it does progress stops.

Instead of throwing a set of deliverables at the next team down the line, bring people together. Embed testers in the development team. Have members of the development team rotate through Ops to help with deployment and operation for a period of time. It actually takes less time to work together than it does to create sufficient documentation to achieve a perfect handoff.

True Responsiveness over the Illusion of Control

Ultimately all these statements are about creating responsive systems.

When we design processes that attempt to corral reality into a neat little box, we set ourselves up for failure. Such systems are brittle. We may feel in control, but it’s an illusion. The real world is not constrained by our imagined boundaries. There are surprises just around the corner.

We can’t control the surprises. But we can be ready for them.

Where Have I Been?

Oh, hai internets. It’s been a while. Did you miss me?

Let me tell you what I’ve been up to.

In the fall of 2012 I shut down my consulting practice (Quality Tree Software) and my studio (Agilistry), and took a job with Pivotal. Actually, to be precise, I joined Pivotal Labs; Pivotal did not even exist in 2012.

Pivotal came into existence in April 2013 as a spin out from EMC and VMWare. Labs is part of Pivotal, but we are more of a product company focused on cloud and data than a services company. We work on Cloud Foundry, an open source Platform-as-a-Service (PaaS), and have our own distribution as well as our hosted service. The other side of our business is data. Our Big Data Suite includes GPDB (an MPP database), HAWQ (SQL on Hadoop), and Gemfire (an in-memory data grid).

My role at Pivotal has evolved in the time I’ve been there.

For the first couple years, I was the Director of Quality Engineering on Cloud Foundry. It’s a title I swore I would never take again. But my job was different than you might imagine. I did not direct the efforts of quality engineers. Rather, I paid attention to our feedback cycles. Teams own their tests, their CI pipelines, and ultimately the quality of their deliverables. I just helped connect the dots. By the way, if you want to know more about quality and testing on Cloud Foundry, I did get out of the building long enough to give a talk on it at the Wikimedia Foundation. I also gave a talk on the Care and Feeding of Feedback cycles at DOES2014.

In the last few months I moved over to our Data organization in Palo Alto. This changed my commute substantially, so my family and I are moving this summer. That will be an adventure. We’ve been in the same house for 17 years. So wish me luck with that.

Along with the move to our data org, my title changed. We removed the word “quality” from it since what I do does not look anything like traditional quality engineering. So I’m now a director of engineering. But the work I do on a daily basis with our Data teams looks a lot like what I did with Cloud Foundry: I’m deeply involved in hiring, cross-team coordination, improving our release practices, improving builds and CI to make the developer workflow better, and coordinating with our product organization to make sure teams have a steady stream of high value work.

I’m also doing my best to climb the steep learning curve of MPP databases and Hadoop. It helps that I worked at Sybase once upon a time. But that was 20 years ago. So between the fact that I was doing very different work 20 years ago, I’ve forgotten much of what I learned, and things have changed a bit in 2 decades, my prior database experience is only helping me a little in understanding my new context.

I have to say that I love working at Pivotal. I adore the people, am fascinated by the products, and am passionate about the way we work. Coming back to Pivotal was like coming home. (After all, Pivotal Labs is where I learned Agile over a decade ago.)

Some of you have noted that I don’t get out much anymore. I’m not at conferences and I don’t travel much. Since I’m in an inward facing role it’s difficult for me to carve out time to get out into the community. I’d like to see my industry friends more often and I am always honored to be invited to speak. But I turn down the vast majority of speaking invitations. My job takes up all my available time and brain cells.

So that’s what I’m up to and why I’ve been silent here for so long. I do have things to say though. I’ve learned a lot in the last 30 months. And I’m learning more every day. So I hope to carve out time to share what I’m learning here. But no promises about when, exactly, I’ll post.

Adventures in Commuting

I rested my head against the window, drowsy from a long day, full tummy, and steady hum of the train. My husband sat next to me absorbed in a game. It’s a long ride from San Francisco, where we work, to Pleasanton, where we live. I thought I might take a nap.

A disembodied voice crackled, announcing the next station: “West Oakland.”

The train stopped, the doors opened. I watched the passengers. The train car was about half full. A few people got off. Three black youths, a girl and two boys, got on. They were talking animatedly, happy. All three were well-dressed: fashionable clothes that looked new and fit well. Tight black pants. Caps. A little bling. They’d taken the time to look nice, and they were out for a good time.

One of the boys sat while the girl and the other boy stood, balancing by holding onto the straps suspended from the ceiling of the car. They talked and joked together. I felt a little voyeuristic watching them share this private moment of fun, but I couldn’t help myself. I was too taken with their delight and energy. Their faces lit with joy as they laughed. The girl and the boy hung from the straps and swung their weight around, enjoying the interplay of gravity and momentum of the train and their own upper body strength.

The girl clung to the strap then braced her other hand on the side of the train. Walking her feet up the train side, she executed an impressive flip. She did it again, obviously pleased with herself. “You try doing that in heels!” she crowed. She did it again. I noted with admiration that she was wearing boots with 5″ heels.

I guessed the kids to be somewhere between 15 and 20. It’s an in-between age. Young enough to treat a train as a jungle gym but old enough for a night out on their own. I thought about my own kids, about the same age.

The disembodied crackling voice announced the next station: Lake Merritt. The train stopped. The kids stayed on. The train started again. The train abruptly came to a screeching halt. “That’s not good,” the passenger in front of me muttered. I looked around but couldn’t tell why we were stopped. I leaned my head back against the window; my husband stayed immersed in his game. The trains don’t run on time here. Stopping wasn’t unusual.

Two BART police officers stepped onto the car. Another guy, a passenger, trailed behind them. The passenger with the cops said something I couldn’t hear and gestured toward the three youths. He was a mousy white guy in a button-down oxford shirt, corduroy pants, and a fleece vest. He looked like the kind of guy who drives an SUV with Save the Planet bumper stickers.

The first police officer addressed the youths. “There’s been a complaint. I need you to step off the train.”

“What’s the complaint?” the girl asked.

“We didn’t do anything,” the boy said.

“There’s been a complaint,” the police officer insisted. “Please come with me.”

“What’s the complaint?” the girl repeated.

The mousy passenger who had made the accusation faded back. I didn’t see where he went. Now it was just the cops and the kids.

“We got a complaint that you were asking for money.” The cop’s voice carried.

“We didn’t do anything!” the boy repeated, louder.

“Please come with me,” the cop insisted. “This train can’t move until you get off.”

The crowd on the car grew restless, impatient. The two cops flanked the three youths and attempted to escort them off the train. For a minute, it looked like the kids were going to comply. The boy who was seated stood, headed toward the open door, but then changed his mind. “I didn’t do anything. I was just sitting there,” he said. He got back on the train.

The girl’s face had gone from joy to furor. “It’s that racist white guy,” she yelled, loud enough for the whole train car to hear. “We didn’t do anything, but he thinks we did because we’re black.”

A passenger stood up, Asian, “They really didn’t do anything,” he said to the cops. Then to the kids: “Let’s just step off the train so they can make their report.”

Another passenger stood up. A black woman. “Really, officers, these kids didn’t do anything.”

By this time I was on the edge of my seat. It was so unfair. These kids were just out for a good time. They had done nothing besides being full of energy and life. They hadn’t asked for money. They were being harassed by the cops for something they had not done.

I’m still not sure what made me decide to act on my sense of injustice. Maybe I arrogantly thought the police would listen to me, a middle-class, middle-aged, white woman. Maybe I saw the faces of my own kids in the black youths and imagined them being detained on hearsay. Whatever the trigger, I stood up, moving past my husband who looked at me quizzically. I walked up the train aisle to the officer who had been barking orders.

“Sir, they really didn’t do anything. They just got on at the last stop.” My voice was steady. Calm. Quiet. Pitched for the circle of people involved, not for the rest of the train. Internally I was amazed that my adrenaline wasn’t amped up, that my voice was not shaking. I don’t normally confront cops. It is not at all like me to tell a cop how to do his job. I couldn’t believe I was confronting a cop and not even nervous about it.

By this time the crowd on the train was growing antsy. A few were calling for the kids to get off the train so they could get where they were going, but far more were calling for the cops to leave the kids alone. Several more passengers stood up and started shouting at the cops:

“These kids didn’t do anything!”

“Leave them alone!”

“Racist white guy can’t tell blacks apart!”

“It must have been someone else!”

“Let the kids go!”

I felt self-conscious standing in the middle of everything. I’d played my bit part in the unfolding drama. I returned to my seat.

A woman on the other side of the car leaned over toward me, “I could have done that,” she said. She was white like me, graying hair cut shoulder length in a bob. She looked like she could live in the house next door in my white-picket-fence neighborhood.

“I’m sorry?” I was confused by her words. What does that even mean, I could have done that? If you could have why didn’t you? Did she mean that she could not have done what I did?

“I could have done what you did,” she repeated.

I smiled and nodded. I didn’t know what else to say.

My attention turned back to the growing situation. More passengers had stood up to defend the kids. Another BART police officer entered the car. The increasing tension and buzz from the crowd made it hard to hear. I realized that there were several more police officers waiting on the platform. The situation had escalated rapidly. This could become ugly.

“CHECK THE BUCKET,” the girl shouted.

For the first time I noticed that the kids had an amplifier and a bucket with them, and suddenly I saw a missing piece of the puzzle. The kids were street performers. They weren’t out for a night of clubbing. They were dancers.

Sometimes street performers come onto the BART trains. They strut up and down the aisle, put on a show, and ask for money. I personally find being a captive audience to these shows annoying. I never give the performers money because I don’t want to encourage it. But I did not realize that such behavior warranted police intervention on this scale.

Nor did I think that these kids had been dancing for money on this train. I looked around for the accuser but couldn’t see him.

I was incensed by the entire situation. The accuser had disappeared, the situation was escalating, and these innocent kids were now targets.

“This isn’t right,” I told my husband. I stood up, shouldered my backpack in case I decided to leave the train, and pushed past my husband’s knees for the second time. Without even thinking about whether or not my husband would follow me or would be upset with me for getting involved, I stepped into the situation again.

I sidled up to the cop who had originally been barking orders and who was now standing between the kids and the rest of the passengers.

“They really didn’t do anything,” I said into his ear. “They didn’t. This isn’t right. I will be happy to make a statement. What can I do to help?”

The cop turned to me. “They might not have done anything originally,” he said, “but now they are resisting. If they don’t get off the train we will have to arrest them.”

Another police officer entered the train. The mood of the passengers was growing dark, roiling. It became a palpable thing, shifting energies threatening to spill over into violence. My stomach fluttered.

I turned around, realized that half a dozen people had their cell phones out, recording the incident. I idly wondered if I would show up on youtube videos. I decided I didn’t care. This wasn’t right and I wouldn’t stand for it. These kids were being detained, harassed, and possibly arrested all because the mousy guy in the fleece vest didn’t like being bothered by dancers panhandling on BART. These kids might have danced on BART at some point, but not tonight, not on this train. They didn’t deserve this.

Another officer entered the train. It was now five officers to the three kids. The Asian passenger who had originally stepped up to intervene turned to the kids, “Why don’t we get off the train and get this handled. Then we can all get onto the next train.”

The police closed their circle, puffing themselves up, herding the kids to the exit. I followed them to the door. I was about to step off the train. I turned back to see if my husband was paying attention. He was staring at me, his face open, questioning. “This isn’t right,” I mouthed at him. He stood up, following me.

I turned my attention back to the kids. The cops had one of the boys in an arm lock and pressed him up against the wall of the station. It looked like it hurt. They had the girl surrounded. I couldn’t see her, but I could hear her screaming. I heard another voice — the other boy? the Asian passenger who had intervened? a cop? — telling her “This isn’t worth it. Stop struggling.”

I stepped off the train onto the platform.

The cops who had the girl surrounded had her hands behind her back. She was struggling and screaming. They began half pulling, half dragging her down the platform toward the station exit. “LET GO OF ME,” she shouted. “LET GO OF ME LET GO OF ME LET ME GO!”

An officer stood in the middle of the chaos just looking, sizing it all up. He was new on the scene. He was also the only officer not in the middle of the chaos. I walked up to him. “Sir, they didn’t do anything,” I said. I watched his face. Dark skin. Brown eyes. A guy who’d been having a quiet night. He turned to me.

“You’re a witness?” he asked.

“Yes sir. I was on the car. These kids didn’t do anything. This isn’t right.” The girl was still screaming. Each panicked shriek felt like shrapnel in my soul. That could be my daughter, I thought. So very not right. “I will be happy to make a statement,” I said.

“OK,” the cop replied. He took out a notebook, asked me my name. He had to ask me to spell my last name a second time. I handed him my driver’s license to make things easier for him. He started taking notes, then looked up at my husband and me. “I left my recorder in my office,” he said. “Would you be willing to go there to make a statement?”

I agreed immediately. To my surprise, my husband agreed as well. “Of course,” he said. We would probably be an hour late, or more, getting home. This was worth it.

We followed the officer off the platform, past the turnstiles, and through a door marked “authorized personnel only.” He led us down a maze of corridors, past a bank of cubicles, and into his office. As promised, he took out a recorder and interviewed us for a verbal statement. My husband and I each told our version of the events. They lined up. Each of us emphasized that the kids had done nothing wrong.

“Let me just clarify a few points,” the officer said after we finished our statements. “In your opinion, did the officers use excessive force at any time?” I said they had, citing the painful hands-behind-the-back restraining technique and the panicked girl screaming. My husband said they had not, but pointed out that the officers should have just let the kids go when so many passengers testified that the kids were innocent.

The officer asked a few more questions: had the officers used mace, batons, or physical violence on the kids? had they acted unprofessionally?

We finished the interview, the officer escorted us back into the station, said that internal affairs would probably be contacting us in connection with the investigation, then said goodbye.

As we waited for the next train, I played over the evening in my mind.

I realized that the statements we made existed only on a recorder in the cop’s office. The kids wouldn’t know we made the recorded statement. If they got lawyers, the lawyers wouldn’t know. Our testimony might never be heard by anyone involved in the case against the kids.

The more I played over the questions that the officer had asked us, the more I realized that the questions were all about the conduct of the other officers, not about the kids and their innocence.

The internal affairs investigation would, no doubt, turn into a huge deal. The situation had escalated so rapidly, fueled by racial tension. So many people had recorded videos. The train had been halted for 15 minutes or more. Passengers were irate. There would be much attention on the conduct of the cops.

I didn’t care about the cops. I wanted to help the kids.

Somehow, I doubted that I had.

Next time I see dancers on BART, I’m giving them whatever is in my wallet. I’ll think of it as their defense fund.

Endings and Beginnings

When I wrote my blog post a few weeks back celebrating 15 years in business, I had no idea that in a few short weeks I would be making sweeping changes. I knew I was restless in my current work, and also tired of being on the road all the time. But I did not realize how ready I was for a major change.

Hoo-boy, was I ready for change though. Rapid, sweeping change. Folks who follow me on Twitter know that in the last couple of weeks, I’ve: delivered my PragProg book Explore It!, to production; started a new job at Pivotal Labs; and closed down both Agilistry Studio and my consulting practice, Quality Tree Software, Inc.

I’ve posted farewell letters on both the Quality Tree and Agilistry Studio websites. You can read those for details about those closings. Eventually, however, I’ll be taking those websites down. This site will be my primary web presence.

I want to take a moment here to talk about what’s next for me.

My new job means that I won’t be training or consulting any more, nor do I intend to participate in many conferences for a while. There’s just so much for me to do and learn with my new job that I don’t have the mental bandwidth for anything else for the foreseeable future.

That said, I do intend to keep pushing the boundaries of software testing and overall development practices. I believe that the traditional software QA model is fundamentally and irretrievably broken. It’s time for new models. I want to figure out how to articulate those new models simply, and I hope that my work at Pivotal Labs will help me do that.

In the coming months, I might be writing here more. Or not. I’m not sure yet. We’ll just have to see how things go.

So that’s what I’m up to. What changes are on the horizon for you?

Diverse Discussions

So, there’s another round of gender/race drama going down on teh intarwebs.

It seems that there was a conference, BritRuby, slated to run next year in Manchester, England. Only there was some criticism on Twitter of their all-white-guy lineup. Suddenly, without warning, the organizers canceled the conference. The organizers published an official statement on the conference website, but Sean Handley, one of the organizers, offered a much better and more detailed explanation.

I was unaware of BritRuby until it was canceled. Even if I’d known about it, I couldn’t have participated. So if you think this is a post about BritRuby itself, it’s not. I don’t have an opinion one way or another about BritRuby or their speaker lineup.

But here’s why I do care: because I think it’s deeply unfortunate that anyone is blaming a Twitter conversation about diversity for the cancellation of the conference. That’s absurd. The issue is not whether the Twitterverse has an opinion. The issue is that sponsors pulled out.

Hang on, let’s back up a step.

The official statement on the BritRuby website said, “Sadly, BritRuby was used as the arena to air these issues on Twitter and this has fundamentally destroyed any chance we had of addressing these issues.” And there’s a bunch of Twittering about how people were dogpiling on BritRuby.

Figuring things must have gotten really nasty, I went searching for the smoking gun on Twitter.

I couldn’t find it.

Where were the activist bullies? Where were the accusations of sexism or misogyny? I found none of that. Instead I found expressions of regret:

And offers of help:

Plus some defensiveness:

There were no flame fests, no diatribes. The more I dug, the more it seemed like a relatively mild discussion on Twitter got reflected back and turned around until people who expressed their opinion were being attacked viciously in tweets with words like “Shame on you” and “Disgusting” and “You ruined everything.”

It appeared that the mere question of diversity had a polarizing effect, dividing people into Us v. Them without anyone actually having thrown the first stone.

This is the underlying tragedy: we cannot have a discussion about the issue of diversity in technology without teh intarwebs hitting resonant frequency until something falls apart.

Ultimately, that’s why I care and why I am wading into the fray myself. If we cannot talk about diversity issues openly, we cannot address them. So attempting to shut down discussion of the issue, to shame people into silence, is the same as actively opposing diversity.

If you personally choose not to address diversity issues, that’s fine. Don’t address them. Keep your own silence. But don’t expect that others will ignore the issue just because you want to ignore it. Don’t ask them to be silent.

If you become fed up enough to wade into the general mayhem and express your opinions, do a little research first. Don’t fly off the handle at accusations that someone said something. Find the original quotes. Respond to actual statements, not to incendiary reactions to over-reactions to original statements.

What is at risk here isn’t just a conference. It’s something much bigger: the ability to discuss very real, very personal issues in an open forum.

Update thanks to Laurent Bossavit:

The full list:

Still seems pretty mild to me; not nearly enough to bring down a whole conference all by itself.

But wait, it gets better. Laurent then searched to find the reactions to the original comments. It’s those reactions where folks reframed the original comments using trigger words like “sexism” and “racism.” See Laurent’s link for details.

Testing Has No Value

Update Nov 20: minor edits to increase clarity.

Yesterday, Rob Lambert tweeted:

Turns out that’s a statement from the ISTQB Expert Level Syllabus on Test Management. Robert was tweeting it to see what others’ reactions were to the statement.

Now, I hold no love of the ISTQB. I don’t see any correlation between ISTQB certification and tester competence. I think an awful lot of what the ISTQB promotes as “best practice” is at best context-dependent and at worst actively damaging. But that’s a topic for another post, one I probably will never write. It’s just not worth the energy. I don’t feed that which I don’t want to see grow.

But my point is that I don’t put any stock in anything the ISTQB says, so I was absolutely stunned to find that I agreed with a statement in an ISTQB syllabus.

So I replied, “Whodathunk I’d agree w/ istqb. Unless org sells test svcs, its is a means to an end, a way to get info.”

That touched off a small storm of responses, some agreeing, but many more disagreeing. I’ve been replying to the objections on Twitter, but I think this topic needs more than 140 characters.

First, a few clarifications:

1) I am talking specifically about software development.

2) I am talking about business value.

3) Business value is inextricably linked with increased revenue or decreased costs. That’s not decreased costs of software development. It’s overall decreased costs that can be attributed to the use of the final delivered software.

So the value in software development rests in the final result—the delivered product or service—not the process or activities that went into making the result.

Testing does not have value on its own. Neither does programming. I have seen more than one team crank out lots of code and yet be unable to ship a product anyone would buy. For that matter, designing doesn’t have intrinsic value either.

“But wait!” you say. “Testing does have value. Customers pay more for better quality software, so testing has value.”

Well no. First, it’s not clear that customers pay more for better quality software. Customers are less likely to abandon better quality software in a fit of rage. But they don’t necessarily pay more for the kind of quality that comes from having testers do more testing. Further, testing does not automatically lead to better software. In fact, better testing can actually lead to worse software. Finally, as Ron Jeffries pointed out, testing only adds value to the extent that the team uses the information that testing produced in order to improve the software.

But that’s not even the main point here. The main point is that testing and programming and designing and all the other activities that go into making software are all just a means to an end. As UCLA coach John Wooden said, “Never mistake activity for achievement.”

The bottom line is that if you want your work to have value, make sure that the software you’re working on ships and satisfies customers. The final result is a whole team effort. The team succeeds or fails together. And the value of the work you’re doing is stuck at 0 until the software you’re working on is in the hands of paying customers.

Why I Won’t Go Back

I was going through some of my old notes yesterday and stumbled across a sketch I made of a diagram of effects showing how test managers become ineffectual. (I re-created it in Illustrator since no one but me could have read my original sketch.)

I’d like to say that back when I was a test manager I never succumbed to this cycle. Alas, I’m sure I did. It’s almost inevitable.

Reading through those old notes was a good reminder about why I don’t manage independent QA/Test groups anymore.

I’d much rather work with organizations to integrate testing activities throughout the cycle and across all roles. The traditional role of a test manager in organizations that practice test-last development is a miserable job. Add in a sprinkle of dysfunctional QA blaming while simultaneously undercutting any efforts the QA manager makes to increase quality? The job becomes a recipe for depression and substance abuse.

Fortunately, I escaped in time.

Bugs Spread Disease

I have wasted countless hours of my life arguing to fix bugs in bug triage meetings.

Bug advocacy is a core skill for testers in traditional software development organizations that follow code-then-test practices. Over time, I got reasonably good at it. I could explain to both business and technical stakeholders not only the symptoms of the bug and steps to reproduce it, but also the corresponding impact to users. I could help my internal stakeholders see the connection between the risk that the bug represented and the core value of whatever system we were working on. In short, I became adept at making a business case to fix bugs.

It was awful. I hated every minute of it.

Every bug triage meeting was torture. They were long and contentious. I didn’t understand then, nor do I understand now, why an organization would bother spending so much time and money on testing if they weren’t going to do anything with the information that testing revealed.

These days I choose to do hands-on-the-keyboard work only in organizations where they value clean code so highly that I don’t have to draw on those bug advocacy skills.

But even now, there is a mistaken notion in the software development community that bugs are inevitable, that you can’t fix them all, and that it makes sense to make a fix/defer decision on bugs based on some notion of ROI.

This line of thinking is a deathtrap.

It might be a slow, lingering death. It might, in fact, be such a slow death that it’s not apparent how each defer decision is bringing death for the product, or company, a little closer. But once we start down the apologists path of deferring “minor” bugs in mountainous inventories of defect backlogs, we’re inviting a sinister guest into our midst.

If you think I’m being overly dramatic, consider this: two of the companies where I spent the most time in bug triage meetings are now dead. So I think I know a deathtrap when I see it. Let me share their stories.

One of the companies sold software directly to consumers. The other was a paid service with a software component distributed through large OEMs. In both cases we “didn’t have time” to fix all the bugs. There were always higher priorities. We had to get features out. We had to make deadlines. We had to respond to customer demands.

In both cases the executive staff impressed on us that every day we delayed a release incurred a massive cost. They pushed us to be pragmatic. Reasonable. Make good business decisions about tradeoffs.

The executives were not wrong. Both companies were in rapidly changing markets where speed was critical. In the case of the company that sold to OEM partner companies, the deadlines were incredibly real. Miss the drop date by even a single day, and we wouldn’t be on the platform. That meant we wouldn’t see revenue on that platform for another 6 months.

Under such conditions it’s easy to see how we made the decisions we did.

But remember, both companies are dead.

It wasn’t the bugs that killed us directly. Rather, the bugs became a pervasive infection. They hampered us, slowing down our productivity. Eventually we were paralyzed, unable to make even tiny changes safely or quickly.

Whenever we considered the question of whether to defer or fix a bug, we looked at the obvious costs associated with releasing software with known bugs: annoyed customers, tech support, and the overhead associated with releasing patches for the most egregious cases.

We were right about those costs. So we weren’t surprised when the resulting interruptions and escalations forced us to split our efforts between new development and maintenance. We just hired more people to keep up with the increased demands and told ourselves that it was the cost of doing business in the software industry.

However, it was the hidden costs that drained us of life: hours lost arguing about bugs in bug triage meetings; hours lost stumbling over the same known issues again and again; hours lost fighting with a fragile and error-prone code base to make even small changes; hours lost cataloging and categorizing the backlog. It was demoralizing and immensely expensive.

All those hours leached away, drained our capacity. We could see it happening but were powerless to stop it. By the time we realized how much trouble we were in, it was too late. The infection had taken root and nothing short of a total overhaul of the entire code base would make things better.

The consumer software company eventually died because a more nimble competitor came along. We simply could not keep up with the changing market demands in a rapidly evolving field.

The OEM partner died because we just couldn’t generate enough revenue. What little revenue we had disappeared as one-by-one, the large OEMs dropped us. The OEM’s internal staff had no faith in our software so when their customers complained to their tech support, the OEM tech support blamed our software.

There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little “cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again, you will spend hours upon hours on it.

This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill. They do it slowly, painfully, but relentlessly.

Want your software to evade that horrific death?

Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified; attend to your broken windows early.

And whatever you do, don’t accept the apologist’s excuse that bugs in production are inevitable, so the only pragmatic approach is to make a cost/benefit decision on whether or not to fix them. The cost is almost always far higher than anyone might guess.

I have been blogging even less than usual because all my writing time is going into a book that hasn’t been formally announced yet. I’ll have official news soon.

Happy Birthday, Quality Tree Software

Last month marked the 15 year anniversary of my decision to go into business for myself.

Over the last decade and a half I have learned so very much, and had the opportunity to work with so many amazingly cool people. I feel immensely grateful. I’ve also had my challenges: strategic and financial missteps that cost me dearly. The cumulative result has been a carnival ride of emotions: panic, joy, mounting despair, satisfaction, panic, glee, giddy bouncy happy glee, disappointment, hope.

This being in business stuff is not for the faint of heart.

Because I’ve been in business so long, aspiring consultants sometimes ask me for advice. So I’ll tell you what I tell them:

Be prepared to work. Hard. Harder than you have in your life, and for less pay.

Some people think being a consultant is easier or cushier or more lucrative than having a “real job.” They see the hourly or daily rates that consultants charge, do the math to calculate an equivalent salary, and see buckets of shiny money. Or they equate independence with getting to work only when they want to, and on whatever they want to.

Certainly there are independent consultants who have structured their businesses to optimize for wealth, freedom, or both. You might succeed at doing that. But you need to know before you start that it’s a really long road between hanging out your shingle and spending 6 months every year lying on a beach.

Clients don’t just line up automatically ready to write you big fat checks for doing whatever you feel like. First, you have to find clients. When you’re first starting out that will be harder than you think. Then, in order to land your first few clients, you are likely to have to take gigs that aren’t exactly your ideal.

Out of your first 12 months in business, you will spend about a quarter your days doing billable work, the other three quarters trying to get billable work, and the other half of your days splitting your time between attending to the minutia of running a business and connecting with people in ways that won’t get you billable work immediately but might in 18 months.

Yes, I am aware that is 150% of your time. That’s roughly what it takes.

Later when you’re more established, the balance will shift. You’ll have more billable work and have to spend less time and energy scrambling to get your next gig. So it goes from 25-75-50 to something more like 50-50-50. Or, if you opt for a business model with longer engagements (e.g. contracting) you might end up at 75-25-50. If you want a business that involves shorter engagements and more time to work on pet projects (training, short term consulting), you might end up with a ratio that looks like 25-25-100.

Notice that the overall effort is still 150%. If you want a lifestyle business where you can put in 75% of the effort you would put into a job working for someone else, expect to get something like 10% of the revenue.

The bottom line is that running a business, any business, is hard.

I am friends with a variety of small business owners across a range of industries. We all have one thing in common: we work our backsides off to make our businesses go.

What people who don’t run businesses don’t understand is that the part of the business they see is just the front of the stage. They don’t see behind the scenes: the long nights spent juggling numbers, trying to predict and balance cash flow. They don’t see all the preparation that goes into the delivery of the product or service. They don’t see the duck legs frantically paddling below the surface; they just see the serene quacker gliding across the water.

As a business owner, there will always something that needs your attention urgently. Stuff happens. Problems crop up. Things don’t go as planned. So between handling emergencies, serving existing clients, meeting new clients, reviewing contracts, writing up the statements of work or proposals, growing your own skills so that you stay relevant in an ever changing market, invoicing, managing logistics, developing content, and attending or presenting at conferences, you will not have an actual vacation for years.

You might travel on business and squeeze out a few hours, or maybe even days, to see the sights. You might take a day off here and there when you’re too sick, or too tired, to continue. But running a business is a 7-day-a-week job. Your business will always be on your mind. You can’t escape it.

The good news is that when you figure out how to make your consulting practice fly, the rewards of all that work are definitely worth it. You get to follow your interests, work with a wider variety of people, help solve a wider variety of problems, and generally have more autonomy than when you’re working for someone else.

Even the cost of that autonomy, that you are wholly and solely responsible for finding ways to ensure you get a paycheck, is a reward in and of itself. It means that you have made a direct connection between the value you offer and the monetary measure of that value. It’s a connection few who work for others really get to make.

So the last 15 years have been amazing, humbling, fulfilling, exhausting, rewarding, challenging, and terrifying all rolled together.

I sometimes say that if I knew back when I started my company in 1997 what I know now, I probably would not have had the guts to do it. Yet I’m very glad I did. So if I could know not just what hardship lay ahead but also what satisfaction and joy there would be, then yes, I would do it all over again.

You can do it too, if that’s what you want.

It’s a wild ride. Your particular path will be different from anyone else’s, so you’ll have to find your own way. The whipsaw ups and downs and hard turns might make you a little green around the gills for a while. But if you can hold it together long enough, you’ll learn how to roll with it.

Being independent, or starting a company, isn’t for everyone.

Is it for you?

If you want to own your own professional destiny so badly that you’re willing to do whatever (legal, ethical, but exhausting) work it takes to make it through those incredibly rough first few years, then yes, yes indeed. It might be just the thing.