Test Obsessed

When I wrote my blog post a few weeks back celebrating 15 years in business, I had no idea that in a few short weeks I would be making sweeping changes. I knew I was restless in my current work, and also tired of being on the road all the time. But I did not realize how ready I was for a major change.

Hoo-boy, was I ready for change though. Rapid, sweeping change. Folks who follow me on Twitter know that in the last couple of weeks, I’ve: delivered my PragProg book Explore It!, to production; started a new job at Pivotal Labs; and closed down both Agilistry Studio and my consulting practice, Quality Tree Software, Inc.

I’ve posted farewell letters on both the Quality Tree and Agilistry Studio websites. You can read those for details about those closings. Eventually, however, I’ll be taking those websites down. This site will be my primary web presence.

I want to take a moment here to talk about what’s next for me.

My new job means that I won’t be training or consulting any more, nor do I intend to participate in many conferences for a while. There’s just so much for me to do and learn with my new job that I don’t have the mental bandwidth for anything else for the foreseeable future.

That said, I do intend to keep pushing the boundaries of software testing and overall development practices. I believe that the traditional software QA model is fundamentally and irretrievably broken. It’s time for new models. I want to figure out how to articulate those new models simply, and I hope that my work at Pivotal Labs will help me do that.

In the coming months, I might be writing here more. Or not. I’m not sure yet. We’ll just have to see how things go.

So that’s what I’m up to. What changes are on the horizon for you?

So, there’s another round of gender/race drama going down on teh intarwebs.

It seems that there was a conference, BritRuby, slated to run next year in Manchester, England. Only there was some criticism on Twitter of their all-white-guy lineup. Suddenly, without warning, the organizers canceled the conference. The organizers published an official statement on the conference website, but Sean Handley, one of the organizers, offered a much better and more detailed explanation.

I was unaware of BritRuby until it was canceled. Even if I’d known about it, I couldn’t have participated. So if you think this is a post about BritRuby itself, it’s not. I don’t have an opinion one way or another about BritRuby or their speaker lineup.

But here’s why I do care: because I think it’s deeply unfortunate that anyone is blaming a Twitter conversation about diversity for the cancellation of the conference. That’s absurd. The issue is not whether the Twitterverse has an opinion. The issue is that sponsors pulled out.

Hang on, let’s back up a step.

The official statement on the BritRuby website said, “Sadly, BritRuby was used as the arena to air these issues on Twitter and this has fundamentally destroyed any chance we had of addressing these issues.” And there’s a bunch of Twittering about how people were dogpiling on BritRuby.

Figuring things must have gotten really nasty, I went searching for the smoking gun on Twitter.

I couldn’t find it.

Where were the activist bullies? Where were the accusations of sexism or misogyny? I found none of that. Instead I found expressions of regret:

Nice speaker lineup for @britruby. Except for the 100% white guys part.

— Josh Susser (@joshsusser) November 17, 2012

And offers of help:

@britruby I’m sure we could hook you up with people who’d be interested in helping be a part of organizing and would help drive diversity.

— ashe dryden (@ashedryden) November 17, 2012

Plus some defensiveness:

@raganwald @anthonycgreen Well said… #BritRuby will keep fighting this issue but stands firm in its views on how speakers are selected.

— British Ruby Conf (@BritRuby) November 17, 2012

There were no flame fests, no diatribes. The more I dug, the more it seemed like a relatively mild discussion on Twitter got reflected back and turned around until people who expressed their opinion were being attacked viciously in tweets with words like “Shame on you” and “Disgusting” and “You ruined everything.”

It appeared that the mere question of diversity had a polarizing effect, dividing people into Us v. Them without anyone actually having thrown the first stone.

This is the underlying tragedy: we cannot have a discussion about the issue of diversity in technology without teh intarwebs hitting resonant frequency until something falls apart.

Ultimately, that’s why I care and why I am wading into the fray myself. If we cannot talk about diversity issues openly, we cannot address them. So attempting to shut down discussion of the issue, to shame people into silence, is the same as actively opposing diversity.

If you personally choose not to address diversity issues, that’s fine. Don’t address them. Keep your own silence. But don’t expect that others will ignore the issue just because you want to ignore it. Don’t ask them to be silent.

If you become fed up enough to wade into the general mayhem and express your opinions, do a little research first. Don’t fly off the handle at accusations that someone said something. Find the original quotes. Respond to actual statements, not to incendiary reactions to over-reactions to original statements.

What is at risk here isn’t just a conference. It’s something much bigger: the ability to discuss very real, very personal issues in an open forum.

Update thanks to Laurent Bossavit:

@testobsessed Here’s a gist of *all* criticism leveled at BritRuby during 24h between “white guys” twt & cancellation: gist.github.com/4108671

— Laurent Bossavit (@Morendil) November 19, 2012

The full list:

Still seems pretty mild to me; not nearly enough to bring down a whole conference all by itself.

But wait, it gets better. Laurent then searched to find the reactions to the original comments. It’s those reactions where folks reframed the original comments using trigger words like “sexism” and “racism.” See Laurent’s link for details.

@testobsessed Now compare it to the reactions (and I’ve just sampled the early ones): bit.ly/Qohbtz

— Laurent Bossavit (@Morendil) November 19, 2012

I was going through some of my old notes yesterday and stumbled across a sketch I made of a diagram of effects showing how test managers become ineffectual. (I re-created it in Illustrator since no one but me could have read my original sketch.)

I’d like to say that back when I was a test manager I never succumbed to this cycle. Alas, I’m sure I did. It’s almost inevitable.

Reading through those old notes was a good reminder about why I don’t manage independent QA/Test groups anymore.

I’d much rather work with organizations to integrate testing activities throughout the cycle and across all roles. The traditional role of a test manager in organizations that practice test-last development is a miserable job. Add in a sprinkle of dysfunctional QA blaming while simultaneously undercutting any efforts the QA manager makes to increase quality? The job becomes a recipe for depression and substance abuse.

Fortunately, I escaped in time.

I have wasted countless hours of my life arguing to fix bugs in bug triage meetings.

Bug advocacy is a core skill for testers in traditional software development organizations that follow code-then-test practices. Over time, I got reasonably good at it. I could explain to both business and technical stakeholders not only the symptoms of the bug and steps to reproduce it, but also the corresponding impact to users. I could help my internal stakeholders see the connection between the risk that the bug represented and the core value of whatever system we were working on. In short, I became adept at making a business case to fix bugs.

It was awful. I hated every minute of it.

Every bug triage meeting was torture. They were long and contentious. I didn’t understand then, nor do I understand now, why an organization would bother spending so much time and money on testing if they weren’t going to do anything with the information that testing revealed.

These days I choose to do hands-on-the-keyboard work only in organizations where they value clean code so highly that I don’t have to draw on those bug advocacy skills.

But even now, there is a mistaken notion in the software development community that bugs are inevitable, that you can’t fix them all, and that it makes sense to make a fix/defer decision on bugs based on some notion of ROI.

This line of thinking is a deathtrap.

It might be a slow, lingering death. It might, in fact, be such a slow death that it’s not apparent how each defer decision is bringing death for the product, or company, a little closer. But once we start down the apologists path of deferring “minor” bugs in mountainous inventories of defect backlogs, we’re inviting a sinister guest into our midst.

If you think I’m being overly dramatic, consider this: two of the companies where I spent the most time in bug triage meetings are now dead. So I think I know a deathtrap when I see it. Let me share their stories.

One of the companies sold software directly to consumers. The other was a paid service with a software component distributed through large OEMs. In both cases we “didn’t have time” to fix all the bugs. There were always higher priorities. We had to get features out. We had to make deadlines. We had to respond to customer demands.

In both cases the executive staff impressed on us that every day we delayed a release incurred a massive cost. They pushed us to be pragmatic. Reasonable. Make good business decisions about tradeoffs.

The executives were not wrong. Both companies were in rapidly changing markets where speed was critical. In the case of the company that sold to OEM partner companies, the deadlines were incredibly real. Miss the drop date by even a single day, and we wouldn’t be on the platform. That meant we wouldn’t see revenue on that platform for another 6 months.

Under such conditions it’s easy to see how we made the decisions we did.

But remember, both companies are dead.

It wasn’t the bugs that killed us directly. Rather, the bugs became a pervasive infection. They hampered us, slowing down our productivity. Eventually we were paralyzed, unable to make even tiny changes safely or quickly.

Whenever we considered the question of whether to defer or fix a bug, we looked at the obvious costs associated with releasing software with known bugs: annoyed customers, tech support, and the overhead associated with releasing patches for the most egregious cases.

We were right about those costs. So we weren’t surprised when the resulting interruptions and escalations forced us to split our efforts between new development and maintenance. We just hired more people to keep up with the increased demands and told ourselves that it was the cost of doing business in the software industry.

However, it was the hidden costs that drained us of life: hours lost arguing about bugs in bug triage meetings; hours lost stumbling over the same known issues again and again; hours lost fighting with a fragile and error-prone code base to make even small changes; hours lost cataloging and categorizing the backlog. It was demoralizing and immensely expensive.

All those hours leached away, drained our capacity. We could see it happening but were powerless to stop it. By the time we realized how much trouble we were in, it was too late. The infection had taken root and nothing short of a total overhaul of the entire code base would make things better.

The consumer software company eventually died because a more nimble competitor came along. We simply could not keep up with the changing market demands in a rapidly evolving field.

The OEM partner died because we just couldn’t generate enough revenue. What little revenue we had disappeared as one-by-one, the large OEMs dropped us. The OEM’s internal staff had no faith in our software so when their customers complained to their tech support, the OEM tech support blamed our software.

There’s a lesson here about the real cost of tolerating bugs, of supporting practices that involve triaging defects. The cost of carrying a bug is far far greater than most people realize. That trivial little “cosmetic” issue will cost a little time to fix. If you have to argue about whether or not to fix it, then track it, then revisit it, then prioritize it, then argue again about whether to fix it or defer it yet again, you will spend hours upon hours on it.

This is why I have no patience for the “bugs are inevitable; you can’t fix them all” attitude. Bugs kill. They do it slowly, painfully, but relentlessly.

Want your software to evade that horrific death?

Cancel all the bug triage meetings; spend the time instead preventing and fixing defects. Test early and often to find the bugs earlier. Fix bugs as soon as they’re identified; attend to your broken windows early.

And whatever you do, don’t accept the apologist’s excuse that bugs in production are inevitable, so the only pragmatic approach is to make a cost/benefit decision on whether or not to fix them. The cost is almost always far higher than anyone might guess.

A reader wrote me to ask:

I know what metrics to use to manage a traditional phased project. But what metrics do you use on Agile projects?

I started drafting my answer in a private email but I decided that it was time to put my answer on the record.

This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue using the same metrics they were using for traditional projects even though the process is fundamentally different.

The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful.

• Requirements coverage is meaningless if the definition of “Done” includes tested. The number will always be 100%. If it’s not 100%, you have a deeper problem that metrics will not help you solve.
• Test pass/fail ratios are not useful. If a test that was passing starts failing, the team stops and fixes the regression right away.
• Counting the defects found internally is not particularly helpful, unless it’s a big number, and then we stop to figure out why there are so many bugs.
• Traditional QA defect ratio-based metrics, especially defect detection efficiency, can be actively harmful. It’s particularly horrific if the testers are being judged based on DDE. In every situation I’ve seen and heard about where that happened, the testers spent more time arguing about what was and was not a bug than actually helping to move the project forward.

So that’s what I don’t do. Before I launch into the metrics that I do use, there are some caveats I have to tell you.

First, I only use metrics to get a 50,000 foot view on what’s going on. They provide an indicator that there’s something to investigate, but they don’t give me enough understanding to make critical decisions. To make important decisions, I have to dig into the details.

Second, I do not use metrics to compare teams or individuals. Ever. This is important. The best way to screw up the usefulness of any process metric is to use it to judge people. This is a basic principle of metrics and has nothing to do with the development process. But apparently it’s something that needs repeating; I still see managers trying to use velocity or defect metrics to compare teams.

I even saw one manager institute the notion of “personal velocity” on an Agile team. He thought it was great. He wanted to know who the solid producers were, and he thought it led to greater attention to individual responsibility. But he was blind to the side effects: people did not help each other because they knew their own personal productivity would decline if they spent their time helping others.

Third, I am much more focused on qualitative than quantitative measures. Counting leads to the illusion that we can understand something because we can quantify it. But numbers can be very misleading. Worse, counting leads to perverse incentives all too often. Even when managers are not assessing people based on process metrics, counting things affects behavior.

Given all that, here’s the list of what I do use:

1. The core measure I use is velocity, or Running Tested Features as Ron Jeffries calls it. Note that the “Tested” part is important: velocity is only meaningful if Tested is part of the definition of Done.
2. I sometimes look at cycle time (the time it takes from when a story moves from To Do into In Progress to the time it’s Done). If the team is having difficulty producing Running Tested Features, measuring the cycle time of the simplest possible change is an enlightening diagnostic. If taking a tiny, self-contained, non-risky change all the way to Done takes more than an hour, then it’s not at all surprising the team can’t get real stuff completed. At that point I start digging to find out what the impediments are.
3. I use the Continuous Integration system to tell me about the health of the build. Sometimes if the build seems to be red a lot I count the time the build stays red, or the time the build has been green. (Think of a big poster: “Days since last build fail: 15″) But any time you count build metrics there is a risk of affecting checkin behavior. I’ve seen a team that is so scared of breaking the build that they stopped checking in altogether. Clearly this is not ideal.
4. I do measure code coverage on the unit tests. I don’t have any illusion that code coverage tools can actually tell me anything about how good the testing was, but if there are large swaths of code with no unit tests, I view that as technical debt and start paying it down.
5. I do pay attention to defects reported from the field. If there are enough of them, counting might be useful. But counting isn’t necessary. One of the more effective things I’ve seen is a support group that created visibility around field problems by posting the big ones on a big visible chart prominently located where everyone (testers, developers, executives, etc.) would see it.

Dale Emery has taught me an enormous amount about using resistance as a resource.

I’m grateful. I use his ideas every time I set foot in a classroom or start consulting with a new client.

In particular, I channel my inner Dale whenever discussing any of the various controversial things I advocate, such as:

The whole team is responsible for testing and quality, not just QA or the designated testers.

If the regression tests aren’t automated and the team is having a hard time finishing all the testing within a sprint, have programmers help execute them.

Do away with traditional (and often punitive) defect metrics like % missed by phase. Focus instead on metrics related to accomplishments: story points completed, cycle time, and test coverage.

In many organizations these suggestions fly in the face of their accepted “best practices.” Such ideas also tread on political toes. So one response I hear a lot is: “That’s a nice theory, but it won’t work here.”

Before learning techniques for reframing that resistance into a resource, I would end up in a position based argument that amounted to the professional equivalent of “Will too!” “Will not!” “Will too!” “Will not!” Not useful.

Dale’s nicer and wiser than I am, so even when I don’t handle the interaction as well as I would like, by leaning on Dale’s techniques, I certainly handle the conversation better than I otherwise would have. (Although I’m far from perfect at this. Sometimes people succeed in pushing my buttons in such a way that I forget everything I know about how to communicate effectively.)

The first thing I need to know is whether the results of doing whatever it is would be useful. So I ask something like:

Do you think this practice could help improve things?

If I hear “No,” as the reply, we have a fundamental and possibly insurmountable difference in perspective. Nothing I can say will make them try the practice if they do not believe there is any value in it.

I can explore their reasoning. I can say, “That’s interesting. Why not?” But if someone flat out does not believe that a practice I advocate will help, and I disagree even after listening to their reasoning, there is a good chance that I will not be able to help them. Further discussion will cause more harm than good. The best thing for me to do is to stop.

On the other hand if I hear “Yes…but…” then we have a different conversation. First I have to understand what follows the “but…” Often it’s:

It won’t work here.

At this point I am tempted to ask, “Why not?”

But I don’t.

“Why not?” won’t get us anywhere. We’ll end up running down a rathole of excuses starting with “our context is different.” (And of course, they’re right. Their context is different. Every context is different.)

So instead of asking “Why not?” I flip it around. I ask:

What would have to change for this practice to work here?

Now we get a list of objections, but each one is framed as a neat little impediment statement.

It would need to allow for this inherent complexity in our situation.

We’d need to allow time for it.

We’d need executive support.

We’d need money in the training budget.

We’d need to get the programmers to buy in.

We’d need the QA manager to agree.

And we can work on each of those impediments in much the same way, following the trail of reasons why this is a nice theory that can’t possibly work in their real world context all the way down to the bottom.

In what way would it have to accommodate the complexity?

What would have to happen in order to make time for it?

What would have to happen in order to get executive support?

What would have to happen in order to get budget money?

What would have to happen in order for the programmers to buy in?

What would have to happen in order for the QA manager to agree?

The answers usually reveal perfectly practical steps. We can talk to the people in a position of authority or influence who can get us resources, training, budget money. We can try a small pilot. We can experiment with variations.

The simple reframe from “Why not?” to “What would have to change?” opens up possibilities. What could have become an argument becomes instead a brainstorming session. The result is a chain of steps we can take to go from where we are now to where we want to be.

Happy New Year!

A funny thing happened on my way to inbox 0 last week: I wrote a book in 4 days.

I didn’t mean to. And actually it’s not true to say that I wrote it in just 4 days. I assembled it in 4 days; I wrote it over 15 years. Allow me to present There’s Always a Duck, now available on Leanpub.

To fully explain, I need to back up a step.

Last Thursday I learned that Laurent Bossavit, who I admire tremendously, had published a work-in-progress book, The Leprechauns of Software Engineering, on Leanpub. Leanpub is a relatively new service designed to make it easy to publish before your book is complete so you can get feedback while you write. Their motto is “publish early, publish often.”

So I immediately purchased Laurent’s book. I found it to be a delightful read. In it he chronicles his attempts to track down the source of some of our most cherished beliefs: the cost of change curve, 10x productivity differential between star programmers and average programmers, etc.

Laurent’s current draft is 79 pages with many more sections outlined. And the nice thing about the way Leanpub works is that Laurent can keep writing, and I can re-download the book any time. Further, Laurent can notify everyone who bought the book when he’s made a substantial addition. I’m really looking forward to future drafts.

Since I hadn’t heard of Leanpub before, I was intrigued. I’ve investigated various other self-publishing channels including CreateSpace and SmashWords. But Leanpub seemed different. So I watched their introductory video, an XtraNormal animated short. Within a minute I was laughing out loud. 2 minutes into the 10 minute video I made myself a Leanpub account.

Leanpub made it absurdly easy to turn my blog into a book. They imported my content from my RSS feed and converted it from HTML into Markdown (the markup language they use for publishing). They put the resulting manuscript into a DropBox folder. I already use DropBox, so getting set up was absolutely trivial.

The result: within a few minutes of signing up, I had a 300 page book of my blog posts organized chronologically.

I started sifting through the content, deciding what would go into a book and rearranging the posts into chapters by topic. By Thursday evening I had a draft.

On Friday I had every intention of attending to my backlog of To Dos. But the book called to me. “I’ll just make a few tweaks,” I told myself.

As I continued arranging the content, I realized that some of my older content hadn’t been imported. Some of it was still on my blog but just wasn’t in the RSS feed. I manually pulled in a handful of older posts that I wanted to include in the book.

But I realized some of my oldest content was missing from my blog. Then I remembered that I’d purged all the really old content from my site and I discovered that I didn’t have backups. Whoops!

Down the rabbit hole I went, digging up all my old stuff from The Internet Wayback Machine.

By this time I was feeling guilty about how much time I was spending on an unscheduled project. Thanks to Leanpub’s book announcement page and a few tweets, I had 30 people who had signed up to be notified when the book went live by Friday afternoon. I resolved to hold off on working on the book until at least 50 people indicated interest. So I set the book aside and worked on an overdue client proposal.

My resolution lasted all of 12 hours. Saturday morning found me hunkered over my keyboard, selecting and arranging content. By late Saturday night the book had come together into a cohesive draft. It just needed a good cover, a little more new prose, and another editing pass. I went to sleep at 1AM, tired but happy.

I awoke Sunday possessed with the idea of finishing. It was just SOOOO close. So I spent most of Sunday polishing the final bits.

The cover took a little longer than I had anticipated. I knew I had the perfect picture for it, a picture I took of a heated duck pond in front of the Finlandia concert hall in Helsinki during winter. But I couldn’t find the picture. My husband saved me: he found a copy of it on one of our old backup drives. Then I had to figure out how to reduce the image size so that a 500K download didn’t balloon to 4MB just for the pretty cover shot.

Despite the delays, it all came together within a few hours and I hit “Publish” on Sunday around 3PM.

So that’s how I published a book in 4 days.

Of course the marvelous thing about Leanpub is that while I’ve published, I can also update. I can fix mistakes (I’ve found a couple small wording glitches already). And I can even add entirely new content. So hitting Publish wasn’t much more nervewracking than publishing a blog post.

And yet it was.

This is a BOOK. An actual honest to goodness BOOK. The running joke between me and my friends for years has been “How’s that book coming?” I’ve been working on various books off and on for years. I’ve abandoned most of those projects. So this is a momentous occasion. Even if it is a self-published eBook, it’s still an important step.

Now that I’ve gotten the first one done, there will be more. I suspect that 2012 will be my year of publishing. I have other things in the works that I’m not ready to talk about yet.

2012 is off to a great start!