Question from the Mailbox: What Metrics Do You Use in Agile?

A reader wrote me to ask:

I know what metrics to use to manage a traditional phased project. But what metrics do you use on Agile projects?

I started drafting my answer in a private email but I decided that it was time to put my answer on the record.

This is a great question. Too many organizations transitioning to Agile don’t ask it. They continue using the same metrics they were using for traditional projects even though the process is fundamentally different.

The problem is that traditional measures don’t work in Agile. Sometimes they’re actively harmful.

  • Requirements coverage is meaningless if the definition of “Done” includes tested. The number will always be 100%. If it’s not 100%, you have a deeper problem that metrics will not help you solve.
  • Test pass/fail ratios are not useful. If a test that was passing starts failing, the team stops and fixes the regression right away.
  • Counting the defects found internally is not particularly helpful, unless it’s a big number, and then we stop to figure out why there are so many bugs.
  • Traditional QA defect ratio-based metrics, especially defect detection efficiency, can be actively harmful. It’s particularly horrific if the testers are being judged based on DDE. In every situation I’ve seen and heard about where that happened, the testers spent more time arguing about what was and was not a bug than actually helping to move the project forward.

So that’s what I don’t do. Before I launch into the metrics that I do use, there are some caveats I have to tell you.

First, I only use metrics to get a 50,000 foot view on what’s going on. They provide an indicator that there’s something to investigate, but they don’t give me enough understanding to make critical decisions. To make important decisions, I have to dig into the details.

Second, I do not use metrics to compare teams or individuals. Ever. This is important. The best way to screw up the usefulness of any process metric is to use it to judge people. This is a basic principle of metrics and has nothing to do with the development process. But apparently it’s something that needs repeating; I still see managers trying to use velocity or defect metrics to compare teams.

I even saw one manager institute the notion of “personal velocity” on an Agile team. He thought it was great. He wanted to know who the solid producers were, and he thought it led to greater attention to individual responsibility. But he was blind to the side effects: people did not help each other because they knew their own personal productivity would decline if they spent their time helping others.

Third, I am much more focused on qualitative than quantitative measures. Counting leads to the illusion that we can understand something because we can quantify it. But numbers can be very misleading. Worse, counting leads to perverse incentives all too often. Even when managers are not assessing people based on process metrics, counting things affects behavior.

Given all that, here’s the list of what I do use:

  1. The core measure I use is velocity, or Running Tested Features as Ron Jeffries calls it. Note that the “Tested” part is important: velocity is only meaningful if Tested is part of the definition of Done.
  2. I sometimes look at cycle time (the time it takes from when a story moves from To Do into In Progress to the time it’s Done). If the team is having difficulty producing Running Tested Features, measuring the cycle time of the simplest possible change is an enlightening diagnostic. If taking a tiny, self-contained, non-risky change all the way to Done takes more than an hour, then it’s not at all surprising the team can’t get real stuff completed. At that point I start digging to find out what the impediments are.
  3. I use the Continuous Integration system to tell me about the health of the build. Sometimes if the build seems to be red a lot I count the time the build stays red, or the time the build has been green. (Think of a big poster: “Days since last build fail: 15″) But any time you count build metrics there is a risk of affecting checkin behavior. I’ve seen a team that is so scared of breaking the build that they stopped checking in altogether. Clearly this is not ideal.
  4. I do measure code coverage on the unit tests. I don’t have any illusion that code coverage tools can actually tell me anything about how good the testing was, but if there are large swaths of code with no unit tests, I view that as technical debt and start paying it down.
  5. I do pay attention to defects reported from the field. If there are enough of them, counting might be useful. But counting isn’t necessary. One of the more effective things I’ve seen is a support group that created visibility around field problems by posting the big ones on a big visible chart prominently located where everyone (testers, developers, executives, etc.) would see it.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

9 Responses to Question from the Mailbox: What Metrics Do You Use in Agile?

  1. Adam Yuret February 23, 2012 at 5:10 pm #

    Great post,

    I have a personal bias against the “V-word” as it conjures up images of points abuse in my mind. But these are all great metrics by which ti measure an agile project.

    I’d like to add a squishier metric from Jim Benson & Tonianne DeMaria Barry’s Personal Kanban of “Relative State of Well Being”. Adding a way to visualize the way individuals feel about the work they’re completing can be a very useful method of optimizing team harmony and therefore effectiveness.

    One way to do this is to suggest team members add smiley-faces to task-board stickies, or even set aside a section of the board for tasks that proved difficult. Team members can throw those stickies into those areas as they finish them and have a visible way of identifying pain points. From an estimation perspective this can be helpful in that tasks that were estimated low or seemed easy during the planning phase will occasionally find their way into that section.

    Not sure if that’s very metricky but it’s my 2 cents. :-)

  2. Jesper L Ottosen February 23, 2012 at 8:42 pm #

    I reckon you can get as much overboard in metrics for agile – as in metrics and measurements for waterfall projects. as also illustrated here: http://exploringuncertainty.com/blog/archives/419

    One of the points in my http://www.thetestingplanet.com/2011/07/a-little-track-history-that-goes-a-long-way/ is to show two tracking “objects”:
    - progress / velocity – compared to an average trend / pattern
    - cycle time / defect throughput – trend /pattern over time

    /Jesper

  3. Ilya Kurnosov March 2, 2012 at 8:40 pm #

    Loved your pitch “do not use metrics to compare teams or individuals. Ever.” I believe it should be on the first page of any book on metrics.

    I suspect that the problem is not that “traditional measures don’t work in Agile”. Likely, they didn’t work for traditional projects either. I can recall congruent idea spelled in “Testing Computer Software” by Kaner et al. almost 13 years ago: “If you work in a company that will use performance measures against individual testers, don’t collect the data.”

    By the way, I’m pretty sure that your points (at least 4 and 5) can be successfully aplied to traditional projects as well.

  4. Vikram Kapoor March 3, 2012 at 3:53 pm #

    Hi

    Good piece! You mention “velocity is only meaningful if Tested is part of the definition of Done”. Tested by whom? The team or the (proxy) PO?

    Thanks a lot. Keep on writing!
    Vikram.

  5. Mike Dwyer March 17, 2012 at 3:05 pm #

    YOU ROCK!

    Always enjoy your insight and your thoughts, but most of all the style you communicate it in.

  6. Phil March 19, 2012 at 12:35 am #

    You mention cycle time, as one possible way to measure Agile. With Agile, formal requirements have changed to user stories, and the velocity of a user story, dictates cycle time. But I’m wondering what you think of in terms of the quality of the user story. The specificity and clarity of the user story, or other attributes of the user story (I assume from the product owner) would have a significant influence on the cycle time. What ideas or recommendations do you have for measuring user story quality as it impacts overall agile development effectiveness?

  7. testobsessed March 26, 2012 at 10:32 pm #

    Hi Phil,

    I don’t recommend measuring the story quality at all. First, a story is just a placeholder for a conversation. So the story-as-written will necessarily be incomplete. Second, the only measure that matters is whether or not the story is clear enough to accept into the queue of work to be done. If the story is vague, nebulous, or too big to fit into a sprint, it needs to be refined. If it’s clear, with clear acceptance criteria, and is small enough to complete within a single sprint, it can go into a sprint backlog. (Obviously, if you’re doing Kanban you won’t have the same notion of sprint, but the notion of not accepting a story into the backlog of work to do still applies.)

    Hope that helps…

    Elisabeth

  8. lokesh April 18, 2012 at 9:28 am #

    You mention cycle time, as one possible way to measure Agile. With Agile, formal requirements have changed to user stories, and the velocity of a user story, dictates cycle time. But I’m wondering what you think of in terms of the quality of the user story. The specificity and clarity of the user story, or other attributes of the user story (I assume from the product owner) would have a significant influence on the cycle time. What ideas or recommendations do you have for measuring user story quality as it impacts overall agile development effectiveness? I will use automation through vbscript. Please visit: EnterWebHub for more details.

Leave a Reply