I Prefer This Over That

A couple weeks ago I tweeted:

Apparently it resonated. I think that’s more retweets than anything else original I’ve said on Twitter in my seven years on the platform. (SEVEN years? Holy snack-sized sound bytes! But I digress.)

@jonathandart said, “I would love to read a fleshed out version of that tweet.”

OK, here you go.

First, a little background. Since I worked on Cloud Foundry at Pivotal for a couple years, I’ve been living the DevOps life. My days were filled with zero-downtime deployments, monitoring, configuration as code, and a deep antipathy for snowflakes. We honed our practices around deployment checklists, incident response, and no-blame post mortems.

It is within that context that I came to appreciate these four simple statements.

Recovery over Perfection

Something will go wrong. Software might behave differently with real production data or traffic than you could possibly have imagined. AWS could have an outage. Humans, being fallible, might publish secret credentials in public places. A new security vulnerability may come to light (oh hai, Heartbleed).

If we aim for perfection, we’ll be too afraid to deploy. We’ll delay deploying while we attempt to test all the things (and fail anyway because ‘all the things’ is an infinite set). Lowering the frequency with which we deploy in order to attempt perfection will ironically increase the odds of failure: we’ll have fewer turns of the crank and thus fewer opportunities to learn, so we’ll be even farther from perfect.

Perfect is indeed the enemy of good. Striving for perfection creates brittle systems.

So rather than strive for perfection, I prefer to have a Plan B. What happens if the deployment fails? Make sure we can roll back. What happens if the software exhibits bad behavior? Make sure we can update it quickly.

Predictability over Commitment

Surely you have seen at least one case where estimates were interpreted as a commitment, and a team was then pressured to deliver a fixed scope in fixed time.

Some even think such commitments light a fire under the team. They give everyone something to strive for.

It’s a trap.

Any interesting, innovative, and even slightly complex development effort will encounter unforeseen obstacles. Surprises will crop up that affect our ability to deliver. If those surprises threaten our ability to meet our commitments, we have to make painful tradeoffs: Do we live up to our commitment and sacrifice something else, like quality? Or do we break our commitment? The very notion of commitment means we probably take the tradeoff. We made a commitment, after all. Broken commitments are a sign of failure.

Commitment thus trumps sustainability. It leads to mounting technical debt. Some number of years later find themselves constantly firefighting and unable to make any progress.

The real problem with commitments is that they suggest that achieving a given goal is more important than positioning ourselves for ongoing success. It is not enough to deliver on this one thing. With each delivery, we need to improve our position to deliver in the future.

So rather than committing in the face of the unknown, I prefer to use historical information and systems that create visibility to predict outcomes. That means having a backlog that represents a single stream of work, and using velocity to enable us to predict when a given story will land. When we’re surprised by the need for additional work, we put that work in the backlog and see the implications. If we don’t like the result, we make an explicit decision to tradeoff scope and time instead of cutting corners to make a commitment.

Aiming for predictability instead of commitment allows us to adapt when we discover that our assumptions were not realistic. There is no failure, there is only learning.

Safety Nets over Change Control

If you want to prevent a given set of changes from breaking your system, you can either put in place practices to tightly control the nature of the changes, or you can make it safer to change things.

Controlling the changes typically means having mechanisms to accept or reject proposed changes: change control boards, review cycles, quality gates.

Such systems may be intended to mitigate risk, but they do so by making change more expensive. The people making changes have to navigate through the labyrinth of these control systems to deliver their work. More expensive change means less change means less risk. Unless the real risk to your business is a slogging pace of innovation in a rapidly changing market.

Thus rather than building up control systems that prevent change, I’d rather find ways to make change safe. One way is to ensure recoverability. Recovery over perfection, after all.

Fast feedback cycles make change safe too. So instead of a review board, I’d rather have CI to tell us when the system is violating expectations. And instead of a laborious code review process, I’d rather have a pair work with me in real time.

If you want to keep the status quo, change control is fine. But if you want to go fast, find ways to make change cheap and safe.

Collaboration over Handoffs

In traditional processes there are typically a variety of points where one group hands off work to another. Developers hand off to other developers, to QA for test, to Release Engineering to deliver, or to Ops to deploy. Such handoffs typically involve checklists and documentation.

But the written word cannot convey the richness of a conversation. Things will be missed. And then there will be a back and forth.

“You didn’t document foo.”
“Yes, we did. See section 3.5.1.”
“I read that. It doesn’t give me the information I need.”

The next thing you know it’s been 3 weeks and the project is stalled.

We imagine a proper handoff to be an efficient use of everyone’s time, but they’re risky. Too much can go wrong, and when it does progress stops.

Instead of throwing a set of deliverables at the next team down the line, bring people together. Embed testers in the development team. Have members of the development team rotate through Ops to help with deployment and operation for a period of time. It actually takes less time to work together than it does to create sufficient documentation to achieve a perfect handoff.

True Responsiveness over the Illusion of Control

Ultimately all these statements are about creating responsive systems.

When we design processes that attempt to corral reality into a neat little box, we set ourselves up for failure. Such systems are brittle. We may feel in control, but it’s an illusion. The real world is not constrained by our imagined boundaries. There are surprises just around the corner.

We can’t control the surprises. But we can be ready for them.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

15 Responses to I Prefer This Over That

  1. Kevin O'Shaughnessy May 17, 2015 at 7:45 pm #

    Very well done. This is the greatest small set of value statements that I’ve seen since February 2001.

  2. Aliaksei May 17, 2015 at 11:22 pm #

    “True Responsiveness over the Illusion of Control” is vague. But the other 4 are truly perfect!

  3. Kristian Flikka May 18, 2015 at 8:04 am #

    Very relevant, timely and well articulated – thank you!

  4. Allison May 19, 2015 at 2:20 am #

    Nice list, and I like the theme of “true responsiveness over illusion of control.”

  5. Venkatesh Kiran(Software Test Consultant/Trainer May 21, 2015 at 9:34 am #

    All points are true to the practical level. But how many so-called Agile’s practice these. If incorporated, projects will reap the benefits. Ah! I really like the “idea” of ownership but it should also be in collaborative-mode, else it falls into pseudo-agility. I have seen quite a few agile projects fail (few scraped too!). thanks for the nice post Ms. Hendrickson.

  6. Jacek Gebal May 26, 2015 at 9:05 pm #

    Wow. I’m truly amazed by how nicely you have put all those things to the table.
    Lots of organizations seem to struggle with the uncertainty and risks around delivery.
    The larger the organization, the more they struggle.
    It is really good to observe in large organizations turning from waterfall towards so-called agile, where they actually are really afraid of replacing the procedural, sign-off approach with responsible development life-cycle.
    This is another manifesto for me, that i would be ready to sign under.
    Too often times, developers/QA’s/DevOps are forced to deliver documents than a real-value product.
    We study for years and educate, to be inventive and productive, all the repetitive and predictive work can and should be automated.

  7. Paul Gibler May 29, 2015 at 1:48 pm #

    “Some even think such commitments light a fire under the team. They give everyone something to strive for.

    It’s a trap.”

    Someday more people will understand what this means.

    Very good article.

  8. Geoff Hummel May 29, 2015 at 7:29 pm #

    Some very good points. One thing you missed that needs to be part of the decision making in each of your value statements is assessing Risk (as alluded to by Jacek).

    For example: “Recovery over Perfection.” As we all know no software can be perfect without excessive cost. Deciding where to focus the testing and what issues to fix vs. not-fix needs to be based upon the risk of failure that that issue may cause, i.e. perform an assessment of the probability of the failure and the severity of the effect. Base the assessment upon the expected value of the feature, impacted by the issue, to the company developing the software and the users of the software. Yes – being able to quickly recover from a failure is critical, however your company can get more bang-for-a-buck by focusing on reducing the likelihood of high-cost (risks) failures.

    Apply the same thinking to the other good points you’ve made.

  9. Thomas Ponnet June 1, 2015 at 4:55 am #

    Thank you for this very good and thoughtful explanations that more people should read and understand. I’d like to add that most current practices that violate your approach or preferences are politically motivated.
    At the heart there is often a contractual obligation where even the illusion of control is seen as being better (“See, we did something to reduce the risk for the client”). Planning for the event of a failure or breaking commitment, which may be sensible but is often avoided, due to a fear of lawyers that may sue without (the illusion of) control mechanisms in place. PMs become the messenger of this fear driven approach.
    Collaboration vs handoffs is tricky – I’m with you here but would like to note that it’s sometimes difficult to spread the knowledge to all team members. If you pair a tester with the developer that’s great but the technical writer, deployment, support team, etc need to get critical information as well, regardless if they sit in the same office or around the globe. A context specific solution needs to be found for each company / project which makes this part so hard.

  10. Adam Yuret June 4, 2015 at 2:53 pm #

    Unsurprisingly, I agree with everything here. Great stuff Elisabeth. 🙂

  11. Jeff Sussna June 4, 2015 at 3:02 pm #

    I like these principles very much, with the exception of “Predictability over Commitment”. I have two questions about it:

    1. What is the value of predictability without commitment? If you can reasonably predict an outcome, why not commit to it? If you’re not comfortable committing to it, have you really predicted it?

    2. The above assumes predictibility is in fact achievable. That assumes that history is predictive. In situations of complexity, that assumption is questionable. To me, this is the crux of the problem with things like estimates and velocity. They presume the future is like the past. The very reason we need the other principles is that we’re likely building something where we can’t rely on that presumption holding.

  12. Mike Talks July 23, 2015 at 1:30 am #

    Wow Elisabeth – nice to see a new IT-focused entry on here, and full of interesting ideas.

    I know that “Recovery over Perfection” really hits a painful nerve there, and for me stands out (from bad releases in the long-distant past).

  13. Wilson Mar January 29, 2016 at 4:50 am #

    This is the single wisest tweet I’ve ever read, Elisabeth.
    It captures the gist of the transformation in what we call “Agile”.

    Seriously, I’m going to print it out and put it over my cat picture.

    In fact, I’m going to put in on my coffee cup.

    BTW, it was a real treat to have chatted with you at the Agile conf. Nashville a couple of years ago. Hope we can talk in person again soon.

    / Wilson

  14. Jacek Gebal February 20, 2016 at 10:03 am #

    Hi Elizabeth,
    Would you mind me re-posting that blog entry on company intranet?
    I would surely refer the author and original source.

    Thank you
    Jacek

  15. testobsessed February 21, 2016 at 6:40 pm #

    My preference is generally to have just an abstract and link instead of republishing.

    However if the website is truly internal (that is, not available on the general public internet) and it is not possible to just publish a link to the original, then it’s OK with me if you republish it internally with attribution.

    Thanks for asking!

    Elisabeth

Leave a Reply