Checking Alignment

Let’s start at the beginning. Somebody, somewhere, needs some software.

Maybe we’re serving an internal “customer” who needs a simple bailing-wire-and-duct-tape app to connect system A with completely unrelated (except that they need to be able to share data) system B. Or maybe we’re in a startup that’s trying to Change the World with a grand vision, or perhaps a modest vision, to give people software that makes their lives better.

Either way, we build software because there are people who need it. Let’s call what users need the Actual Need. We want to serve that Actual Need by building a truly kick butt solution.

On Agile teams we use user stories in an attempt to capture actual needs. For example:

As a Banking Customer I want to use my ATM card to withdraw money from an automated banking machine while I’m in Kiev so that I can buy a cup of fabulous local coffee with the local currency, Hryvnia.

This is way better than “The system shall accept a rectangular piece of plastic made to conform with standard …” It humanizes the problem space and puts the user at the forefront.

But user stories aren’t typically written by real users. They’re written by surrogates: Business Analysts or Product Managers or the like. These internal people go out and find the Actual Need and then set Intentions for what we need to build to meet those needs.

And then the Software Development team brings the intentions to life with the Implementation.

That’s software development in a nutshell from gathering requirements through deployment. It’s all about finding the happy place where we are addressing real needs with awesome solutions.

So here’s the big question:

How do we know that our Intentions matched the Actual Need, that the Implementation matched our Intentions, and ultimately that the Implementation matched the Actual Need?

Three Sides of Alignment

If software development projects exhibited mathematical properties then we could count on the relationship between each of these three things being symmetrical. That is, if A = B, and B = C, then mathematically speaking, A must also equal C.

But that doesn’t work with software.

We can set Intentions that, if implemented, would match the Actual Need. And we can communicate those Intentions effectively so that we end up with an Implementation that does what we intended it to do. But that does not mean that users won’t experience any problems with the delivered solution.

As an aside, this is fundamentally why waterfall does not work. Even if we could build the perfect requirements document that perfectly captured the actual needs of the business or our user base, there is no way to ensure that the resulting implementation will be a success. And by the time we release it’s way too late.

Back to my assertion that alignment is not symmetrical in software systems. Consider my story of trying to get Hryvnias in Kiev.

So there I was in Kiev with only a few Hryvnia in my pocket. I needed cash. So I took my trusty ATM card to an AutoBank machine. Actually, I walked by any number of AutoBank machines looking for one that had the right symbols on the front so I could be sure my card would work, and that was built into the wall of a bank so that it wasn’t a fake ATM machines designed to skim info. Yes, I can be paranoid sometimes. So anyway, I think I marched Sarah halfway across the city before I found one I would stick my card into.

Having finally found what I deemed to be a trustworthy ATM, I put in my card and entered my pin. And then I got a scary looking message: “You entered your PIN number incorrectly 3 times.” And the machine ate my card.

Here we see an example of how alignment is not symmetric. The Implementation of the ATM software no doubt matched the Intentions of those who built it. ATM machines are a mature and robust technology after all. And the Intentions addressed the Actual Need. Again, ATM machines are a known quantity at this point. But the Implementation spat in the face of my Actual Need. I was thwarted. Not only was my need for cash not met, but now my card was gone. And I had not entered my PIN number 3 times; I just entered it once. (My guess is that the real issue had to do with whether or not the ATM machine supported my card and not with the actual PIN number.)

But I digress. Back to the point.

We have Actual Need, Intentions, and Implementation. How do we know that all three of these things are in alignment?

The product owner can speculate that the Intentions accurately describe the Actual Need. If the product owner is hands off we often see development teams unilaterally asserting that the Implementation matches the Intentions. Worse, some product owners remain hands off because they want plausible deniability when things go wrong. That’s just…ew. Throughout all this, the business stakeholders can assume that by doing what we set out to do, the Implementation will meet the Actual Need and we’ll all be rich.

And we will be fooling ourselves. Such guesses and speculation allow us to become wrapped up in the illusion of progress.

If we want to know whether our Intentions are in alignment with the Actual Need, Steve Blank would say that we have to get out of our cubes and talk to potential users or customers.

If we want be sure our Implementation matches our Intentions, we have to state those Intentions concretely, with examples and explicit expectations. As long as we’re doing that we might as well go whole hog and do ATDD. It’s the best way I know to drive out ambiguity, clarify assumptions, and provide an automated safety net to alert us any time the Implementation strays from the Intentions. But automated checks aren’t enough. We also have to explore to discover risks and vulnerabilities that would jeopardize the spirit of the Intentions even if not the letter.

Finally, if we want to be sure our Implementation matches the Actual Need we have to watch customer behavior carefully. That means monitoring usage statistics, watching conversions, and generally listening to what the Lean Startup guys have to say on validated learnings.

All of these activities are aspects of testing. And while testers are still important, not everything that involves some aspect of testing should be done by people with QA or Test in their title.

Too often software teams take a narrow view of “testing.” They think (to paraphrase Stuart Taylor) that it’s about checking or breaking. They relegate it to the people with “QA” or “Test” in their title. Typically we only test whether the Implementation meets the Intentions. We speculate about the rest.

And then we’re surprised by failures in production, angry customers, and declining revenue.

The harsh fact is that empirical evidence trumps speculation. Every. Single. Time. And testing, real testing, is all about getting that empirical evidence. It’s not something testers can do alone. There are too many kinds of testing involved in ensuring that all three things are in alignment: Actual Need, Intentions, and Implementation.

And ultimately that’s why testing is a whole team responsibility.

9 Responses to Checking Alignment

  1. Laurent Bristiel October 27, 2011 at 9:09 am #

    Intersting read. Thanks.

    I have only one question : did you end up having your cup of fabulous local coffee ?

  2. Vin D'Amico October 27, 2011 at 1:47 pm #

    Yes, testing is a whole team activity AND the customer is part of the team. Of course, you may have tens, hundreds or even thousands of customers. A small number of them must be engaged early in the process. The team can progressively engage more of them using pilot releases, beta releases or whatever works. In the end, the customers’ opinions about the functionality and the quality of the software are the only opinions that matter.

    Thanks for sharing your thoughts. Like your blog!

  3. Nick Loadholtes October 27, 2011 at 4:37 pm #

    Regarding “empirical evidence”: Can you recommend a resource for what evidence should be collected/analyzed? I’m fully on board with the idea, but with a large system it can be hard to figure out where to start.

  4. testobsessed October 27, 2011 at 5:27 pm #

    Of course! Yum! (Heard there was a particular pastry that I should have gotten as well, but can’t remember the name. Next time!)

  5. testobsessed October 27, 2011 at 5:39 pm #

    I’m guessing you are talking about empirical evidence that tells us if we’re meeting the actual need. That’s the piece of the puzzle that is so often missing and that all too often causes the most churn and grief.

    In answer, I’ll refer you to the Lean Startup movement. In particular I appreciate Eric Ries’ perspective that the measure of progress is validated learning, not eyeballs or clicks or even revenue. Eric Ries calls Google Analytics “vanity metrics” because they make us feel good, but they don’t actually tell us anything real. For an alternative to vanity metrics, here’s a great piece from Ash Maurya on metrics and cohort tracking. And, of course, there’s nothing like near real time feedback from real users. I just saw this video today of Nordstrom’s Innovation Labs building an app in a week while camped out in the middle of the Nordstrom department store in Seattle, WA.

    Hope that helps…

  6. Michael Bolton October 30, 2011 at 3:05 pm #

    Perfect post, but for one thing: the title.

    Testing Alignment. As you point out, alignment needs to be tested (you’d probably say explored), not just checked.

    —Michael B., grinning, running, and ducking

  7. Michael Bolton October 30, 2011 at 3:20 pm #

    Okay, so now I’ll contribute something more valuable, I hope. (Maybe it will help provide an answer to Nick, above.)

    Several years ago, there was a problem at the organization where I worked. We had a huge volume of support calls from particular customers who were having trouble filling out an online form that requested bank account numbers in a certain format. The information to be entered could be obtained by looking at the cheque. The problem was experienced more (by a factor of dozens) by customers of two particular banks than any of the others. My own testing allowed me to come up with a number of theories about the nature of the problem. Those theories suggested that the problem should either not happen any more often than with other banks, or that none of the submissions from those banks should work at all.

    Fortunately, there was a branch of one of the banks within easy walking distance, so one day on my lunch hour I went to that bank and picked up a sample cheque. Sure enough, by looking, I could see how the cheque number was rendered in a format that would make the error happen in exactly the way it did. It turns out that there were a total of six little issues—and slx little refinements that we could make. Any one of those refinements could have lined up implementation with customer need, and reduced the problem to normal background-radiation levels for those two banks. The development manager told me that my report was the best testing work that he had ever seen. Empirical evidence for the win!

    —Michael B.

  8. Karlo Smid November 1, 2011 at 2:22 pm #

    I would like to explain the problem that I encountered in one project. In order to align customer actual needs with intentions and implementation you first has to know who is the customer. My project was run by the government, and it was started because government was pressured and financed by the European Union to cut the expenses in Ministry of Health.
    When project started nobody from the government side did not know what the system should do. They said just build the system according to your vision. When system went into production, and first issues were reported, we found out the real customer, it was person who reported the issues. It was technical coordinator (newly assigned and actually several of them) from Ministry of Health, and they had a vision what new system should do.

    Karlo Smid.

Trackbacks/Pingbacks

  1. Checking Alignment, Redux | Test Obsessed - October 30, 2011

    [...] guessing this is why I failed to explain myself well in my last post. Or at least I am inferring from the response to that last post that there is a gap between what I [...]

Leave a Reply