When a Team Does Not Want to Do the Work that Needs to Be Done

Continuing on the theme of motivation…

Sometimes the work that most needs to be done is not the work a team wants to do. Let’s eavesdrop on a conversation between two leaders in an organization that ships enterprise software for an example of one such situation:

“Have you seen the number of support cases coming in for release <x.y.z>?”
“Yeah, it’s bad.”
“Is the server team working on the bugs?”
“No. They’re working on the next major release, <(x+1).0.0>”
“But…that release won’t come out for another year. Even if we could get it out tomorrow, the customers who are experiencing these failures are not going to do a major upgrade to address bugs.”
“I know.”
“So…why isn’t the team tackling the bugs?”
“They want to work on the new features.”

There are several organizational issues around alignment and prioritization evident in this single conversation, but let’s look at the biggest problem that’s front and center: there are unhappy customers waiting on bug fixes, but the team wants to work on the new features in the next major release.

As a leader, how would you get the team focused on the bug fixes?

There are several unfortunately common approaches that I don’t recommend because they won’t work at all, or won’t work well, or won’t work over the long term. In yesterday’s post I talked about how creating a false sense of urgency doesn’t work. As we will see in what happened next, other approaches that don’t work include ignoring the problem and hoping it will go away on its own, and attempting to coerce the team.

The conversation above happened because the leaders closest to the team were hoping the team would start fixing the bugs on their own, without direction or intervention, and they didn’t.

To be fair: if the organization had a high degree of visibility across the board where everyone is working from the same information, and a high degree of alignment such that everyone has a shared sense of purpose, it’s certainly possible that the team would naturally gravitate to the work that leadership also agreed was the highest priority. (Not guaranteed, but possible.) Alas, at the time this conversation took place, this particular organization had neither shared visibility or alignment. So it’s not surprising that the team did not magically turn their attention to the bugs.

I was a relatively new leader in the organization, and I knew I had to push the issue. So I did.

It’s worth noting that the organization had a history of leaders who waffled between avoiding conflict and inciting major blow ups. Leadership interventions often resulted in major drama. The leaders involved in the situation so far were hesitant to wade in because things were not yet dire enough to warrant a knock-down-drag-out fight.

When I pushed the issue, an engineering leader I’ll call B (not his real name or initial) called a meeting with the team and told them to “do it or else.” The team members—all quite senior and all with deep specialized knowledge—responded that they did not appreciate B’s stance, and that they would be brushing up their resumes. In short, B gave an ultimatum, and the team called his bluff. And I gained insight into why leadership interventions often resulted in drama.

It was time to change the approach. Here’s what we did next.

Step 1: Have the “Here’s the Problem We Need to Solve” Conversation

By the time I became directly involved in the situation, tempers were running high on all fronts. I asked for a meeting with the team, and started with an apology for how things had been handled to date.

Then I said: “Here’s the problem we need to solve: we have multiple customers running version x.y.z who are experiencing serious problems. It’s not a crisis yet. But we can’t wait for version (x+1).0.0 to do something.”

And then I sat back and listened.

Sensing that I wasn’t there to issue an ultimatum, the programmers opened up, and I learned two critically important things:

1) The programmers admitted that working through the bugs would be a painful slog and they didn’t want to do it.

2) They didn’t think it was that important because the new architecture they were working on would eliminate the source of most of the bugs. They genuinely believed that the highest value work they could do would be deliver the new release as soon as possible.

When I further explained that even if we could deliver the new release the next day, customers would still demand that we patch the previous release, the team saw the flaw in their logic. The new architecture was incompatible with the older code base. No matter what we did, we were going to have to address the bugs in the older, cruftier, more error prone code base.

The thing about the “here’s the problem we need to solve” conversation is that it’s an actual discussion. I wasn’t talking to the team, I was talking with them. I gave them a piece of the context they didn’t have; they gave me a piece of the context I didn’t have.

Together we came to the conclusion that we had to do something in the current release, and not wait for the new architecture.

Step 2: Create Momentum

Although every programmer who worked on that product had access to the bug reports coming in from customers, they usually only saw the issues one at a time. They rarely looked at the entire support backlog to see the magnitude of the pile of open issues or the patterns. 

On the rare occasion when a programmer did look at the large pile of customer reported bugs, it seemed like an overwhelming pile of unsolved problems across all areas of the product (the server being just one). Any given programmer wading into the pile would find themselves up to their eyeballs in issues they didn’t know anything about. Anyone who ventured into the big pile ended up throwing up their hands in despair before finding something else to do where they could feel productive.

We were in a self-reinforcing cycle: the pile of open issues felt insurmountable, so no one wanted to tackle it, so it grew larger, and the larger it grew the more insurmountable it felt. We had to break the cycle.

The solution wasn’t just to get people access to the data; they had that. Rather, we needed to create visibility that would support a sense of momentum.

We started tracking open customer issues by team, and by severity. We set a goal of driving down the open highest severity customer issues for each team down to 0. Product leadership partnered with support to make sure each team knew about the support cases that were relevant to their part of the code base. Individual product managers prioritized the most critical bug fixes and made sure they were delivered in maintenance releases. Every week the numbers went down. Eventually the effort took on a life of its own. The pile felt surmountable; over time we conquered it.

Note that this is not a story about how awesome metrics are for driving behavior in an organization. That would be a dangerous lesson to take from this story. Metrics are a double-edged sword. If you manage people by numbers, you get number-based compliance. High severity issues end up reclassified as “medium” with little to no justification. People squabble over whether or not something is a bug just because they want their numbers to look better.

When we started sharing a dashboard of bug stats, we made sure that no one was rewarded for achieving a low number, or penalized for having a high number. We celebrated together when the numbers went down, but no one’s performance assessment was tied to the numbers. Further, the numbers were just a proxy for the list of issues. Although we all looked at the same dashboard of numbers, the real information was in the list of issues rather than the count of how many issues. Week after week, seeing the list get smaller felt much better than just seeing a number tick down.

It’s All About Transparency and Respect

As a leader, your job is to inspire people to do their best, working toward a common objective. You could try “rah rah” cheerleading, or coercion, but neither approach will forge the same kind of relationship or results as creating a culture of transparency based on mutual respect.

Start by assuming that the people you are leading are smart and well-intentioned. They will do the very best they can under the circumstances to deliver the highest value they’re able.

When you need to intervene to change course, assume it’s not because people aren’t doing their job. Instead, assume that your team does not have something they need, or that they see the situation differently from you. Then have an open conversation. Start with “here’s the problem we need to solve.” Make sure you are aligned on the problem statement. Then work together, sharing information and perspectives, to determine how to move forward.

There may come a time when no matter how much information you share, the team still disagrees about the best path forward. If that happens, it’s OK to say: “I’m going to need you to trust me on this one. We need to go this way…” 

That moment is the ultimate test of leadership: if you lead, will they follow?

By having treated the team with respect, not asking for blind obedience but rather including them at the table when making a decision, you are very likely to have earned their respect, and they are much more likely to follow you when you most need them to trust your judgment.

Momentum > Urgency

Let’s talk about motivation. More specifically let’s talk about what leadership can do to help motivate a team.

Once upon a time, a director of product management and I were butting heads. Let’s call him J (not his real name, or even his real initial).

J was unhappy with the pace at which the developers were delivering. His usual approach to hastening development was to set arbitrary deadlines to create a sense of urgency.

It wasn’t working.

Despite J’s best efforts at inspiring the developers to work harder and faster, progress was moving too slow for his liking.

I was in an engineering leadership role, so J tried pushing on me to push harder on the engineers. Things came to a head when J said to me, in a tone of utter exasperation, “Come on! You know programmers. You have to light a fire under them to get anything done!”

Well, no. I don’t know that. In fact, I don’t think that’s true at all.

Granted, pressure can expedite delivery. However it comes at a cost, and it’s a cost that’s difficult to measure. Pressure and urgency typically lead people to cut corners. That, in turn, can lead to an illusion of speed that comes crashing down later with delays or poor results. But it might not. It could take months or years for problems to arise out of those cut corners. Further, people who operate under pressure for too long burn out. But burnout doesn’t happen right away.

So from J’s point of view, when he was unhappy with the productivity of a team in the past, he ratcheted up the pressure and got the result he wanted. If there was a downside to his approach, he probably never saw it. He was too far away from the work and from the people doing it.

While I don’t know if J left a trail of destruction in his high-pressure-wake before, I have seen the mess that others who used similar tactics to J left behind. In fact, I’ve spent a good part of my career cleaning up after those messes.

What I’ve learned is that if we want things to go fast, a sense of momentum is much more effective than a sense of urgency.

Consider another project with a very different team dynamic.

In 2008 or so, I was a programmer on the BringLight development team at Pivotal Labs. Drew McManus was both a company founder and our product manager.

We used Pivotal Tracker to manage our work, so we broke the work into bite-sized chunks, tracked as stories. Drew prioritized the stories. When we delivered the work, Drew tried out the new capabilities, and accepted or rejected the story within minutes. Even when he was out of the office for meetings, he would check the tracking tool frequently to see if there was anything for him to accept.

Drew’s responsiveness had a palpable effect on the whole team. We became a delivery machine.

If we were close to delivering a story and had a few minutes before lunch or the end of the day, we’d push through and deliver the story. We knew it made a difference. More often than not, Drew would have done his acceptance before we got back to the keyboard. There were times Drew accepted a story while sitting in a coffee shop, then demoed that new story to a VC just minutes later.

Each acceptance fueled the next delivery. We never rushed. We never cut corners. (Drew was just as good at rejecting stories as accepting them.) Instead, we worked with a sense of focus and purpose. It also helped that Drew had a strong overarching vision of what we were building, so as we broke down stories to bite-sized pieces, we could still connect them to the larger vision.

Drew and I are still in touch, so I know how the story ended. He and his partners eventually sold the company. Before that, the code we wrote ran in production for years with no substantial defects. After go-live Drew hired a solo part time developer to handle routine ongoing maintenance.

BringLight is not the only time I’ve seen a team benefit from a sense of momentum, it just happens to be one of my favorite stories.

Every time I’ve seen a team achieve a sense of momentum, several things are true:

  • Everyone on the team has a shared understanding of the big picture and what “good” looks like so they don’t burn cycles on unnecessary or non-useful work.
  • The team breaks the work into small pieces (a couple days of work at most) that still represent incremental business value (even if not user-facing features).
  • The team embraces the technical practices necessary to ensure that the code they deliver does what they intended it to do.
  • There’s an engaged and responsive product manager* who is considered part of the team, and who does hands-on acceptance of the work.
  • When the work is “accepted,” it’s done. The entire team can stop thinking about it.
  • The work, and the status of the work, is visible to everyone.
  • The team as a whole has a strong sense of partnership and trust, so the visibility never becomes a mechanism for micromanagement.

There is nothing on that list that’s easy, but it’s worth the effort. Creating a sense of momentum remains the very best way I know to enable long term sustainable delivery. Even improving along a subset of the dimensions in the list can yield huge benefits.

(Oh, and if you’re waiting to hear the end of the story about J, I’m not going to go into details. Suffice it to say that J and I never did figure out how to partner effectively and we didn’t work together for too much longer. In the fullness of time, some time after J left, the team he was trying to pressure did eventually improve their ability to deliver…by developing a sense of momentum.)

* In some cases the person doing acceptance may not have a product manager title or report into the product organization. They might be an engineering manager or a tester. The key thing isn’t the title of the person doing acceptance, but rather the extent to which “accepted” means “done.” If someone in QA accepts a story, but later the team is forced to redo the work because someone in a position of authority had a different vision and is dissatisfied, “accepted” stops meaning anything.

Welcome to my Happy Place

When I tell friends about my new obsession with my Oculus Quest, the first thing they usually ask is what apps I use. It’s a little hard to explain that one of my favorite apps is the home screen.

Seriously. Let me explain.

When you enter the Oculus Quest you find yourself in a geodesic dome. Being VR, it’s a fully immersive experience. Every direction you turn there’s something to look at. Overall, the space is lovely. Wood floors, cozy fire, mid-century modern furniture, and a gorgeous view. What’s not to like?

Cozy fireplace on the Oculus Quest home screen
Desk (with an appropriate amount of shelf space)
Chair arrangement with coffee table perfect for a conversation or game

(OK, the absence of any doors in or out is a little creepy if you think about it too hard, but let’s not.)

Of course the first thing I wanted to do when I landed in the home screen was to walk around and maybe hang out on the furniture. Small problem. The furniture isn’t real (obviously). Also, you can’t just wander freely: you are constrained to the space inside your “guardian.”

Let me backup and explain how the “guardian” system works. When you start using the Quest, it prompts you to choose either a stationary or room scale experience. If you choose stationary, it sets up a circular boundary around you. When using the device, if you reach your hands or move your head close to the boundary, a grid (reminiscent of the Star Trek holodeck grid) appears. If you move your head outside the boundary it stops showing you the virtual world and shows you a grainy black and white image of the real world.

When you choose a roomscale experience, the guardian border works the same, but you use the hand controller to “draw” your own boundary. You need about 6ft x 6ft minimum, but you can go much bigger if you have more clear space. Here you can see the outline of my guardian, showing where I can go in the dome.

The guardian shows where I can safely walk within the virtual space

The geodesic dome is probably about 40ft across, give or take. So you would need a lot of space if you wanted to roam freely. I haven’t tried to test the limits, so the device might not even let you draw a guardian that big. But unless you happen to have an empty warehouse available to you, it’s a purely hypothetical question.

As it turns out, I do happen to have a very long living room. Not 40″ but I can get a guardian that’s about 8ft x 18ft. And after some experimentation, I discovered that I could orient my view with respect to the guardian in such a way that I actually could get close to the furniture. With a little bit more work, I can orient the view so that the virtual couch is in the same location as my real couch.

Let’s go over to the fire, shall we?

Here’s what the view looks like from the couch.

What a fabulous place to just sit

So why do I like the home screen so much? Because with a little finagling to get everything lined up Just So, I get to sit on my actual (comfortable) couch next to a cozy animated fire, and just be. Not check my cell phone or my email or my twitter, not be distracted by a mess I should be cleaning up or an undone To Do. Just exist. Enjoy the present moment. Quiet my busy mind.

Sometimes Ellie, our 100lb rescue pup, will see me on the couch and decide to join me. So I’ll be staring at a virtual fire scritching my real-but-invisible-to-me dog. Sitting there for 15 minutes, petting Ellie, enjoying the environment, and thinking about nothing in particular leaves me in a good mood for the rest of the day.

But wait, there’s more.

After a little more experimentation I discovered that I could actually walk outside the dome. Below is a view from just behind the couch.

It’s a lovely evening in Oculus Land. Let’s enjoy the view!

Notice the details in the art. The back of the couch is a wooden frame. Someone (or several someones) put a lot of care into every single detail in the home screen. Even parts of the scene that there is no reason to think a user would ever see are carefully crafted.

The one thing you can’t do is interact with any of the objects in the home screen. So when I want to do something more active than admire the view, I reach for one of my favorite apps. More on those in another post.

Comments are still off for now.

My New Hobby: VR

In October last year I visited my therapist for a regularly scheduled check in. Things had been a little stressful, so we talked about ways to relax.

“How do you quiet your busy mind?” she asked.

It’s a great question, and it’s not the first time she asked it. A few months prior when she asked it, my answers prompted her to recommend a meditation device that she had found some of her patients responded to well. So I’d borrowed a device from her to try out, and ultimately bought my own. That had worked very well for a while because it gave me the real time feedback I needed to stay in the zone and quiet my busy mind instead of just giving it more space to wind itself up.

I waved my hands. “Oh, you know, the usual,” I said. “Meditation. Walking.”

She nodded.

“But they’re not working particularly well at the moment,” I added.

Given that my therapist turned me onto the Muse device, I expected her to offer some sage advice about how to improve my meditation practice with it. Instead, she nodded again, and then asked a question I never thought I would hear come out of a therapist’s mouth: “Have you tried VR?”

Have I tried what now?

At the time, my impression of virtual reality was that it was yet another gaming platform, and relatively nascent. My incredibly limited personal experience with VR involved a cardboard frame, a mobile device, and a video of a roller coaster that I could not see very well. It was…underwhelming.

I knew there were commercial VR rooms cropping up with immersive higher fidelity experiences that I was certain would be far better than the home experience, but as far as I knew they were all in the shoot-at-things genre. I’m not into first person shooters: I don’t like violence and don’t have the reflexes for real-time games requiring lightning fast response times. I’d tried laser tag once back in the day and hated it, so I did not find the idea of a more immersive virtual war game experience even remotely appealing.

Further, in my personal life I tend to be a technology laggard. While I own the usual complement of electronic devices for someone in tech (two smart phones and a tablet plus a couple computers I use actively and a small stash of ancient laptops I need to clear off and recycle Some Day), that’s where the tech stops for me in my personal life. I want my home to be dumb. You are not going to find a Ring by our door or an Alexa in our living room, nor do I want connected light bulbs controlled by an app.

So VR? Yeah. Totally not my thing, I thought. 

Also not at all something I expected a therapist to suggest might help with mental health.

I explained all that to my therapist, and she shook her head. “It’s not what you think it is,” she said. And then she told me about studies in using VR to reduce chronic pain and thus reduce the need for pain medications. She told me about applications for play therapy for kids (a particular area of interest for her). And she told me that in general VR is such a hot field in medicine there’s a whole conference dedicated to it.

“Try swimming with dolphins,” she said. “I guarantee you that you’ll feel like you just took a 2 week vacation after 15 minutes.”

So I bought myself an Oculus Quest at her suggestion. And as I tweeted shortly after:

In my early days with the technology I was spending about an hour a day immersed in other realities. My therapist was right. A 15 minute session in the morning left me with the same kind of relaxed and ready-for-almost-anything feeling as a multi-day vacation.

So as I indicated in my previous post, now that I have some time on my hands I’m starting to experiment with creating content for VR. But those experiments, along with a list of the apps I’ve been enjoying, will have to be a subject for another post.

(Comments are still off for now.)

Hello, 2020!

It’s the start of a new year, and for me it’s the start of a new adventure. As I posted on Twitter:

In case you didn’t know, the VMware / Pivotal merger closed on December 30. The merger makes a world of sense, and I am genuinely excited for my Pivotal colleagues as they embark on a new journey with Tanzu.

I will forever be grateful for my time at Pivotal. I worked with amazing people across most of Pivotal’s product areas (Cloud Foundry, Greenplum, Apache Geode / GemFire, and PKS). Along the way, I learned and grew so much. However this post isn’t about my time at Pivotal, it’s about what comes next.

As we start the next decade[1] and I come up for air after 7 years of not getting out much, I wanted to take some time to explore before diving headlong into something new. Although I truly do not know yet where my adventures in 2020 will take me, there are a few things I do know.

  • There will be naps. I wasn’t kidding about that. I have napping places set up all around my house. I’ve missed naps.
  • I’ll be getting out a little bit more than I have been in the last several years. I still won’t be on the conference circuit per se, but I’ll definitely be at Agile Testing Days in Chicago in June. I’m really looking forward to it. ATD is one of the finest conferences around. They always have great a great speaker lineup and put so much heart into creating an inclusive and fun atmosphere.
  • I’ll be experimenting with making content for VR.

On that last point…

If you follow me on Twitter you’ve probably seen my tweets about my Oculus Quest. VR is my new hobby. I’ve been tremendously enjoying consuming content for VR (apps and videos). Now that I have some time, I want to try my hand at creating content. For now, it’s just for fun, and nothing I would want to share. However, enough people have asked me about what I’m doing with VR, what apps I use, etc. that I figure this blog is a good place to share. So watch this space…

[1] Thank you @11rcombs for pointing out ISO 8601 section 4.11. Zero-based numbering for the win.

(Note: I’ve disabled comments on this blog because I just don’t have the energy to moderate them. Long term, I don’t know if I’ll be turning comments back on, or just engaging with folks in other ways. For now if you want to engage with me about this post please reach out on Twitter.)

Checking In

Well, so this is awkward. I kind of disappeared. I’ve been focusing all my time, energy, and attention on my day job at Pivotal. It’s been fun! And wild! We went public and I got to be part of the team that stood on the podium at the New York Stock Exchange! A bucket list item I didn’t even know I had! Along the way I’ve gotten to work with some amazing folks on super interesting products.

I’m just popping back in now to update my wordpress theme. Some very kind folks let me know that my site was kinda borked and that made them sad. And here I am at SFO with time on my hands because of a 2 hour flight delay. It was this or try to plow through another hundred messages in my embarrassingly and perpetually overflowing inbox. And oh, the yaks I had to shave to get to this point, with a new and hopefully unborked theme! Three password recovery workflows. The discovery that my old theme is no longer being maintained, apparently. Searching for new themes. Getting overwhelmed. Settling on the default theme that comes with any wordpress installation. It was a whole herd of yaks! My flight will be boarding before I know it.

As long as I’m here, I’ll confess that I don’t quite know what to do with this blog. Apparently it’s useful at least to some folks who take the time to notify me when things are broken. For that matter, it’s been useful to me to have a public repository of things I’ve written: I find myself foraging among my old writings periodically and sending along links to folks instead of writing blog length emails. But I don’t even look at comments anymore. (There are 571 in moderation. I’ll bet 568 of them are spam. So yeah. Not gonna look. Really sorry, other three people!) And I haven’t been writing anything I’d want to post. So it’s a static repository. WordPress is totally overkill, but migrating it to anything else would require more time and energy than I have right now.

So here we are. A weak update just to say “yo, maybe the css will work in chrome now?”

Maybe now that I’ve recovered all the passwords and found my admin page I might post again? We’ll see. In the meantime, please do continue to let me know if you experience problems with the blog. Thanks!

I Prefer This Over That

A couple weeks ago I tweeted:

Apparently it resonated. I think that’s more retweets than anything else original I’ve said on Twitter in my seven years on the platform. (SEVEN years? Holy snack-sized sound bytes! But I digress.)

@jonathandart said, “I would love to read a fleshed out version of that tweet.”

OK, here you go.

First, a little background. Since I worked on Cloud Foundry at Pivotal for a couple years, I’ve been living the DevOps life. My days were filled with zero-downtime deployments, monitoring, configuration as code, and a deep antipathy for snowflakes. We honed our practices around deployment checklists, incident response, and no-blame post mortems.

It is within that context that I came to appreciate these four simple statements.

Recovery over Perfection

Something will go wrong. Software might behave differently with real production data or traffic than you could possibly have imagined. AWS could have an outage. Humans, being fallible, might publish secret credentials in public places. A new security vulnerability may come to light (oh hai, Heartbleed).

If we aim for perfection, we’ll be too afraid to deploy. We’ll delay deploying while we attempt to test all the things (and fail anyway because ‘all the things’ is an infinite set). Lowering the frequency with which we deploy in order to attempt perfection will ironically increase the odds of failure: we’ll have fewer turns of the crank and thus fewer opportunities to learn, so we’ll be even farther from perfect.

Perfect is indeed the enemy of good. Striving for perfection creates brittle systems.

So rather than strive for perfection, I prefer to have a Plan B. What happens if the deployment fails? Make sure we can roll back. What happens if the software exhibits bad behavior? Make sure we can update it quickly.

Predictability over Commitment

Surely you have seen at least one case where estimates were interpreted as a commitment, and a team was then pressured to deliver a fixed scope in fixed time.

Some even think such commitments light a fire under the team. They give everyone something to strive for.

It’s a trap.

Any interesting, innovative, and even slightly complex development effort will encounter unforeseen obstacles. Surprises will crop up that affect our ability to deliver. If those surprises threaten our ability to meet our commitments, we have to make painful tradeoffs: Do we live up to our commitment and sacrifice something else, like quality? Or do we break our commitment? The very notion of commitment means we probably take the tradeoff. We made a commitment, after all. Broken commitments are a sign of failure.

Commitment thus trumps sustainability. It leads to mounting technical debt. Some number of years later find themselves constantly firefighting and unable to make any progress.

The real problem with commitments is that they suggest that achieving a given goal is more important than positioning ourselves for ongoing success. It is not enough to deliver on this one thing. With each delivery, we need to improve our position to deliver in the future.

So rather than committing in the face of the unknown, I prefer to use historical information and systems that create visibility to predict outcomes. That means having a backlog that represents a single stream of work, and using velocity to enable us to predict when a given story will land. When we’re surprised by the need for additional work, we put that work in the backlog and see the implications. If we don’t like the result, we make an explicit decision to tradeoff scope and time instead of cutting corners to make a commitment.

Aiming for predictability instead of commitment allows us to adapt when we discover that our assumptions were not realistic. There is no failure, there is only learning.

Safety Nets over Change Control

If you want to prevent a given set of changes from breaking your system, you can either put in place practices to tightly control the nature of the changes, or you can make it safer to change things.

Controlling the changes typically means having mechanisms to accept or reject proposed changes: change control boards, review cycles, quality gates.

Such systems may be intended to mitigate risk, but they do so by making change more expensive. The people making changes have to navigate through the labyrinth of these control systems to deliver their work. More expensive change means less change means less risk. Unless the real risk to your business is a slogging pace of innovation in a rapidly changing market.

Thus rather than building up control systems that prevent change, I’d rather find ways to make change safe. One way is to ensure recoverability. Recovery over perfection, after all.

Fast feedback cycles make change safe too. So instead of a review board, I’d rather have CI to tell us when the system is violating expectations. And instead of a laborious code review process, I’d rather have a pair work with me in real time.

If you want to keep the status quo, change control is fine. But if you want to go fast, find ways to make change cheap and safe.

Collaboration over Handoffs

In traditional processes there are typically a variety of points where one group hands off work to another. Developers hand off to other developers, to QA for test, to Release Engineering to deliver, or to Ops to deploy. Such handoffs typically involve checklists and documentation.

But the written word cannot convey the richness of a conversation. Things will be missed. And then there will be a back and forth.

“You didn’t document foo.”
“Yes, we did. See section 3.5.1.”
“I read that. It doesn’t give me the information I need.”

The next thing you know it’s been 3 weeks and the project is stalled.

We imagine a proper handoff to be an efficient use of everyone’s time, but they’re risky. Too much can go wrong, and when it does progress stops.

Instead of throwing a set of deliverables at the next team down the line, bring people together. Embed testers in the development team. Have members of the development team rotate through Ops to help with deployment and operation for a period of time. It actually takes less time to work together than it does to create sufficient documentation to achieve a perfect handoff.

True Responsiveness over the Illusion of Control

Ultimately all these statements are about creating responsive systems.

When we design processes that attempt to corral reality into a neat little box, we set ourselves up for failure. Such systems are brittle. We may feel in control, but it’s an illusion. The real world is not constrained by our imagined boundaries. There are surprises just around the corner.

We can’t control the surprises. But we can be ready for them.

Where Have I Been?

Oh, hai internets. It’s been a while. Did you miss me?

Let me tell you what I’ve been up to.

In the fall of 2012 I shut down my consulting practice (Quality Tree Software) and my studio (Agilistry), and took a job with Pivotal. Actually, to be precise, I joined Pivotal Labs; Pivotal did not even exist in 2012.

Pivotal came into existence in April 2013 as a spin out from EMC and VMWare. Labs is part of Pivotal, but we are more of a product company focused on cloud and data than a services company. We work on Cloud Foundry, an open source Platform-as-a-Service (PaaS), and have our own distribution as well as our hosted service. The other side of our business is data. Our Big Data Suite includes GPDB (an MPP database), HAWQ (SQL on Hadoop), and Gemfire (an in-memory data grid).

My role at Pivotal has evolved in the time I’ve been there.

For the first couple years, I was the Director of Quality Engineering on Cloud Foundry. It’s a title I swore I would never take again. But my job was different than you might imagine. I did not direct the efforts of quality engineers. Rather, I paid attention to our feedback cycles. Teams own their tests, their CI pipelines, and ultimately the quality of their deliverables. I just helped connect the dots. By the way, if you want to know more about quality and testing on Cloud Foundry, I did get out of the building long enough to give a talk on it at the Wikimedia Foundation. I also gave a talk on the Care and Feeding of Feedback cycles at DOES2014.

In the last few months I moved over to our Data organization in Palo Alto. This changed my commute substantially, so my family and I are moving this summer. That will be an adventure. We’ve been in the same house for 17 years. So wish me luck with that.

Along with the move to our data org, my title changed. We removed the word “quality” from it since what I do does not look anything like traditional quality engineering. So I’m now a director of engineering. But the work I do on a daily basis with our Data teams looks a lot like what I did with Cloud Foundry: I’m deeply involved in hiring, cross-team coordination, improving our release practices, improving builds and CI to make the developer workflow better, and coordinating with our product organization to make sure teams have a steady stream of high value work.

I’m also doing my best to climb the steep learning curve of MPP databases and Hadoop. It helps that I worked at Sybase once upon a time. But that was 20 years ago. So between the fact that I was doing very different work 20 years ago, I’ve forgotten much of what I learned, and things have changed a bit in 2 decades, my prior database experience is only helping me a little in understanding my new context.

I have to say that I love working at Pivotal. I adore the people, am fascinated by the products, and am passionate about the way we work. Coming back to Pivotal was like coming home. (After all, Pivotal Labs is where I learned Agile over a decade ago.)

Some of you have noted that I don’t get out much anymore. I’m not at conferences and I don’t travel much. Since I’m in an inward facing role it’s difficult for me to carve out time to get out into the community. I’d like to see my industry friends more often and I am always honored to be invited to speak. But I turn down the vast majority of speaking invitations. My job takes up all my available time and brain cells.

So that’s what I’m up to and why I’ve been silent here for so long. I do have things to say though. I’ve learned a lot in the last 30 months. And I’m learning more every day. So I hope to carve out time to share what I’m learning here. But no promises about when, exactly, I’ll post.

Adventures in Commuting

I rested my head against the window, drowsy from a long day, full tummy, and steady hum of the train. My husband sat next to me absorbed in a game. It’s a long ride from San Francisco, where we work, to Pleasanton, where we live. I thought I might take a nap.

A disembodied voice crackled, announcing the next station: “West Oakland.”

The train stopped, the doors opened. I watched the passengers. The train car was about half full. A few people got off. Three black youths, a girl and two boys, got on. They were talking animatedly, happy. All three were well-dressed: fashionable clothes that looked new and fit well. Tight black pants. Caps. A little bling. They’d taken the time to look nice, and they were out for a good time.

One of the boys sat while the girl and the other boy stood, balancing by holding onto the straps suspended from the ceiling of the car. They talked and joked together. I felt a little voyeuristic watching them share this private moment of fun, but I couldn’t help myself. I was too taken with their delight and energy. Their faces lit with joy as they laughed. The girl and the boy hung from the straps and swung their weight around, enjoying the interplay of gravity and momentum of the train and their own upper body strength.

The girl clung to the strap then braced her other hand on the side of the train. Walking her feet up the train side, she executed an impressive flip. She did it again, obviously pleased with herself. “You try doing that in heels!” she crowed. She did it again. I noted with admiration that she was wearing boots with 5″ heels.

I guessed the kids to be somewhere between 15 and 20. It’s an in-between age. Young enough to treat a train as a jungle gym but old enough for a night out on their own. I thought about my own kids, about the same age.

The disembodied crackling voice announced the next station: Lake Merritt. The train stopped. The kids stayed on. The train started again. The train abruptly came to a screeching halt. “That’s not good,” the passenger in front of me muttered. I looked around but couldn’t tell why we were stopped. I leaned my head back against the window; my husband stayed immersed in his game. The trains don’t run on time here. Stopping wasn’t unusual.

Two BART police officers stepped onto the car. Another guy, a passenger, trailed behind them. The passenger with the cops said something I couldn’t hear and gestured toward the three youths. He was a mousy white guy in a button-down oxford shirt, corduroy pants, and a fleece vest. He looked like the kind of guy who drives an SUV with Save the Planet bumper stickers.

The first police officer addressed the youths. “There’s been a complaint. I need you to step off the train.”

“What’s the complaint?” the girl asked.

“We didn’t do anything,” the boy said.

“There’s been a complaint,” the police officer insisted. “Please come with me.”

“What’s the complaint?” the girl repeated.

The mousy passenger who had made the accusation faded back. I didn’t see where he went. Now it was just the cops and the kids.

“We got a complaint that you were asking for money.” The cop’s voice carried.

“We didn’t do anything!” the boy repeated, louder.

“Please come with me,” the cop insisted. “This train can’t move until you get off.”

The crowd on the car grew restless, impatient. The two cops flanked the three youths and attempted to escort them off the train. For a minute, it looked like the kids were going to comply. The boy who was seated stood, headed toward the open door, but then changed his mind. “I didn’t do anything. I was just sitting there,” he said. He got back on the train.

The girl’s face had gone from joy to furor. “It’s that racist white guy,” she yelled, loud enough for the whole train car to hear. “We didn’t do anything, but he thinks we did because we’re black.”

A passenger stood up, Asian, “They really didn’t do anything,” he said to the cops. Then to the kids: “Let’s just step off the train so they can make their report.”

Another passenger stood up. A black woman. “Really, officers, these kids didn’t do anything.”

By this time I was on the edge of my seat. It was so unfair. These kids were just out for a good time. They had done nothing besides being full of energy and life. They hadn’t asked for money. They were being harassed by the cops for something they had not done.

I’m still not sure what made me decide to act on my sense of injustice. Maybe I arrogantly thought the police would listen to me, a middle-class, middle-aged, white woman. Maybe I saw the faces of my own kids in the black youths and imagined them being detained on hearsay. Whatever the trigger, I stood up, moving past my husband who looked at me quizzically. I walked up the train aisle to the officer who had been barking orders.

“Sir, they really didn’t do anything. They just got on at the last stop.” My voice was steady. Calm. Quiet. Pitched for the circle of people involved, not for the rest of the train. Internally I was amazed that my adrenaline wasn’t amped up, that my voice was not shaking. I don’t normally confront cops. It is not at all like me to tell a cop how to do his job. I couldn’t believe I was confronting a cop and not even nervous about it.

By this time the crowd on the train was growing antsy. A few were calling for the kids to get off the train so they could get where they were going, but far more were calling for the cops to leave the kids alone. Several more passengers stood up and started shouting at the cops:

“These kids didn’t do anything!”

“Leave them alone!”

“Racist white guy can’t tell blacks apart!”

“It must have been someone else!”

“Let the kids go!”

I felt self-conscious standing in the middle of everything. I’d played my bit part in the unfolding drama. I returned to my seat.

A woman on the other side of the car leaned over toward me, “I could have done that,” she said. She was white like me, graying hair cut shoulder length in a bob. She looked like she could live in the house next door in my white-picket-fence neighborhood.

“I’m sorry?” I was confused by her words. What does that even mean, I could have done that? If you could have why didn’t you? Did she mean that she could not have done what I did?

“I could have done what you did,” she repeated.

I smiled and nodded. I didn’t know what else to say.

My attention turned back to the growing situation. More passengers had stood up to defend the kids. Another BART police officer entered the car. The increasing tension and buzz from the crowd made it hard to hear. I realized that there were several more police officers waiting on the platform. The situation had escalated rapidly. This could become ugly.

“CHECK THE BUCKET,” the girl shouted.

For the first time I noticed that the kids had an amplifier and a bucket with them, and suddenly I saw a missing piece of the puzzle. The kids were street performers. They weren’t out for a night of clubbing. They were dancers.

Sometimes street performers come onto the BART trains. They strut up and down the aisle, put on a show, and ask for money. I personally find being a captive audience to these shows annoying. I never give the performers money because I don’t want to encourage it. But I did not realize that such behavior warranted police intervention on this scale.

Nor did I think that these kids had been dancing for money on this train. I looked around for the accuser but couldn’t see him.

I was incensed by the entire situation. The accuser had disappeared, the situation was escalating, and these innocent kids were now targets.

“This isn’t right,” I told my husband. I stood up, shouldered my backpack in case I decided to leave the train, and pushed past my husband’s knees for the second time. Without even thinking about whether or not my husband would follow me or would be upset with me for getting involved, I stepped into the situation again.

I sidled up to the cop who had originally been barking orders and who was now standing between the kids and the rest of the passengers.

“They really didn’t do anything,” I said into his ear. “They didn’t. This isn’t right. I will be happy to make a statement. What can I do to help?”

The cop turned to me. “They might not have done anything originally,” he said, “but now they are resisting. If they don’t get off the train we will have to arrest them.”

Another police officer entered the train. The mood of the passengers was growing dark, roiling. It became a palpable thing, shifting energies threatening to spill over into violence. My stomach fluttered.

I turned around, realized that half a dozen people had their cell phones out, recording the incident. I idly wondered if I would show up on youtube videos. I decided I didn’t care. This wasn’t right and I wouldn’t stand for it. These kids were being detained, harassed, and possibly arrested all because the mousy guy in the fleece vest didn’t like being bothered by dancers panhandling on BART. These kids might have danced on BART at some point, but not tonight, not on this train. They didn’t deserve this.

Another officer entered the train. It was now five officers to the three kids. The Asian passenger who had originally stepped up to intervene turned to the kids, “Why don’t we get off the train and get this handled. Then we can all get onto the next train.”

The police closed their circle, puffing themselves up, herding the kids to the exit. I followed them to the door. I was about to step off the train. I turned back to see if my husband was paying attention. He was staring at me, his face open, questioning. “This isn’t right,” I mouthed at him. He stood up, following me.

I turned my attention back to the kids. The cops had one of the boys in an arm lock and pressed him up against the wall of the station. It looked like it hurt. They had the girl surrounded. I couldn’t see her, but I could hear her screaming. I heard another voice — the other boy? the Asian passenger who had intervened? a cop? — telling her “This isn’t worth it. Stop struggling.”

I stepped off the train onto the platform.

The cops who had the girl surrounded had her hands behind her back. She was struggling and screaming. They began half pulling, half dragging her down the platform toward the station exit. “LET GO OF ME,” she shouted. “LET GO OF ME LET GO OF ME LET ME GO!”

An officer stood in the middle of the chaos just looking, sizing it all up. He was new on the scene. He was also the only officer not in the middle of the chaos. I walked up to him. “Sir, they didn’t do anything,” I said. I watched his face. Dark skin. Brown eyes. A guy who’d been having a quiet night. He turned to me.

“You’re a witness?” he asked.

“Yes sir. I was on the car. These kids didn’t do anything. This isn’t right.” The girl was still screaming. Each panicked shriek felt like shrapnel in my soul. That could be my daughter, I thought. So very not right. “I will be happy to make a statement,” I said.

“OK,” the cop replied. He took out a notebook, asked me my name. He had to ask me to spell my last name a second time. I handed him my driver’s license to make things easier for him. He started taking notes, then looked up at my husband and me. “I left my recorder in my office,” he said. “Would you be willing to go there to make a statement?”

I agreed immediately. To my surprise, my husband agreed as well. “Of course,” he said. We would probably be an hour late, or more, getting home. This was worth it.

We followed the officer off the platform, past the turnstiles, and through a door marked “authorized personnel only.” He led us down a maze of corridors, past a bank of cubicles, and into his office. As promised, he took out a recorder and interviewed us for a verbal statement. My husband and I each told our version of the events. They lined up. Each of us emphasized that the kids had done nothing wrong.

“Let me just clarify a few points,” the officer said after we finished our statements. “In your opinion, did the officers use excessive force at any time?” I said they had, citing the painful hands-behind-the-back restraining technique and the panicked girl screaming. My husband said they had not, but pointed out that the officers should have just let the kids go when so many passengers testified that the kids were innocent.

The officer asked a few more questions: had the officers used mace, batons, or physical violence on the kids? had they acted unprofessionally?

We finished the interview, the officer escorted us back into the station, said that internal affairs would probably be contacting us in connection with the investigation, then said goodbye.

As we waited for the next train, I played over the evening in my mind.

I realized that the statements we made existed only on a recorder in the cop’s office. The kids wouldn’t know we made the recorded statement. If they got lawyers, the lawyers wouldn’t know. Our testimony might never be heard by anyone involved in the case against the kids.

The more I played over the questions that the officer had asked us, the more I realized that the questions were all about the conduct of the other officers, not about the kids and their innocence.

The internal affairs investigation would, no doubt, turn into a huge deal. The situation had escalated so rapidly, fueled by racial tension. So many people had recorded videos. The train had been halted for 15 minutes or more. Passengers were irate. There would be much attention on the conduct of the cops.

I didn’t care about the cops. I wanted to help the kids.

Somehow, I doubted that I had.

Next time I see dancers on BART, I’m giving them whatever is in my wallet. I’ll think of it as their defense fund.

Endings and Beginnings

When I wrote my blog post a few weeks back celebrating 15 years in business, I had no idea that in a few short weeks I would be making sweeping changes. I knew I was restless in my current work, and also tired of being on the road all the time. But I did not realize how ready I was for a major change.

Hoo-boy, was I ready for change though. Rapid, sweeping change. Folks who follow me on Twitter know that in the last couple of weeks, I’ve: delivered my PragProg book Explore It!, to production; started a new job at Pivotal Labs; and closed down both Agilistry Studio and my consulting practice, Quality Tree Software, Inc.

I’ve posted farewell letters on both the Quality Tree and Agilistry Studio websites. You can read those for details about those closings. Eventually, however, I’ll be taking those websites down. This site will be my primary web presence.

I want to take a moment here to talk about what’s next for me.

My new job means that I won’t be training or consulting any more, nor do I intend to participate in many conferences for a while. There’s just so much for me to do and learn with my new job that I don’t have the mental bandwidth for anything else for the foreseeable future.

That said, I do intend to keep pushing the boundaries of software testing and overall development practices. I believe that the traditional software QA model is fundamentally and irretrievably broken. It’s time for new models. I want to figure out how to articulate those new models simply, and I hope that my work at Pivotal Labs will help me do that.

In the coming months, I might be writing here more. Or not. I’m not sure yet. We’ll just have to see how things go.

So that’s what I’m up to. What changes are on the horizon for you?