Jason Huggins has very kindly pointed me to two more places where Developer-Testers/Tester-Developers (DT/TD) hang out. Interestingly, both were in London.

• Google hosted LTAC (the London Test Automation Conference) in September 2006. Antony Marcano mentioned LTAC on his blog. And the LTAC talks are available on Google Videos. And there’s a mail list.
• ThoughtWorks sponsored CITCON (the Continuous Integration and Testing Conference), in Chicago and also in London. I had trouble finding online content relating to the conference. The citconf.com site points back to the thoughtworks.com main site, and most of the blog entries I found said, “looks like it will be fun, I’m going/I might go/you should go.” But no one I found said, “I went, it was great, and here’s what happened…” I found pictures though. And there’s a mail list.

(What makes London the hotbed of DT/TD activity? Hmmm.)

Anyway…

Chris McMahon and I have been talking about pulling together a small peer conference in the SF Bay Area. And the more we talked, the more excited we got about it. So today we said: “let’s just do it and see what happens.” And we set a date. So…without further ado…I’m pleased to announce…

Bay Area Developer-Tester/Tester-Developer (DT/TD) Summit
Saturday, February 24
A peer-driven gathering of developer-testers and tester-developers to share knowledge and code.

Location: SF Bay Area, exact location TBD. If you have space you’re willing (and authorized) to lend us, we’d like to talk to you.

This is a small, peer-driven, non-commercial, invitation-only conference in the tradition of LAWST, AWTA, and the like. The content comes from the participants, and we expect all participants to take an active role. We’re seeking participants who are testers who code, or developers who test.

Our emphasis will be on good coding practices for testing, and good testing practices for automation. That might include topics like: test code and patterns; refactoring test code; creating abstract layers; programmatically analyzing/verifying large amounts of data; achieving repeatability with random tests; OO model-based tests; and/or automatically generating large amounts of test data.

These are just possible topics we might explore. The actual topics will depend on who comes and what experience reports/code they’re willing to share.

If we can get donated space, the cost to participate will be $0. If we can’t get donated space the cost will be a nominal fee (~$50) intended to help us defray expenses.

Participants will be responsible for their own travel expenses.

Proposed Agenda:

• Timeboxed group discussions: “Essential attributes of a tester-developer and developer-tester (differences and similarities)” and “What tester-developers want to learn from developers; what developer-testers want to learn from testers.”
• Code Examples/Experience Reports (we figure we have time for 3 of these)
• End of day discussion: Raising visibility for the role of a DT/TD, building community among practitioners

If you’re interested in participating, send me an email answering these questions:

• Which are you: a tester who codes or a developer who tests?
• How did you come to have that role?
• What languages do you usually program tests in?
• What do you hope to contribute to the Bay Area DT/TD summit? Do you have any code or examples that you’d like to share? (Please note that you should not share anything covered by a non-disclosure agreement.)
• What do you hope to get out of the Bay Area DT/TD summit?

My own goals with doing this are:

• Learn from others better ways of programming automated tests
• Meet others in the DT/TD role
• Build community

I spent this last weekend at AWTA, the Austin Workshop on Test Automation. Our official topic was Open Source Testing Frameworks. Before the meeting, I figured we’d discuss experience reports about how folks used Watir/Selenium/xUnit/CruiseControl/etc. and Ruby/Perl/Python/etc. to cobble together totally automated, lights out, acceptance testing solutions.

We did discuss topics like that. Bob Cotton showed us some very cool code integrating SeleniumRC and rspec. Jeff Fry showed some excellent Ruby code that generated tests on the fly. Some folks got together and spiked an integration of Selenium and Watir. (Brian Marick dubbed it “Mineral Watir.” Oooh. Bubbly.) Paul Rogers and David Crosby both spent time pairing with me to improve my Web 2.0 testing example, digging into JSUnit, Selenium, and the unittest.js library from Prototype. (Yes, I’ll get around to posting code sometime soon.)

And we did talk about how any given Open Source test automation approach intevitably involves integrating multiple pieces.

But what I didn’t expect to happen is that the discussion would turn to how developers are becoming testers and testers are becoming developers. In retrospect, I should have known it would happen. The subject is bound to come up when a bunch of developers who are passionate about testing hang out with a bunch of testers who spend a lot of their time writing code.

And it seems we weren’t the only ones thinking about this topic. Chris McMahon kindly alerted me to Steve Rowe’s excellent post about Test Developers.

So this has me thinking.

Tester-developers and developer-testers aren’t new. But we seem to be a hidden sub-community. We sometimes have titles like “Tool Smith,” but more often we have titles that look like most other tester or developer titles: “Senior Test Engineer” or somesuch.

But the role is growing.

I think that’s partly because large, successful companies are leading the way. Microsoft has a special job class and title: “Software Development Engineer in Test.” And I have heard unsubstantiated rumors that interviewing for a test position with Google involves answering questions about algorithm design.

I also think Agile is contributing to the growth in the role. Agile teams tend to blur the distinction between jobs, particularly developers and testers. Agile developers are test infected, and Agile testers tend to spend time mucking about in code. As Agile grows, so the hyphenated tester-developer/developer-tester role grows.

So, if the number of people doing both test and development growing, where are we gathering as a community?

One gathering place is online communities.  The agile-testing mail list is chock full of folks who both test and develop.

Another natural place for a community to gather is at a conference.  AWTA was a gathering place on a small scale. But when I look at the major conferences, they’re pitched to Developers (SDWest), Testers (STAR), or Everyone (Agile, Better Software).

I want more opportunities to share experiences, write code together, and learn from each other.

So, if you’re a tester-developer/developer-tester, where do you go to share test code, talk about test refactoring strategies, or exchange notes on test frameworks and scripting languages?
Oh, and if you have a blog related to ( testing && development ), please post a link in the comments!

Public Course: Developing Agile Teams

I’m happy to announce that my colleague Dale Emery and I will be hosting a public offering of our course Developing Agile Teams.

When: March 1 – 2, 2007
Where: Mountain View, CA
Cost: US$799

This is a methodology-agnostic workshop in which we focus on the core values and practices that make a team Agile.  This experiential course centers around an extended simulation, giving participants an opportunity to practice the skills we’re teaching.  For more information, see the extended course description.  Or reserve your seat now.

Agile2007: Call for Proposals, and for Reviewers

Got an Agile Experience Report you want to share?  Maybe you have an idea you’d like to explore in a Discovery Session?  Or perhaps you have a Tutorial you’d like to present?  The Agile2007 Call for Proposals is now open.  We’re actively looking for various types of sessions.  Please consider submitting a proposal!

Also, I am the Tutorial Track chair and am actively seeking reviewers for tutorials.  All proposals are peer-reviewed, and that means we need lots of reviewers.  The work involves reading proposals and providing comments.  It doesn’t take too much time, and you’ll be contributing to the overall quality of the conference.  Interested?  Please email me.  (And if we haven’t met yet, please tell me a little about yourself and your experience with Agile.)

New Website Design
I finally got the latest version of the www.qualitytree.com website up this weekend.  Hop on over and take a look!  And I’d love your feedback…drop me an email or a comment and tell me what you think.

A long time ago, when I was employed at a software company, I participated in an all-engineering offsite meeting in which we attempted to figure out what we needed to do to improve.

First exercise: we went around the room, round-robin style, and everyone contributed an idea.

A Tester said the Developers should write better Specifications.

A Developer said Marketing should give them Better Requirements.

Another Tester said the Developers should be doing More Unit Testing. (Yes, I was in that group, but it wasn’t me, I swear…OK, maybe it was.)

Someone else said the Managers should do Better Planning.

I knew things were really getting silly when someone said that what was really needed was Better Health Coverage and someone else said Bigger Cubicles. The intent of the meeting was to figure out how we could improve, not to have a two day whine festival.

But what struck me was that no one suggested something that could be accomplished by their own group. No one ever said, “I’d really like to do better at my own work by…” Every single idea was an idea that Someone Else would have to implement. At the end of two days I felt completely demoralized. We had a long to do list, much of it Action Items for management, and nothing that seemed likely to actually happen. Indeed, six months after the meeting, nothing had changed.

Since then, I’ve had the opportunity to lead similar offsite meetings. Whenever I lead such a meeting, I remember how creating a long To Do List for Other People didn’t work well, nor did hosting a Whine Festival.

Fortunately, I’ve discovered a way to avoid the Other People’s To Do List trap.

I ask participants to focus on what they can do. Not what they want someone else to do, but what they, themselves can do. And then I teach them the We Reframe.

The We Reframe involves transforming “They” statements like “They don’t give us documentation” into “We” statements like “We don’t have the information we need to do our jobs effectively”.

The transformation is straightforward. You change the sentence from what someone else isn’t doing to the effect it has on you. Some more examples:

“They aren’t doing unit testing” becomes “We’re getting software that is not stable enough to test.”

“They don’t give us clear requirements” becomes “We don’t know what we’re supposed to build.”

“They aren’t planning” becomes “We feel pulled in too many directions at once.”

This seems like a subtle difference. But reframing the problem in terms of the effect it has on you allows you to own the solution. It frees you from trying to figure out what someone else should do differently and allows you to focus on how you can adapt to the situation.

If we’re getting software that isn’t stable enough to test, what can we do about it? We could reject it. We could provide a set of minimum acceptance tests to the developers. We could integrate our test efforts with the developers’ test efforts, eliminating the handoff that isn’t working. There are a number of things we could do. But if we define the problem as “They aren’t doing unit testing,” there’s very little we can do. We can’t force them to write unit tests. Even if we could, we can’t force them to write good ones that prevent the problem we’re actually experiencing.

The reframe worked well at helping the teams I facilitated come up with an actionable list. And I discovered another profound result of the We Reframe that I hadn’t expected.

By focusing on things they can control directly, participants discovered that they have more power to solve their own problems than they originally thought.

The result is more than a list of things they can start doing immediately. It’s a general change in attitude.

So the next time you feel frustrated because They aren’t doing something, try the We Reframe to see how many things you can do without any help at all from Them.

Oh, and by the way: They are likely to notice when you change how you work, and that might lead Them to consider changing too. It’s worth a shot.

Dave Simcox just brought something important to my attention: I hadn’t included contact information on my blog.

When I was using one of the standard templates, I had a page with “About me” information, including contact info.  But when I switched to a custom template of my own design, I forgot to include that.  My blog isn’t intended to be anonymous, but it might as well have been.  Whoops.

So, for those of you who are new here, allow me to introduce myself.  Hi.  I’m Elisabeth Hendrickson.  I live, eat, breathe, and sleep software testing, just in case you couldn’t tell by the title of this site.

I’ve had a pseudo-blog going over at my qualitytree.com company site since 1997, back before the term “blog” had been coined.  I’ve now migrated all those old posts over here along with a bunch of other content like my papers and presentations.  Check out the archives.

I started working in the software industry back in the 80s.  Yes, that makes me an old timer, although I don’t feel like it.  I’m always shocked to realize that people with several years of experience weren’t even born yet when I wrote my first computer program in 1980 (in BASIC on a TRS-80 with 4K of memory. woot.).

Over the last 20 years, I’ve been a technical writer, tester, test automator, manager, director, and entrepreneur.  I founded my company in 1997, took a break for a couple years to play the Silicon Valley Lottery along with everyone else during the DotCom bubble, then returned to running my own company in 2001.  I’ve intentionally kept Quality Tree Software small so I can continue to be hands on.  So when I’m not on the road at client sites, I spend time testing, coding, and writing automated tests just for fun.  I realize that makes me a little wacky by most normal standards.

Now, back to Dave.  Dave gave me a really important piece of feedback when he pointed out to me that I’d neglected to include any real identifying information about myself.  I appreciate that.  So, to return the favor, I want to let y’all know that Dave is a recruiter trying to fill an opening for a Senior Lead QA Engineer based in Boston, MA.  It sounds like an interesting role.  If you’re interested, please contact Dave directly.

Once upon a time, I worked on a project where the developer protested “SCOPE CREEP!” to every bug report I filed.

Sadly, the two of us built up a lot of animosity arguing over whether or not the bugs I found were bugs or enhancements. I reasoned that I was testing conditions that were likely to occur in the real world, and “not crashing” did not count as an enhancement. The programmer argued that he’d done what he’d been asked to do and that it was too late to add more work to his plate. “No one said anything about the software being able to handle corrupt data!” he snapped.

Much to the programmer’s chagrin, management tended to agree with my assessments. The programmer ended up doing a whole lot of rework, grousing the whole time.

I later realized that the programmer thought I was making up new requirements as I went along. Seriously. He thought my tests were unfair.

Of course, that’s not what I intended.

The way I saw it, my testing was revealing answers to questions no one had thought to ask before: What if this file is locked? What if that connection is broken? What if the data is corrupted? If I’d been involved in the project earlier, I would have asked the questions earlier. But this was a waterfallish project, and testing happened at the very end of the process.

I know I’m not alone in having had a project like that. Some situations are even more dysfunctional. “They yell at me when I find bugs,” lamented one tester, “and they yell at me for not finding bugs if a user finds a problem in the field. I can’t win.”

Can’t win. Can’t break even. Can’t quit. At least not without dire consequences, like unemployment. Sometimes testing feels like Ginsberg’s restated laws of thermodynamics. The only way out of the trap is to change the game.

Fortunately, testers have the opportunity to change the game when we’re part of the team from the beginning, like on Agile projects. We can take those same testing skills we apply at the end of the project that enable us to find good, deep bugs, and use them to elicit specific acceptance criteria with good examples.

In fact, this is one of the most important ways people with testing expertise can help Agile teams.

But one stumbling block for some testers new to Agile his how, exactly, to do this.

First, you have to be able to imagine tests based on only the sketchiest idea of what the software under development is supposed to do.

Since this is the situation most testers work in, we’ve already honed that skill. We know how to take a hand-wavy set of statements like “users belong to groups; groups have permissions; permissions allow for create, read, update, and/or deleting of floozibitzes” and turn them into 193 test cases. So let’s assume that as a tester, you know how to identify potentially interesting conditions, actions, sequences, configurations, and such.

Now let’s explore how to leverage that skill at the beginning of a project, transforming tests into questions that will prompt stakeholders to discuss acceptance criteria in concrete terms.

First, remember the anatomy of a test. Tests generally involve some setup (“Log in as a user with read only permissions”), one or more actions (“Double click the floozibit record”), and one ore more expected results (“Verify the Edit button does not appear”).

Let’s rephrase that as a sentence: “Given this setup, take some action, and verify the expected results.”

With traditional tests, we know the expected results in advance. But when we’re still exploring requirements, we’re trying to determine the expected behavior. We only have our imagined setup and actions to work with. But we can get some pretty detailed information if we ask the question in the form: “If we have this setup and take these actions, what do you expect to happen?”

Let’s take a couple examples. Where you might be tempted to design a test around a condition like corrupted data or NULL values, instead frame it as a question:

“How should the software respond if it encounters corrupted data?”

“What if the data in this field were NULL in the database?”

Of course, sometimes when we ask that question, we get an answer like “Gosh, I dunno.” That doesn’t help much. Fortunately, testers are also good at figuring out what a reasonable expected result might be based on all kinds of things like past versions of the software, comparable products, and past experience. We can suggest some possible expected results in our questions: “If we have this setup and take these actions, should the software do this or that?”

“If the software encounters corrupt data, is it better for it to attempt a repair or discard the data or something else?”

“If the data in this field is NULL in the database, should the software turn that into a zero-length string?”

Of course, just because we can imagine a test doesn’t automatically make it interesting. Just as you might offer potential real world use cases to make a bug report more compelling, you can offer use cases to make a condition more interesting when eliciting requirements:

“That could happen if the database were improperly restored, or with data migrated from a legacy database.”

“What if a user bookmarkes a page for an item that is later deleted?”

Where you might define negative tests, instead ask about allowed/disallowed actions:

“Should a user with Delete Account permissions be able to delete the last Administrative Account?”

Traditional software development processes define the relationship between requirements and tests by insisting that tests be written from requirements. As a result, bad requirements lead to bad tests, or to a whole lot of arguing about what is and is not in scope.

Forget the arguing. Let’s focus on gaining alignment, and gaining it early. And as testers, we’re in a unique position to help the team do just that.
Anyone who is good at designing tests can use that skill to frame a wide variety of specific, thoughtful, and thought-provoking questions around expected behavior. And that means everything you know about testing can contribute to establishing a well-defined, specific, shared understanding of what the team is building.

I’m in the process of migrating a bunch of content from my corporate website over here for reasons which will become apparent when I finally publish the newest version of my corporate website. But that’s a topic for another day.

Migrating all this content has me looking at stuff I haven’t really looked at in a while. And it has me reminiscing. I figure January is as good a time as any to reminisce about presentations and conferences past.

The last file I uploaded were the slides to “Why Are My Pants on Fire?”, a presentation I gave at the SM/ASM 2002 conference.

And that reminded me that I owe Tim Lister a big thank you for the title. And therein lies a story.

Back in 2000, I was working for a company I shall choose not to name. But it wouldn’t matter if I did name it since hardly anyone has heard of it.

While working at this unnamed company, I managed a group whose title was “Quality Engineering” but whose function was much more on the testing side of things.

Like most DotComs at the time, we were under constant pressure to bring the software to market faster.

You’re probably thinking, “Schedule pressure? Right. It’s software. What else is new?” This was different, as I discovered when management made it clear they’d spend just about any amount of money if it meant we could ship a week earlier.

The time pressure meant we took some crazy shortcuts. And that meant I was in crisis mode most of the time. I spent my days chasing down bugs and having nutty debates about whether or not we could ship software that, under certain circumstances we couldn’t quite pin down, would cause a Blue Screen of Death.

A year or so into the job, I took some time out to attend a conference. Talking with others about DotCom insanity gave me a much needed sense of perspective. And it felt rather cathartic.

That’s when I found myself sitting at a table with Tim Lister, Jim Highsmith, and some other luminaries I can’t recall just at this moment. I was, quite frankly, more than a little starstruck. Tom DeMarco and Tim Lister’s book Peopleware changed my life, or at least my career, and this was the first time I met Tim Lister in person.

You’d think being a bit starstruck would keep me quiet. But no. Remember I was feeling cathartic. I talked about my job. A lot. Too much. And much of what I said probably sounded like whining. Let’s be honest. I really was whining. Tim listened for a while, then stopped me.

“Let me get this straight,” he began. “You have a greenfield project with no legacy users to support, right?”

I nodded.

“The code is written in a modern programming language, you still have the original source code, and it’s even kept in version control?”

I nodded again.

“You have no legacy data you’re supposed to import into the system?”

I nodded again.

“SO WHY ARE YOUR PANTS ON FIRE?!?” he demanded.

Why indeed. His question had the desired effect: I stopped whining.

I pondered that question over the next year. I considered Tim’s words each time we had yet another crisis, whenever customers complained about devastating bugs, and every time a DOA build was delivered into test.

Then I left the company. I got tired of the smell of singed pants. I got tired of the crises. And I realized that I couldn’t do anything more to help. I’d done everything I knew how to do, it hadn’t worked, and I was done.

Turns out I had great timing. I didn’t know it at the time, but things were about to go from chaotic to dismal. A couple months after I left they laid off half the staff. A month later, they laid off another big chunk of folks. And a month after that they closed their doors.

That was 2001, the year Silicon Valley turned into a ghost town. You could tell no one in Silicon Valley was working by the shocking lack of traffic on the major freeways: commute times dropped precipitously for the lucky few still employed.

By then, I was back into consulting and training, and my commute involved airports. Lots of airports. And with each new client I visited, I kept pondering Tim’s question.

When you think about a question for that long, eventually answers emerge. About two years after Tim asked me that fateful question, I turned my answers into the “Why Are My Pants on Fire?” presentation. Now it’s one of my most frequently requested talks.

So, that’s the story of how Tim Lister inspired me to transform my miserable whining into useful answers that became a well-received and oft-requested presentation. I’m grateful to Tim for the title and for his wisdom, but most of all, I’m grateful to him for making me see I was at least partly responsible for igniting my own trousers.

When interviewing candidates for testing positions, I often describe a program, then ask the candidate to tell me how they would test it.  I recall one candidate who came up with a couple test cases, then sat back in his chair and said, with a relaxed smile on his face, “and that’s that.”

“That’s that?” I asked.

“Yup,” he replied.  “That’s all I’d need to test.”

He didn’t get the job.  Complacency is dangerous in a tester.

I expect testers to think about the various ways the software might be used or misused.  And I look for that knack when I interview.

A couple test cases just can’t cover all the potential variations in data, sequences, timing, configurations, and operating conditions that might affect the behavior of the software.  A tester who brushes his hands together declaring, “That’s enough” after two test cases isn’t even aware there could be more.  If he’d made an agonized face and said, “there should be more to test but I can’t think of what it is right now,” it would have been better than a self-satisfied “that’s that.”

Out in the real world, real users will do things even the most diabolical tester would be hard-pressed to imagine.  They’ll try to “undo” actions by hitting back or refresh in the browser; they’ll move message windows all the way off the screen “to get them out of the way” and then leave them there; and they’ll power down the machine to silence an “annoying” (but critical) alert.  A tester who can’t think beyond the most mundane tests has no hope of discovering how the software might behave when subjected to such treatment.

At best, a “that’s that” tester will provide extremely limited information about how the software will behave under optimal conditions.  Unfortunately, it’s all too likely that superficial testing will instill the team with a false sense of confidence.  In the worst case scenario, the team will ship seriously flawed software with absolutely no idea how badly it will fail in the field.  It’s a disaster waiting to happen.

Unfortunately, there are some people who think that the entire role of a tester is to verify that the implementation matches the written requirements and/or specifications.  Whether or not they realize it, such people are advocating “that’s that” testing as general policy.

I wish I could say that all such people I’ve encountered were old-school traditionalists who also think Big Design Up Front is super cool.  Unfortunately, there are plenty of Agilists who think testers should stick to executing overly simplistic Acceptance Tests.

“If it isn’t part of the Story Acceptance Criteria,” they say, “we shouldn’t be testing for it.  It’s out of scope.”

When they’re feeling generous, they add, “Make a new Story for it.”

It seems to me that such an approach to Agile development isn’t terribly Agile.

Agilists value working software over comprehensive documentation.  Tests tell us how well the software works under various conditions.  So it’s not surprising that all Agilists I’ve spoken to claim to value testing.  And I believe that most actually do.  Unfortunately, some only value those tests that demonstrate the software “works.”

So to those who think tests should only show that the software “works,” I ask: If you limit the scope of testing to demonstrating the software “works,” how are you going to discover the circumstances under which it doesn’t?