Back in August, I called for volunteers to help me experiment with online training. Many folks expressed interest in what I’m trying to do. So I thought I’d write a little status report.

First, let me summarize what I’m trying to accomplish.

Over the years I have received several inquiries asking if I have online versions of my instructor-led testing classes available. I always say “no.” That’s because my classes are interactive, collaborative, and experiential. Participants don’t just do exercises. They take on roles in simulations, play with toys, work in small groups on realistic testing problems, and generate ideas in big group brainstorms. A typical webinar structure doesn’t even come close to supporting that style of training.

But these days virtual collaboration solutions do much more than just let a presenter display slides and talk. So I began to wonder…maybe the technology is finally ready to support the kind of training I do?

I started by defining what I would really need in a virtual classroom, given that I want my online classes to feel as interactive and spontaneous as in real-space. And that means the following:

• Supports at least 6 and preferably 10 or more people connecting from around the world.
• Supports Firefox and Mac seamlessly. (IE-only solutions need not apply.)
• Has integrated VoIP so there’s no phone line needed, and no need to make international phone calls.
• Has integrated video that allows participants to see each other, all at once, in realtime.
• Allows for multiple simultaneous speakers so the conversation can flow at least as smoothly as on a telephone call (no “pass the microphone” metaphor).
• Allows participants to drive a shared browser so they can collaborate on web-based exercises, such as testing a web app.
• Allows participants to capture their individual insights and contribute ideas in a shared workspace, visible to all, in text and drawings.

And of course the performance and reliability has to be at an acceptable level. I don’t know how to quantify what constitutes “acceptable.” But I do know that every time I have to say “I’m sorry, I didn’t catch that. The sound dropped out. Could you repeat what you said?” I’m going to get progressively grumpier. Ultimately my intent would be to offer commercial public classes online, and I can’t ask people to pay for classes when they can only hear 50% of the conversation.

So to figure out if the technology was ready for prime time, I held a series experiments. Many were small, involving just me and one other person. Two were larger experiments with six and five people respectively (including me).

The results were mixed. I’m still cautiously optimistic, but not completely convinced the technology is ready for this kind of usage.

In each of the two big experiments - held on two different virtual meeting solutions - the sequence of events was remarkably similar:

1. Got everyone connected. That involved a lot of rounds of “Can you hear me now?” and “I can’t see you!” and “How do I…?”
2. Discovered that the VoIP audio was flaking out, and took steps to throttle back on the video bandwidth in an attempt to correct the problem.
3. Did a round of introductions.
4. Did an online version of a short exercise that I use in real classroom settings.
5. Debriefed the exercise.
6. Debriefed the overall experience.

So far, the lessons I’ve learned include…

Have Two Machines
As the meeting leader, I find it incredibly useful to be logged into the meeting twice, once as Host, and once as a normal participant. This ensures that I see what the participants see. And it gives me a safety net so that if one machine is experiencing technical difficulties, I have another way to connect with participants.

The Webcams Are Important
When I started this process, I wondered if webcams were actually necessary or if they were gold-plating. I discovered that the webcams are essential to create the kind of collaborative atmosphere I want to create. As one of the participants in the first experiment put it: being able to see the other people created a bond. Amazing how a 1″x1″ 1-frame-per-second-stop-motion-animation of a person contributes so much to forging a connection.

Let Participants Type
In the first experiment I ran with other people, I did the same things I do in a classroom setting. So when I shared the web page with the exercise we were going to work through together as a group, I drove, just as I would if I were projecting the exercise onto a screen at the front of the room. And when we debriefed the exercise, I typed. Finally one of the participants said, “Wait a minute…we all have keyboards, so why are you the only one typing?” Doh! Of course! That’s why the whiteboard is a shared resource: so everyone can contribute insights without having me act as a scribe. It’s why I wanted a shared collaborative space to begin with. Silly me! I was too accustomed to the physical constraints of laptops and projectors and pens and flipcharts.

So I made the participants type more in the second experiment, and I think the results were better. (Not perfect, but better.) I will continue to find ways to ensure all participants have the opportunity to drive and type. I think that this is actually one way in which a virtual venue could be MORE effective than a real classroom.

Watch the Momentum
In the real world when I run extended exercises, the participants tend to have a fair amount of momentum. My problem usually isn’t keeping the participants engaged in the work, but getting them to stop. In a virtual context, I have observed that the discussion seems to grind to a halt much more easily. I’m not sure why yet. But it’s something I need to watch and find ways to address.

Keep the Group Small
Initially I thought I might be able to handle groups of up to a dozen people in a virtual setting. Now I’m thinking eight. It takes more effort and attention than I’d originally imagined to ensure that everyone is participating and engaged, to keep track of who’s saying what, and to monitor the technology to make sure all is well. Also, I think there are still issues with scaling the number of participants, and even eight might push the limits of the technology too much. Maybe six is the right number.

Moving forward, I need to:

• Make the whole experience smoother. That means I need to become much more familiar with all the options and controls so everything is second nature (no more “Now where was that menu option…?”). I think that some of the awkwardness the group felt in the sessions had more to do with my blunders than with real limitations in the technology. So I need to spend more time rehearsing.
• Hold a separate one-on-one session with each participant to check their audio and visual and practice the mechanics of using the technology in advance of the actual session. And that means developing a set of good and preferably relevant-to-the-topic Getting Started one-on-one exercises.
• Experiment with different exercise structures to see if some are better for keeping the momentum going than others. Also experiment with overall course structures. My in-person classes have a blend of individual, small group, and big group work, but emphasize working in groups. I suspect the optimal balance for online training is different. So I plan to experiment with incorporating more self-paced individual work done in preparation for the interactive online sessions.

But before I get too far ahead of myself, I have to decide how much (more) time and money I want to invest in this endeavor. To do that, I need two vital pieces of information:

1. Can the technology really and truly do what I need it to do? My experiments to date have been promising but inconclusive. So in the next few weeks I’ll be running one more big experiment to test performance and audio/video quality with more people. If you’re interested in participating, watch this space for details…
2. Will anyone pay money for it if I can make it work? So far I’ve been much more focused on the question “Is it possible?” than on the question “Does anyone want it?” But I need to assess interest. And you can help. If you would be interested in seeing my company offer online experiential training as a for-pay service, drop a note in the comments or send me an email. (No obligation. No sales person will call. I will not add you to my mail list unless you ask me to. Your feedback just helps me decide if there’s enough public interest to pursue this.)

Oh, and if you have additional questions or comments, feel free to drop a note in the comments about those too!

I spoke too soon. Last night, I wrote:

I think I’ve tested it. I think I’ve fixed all the (sufficiently important) bugs. But that’s just me wearing my developer hat saying “It works on my machine.” If you happen to notice any problems with the new WordPress theme, I’d be most grateful if you could let me know. Oh, and you can let me know if you like it too. Thanks!

Within minutes of posting that, I realized that the graphics were jaggy and ugly on a PC. Hmph. So I pulled down the new look and the “New Look” post, and went to bed grumbling about ugly PCs vs. beautiful Macs.

Depending on your perspective, I either pulled it down too fast, or not fast enough. By the time I pulled it, Google had already indexed it, and feed readers all over had already gotten the new post via RSS. But when folks went to the site, they got the old look, and no sign of the “New Look (Beta)” post. Confusing to say the least.

But if I’d done things differently, I wouldn’t have gotten Zach’s hilarious email this morning chronicling his adventures trying to find the mysterious “New Look” post.

Anyway…

What I suspect (um, hope) you all don’t know is that I spent a fair amount of time last night fixing functional problems with the site through the tried-and-true, but oh-so-cowboy, process of take-it-live, find-the-fatal-error, take-it-down, fix, and repeat. So by the time I posted the “New Look (Beta)” message I had already solved numerous very real problems and just wanted to be done.

Bugs showing up in production that didn’t show up in the development and test configuration? How could that be?

In trying to create a development and test environment on my local network, I downloaded WordPress and installed it on my Mac. And I then used the Default theme as a starting place. You see, I don’t actually know PHP, and am relatively unskilled at WordPress theme creation. So starting with something that works well, then tweaking it until I got the look I wanted, seemed like a good way to go. (Some of you may be thinking, “Wow, this site looks nothing like the default.” Behold the power of CSS. That, and I learned an awful lot over the last couple of days about how WordPress works. But I digress.)

When I took the new theme live for the first time last night, I discovered my fatal error. The version of WordPress on my local machine is 2.3.1. The version of WordPress installed by my host, GoDaddy, is 2.0.4. And the new theme was incompatible with the version 2.0.4 of WordPress. I got fatal errors all over the place.

So at this point (about 10PM my time), I decided to do something that I would never do if this had been a real software project with real consequences. I decided the fastest way to get from broken to working would be to:

1. Look at the live site to get the error message.
2. Change back to the old theme while I worked to fix the bug.
3. Grep the local files to figure out where the offending code was coming from.
4. Fix each instance of offending code and test locally.
5. Upload the fixes.
6. Change the presentation to the new theme and go back to step 1. Repeat till done.

Cowboy, yes. But it worked. Within 15 minutes I’d fixed all the fatal errors and the site was actually functioning. Like I said: not recommended for real projects, but this is my blog, not a mission critical web app. And it would have taken me at least an hour to downgrade my local WordPress instance and do this the proper way. I didn’t have an hour.

After all that, I was so proud that it worked, I overconfidently posted the “New Look (Beta)” message. And only then did I finally got around to looking at the new site on a PC. In retrospect, that was the wrong order to do those things in. In any case, looking at the new design on a PC was just depressing, and I’d had enough for one night.

This morning, I figured out that although Macs make scaled-down images look nice in a browser, PCs don’t. I’d taken a shortcut and rather than resizing the images properly as I tweaked the design, I just used the WIDTH and HEIGHT attributes in the IMG tag to change their size. The result was that the images looked oh-so-1985 jaggy on a PC.

This whole experience underscores the importance of cross-browser and cross-functional testing before taking a site live. I knew that. Really I did. But I ignored it because my local network just isn’t set up to make this kind of thing easy. I knew it would be a little bit of a hassle to get a PC to surf to WordPress hosted on my Mac, and I couldn’t be bothered. I was overconfident about my ability to create designs that work cross-browser and cross-platform. And I was willing to take the risk if I was wrong. Mostly, I was just being impatient. I wanted to get the new look live before I went to bed, and I was tired and wanted to sleep.

These are all really bad reasons not to test. But then this is why I develop my own websites instead of outsourcing the effort. I need these reminders of how easy it is to choose to do the wrong thing for bad - but legitimate sounding at the time - reasons.

You see, I hear “we can’t do that kind of testing because it’s too hard” from my clients on a regular basis. And it would be too easy for me to become all judgmental and say “it’s not that hard; you’re just lazy.” But I can’t say that because I make the same dang mistakes in my own development. So instead of grumbling about lazy developers, I say “I know it’s hard. But it’s important. So instead of telling me that it can’t be done, please tell me what it would take to make it more feasible, and we’ll work on that.”

Oh, and for the record I’ve now set up my Mac and PC so I can surf to the test site running on the Mac from the PC easily. (Mostly that involved adding entries to the hosts file on the PC.) Still haven’t downgraded my local WordPress instance, but I will before I do any more serious theme development. Next time will be different. But since I tend to do these big site redesigns about once a year, next time is also a long way off.

Unless y’all absolutely hate the new look. So tell me what you think…

So I’m working on course materials for a class I’ll be teaching next week on Acceptance Test Driven Development.

I decided to try using Keynote on my trusty no-longer-all-that-new Mac instead of PowerPoint. I find PowerPoint on the Mac incredibly annoying. Whenever I want to edit a presentation on the Mac that I created on the PC (like, all of the presentations I have), I get Converting Metadata messages all over the place. When editing a 300 slide file, making even the smallest change can take forever. So, Keynote seemed like a better alternative.

Now I should explain that although most of my slide decks look reasonably simple, I tend to use a lot of animations for certain types of things - like bringing a workflow to life.

And I also use the presenter notes area as a place to put the meat of the materials for participants so that when I print books with both the slide and the presenter notes, there’s a good amount of text. The result are course notes that read kind of like a book with a whole lot of illustrations.So in other words, I am not your typical lightweight presentation software user. And I’ve crashed PowerPoint a lot as a result.

But I had high hopes for Keynote. For starters, it’s Mac software. Built for Macs. None of this ported over from Windows crap.

Alas, I was doomed to disappointment.

First, when I attempted to do some fiddly edits on the text under a page with a ton of animations, Keynote died a horrible death. It didn’t exactly freeze entirely, but I could no longer edit the content. I could attempt to save/export my file, but every time I did - no matter what format - it told me that the file was corrupt. Sadly, I’ve gotten out of the habit of saving every 30 seconds, so I lost a fair amount of work.

When I finally recovered everything an hour later, I vowed not to make the same mistake again. I not only started saving rabidly every 30 seconds, but I also checked everything into my subversion repository. However, when a subsequent commit failed with the message “Directory …/.svn containing working copy admin area is missing” I learned that there’s an interaction bug between Keynote and Subversion. I used the workaround kindly published by Daniel Sadilek (bless you, dude). And I was back in business. Annoyed, but back in business.

But here’s the thing. I’m already annoyed, and now I’m looking at my .key file for a 27 slide file, and it’s 3.8Mb. What??? I know I’m abusive to presentation software. I understand that my usage is a tad on the abnormal side. But 3.8Mb for 27 slides? Slides with graphic drawings and text, no photos, no movies, no sounds? That’s nuts. With a little experimentation I discovered that more than half that is due to two slides that make extensive use of animations. And I’m betting that those same animations had something to do with Keynote dying a horrible death when I tweaked the text underneath. Someday when I have more time I’ll see if I can reproduce the bug.

Let’s just say I’m re-evaluating my decision to use Keynote. PowerPoint is looking pretty good right now. Sigh.

And this is also a lesson re-learned about software testing: there’s nothing like a real user to find real problems. For Agile projects, that means automated unit and acceptance tests are necessary but not sufficient. Make sure real humans try the software under real world conditions. Given the topic of the course materials I was working on, this was a timely reminder…though a far more painful one than I really wanted.