A Place to Put Things

At the AA-FTT workshop last October, I did this lightning talk titled “A Place to Put Things.”

In it, I propose standardizing on places to put different kinds of information associated with automated functional tests.

It seems to me that one of the key success factors for the xUnit family of unit testing frameworks is that they gave us just 5 places to put code related to unit tests: setup, test, teardown, suite setup, suite teardown. That simple organization has a powerful focusing effect, enabling (or, perhaps, forcing) developers to narrow their attention down to just the code needed to create one little itty bitty unit test at a time.

Functional testing frameworks have no such common, standardized structure.

FIT has given us something close with the notion of a test in natural language in a table and fixture code to hook that test to the software under test. If we extend that model a little to include the idea that we may well be testing against an external interface, like a web interface, where a driver, like Watir or SeleniumRC, would be handy, we end up with 3 big categories of things:

  • Tests: scenarios describing the actions and expectations, expressed in natural language with keywords
  • Fixtures: code that connects the keywords in the test to actions in the software under test
  • Drivers: libraries like Watir, SeleniumRC, Perl’s Win32::GUI, etc. that know how to address external interfaces such as Web interfaces, thick client GUIs, command line interfaces, soap/XML calls, etc.

That’s the direction I think test automation tools in general are headed, and it’s an important evolution with profound implications. However, I’m still figuring out how to explain how this structure differs from what traditional tools offer, and the significance of those differences.

At the very end of the “A Place to Put Things” video, there’s a little exchange between two of the participants in the workshop. A woman’s voice says, “She just pulled that together in like a minute!” That’s Jennitta Andrea speaking. She was the co-organizer of the AA-FTT workshop.

In response, Ward Cunningham says, “Oh, no. I think she’s been pulling that together for the last 3 years.”

Ward’s right. And I’m still pulling together my ideas and figuring out how to articulate them. So while you don’t see much evidence of it on my blog, I am actually spending a fair amount of time writing, and rewriting. More soon. I hope.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

11 Responses to A Place to Put Things

  1. Shrini Kulkarni May 31, 2008 at 11:01 am #

    >>we end up with 3 big categories of things:

    Where do COTS type GUI automation tools like QTP go among these 3 categories?

    Drivers? or Fixtures? or Tests? A bit in each of these catetories or a 4th new category?

    Shrini

    Elisabeth responds:

    COTS type GUI test automation tools could be drivers. Borland demonstrated their new Silk Agent technology that allows users to call the agent from inside any Java program at STAREast.

    However, most of the traditional GUI test automation tools hopelessly intermingle all three categories of things into one file: the automated test script. That’s one of the reasons those scripts are so fragile. And it’s one of the reasons I don’t consider such tools applicable for an Agile context.

  2. Jeffrey Fredrick May 31, 2008 at 12:50 pm #

    I was struck by the similarity of what you’re describing and a talk that PJ and I have submitted to SDBP on the architecture of CruiseControl. Though we didn’t use the terminology a big thrust of the talk is about how using Inversion of Control gives you “a place to put things”: bootstrapping, check for modifications, build, publish. Each of those places is an interface and that makes it an easy system to work with.

    Not sure where to take the similarity from here but the resonance seemed worth mentioning.

  3. Pekka Laukkanen May 31, 2008 at 2:24 pm #

    Great post, couldn’t agree more! I normally don’t want to leave “me too” comments, but in this particular case I can make an exception. I also might be able to guess what you mean with “more soon”. =)

  4. Shrini Kulkarni June 1, 2008 at 6:46 am #

    >>we end up with 3 big categories of things:

    Where do COTS type GUI automation tools like QTP go among these 3 categories?

    Drivers? or Fixtures? or Tests? A bit in each of these catetories or a 4th new category?

    Shrini

  5. Steve Freeman June 1, 2008 at 1:28 pm #

    On the last project I did any serious FIT work, the terminology we used was Fixtures and Fittings. I think Fittings were about equivalent to your Drivers, they made the Fixtures “fit” the system, and it’s a cute pun.

  6. Danny Faught June 9, 2008 at 3:43 pm #

    Great point, Elisabeth. I think it’s very useful to separate things into test cases, fixtures, and drivers. Some day we’ll commonly be able to plug in a new driver or fixture without changing the other components. Note that since “fixture” comes out of the unit testing world, you may have an uphill battle teaching black box testers that term.

    There could be confusion about “drivers”, which has been applied in many different ways, possibly including test management functions (I recently split drivers and test case management into two different pages on testingfaqs.org).

    I have a question for you. A client recently asked me to implement a subroutine mechanism in a fixture. They want to be able to essentially build new keywords using existing keywords. Would these subroutines be part of the test case realm (they most likely will be stored in the version control near the test cases) or the fixture (which is functionally how they’ll be treated)?

  7. Benny Daon June 15, 2008 at 12:30 am #

    A great Post. The only point I beg to differ is the language in which we write testing scenarios.
    Testing scenarios written in natural language are ambiguous making automatic testing unreliable. As we are all pros, coders and quality experts alike, I prefer to use a simple scripting language – like python – for documenting project tests. It will also solve the issue of where to store them – together with all the other source under source control.

  8. Isaac Howard June 19, 2008 at 11:37 am #

    Question: In your video you mention ‘multi-mode’. Could you describe this more? I am unfamiliar with the term in relation to tests/testing.

    Thanks.

  9. Elisabeth Hendrickson July 1, 2008 at 2:39 am #

    Hi Isaac,

    By “multi-mode” in this talk, I meant the ability to articulate tests in more than just tables. Some tests lend themselves to tables. Other tests are better expressed with models or pictures or step-by-step sequences. I find tables useful, but sometimes a little limiting, just as some documents are best created in Excel while others would be better created in Word or PowerPoint. It all depends on what form best captures the ideas (or, in this case, tests) that you’re trying to articulate.

    So I can imagine a world in which I can express tests in whatever form is best suited to each particular test, and the automation framework can read & execute the tests no matter whether they’re expressed in pictures with annotated expectations (as in Brian Marick’s experiments with annotated OmniGraffle diagrams, see http://www.exampler.com/blog/2007/07/13/graphical-workflow-tests-for-rails/ ) or FIT tables.

    Thanks,

    Elisabeth

  10. Basim Baassiri February 6, 2009 at 12:37 pm #

    To comment on how QTP can fit into the 3 categories

    I’ve been using the following for my own framework for web applications
    1. Tests are written using EXCEL with keywords and params
    For Example
    Navigate
    ClickLink
    etc

    2. I have a class written in VBScript that reads in the tests (EXCEL) and passes it on the “Fixtures” (code to determine what to do)

    3. QTP is the driver to act upon the application under test

    I hope this helps this first commenter and anyone who uses QTP

  11. aidy lewis April 10, 2009 at 12:27 pm #

    Powerful stuff: But in my opinion keywords are not natural language. Most applications undergo a sequence of actions to achieve a goal and neither I nor the business speak in keywords. This is why I prefer to use a natural language parser like Rspec Cucumber.