From the Mailbox: What's the Definition of Testing?

A reader wrote to ask me, “My team and I have been discussing the definition of ‘test’ and ‘testing.’ What’s your definition?”

There are almost as many definitions of software testing as there are books on the topic.

  • In his 1979 book The Art of Software Testing, Glenford Myers said, “Testing is the process of executing a program with the intent of finding errors.”
  • Kaner, Falk and Nguyen agreed in their 1999 book, Testing Computer Software (2nd Ed): “The purpose of testing a program is to find problems in it.”
  • In his 1983 book The Complete Guide to Software Testing, Bill Hetzel said, “Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.”
  • The 1990 IEEE Standard Glossary of Software Engineering Terminology (Std 610.12-1990) says testing is, “The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.”

But whatever definition of testing you subscribe to, the result of testing is information: information about bugs, discrepancies, misunderstandings, current status, and software behavior under a variety of conditions.

Dale Emery and I discussed this at length and decided that we agree that: Testing is a process of gathering information by making observations and comparing them to expectations.

(Please note that this definition is intentionally vague about where those expectations come from.)

That leaves the question about the definition of the word “test.”

There are numerous definitions including the one in the IEEE glossary (Standard 610.12) that defines “test” as “An activity in which a system or component is executed under specified conditions, the results are observed or recorded, and an evaluation is made of some aspect of the system or component.”

However, Dale and I find a slight reframe helpful: A test is an experiment designed to reveal information, or answer a specific question, about the software or system.

Anyone have definitions for “Testing” and “Test” that you like better?

Subscribe

Subscribe to our e-mail newsletter to receive updates.

12 Responses to From the Mailbox: What's the Definition of Testing?

  1. Anthony June 20, 2007 at 7:55 am #

    I totally agree with your definition of Test and Testing, to me they allow broader usage of test and not only to find problems or errors

  2. Chris McMahon June 20, 2007 at 12:39 pm #

    I’ve always thought of testing as being closely related to the Scientific Method. A scientific experiment has a hypothesis, an experiment to attempt to falsify the hypothesis, and the results of the experiment.

    Similarly, a software test has an objective, a procedure, and an expected result.

    I realize that both of the definitions above are overly simple. But it is a great way to frame the conversation. I have worked with testers who had no concept of testing *for* any particular feature. They were punching buttons hoping something bad happened.

    The concept that a test is comprised of an objective (something to be tested), a procedure (the means by which that something is exercised), and an expected result (an idea about correct behavior of that something) is a powerful way to think about software testing.

    At least it’s a really good start. More advanced testing techniques follow from this basic idea.

  3. Jared June 21, 2007 at 6:34 am #

    If I observe the product, but am never able to find out what the expectation should be, did I test it? If I leave my notes lying around, and months later someone else comes along who *does* have an expectation and knows what a good and bad result are, did *they* test it?

  4. B Kuhn June 21, 2007 at 8:53 am #

    I like you’re re-framed definition, along with the one that Chris offered. One thing I would add – testing is a vital part of the overall verification process. Do the results of the experiments (tests) match the specifications (design, requirements, etc…)? This is part of the answer that testing should provide. The majority (note that I’m not saying 100%) of our expectations as testers should be based on the specs.

    Testing also provides valuable information to project management and stakeholders about the overall quality of the system, which is in turn used in decision making processes.

    BTW – I just found your blog and I am enjoying the articles!

  5. Shrini June 21, 2007 at 9:39 am #

    I attempted to do something like this here

    http://shrinik.blogspot.com/2005/08/definitions-of-software-testing.html

    For now, I settle to James Bach’s definition –
    “Testing is an act of questioning a product in order to evaluate it”

    or this one “Testing is an infinite process of comparing invisible to ambigious in order to avoid unthinkable happening to the annonymous”

    Or this one – “Testing is a process of infinite search for problems”

    Or Cem Kaner’s definition:
    “Testing is an Empirical technical investigation done to provide stakeholders, information about quality of a product or a service.”

    Shrini

  6. Shrini June 21, 2007 at 9:41 am #

    What about a testcase or test case then?

    Is a test case (or it is testcase) an instance of test?
    just as an example of a test?

    Shrini

  7. Jeff Fry June 21, 2007 at 12:54 pm #

    Hey Elisabeth, thanks for another great post!

    I’ll add James Bach’s and Cem Kaner’s definitions from http://www.satisfice.com/blog/archives/43

    “Testing: questioning a product in order to evaluate it (Bach version); technical investigation of a product, on behalf of stakeholders, with the objective of exposing quality-related information of the kind they seek (Kaner version).”

    I realized I had more to say about all this than would comfortably fit in this comment box so I have posted a longer reply here:
    http://testingjeff.wordpress.com/2007/06/21/definitions-of-testing/

  8. Vaibhav June 27, 2007 at 1:10 am #

    A test is to analyze the result of some activity with the clear cut defined expected result of the system under test. If I say I want to test for the cleanliness of the table, then I know what degree of cleanliness I want. Is it just the shine I want or Is it just want a clean table I want with no dust or paper on it. And how I perform a test is testing.
    Test case is any case which contains test and the required environment, settings, tools to perform testing. Like for the table example if I say test is – just to verify whether the table is clean or not. But a test case would be to verify this with a magnifying glass, or with some clean cotton, under some set of conditions or some particular environment.

    Vaibhav Agarwal
    vaibhav@crestech.in
    http://www.crestech.in

  9. Debasis Pradhan July 2, 2007 at 6:27 am #

    I agree to your definition for testing. Personally, I follow James Bach’s and Cem Kaner’s definition for testing. But I must admit that I was equally impressed by your definitions too. I have written a whole new blog entry on the definition(s) of testing here: http://software-testing-zone.blogspot.com/2007/07/software-testing-definition.html , in case you might want to take a look!

    Regards,
    Debasis.

  10. Matthew Heusser July 5, 2007 at 5:30 am #

    Hey there. I think this is a great discussion Brian Marick recently wrote that: “I’m not convinced that automated business-facing (acceptance) tests are, in the presence of good programmer testing, all that incredibly useful as tests—as, that is, regression tests, tests that will find bugs created when a programmer changes something here that causes a failure over there.” (http://www.exampler.com/blog/2007/06/01/two-conference-ideas/)

    He goes on to recommend other uses for customer facing acceptance tests, and they are interesting – but they aren’t really experiments designed to reveal information about the system type benefits, as much as documentation and communication type benefits.

    Which leads me to think that we should stop calling them tests …

    🙂

    I don’t have any easy answer here, but I don’t think the vagaries and inconsistencies we have as a field about testing are doing us any favors. If we could just get a little more specific – not perfect, just slightly better terminology – I think as a field we could get a little more street cred. (You have to admit, testers are famous for our insecurity about our street cred when compared to, say, devs …)

  11. Gieno July 22, 2007 at 9:15 pm #

    Testing is done to find information.

  12. Muhammad Hammad Khan December 2, 2007 at 12:11 am #

    I agree with the defination provided by chris McMahon
    Testing of systems can be of two types (i)Known System(ii)Unknon system
    for known system “Testing is the process of taking series of measurements through certain systematic procedure and then comparing them to expected values” (to verify)
    for unknown system “is the process of determining the mathematical model of a system by applying known input signals and then observing output response” (to determine the transfer function of the system)