Testing Has No Value

Update Nov 20: minor edits to increase clarity.

Yesterday, Rob Lambert tweeted:

Turns out that’s a statement from the ISTQB Expert Level Syllabus on Test Management. Robert was tweeting it to see what others’ reactions were to the statement.

Now, I hold no love of the ISTQB. I don’t see any correlation between ISTQB certification and tester competence. I think an awful lot of what the ISTQB promotes as “best practice” is at best context-dependent and at worst actively damaging. But that’s a topic for another post, one I probably will never write. It’s just not worth the energy. I don’t feed that which I don’t want to see grow.

But my point is that I don’t put any stock in anything the ISTQB says, so I was absolutely stunned to find that I agreed with a statement in an ISTQB syllabus.

So I replied, “Whodathunk I’d agree w/ istqb. Unless org sells test svcs, its is a means to an end, a way to get info.”

That touched off a small storm of responses, some agreeing, but many more disagreeing. I’ve been replying to the objections on Twitter, but I think this topic needs more than 140 characters.

First, a few clarifications:

1) I am talking specifically about software development.

2) I am talking about business value.

3) Business value is inextricably linked with increased revenue or decreased costs. That’s not decreased costs of software development. It’s overall decreased costs that can be attributed to the use of the final delivered software.

So the value in software development rests in the final result—the delivered product or service—not the process or activities that went into making the result.

Testing does not have value on its own. Neither does programming. I have seen more than one team crank out lots of code and yet be unable to ship a product anyone would buy. For that matter, designing doesn’t have intrinsic value either.

“But wait!” you say. “Testing does have value. Customers pay more for better quality software, so testing has value.”

Well no. First, it’s not clear that customers pay more for better quality software. Customers are less likely to abandon better quality software in a fit of rage. But they don’t necessarily pay more for the kind of quality that comes from having testers do more testing. Further, testing does not automatically lead to better software. In fact, better testing can actually lead to worse software. Finally, as Ron Jeffries pointed out, testing only adds value to the extent that the team uses the information that testing produced in order to improve the software.

But that’s not even the main point here. The main point is that testing and programming and designing and all the other activities that go into making software are all just a means to an end. As UCLA coach John Wooden said, “Never mistake activity for achievement.”

The bottom line is that if you want your work to have value, make sure that the software you’re working on ships and satisfies customers. The final result is a whole team effort. The team succeeds or fails together. And the value of the work you’re doing is stuck at 0 until the software you’re working on is in the hands of paying customers.

Subscribe

Subscribe to our e-mail newsletter to receive updates.

13 Responses to Testing Has No Value

  1. Bob Marshall November 14, 2012 at 7:32 am #

    A great post. And very congruent with my (default) position that “All testing (of the Inspection variety, at least) is waste.”

    So, it pains me to disagree with you on one thing (and with the ISTQB statement you quote).

    Here’s my line of thinking:
    1. Effective inspection provides information on the state of the product under test. This information is of little or no direct value to potential customers of the product.
    2. Effective inspection also provides information on the state of the process used to produce the product under test. (“Real QA”). This information would be of huge value if the producer (organisation) knew what to do with it.
    3. As the (“process”) capability of the producer organisation grows, its process improves such that inspection becomes less and less necessary (as the process produces fewer and fewer defects), so its need for inspection (and the value it brings) diminishes proportionately. A neat paradox?

    - Bob

  2. José November 14, 2012 at 8:45 am #

    I don’t agree at all:

    1. First of all, PROPER testing DOES have business value:
    1.1 it can increase revenue due to the perceived quality of the product (lesser rate of unsatisfied customers searching for alternatives)
    1.2 it can reduce costs thanks to early detection of bugs, increased productivity thanks to no need to manually re-test prior functionalities or confidence in the test system detecting any change that would cause an error, reduced number of bugs reaching production, so less money needed in customer care and urgent bug fixing

    2. It is evident that ANY activity that is not guided by some business value objective has no business value. But that’s completely different than saying that NO activity has any business value as you seem to say.

    More in http://www.openinput.com/post/35695918291/los-tests-no-tienen-valor (in spanish, sorry)

    Customers and managers are prone to see tests as superfluous and not adding value, so articles like this are the last thing we need.

    - José

  3. Dave Van den Eynde November 14, 2012 at 9:00 am #

    Testing does not add value, but it reduces the cost. The cost of spending time in QA, the cost of providing support, the cost of doing business.

    I can’t believe you’re this short-sighted.

  4. Chris November 14, 2012 at 9:47 am #

    Well, yes. In that sense the phrase “Testing, as an activity by itself, has no business value” is like saying “Umpiring a tennis match without any players has no value”. Am I missing the point of the quote?

  5. Chris Millar November 14, 2012 at 8:03 pm #

    Thanks for the post, I’ve chewed on it quite a bit today.

    I work in the defence industry. When I am testing software I am mainly motivated by wanting it to work in deployment, or preventing it from being deployed if I find a sufficiently serious error. I’d argue this is tangential to business value.

    You say that an activity such as design has no business value by itself, but I would argue that this is not the case. Can it not have inherent qualities that give it value? Consider Leonardo da Vinci’s designs for helicopters – are they worthless because they did not generate any sales?

    An activity such as test is something that can be learned from. A good test strategy or test report can have value by itself. It can be learned from to promote better testing throughout the business or to improve the state of testing in the industry as a whole.

  6. Joe Strazzere November 14, 2012 at 8:42 pm #

    Agreed!

    Businesses don’t want testing. They want systems that work well enough to make money. Sometimes that requires more testing, sometimes less, sometimes none at all.

    You can do a lot of testing and provide no business value at all (for example, if the software doesn’t ship).

    You can do some testing and still provide some business value (for example, if the software is good enough to ship and make money).

    We as professionals need to understand that, and in light of that, provide “just enough” testing. That will maximize our value to our companies.

  7. Ian E. Savage November 19, 2012 at 1:07 pm #

    Testing has value in my company, at our level of (practice) maturity, with our product set.

    And, I think, my company will always need product quality data for our deployment decisions. So I must take dissenting position.

    While the value may be just peace-of-mind, it has value for important stakeholders.

    Regards, Ian

  8. Aurelie F. November 21, 2012 at 4:19 am #

    The wording is pretty bad indeed.

    Testing has business value: You can demonstrate it by the negative, simply pointing at major software failures and their business impact. You can also demonstrate it by pointing at the success of “quality” products. Never in the past have we seen so much focus on quality: Testing is not just about finding bugs, but measuring quality attributes. In this era of consumerism, quality drives business value: Just ask Apple and its shareholders, or the New York Times when they introduced their 1 star application…

  9. Olle Morell November 26, 2012 at 9:08 am #

    Well, unless you sell testing services, testing has of course no business value.

    For software, testing is usually a rather large and expensive part of the process to create something that actually has business value, and considering that we humans are very prone to make errors it is unlikely that we will ever be able to do away with it. In fact, testing well and efficiently is one of the more important parts of modern SW development.

    Saying that testing has a business value of its own for a product-creation company is like saying that the act of hammering has a business value for a carpenter. It hasn’t. It does not matter for the customer how the nails get into the wood. It’s the end product with various properties (hard or soft) that has a business value.

  10. Shrini Kulkarni December 30, 2012 at 4:18 pm #

    >>> The bottom line is that if you want your work to have value, make sure that the software you’re working on ships and satisfies customers.

    I am not sure as a tester I have direct control over ship date of software and its impact on customers. While I can, as a tester, to best of my ability, investigate and report potential issues that can annoy and irritate customers – I do not directly control what actually gets shipped to the customer. How can I, then make my work valuable in the terms that you define here. Does this mean my work as a tester is “worthless” in terms of business value?

    I am not suggesting, also, that it (testing) does provide business value. This issue is on a slippery slop. Cannot settle it one way or the other.

    >>> Finally, as Ron Jeffries pointed out, testing only adds value to the extent that the team uses the information that testing produced in order to improve the software.

    True. Well said. Then why ask question or deliberate on “testing has or does not have business value. Business value comes from elsewhere right?

    >>>> The main point is that testing and programming and designing and all the other activities that go into making software are all just a means to an end.

    So – we should not be asking if any activities (means to end) have or does not have business value. That would be a pointless question.

    Shrini

  11. Ralf February 20, 2013 at 7:49 pm #

    This is a nice discussion. My opinion is that the sentence “Testing as an activity by itself has no value!” is true – but only then when you don’t concider the outputs which you get from the testing work. Adressing the outputs and results from the testing work to the right people or processes, testing ofcourse has value! That is the value!
    But when it’s not clear why testing is necessary, what kind of testing is necessary, what the mission of the testing work is and what to do with the outputs and results of testing it is indead useless and has no value!
    I mean: You can also say something like that: “Breathing as an activity by itself has no value!” Also this might be true. But also here the result of breathing is important! You need the air to live, don’t you?
    Another example: “Walking as an activity by itself has no value!” might also be true, when you don’t know from where you come and where you will walk to! (= result/output/purpose – this is cucial and this is the value!)

    And I also think as Shrini mentioned, that we should not ask needless questions like these! Whatever for?

    kind regards from Germany
    Ralf

  12. jlottosen February 23, 2013 at 9:47 pm #

    Provocative, but has an important point – it reminds me that:

    The product is a solution. If the problem isn’t solved, the product doesn’t work. (Even of all the tests are green).

    Each day, business decision makers depend on test information for help in making business decisions

    which I ahve added to: “Acceptance criteria are more than what can be measured”
    http://jlottosen.wordpress.com/2013/02/03/more-than-what-can-be-measured/

    Thanks!

  13. R Madala November 4, 2014 at 2:24 pm #

    Reading the above post in the end of 2014 feels pretty strange. Software testing has not only become an integral part of software development life cycle, but has carved out its only niche in the software domain. It’s a matter of time before it has its own set of quality standards on which there is a huge debate going on, refer this link: http://www.evoketechnologies.com/blog/new-software-testing-standards/ and businesses these days are willing to part away with the extra revenue, which is required to undertake comprehensive software testing. I completely disagree that software testing has no value, at least in the year 2014 :)

Leave a Reply