Do Testers Have to Write Code?

For years, whenever someone asked me if I thought testers had to know how to write code, I’ve responded: “Of course not.”

The way I see it, test automation is inherently a programming activity. Anyone tasked with automating tests should know how to program.

But not all testers are doing test automation.

Testers who specialize in exploratory testing bring a different and extremely valuable set of skills to the party. Good testers have critical thinking, analytical, and investigative skills. They understand risk and have a deep understanding where bugs tend to hide. They have excellent communication skills. Most good testers have some measure of technical skill such as system administration, databases, networks, etc. that lends itself to gray box testing. But some of the very best testers I’ve worked with could not have coded their way out of a For Loop.

So unless they’re automating tests, I don’t think that testers should be required to have programming skills.

Increasingly I’ve been hearing that Agile teams expect all the testers to know how to write code. That made me curious. Has the job market really shifted so much for testers with the rise of Agile? Do testers really have to know how to code in order to get ahead?

My assistant Melinda and I set out to find the answer to those questions.

Because we are committed to releasing only accurate data, we ended up doing this study three times. The first time we did it, I lost confidence in how we were counting job ads, so we threw the data out entirely. The second time we did it, I published some early results showing that more than 75% of the ads requested programming skills. But then we found problems with our data, so I didn’t publish the rest of the results and we started over. Third time’s a charm, right?

So here, finally, are the results of our third attempt at quantifying the demand for programming skills in testers. This time I have confidence in our data.

We surveyed 187 job ads seeking Software Testers or QA from across the 29 states in the US posted between August 25 and October 16, 2010.

The vast majority of our data came from Craigslist (102 job ads) and LinkedIn (69 job ads); the rest came from a small handful of miscellaneous sites.

The jobs represent positions open at 166 distinct, identifiable companies. The greatest number of positions posted by any single company was 2.

Although we tried to avoid a geographic bias, there is a bias in our data toward the West Coast. (We ended up with 84 job listings in California alone.) This might reflect where the jobs are, or it could be because we did this research in California so it affected our search results. I’m not sure.

In order to make sure that our data reflected real jobs with real employers we screened out any jobs advertised by agencies. That might bias our sample toward companies that care enough to source their own candidates, but it prevents our data from being polluted by duplicate listings and fake job ads used to garner a pool of candidates.

Based on our sample, here’s what we found:

Out of the 187 jobs we sampled, 112 jobs indicate that programming of some kind is required; an additional 39 jobs indicate that programming is a nice to have skill. That’s just over 80% of test jobs requesting programming skill.

Just in case that sample was skewed by including test automation jobs, I removed the 23 jobs with titles like “Test Automation Engineer” or “Developer in Test.” Of the remaining 164 jobs, 93 required programming and 37 said it’s a nice to have. That’s still 79% of QA/Test jobs requesting programming.

It’s important to understand how we counted the job ads.

We counted any job ad as requiring programming skills if the ad required experience or knowledge of a specific programming language or stated that the job duties required using a programming language. Similarly, we counted a job ad as requesting programming skills if it indicated that knowledge of a specific language was a nice to have.

The job ads mentioned all sorts of things that different people might, or might not, count as a programming language. For our purposes, we counted SQL and shell/batch scripting as programming languages. A tiny number of job ads (6) indicated that they required programming without listing a specific language by listing broad experience requirements like “Application development in multiple coding languages.” Those counted too.

The bottom line is that our numbers indicate approximately 80% of the job ads you’d find if searching for jobs in Software QA or Test are asking for programming skills.

No matter my personal beliefs, that data suggests that anyone who is serious about a career in testing would do well to pick up at least one programming language.

So which programming languages should you pick up? Here were the top 10 mentioned programming languages (including both required and nice-to-haves):

  • SQL or relational database skills (84)
  • Java, including J2EE and EJBs (52)
  • Perl (44)
  • Python (39)
  • C/C++ (30)
  • Shell Scripting (27) note: an additional 4 mentioned batch files.
  • JavaScript (24)
  • C# (23)
  • .NET including VB.NET and ASP.NET but not C# (19)
  • Ruby (9)

This data makes it pretty clear to me that at a minimum, professional testers need to know SQL.

I will admit that I was a little sad to see that only 9 of the job ads mentioned Ruby. Oh well.

In addition, there were three categories of technical skills that aren’t really programming languages but that came up so often that they’re worth calling out:

  • 31 ads mentioned XML
  • 28 ads mentioned general Web Development skills including HTTP/HTTPS, HTML, CSS, and XPATH
  • 17 ads mentioned Web Services or referenced SOAP and XSL/XSLT

We considered test automation technologies separately from programming languages. Out of our sample, 27 job ads said that they require knowledge of test automation tools and an additional 50 ads said that test automation tool knowledge is a nice to have. (As a side note, I find it fascinating that 80% of the ads requested programming skills, but only about half that number mentioned test automation. I’m not sure if there’s anything significant there, but I find it fascinating nonetheless.)

The top test automation technolgies were:

  • Selenium, including SeleniumRC (31)
  • QTP (19)
  • XUnit frameworks such as JUnit, NUnit, TestNG, etc. (14)
  • LoadRunner (11)
  • JMeter (7)
  • Winrunner (7)
  • SilkTest (6)
  • SilkPerformer (4)
  • Visual Studio/TFS (4)
  • Watir or Watin (4)
  • Eggplant (2)
  • Fitnesse (2)

Two things stood out to me about that tools list.

First, the number one requested tool is open source. Overall, of the number of test automation tool mentions, more than half are for free or open source tools. I’ve been saying for a while that the commercial test automation tool vendors ought to be nervous. I believe that this data backs me up. The revolution I predicted in 2006 is well under way and Selenium has emerged a winner.

Second, I was surprised at the number of ads mentioning WinRunner: it’s an end-of-lifed product.

My personal opinion (not supported by research) is that this is probably because companies that had made a heavy investment in WinRunner just were not in a position to tear out all their automated tests simply because HP/Mercury decided not to support their tool of choice. Editorializing for a moment: I think that shows yet another problem with closed source commercial products. Selenium can’t ever be end-of-lifed: as long as there is a single user out there, that user will have access to the source and be able to make whatever changes they need.

But I digress.

As long as we were looking at job ads, Melinda and I decided to look into the pay rates that these jobs offered.

Only 15 of the ads mentioned pay, and the pay levels were all over the map.

4 of the jobs had pay ranges in the $10-$14/hr range. All 4 of those positions were part time or temporary contracts. None of the ads required any particular technical skills. They’re entry-level button-pushing positions.

The remaining 11 positions ranged from $40K/year at the low end to $130K/year at the high end. There just are not enough data points to draw any real conclusions related to salary other than what you might expect: jobs in major technology centers (e.g. Massachusetts and California) tend to pay more. If you want more information about salaries and positions, I highly recommend spelunking through the salary data available from the Bureau of Labor Statistics.

And finally I was wondering how many of the positions referred to Agile. The answer was 55 of the job ads.

Even more interesting, of those 55 ads, 49 requested programming skills. So while 80% of all ads requested programming skills, almost 90% of the ads that explicitly referenced Agile did. I don’t think there’s enough data available to draw any firm conclusions about whether the rise of Agile means that more and more testers are expected to know how to write code. But I certainly think it’s interesting.

So, that concludes our fun little romp through 187 job listings. I realize that you might have more questions than I can answer. If you want to analyze the data for yourself, you can find the raw data here.

62 thoughts on “Do Testers Have to Write Code?

  1. The scenario in India is “A Tester should know programming” otherwise he does not stand a chance to get a job.

  2. In my country (Netherlands), the majority of testers do not know how to code. I think that’s a shame. I have heard a talk by datawarehouse testers in which they admitted they did not now how to write SQL statements and therefore had to implement a tool. They were proudly explaining the implementation. They could have saved a lot of their client’s money if only they knew SQL.

    I’ve witnessed test managers spend weeks on evaluating tools because the requirement was that no coding was to be done by testers. Many test-tools incorporate a scripting language(like beanshell or groovy) for making it more flexible. I you use e.g. the Python formatter in Selenium, it becomes much more flexible.

    Being able to write code (mainly Python) helps me in understanding basic programming principles such as data types, loops, conditions, functions, classes etc. Knowledge that makes me a better tester. Moreover, it narrows a communication gap between me and programmers on a project. I’ve experienced programmers explaining their code to me, and I was able to ask good questions because I could understand (most of) the code.

    On one project I had to test webservices. In order to better understand webservices, I decided to program a test webservice and write a small client for making automated checks. It’s not only a great way to spend your evenings, but it also provides great insight in how webservices should handle exceptions, provide error messages, validate, handle sessions, use authentication etc.

    Also, I’m using Python a lot for generating test data, which can range from random character strings in boundary bytesizes to database mass insert scripts.

    I’m sure a lot of testing can be done without knowing how to program, but, I’m convinced that being able to program can make you a better tester.

  3. I am fresher of 2015 batch… Does testing requires good programming knowledge..if so what are programing languages i have to learn and which types of testing requires programming knowledge… I am totally confused so to begin my career in testing…please giude me in right path.

  4. If you can write c#, Java etc proficiently why bother being a tester? You could make over 100k knowing c#. I would not expect a tester to REALLY understand serious programming languages. Unless you a referring to a basic understanding of the language where a tester could google a tutorial and understand the basic concepts.

  5. I think increased collaboration with development is the order of the day.
    We all know that collaboration with development teams to encourage up stream quality is a critical aspect of shift-left philosophy.
    Imagine how effective you can be when you speak the dev. language?
    Personally, I find my development background extremely useful in areas such as test automation ,
    DevOps and development quality.
    Consider Test automation , the most obvious extension of a coding know-how that we can apply in our day to day jobs.

    My 2 cents on the topic

  6. I live in Sweden and I see all the time, ads where people are looking for testers and of course, they should always know programming. I think it’s very strange since it’s a difference between a functional tester and technical tester. I am a functional tester, not a developer. We have developers performing automatic tests. When I am working as a test manager, I discuss requirements and such with the developers, but they run and code the tests.

    I think it’s soo unfair and lame that testers should know programming. It’s like saying a developer needs to have X years of study within the area of test. How many developers are familiar with testing? Not many.

    There are several benefits with having a tester who does not go through code. You get a certain quality check from someone who is not into coding.

    I hate that test is an area which is not popular or prioritized and that me as a tester and test manager always comes last.

  7. “Selenium can’t ever be end-of-lifed: as long as there is a single user out there, that user will have access to the source and be able to make whatever changes they need.”

    Don’t agree with this at all, most companies don’t have the staffing or inclination to support open source tools themselves, so availability of the source code may appeal to some developers, but in reality its a false hope. IT would also make open source tools very expensive compared to commercial tools.

  8. Thank you very much. I was looking for this post details and found it today. It is very helpful for me to Plan ahead.

    If your developers says.-“Why do testers need to do SQL Queries”. Is it Testers to be low-scaled?

  9. I am working as Manual tester from 2.6 years in India. Before started job I learned JAVA and I really didn’t like coding. But I heard few Automation Tool does not required any coding.
    Is it true …. if it please let me know that name then I can learn that.

  10. Many tools have attempted the promise of automation with no coding. I’ve never seen any work over the long haul and am quite skeptical. I would not recommend making big career decisions on vendor hype. (I’m really sorry. I know that’s not the answer you wanted.)

Comments are closed.