This is a guest blog post by Daniel Frank, my assistant. Daniel took on the challenge of updating the QA/Test job study for 2011, just in time for making New Year’s resolutions. Enjoy! Elisabeth
It’s been a little over a year since Elisabeth published “Do Testers Have to Write Code,” the results of an in-depth survey of job ads that she and Melinda conducted to see if employers expect testers to program. The resounding conclusion, with 80% of tester job ads requesting some kind of programming skill, was “Yes.”
This year we wanted to see if things have changed, so I conducted the same study again. I also wanted to add a bit more granularity to the study, to see if there were any trends that were missed last time.
I screened the lists with the same basic guidelines as our previous study. That means I restricted my search to the US only. I only counted a job if it was described as a testing/QA position in the job title. I did not include recruiter listings in order to avoid the risk of including duplicate jobs or even fake jobs used to gather pools of applicants.
Our final sample size this year is 164 jobs. That’s a little less than last year. Why?
The lists were sparse. There just aren’t that many job ads out there. Many of the job ads I found were from recruiters or were repeats, with the same company listing the same position several weeks in a row.
The simple fact that I had a hard time finding the same number of ads as last year is interesting information all on its own. From an overall economic standpoint, the country is in no more of a slump than we were in 2010. So why are there fewer listings for testers? Could it be that Alberto Savoia, who recently declared testing dead, is correct? We’ll come back to that question later.
Back to the study…
Like last year, the majority of our jobs came from Craigslist (90) and LinkedIn (64). The rest of them came from a smattering of other sites.
The data includes an even higher proportion of jobs in California than last year: 102 of the listings were in CA, with the remainder divided in small chunks between 28 other states. Unsurprisingly,Texas, Massachusetts, and Washington are the three runners up.
Last year there was some question of whether or not the sample was biased simply because we’re located in California. However, I took extra steps to try and get equal representation. The simple fact is that a search that might find 70 jobs when I filter the location for CA will result in 30 jobs or fewer if I filter for another area. If anything, I’d estimate that California is actually under represented.
I kept track of the job titles. By far the most popular title is “QA Engineer” (99 of the listings). 136 of the titles contained “QA” compared with only 32 containing the word “Test.”
An interesting side note: when I searched for the word “test” in the body of job ads, I found far more developer positions than similar searches for “qa” did. It would seem that at the same time QA/Test positions are requiring more coding skills, developer positions are requiring more testing skills. That might be another interesting job ad survey project.
So how much coding are testers expected to do?
Of the 164 listings, 102 jobs say they require knowledge of at least one programming language, and 38 jobs indicate coding is a nice to have. That’s 140 out of 164, or 85.37% of the sample. That’s an even higher percentage than last year. It’s difficult to say if the 5% uptick represents a real increase in demand, but at the very least it’s fair to say that demand for testers who code remains high.
I used the same criteria that Elisabeth and Melinda used last year. That means that I counted a job as requiring programming if the job required experience in or knowledge of a specific language, or if the job duties mentioned a language. There were 7 jobs which listed broad experience requirements like “must be able to script in multiple languages,” which also counted as requiring programming.
There were some judgment calls to be made about what may or may not count as a programming language. For the purpose of the results here, I counted SQL or other relational database knowledge as a programming language in order to be consistent with last year. However, unlike last year, I tracked proficiency in relational databases separately. This will let me track specific trends more easily in future studies.
One of the questions Elisabeth wanted to answer last year was whether jobs with self-identified Agile organizations required testers to code more than other jobs. This year 46 of of the 58 Agile job ads list programming skills as required or nice to have. That’s 79.31%, which is actually a lot less than last year’s 90%. However, this is one of those places where the small sample size has to be taken into consideration. In 2010, 49 out 55 agile jobs mentioned programming. Today, 46 out of 58 jobs mention it. Just a few jobs result in a 10% variation.
An enduring question about any kind of job is how much it pays. I saw even less mentions of pay this time around. Only 7 jobs even listed it, and 5 of those were button-pushing game testing positions in the $10-$20/hour range. The other two ran around $85,000-$105,000. Most positions simply don’t provide up front salary estimations, so we cannot draw any real conclusions from these data points.
Just for fun, I also noted whenever a job requested a certification. In 164 jobs I found exactly 4 mentions of certification, and not a single one was required. 3 of them were vendor or technology certifications that had nothing to do with testing. And even in the single instance where a testing certification was nice to have, it was the CSTE offered by QAI, rather than the much more hyped ISTQB. So it would seem that testing certifications are not much in demand. The bottom line is that someone looking to improve their marketability would be much better served by upskilling to a new proficiency rather than picking up an irrelevant certification.
And that’s about it for our study. If you’d like to dig through the raw data to look for any trends I may have missed, I’ll be happy to send it to you. Drop me a line.
Now back to the question about the number of QA/Test jobs out there. Could it be that there are fewer QA/Test positions? Was this just a matter of luck and timing, or is there a trend?
Alberto Savoia gave a talk titled “Test is Dead” at GTAC (dressed as the Grim Reaper). He may have used intentionally inflammatory hyperbole to make his point, but that doesn’t change the fact that he had interesting points to make.
Alberto points out that especially in web development, speed is paramount. Further, the biggest challenge isn’t in building “it” right, but in building the right “it.” So the goal is to get a minimum viable product out as quickly as possible, and get fast feedback from real users and customers. Traditional black box testing ends up taking a back seat in this type of development, and these projects often rely heavily on user feedback instead.
At STARWest 2011, James Whittaker of Google gave a talk titled “All That Testing is Getting in the Way of Quality” where he talked about the closest thing to a traditional testing role they have at Google. It’s called the “Test Engineer,” and they spend anywhere from 20%-80% of their time writing code. He also explains how Google utilizes their user bases to do almost all of their exploratory tests. As he puts it, “Users are better at being users than testers are, by definition.”
With James and Alberto’s talks firmly in mind, I can’t help but wonder if the difficulty I experienced in finding job ads that met my criteria is indicative of a sea-change in the industry rather than an anomaly. Could it be that we’re seeing a reduction in the number of QA/Test positions?
What do you think? Are you seeing fewer QA/Test positions in your organization or (if you’re looking) in your job search?