“They never give us enough time to automate our tests, and then they complain at us that we don’t test fast enough!” J. shook her head. “And when I want to hire more people to help automate, they tell me I have too many people already! Management blames me because testing takes too long, but they won’t support me in fixing the problem. What’s wrong with them!?!”
J. is a QA manager in an organization that’s adopting Scrum. She’s frustrated, and understandably so. From her point of view, she’s being squeezed in all directions. The developers are producing releasable code every month. But for her team to run a regression test cycle – mostly manual – takes 6 weeks. That’s too long. Just one test cycle exceeds the sprint length by 2 weeks. J. feels tremendous pressure to reduce the time it takes to test the software. Yet at the same time, she feels like she’s not getting any support to do the one thing she can see that will help reduce the test cycle time: automate the regression tests.
I’ve had some visibility into J.’s situation for some time now. J.’s team has been trying – and failing – to automate the regression suite for the last two years. They aren’t making any headway because as soon as they get one script working, another one breaks. The automation is brittle, error-prone, and incredibly expensive to create and maintain. That’s in part because they’ve been using a cumbersome commercial tool that doesn’t support creating maintainable tests. It’s also because the user interface was not designed with test automation in mind. Many UI elements don’t have IDs, and the ones that do use automatically generated IDs that change with each build. In short, the combination of the tool and the software under test
equals a test automation nightmare. It’s no wonder J.’s team is not making headway.
Yet J. persists. Doing more of the same kind of test automation that’s already failing doesn’t make much sense to me, but she disagrees. “We just need more time!” she says.
The problem is that J. is still thinking in terms of silos. She thinks all testing tasks must be done by QA people using specialized QA tools. It simply would not occur to her to suggest that development help automate tests. Nor does she suggest that developers and testers collaborate on making the UI more testable. Instead, she says, “QA can’t go that fast. Slow down.”
J. doesn’t want to acknowledge that test automation created by a siloed QA team working in isolation to reverse-engineer existing software and automate tests against an untestable UI using proprietary tools accessible only to a few select team members is guaranteed to be incredibly expensive both to create and to maintain, and also ridiculously fragile. In short, her approach just isn’t going to work.
Unfortunately, J.’s story is likely to have an unhappy ending – at least for J. and her team. Her strategy of trying to get development to slow down, and telling management that they can’t release monthly, is backfiring. The development team is already bypassing QA for small changes and getting good results. But J. is undeterred. My past observations tell me that no matter what the reaction of the people around her, she will keep doing the same thing and expect different results.
But maybe, just maybe, by telling J.’s story here, I can help someone, somewhere.
So allow me to repeat the moral of this story:
When QA works in isolation, creating automated tests after the software is theoretically “done,” using proprietary tools that are available only to a select few team members, the results will be a fragile, unmaintainable mess.
For test automation to work well, it must be created in collaboration with the whole team and the resulting test automation code must be treated as code. That means it should be versioned with the source code, executed with each and every build, and created and maintained as part of the overall development cycle rather than as an afterthought.
And when a Test/QA group insists on keeping within their silo when the rest of the organization adopts Agile practices, they will end up bypassed and irrelevant as the rest of the organization finds ways to move forward without their help.
I’ve observed a very similar situation from the development side. When development proactively met with the testers (who were working separately, for “organizational” reasons), we talked through what the obstacles to testing were. One that came up was lack of ids (or stable ids) on HTML elements. When we asked “would it help if we were to guarantee a stable set of element ids for each page?” the testers said were quite pleased and said ‘yes’, but seemed surprised. They’d never thought to ask for them.
I’m guessing this was mostly due to the artificial organizational barrier, but wonder if it might also have been that the testers where so chronically behind that they weren’t able to effectively think ahead to what would make things would make the situation better.
Moving from “we can’t, because ___” to “to be successful, we need ____” is a often such a puzzlingly difficult thing.
This situation is a typical trap when embracing the waterfall model. R&D starts development, releases a Beta, QA tests it and it all breaks down. One more release please. This cycle starts to spin faster as the release date is getting close – and leads to the opposite effect of what a QA Team wants to achieve. By doing internal releases faster and faster, releasing one Beta after another there is no serious regression testing, and no real verification of bugfixes, it’s just really bad for software quality. On one hand, the development department is stressed by fixing nasty bugs that may have influence to architectural changes that could have been overseen while planing the release – nobody is perfect. Those architectural changes at the end of a waterfall driven development process mostly leads to catastrophic results on product quality.
With agile processes on the other hand, it’s possible to embed the QA staff more closely to the development staff. It enables QA Engineers to dive into what is really happening and where are potentials hidden problems. It allows a whole different point of view. Understanding just a little bit about programming and how the development cycles work is quite easy, but it helps very much on daily QA work. Especially Scrum is quite nice to force cooperation of both camps and getting a lot of feedback.
Finally, when doing test automation, there is just no other way than breaking out of the crystal prison called QA Office and get down to dirt. Proprietary tools often promote the “record&playback” functionality to get the attention of C-Level-Decision-makers but mostly fail “in the wild”. On this point it is necessary for the QA Guys to start using a programming language and write automated tests. Computers are incredible fast and precise, there is no way a manual tester could ever achieve the performance of automated tests. Even if it starts with “we’re just doing some small test suites manually” – they will not let you escape from this, manual tests start to grow exponentially on every release. On that point you simply have to team up and start “real work”, there is no alternative to this. Software quality can be assured by manual tests if there are 100, maybe 200 Testcases, but on every test cycle performance, quality and measurable success starts to decrease.
I use a term “Automation Eco system” to indicate the nature of interactions involved in making a successful automation. Unfortunately, many people believe that automation can be done in silo. That is probably the main reason for failure of many automation initiatives – especially in IT services scenario. Today’s’ automation engineers need to come out of their “crystal” Prison (as Martin puts it, above) and interact with developers, ask for testability features, use developers knowledge of code.
Talking of automation ecosystem – one should not forget “stakeholders” – business sponsors. They would be having their own expectations from automation (influenced by Tool vendor’s sales presentation, by and large). If the automation initiative is driven without taking this fact into account, it will be a disaster….
Shrini
J’s situation is typical in a QA hostile environment. An important prerequisite before choosing UI test automation is to have a stable UI (includes stable object identities) and a stable application. Having missed to satisfy this criterion, J may either choose to build that pre-requisite through negotiations with product development team which seems to be a thin possibility.
The options that J may look for in the given environment are:
-> Adopting the right automation tool:
If J has budget to choose another automation tool, J may make a better feasibility analysis and adopt a COTS automation tool (probably like Worksoft Certify), that is capable of providing quick maintenance updates to automation scripts. There is no doubt that to reap the required benefits, the automation approach will need to be handled with care and expertise.
-> Develop a test harness:
If AUT supports, J’s team may build the capability (if absent) to develop a Test Harness/API Testing that can execute tests without UI interaction.
-> Offshore testing:
Many IT Service organizations with operations in Asia are now capable of providing staff with high expertise on QA services at half the cost of a resource in Europe and Americas. This provides an opportunity to double the strength of the QA Team at the same cost and hence the testing cycle can be completed in half the time.