Somewhere I saw a statement that testing today is about 25% of IT budgets with a tendency to increase. Not sure if that’s true but even half of that is a lot of money. So of course managers and financial controllers are looking where they can eek out any spare $s.

Because testing and quality are not tangible or easily understood by outsiders it is tempting to cut and slash. There are no immediate or obvious down sides (50% of testing is still testing right?). Issues appear later and then the correlation back to the cause is spurious and it’s the operational budget then. And we all know testers are just all doom and gloom.

Therefore Test automation is being sold to (mostly non testing) managers to be just as sexy as that bikini model in the gloss magazines. The promises they drive home are simple and simplistic.

• It will save you money because you can do more with less people
• It will accellerate your testing timeframe
• It’s easy and anyone can do it (especially with their tools/training/… whatever is being sold)
• It’s the future (i.e. you don’t want to be caught out not doing it)
• Record and Playback (hopefully died out with the end of the 90ies but still rears it’s head)
• If you’re a tester get on board or get out of the way
• There is no agile/scrum without test automation
• There is no continuuous integration and deployment without test automation
• “Best practice”
• It will give you unicorns and rainows (…ahhh no, haven’t read that anywhere …yet)

Now isn’t that what every manager wants to hear? I can reduce my headcount and everything will be done with a push of a button while decreasing my risk and timeframe!!

If that were all 100% true even I’d go for it without a moment’s hesitation. But because I know what test automation really looks like in practice I can assure you it isn’t true at all. None of it. It’s sales! And I think it borders on the unethical (not sure where sales and advertising should/or need to be ethical).

So let me give you the reason for this rant of a blog post. It is another blog post here http://blog.softed.com/2016/01/11/the-impending-extinction-of-testers

In 2014 we saw the heyday of all the “Testing is Dead” hype. There had been lots of discussion around that and in the end it basically came down to the fact that it was little more than clickbait. Just as a disclaimer, SoftEd is a reputable company and I do think this is not characteristic for them. I also just picked them as an example. The reason I’m upset is because it’s in my own back yard. It threatens the message about test automation me and a lot of other people I know have been trying to send.

The thing is these blogposts hurt us as a profession. They confirm simplistic ideas about testing and test automation. They confuse checking with testing and that makes testing seams easy, able to be done by a monkey or machine for that matter. Managers (and testers) that follow such advice will fail and that again reflects upon testing and automation negatively. Possibly even confirming the preconception that testing is really not worth doing.

So let’s go into details what is claimed in the blog post:

1. “Impeding Extinction of Testers” – If you don’t do test automation you will not have a job as a tester, highlighted by the meteor analogy. As the author points out later it is actually good testing or better methodologies used by actual testers that drive the finding of bugs. All the methods listed and the ones I am thinking of require a human in some form or another. In the foreseeable future nothing will change that (even if we can build self driving cars).
2. “What is the driver of this doom and gloom?  Cost!  The desire to get more bang for your buck.” – The test automation saves money argument. It doesn’t! Test Automation covers a vital aspect of testing (checking!) but it does not replace anything so it is always added cost on top of testing! Savings can be had for things like aiding data creation for testers, having standard processes automated and being able to quickly smoke test environments/builds. But this comes at a high build, maintenance, tool and personell cost, which by far overshaddows personell savings. The thing is the features and advantages that test automation brings can still be desirable even if they increase cost. That depends on what your project is trying to achieve. What I am highlighting is that there is a connection between automation and head count.
3. “As the manager of that team, I would retain one member of the species to write the test scripts and analyse and interpret the results, but down-size (use your preferred euphemism here) the remaining four members.” – Downsizing of resources (direct link to 2.). I think this is what managers want to hear and it is the biggest fallacy. It is just not true. Yes, eventually if you reach a higher maturity grade of your WHOLE SDLC you will be able to streamline resources and save money but in my 20y in IT I’ve never really seen it happen. Mostly because that would necessitate a finite scope and all companies/projects I know end when they are “done” or they keep going with extended scope.
4. “Why would I pay for five testers when the simple act of following a test script can be performed by almost any semi-skilled labour at a significantly lower cost?” – Here the oversimplification of testing and the confusion of checking and testing. i.e. comparing test automation to what I’d call just plain bad testing (or not testing at all). In this scenario test automation might actually do all the things claimed but I’d argue that you have far bigger issues and all you are doing is literally “automating monkeys” and that doesn’t sound very IT or professional to me.
5. “…time that can be spent on tasks that cannot be automated, such as exploratory testing. Such techniques yield a richer crop of bugs…” – a weird one. So we should automate and do away with bad testing and “install” -what I’d call- good testing (provided the author actually knows what exploratory testing is). And then the blog admits that it is actually those methodologies, that find the bugs (and NOT automation!!). I am getting confused. So why am I going to do test automation again?
6. “As a manager, I would want testers who can offer that bit extra beyond their colleagues, who can find those more elusive bugs, who yield better productivity because they automate all the repetitive tasks that they can.” – As above this doesn’t make sense in context. The article is now  saying the value is in doing good testing, which I totally agree with! But I fail to see why test automation should be a precursor to that? I can save the time to do good testing by just stopping the “mundane testing” that is described.
7. There is no detail on what test automation is being talked about. There is automation at a unit testing level, at a API level, UI level, Performance, Security,… all of these are different and require different skills and have distinctly different goals. Some necessitate automation just from a technical perspective. Some is done by developers, some by testers. So when we talk automation and make specific claims shouldn’t we specify exactly what automation we actually mean? (I’ve made myself guilty of this in this blog post too by the way).

Just to re-iterate. The SoftEd blog post is an example here, that just happens to be in my own back yard. These arguments are by no means an exception but a widespread belief in the IT industry. It creates a lot of pain, cost and unmet expectations for companies, people and projects everywhere. This is why I can’t let it slide. We cannot perpetuate this belief. I know SoftEd actually doesn’t act like what what they wrote (and there is a second post that outlines this somewhat better here: http://blog.softed.com/2016/02/15/the-future-of-testing-2/) but the damage is done as long as the article is up. Managers reading that will gladly abuse their confirmation bias, never reading the comments below that refute the claims.

Now, I have commented on the article too. My comment was never approved (i.e. is not visible). Probably best it’s left that way because I did loose my cool somewhat. This post reflects far better what I wanted to say. But I did ask SoftEd to retract the post in the comment and do so here again. As long as it is up it is potentially harming our industry. The second blog post is not likely to change that and I am by no means saying you shouldn’t train test automation or rather let’s call it tool use! Just the opposite. I fully support more people getting into it. We just need to sell it for what it actually is and have some realistic expectations that we communicate to management and stakeholders. Especially those people outside the know.

I’m a huge proponent of test automation but under the right circumstances/context. I don’t want the Snake Oil version sold. What I’d like people (that is testers, managers, developers,…) to do is invest some time researching the topic. I’d advise starting here: http://www.satisfice.com/articles/cdt-automation.pdf and then go from there.

If you have any detailed questions or want to know more please discuss in the comments or contact me directly. I am always glad to help. As will be a lot of other testers out there that I can refer to.

by Oliver Erlewein