On Tuesday night, Steve Willsher (QualIT) gave his best to bring ISO/IEC 29119  closer to our testing hearts at a NZCS TPN presentation. Kudos for him to really have a go at the beast that is standardisation. I am part of the working group and am at least familiar with the documents being produced. The thing is that I’m not (yet?) sold on standardization in testing and IT in general, where the intent is that it applies everywhere. In this post I’d like to share some of my concerns and elicit some debate on the topic.

The first thing I’d like to highlight is the way a standard is “born” (note the use of quotation marks here, as in the title around the word – new.) All going well ISO 29119 as a standard will be released, at it’s earliest, May 2013. This will be after already being on a 2 year journey involving lots of groups, discussions and opinions. In the last 5 years I have seen quite a few revolutions/evolutions in testing and IT SDLC in general. Testing, and IT, is still in it’s infancy and new ideas are cropping up daily, so it eludes me how the standardisation process is not being overrun by current events. The risk of the standard being out of date when it gets released is immense. The timelines for standardization might be okay for other engineering disciplines but IT moves at a faster pace (also accelerating other fields but not quite to this speed). I think the process is too unwieldy.

The four documents created by the ISO 29119 standard are a guideline and the content is not required to be used in it’s entirety. The next step in the standardisation process is to define when someone (or an organisation) can claim to be standard compliant. This would need to be a definition of a minimal subset of the standard being implemented. The timeline for this step is not known yet but certainly will be later than 2013 from what I can deduce.

What I also find problematic is the visability about who can see the work in progress. I can fully understand why documents are only visible to the working group members. Anyone with a vested interest can become a member BUT that is a circular argument of sorts. How do I know when I have a comment but I can’t see what’s going on and just joining the working group on the hunch I might have a comment is a hard ask. Doesn’t this problem water down the effectiveness and quality of the standard as potentially the right people never get involved?

Standards are also a process of compromise from what I gathered from the talk and comments made by committee members. So the standard represents something a lot of people can agree on. That is contrary to a common perception of standards though. We expect a standard to mean it is something special and difficult to achieve; a hurdle that needs to be cleared so that we can depend on it. From what I gathered it might not be that way with software standards (hopefully I am wrong about the other standards!). The problem is similar with test certifications in that the *testing public* projects what they assume about standards to certification and might be quite mistaken in doing so.

Then there’s the whole list of claims about what advantage a standard has (the following is a straight copy from the presentation). Here I just want to pick on two although the second argument is actually two arguments.

1. Standards shorten timeframes
2. Standards reduce cost and complexity

Timeframes and cost go hand in hand. Standards to me are a prescriptive and detailed way of doing something. That in itself can be very good but usually there is also a heuristic way of achieving a goal without a standard. I fail to see how a more prescriptive approach is faster and cheaper than a heuristic and/or exploratory approach. There are certainly projects and situations, where this maybe true but generally I’d disagree for obvious reasons.

So what about complexity? First of all, what degree of complexity are we talking about here? The only one that makes sense is testing complexity and there I’d argue that never changes without changing the application under test, the quality criteria and/or the requirements. So how can standards make it less complex. They might be able to show and deal with complexity in a more prescriptive way but the standard will never be able to do away with it.

But one thing Steve said made me wonder if I’m thinking standards in the right way. I asked how ISO standards in testing should be applied if no other team is following ISO standards?

He said something like “then you don’t need the ISO testing standard”.

I don’t think he meant it quite that way but the gist is in my opinion correct. In the SDLC, often the testing phase is stuck with more precieved rigor, documentation and procedures than any other phase. I’m not saying I know the reason for this but this may be due to a misinterpretation of what testing is and that coming down hard on testing will save the day. The point I’m trying to highlight here is a common one and is only highlighted by standardization. Like the chain is defined by it’s weakest link, so it is in the SDLC. What’s the point in having an ISO standard for testing when, generally speaking, there is very little adherence in any other discipline. What is then the benefit to testing?

On the other side, I think standards are necessary for certain types of testing. I’d like to fly in a plane that also has testing done that adheres to some standards. But standards are certainly no cure (or for that matter the cheapest thing to do.) So I’d like to address stakeholders when talking about standards to please don’t ask to implement them when you don’t have a clue what they are! You’re potentially inviting a bevy of issues and risks into your projects. And, as always, don’t just believe what it says on the package. Listen to your SMEs and reason. Then choose.

So standards have some value in my opinion but as you can see I’m not fully sold. Maybe there needs to be more discussion and awareness on this topic on a much broader front and, who knows, I might be swayed.

Author: Oliver