In every project (well, nearly every one) there comes the moment, when testing gets squeezed for time. Immediately the next question becomes how to cut back testing in a sensible way.
The immediate reaction of many a tester (especially if she went through some kind formal training) goes a little like this:
Use Risk Based Testing!
I agree but sort of don’t…
Yes, assessing risk and focusing test cases or test scope or whatever you have planned to do is a good idea. Assessing risk is commonly seen as an easy and quick task, at least simpler and quicker than executing a number of test cases.
But this belief doesn’t even hold up to a first tentative inspection.
If you actually delve a bit into the field of assessing risk it becomes a quagmire really quickly. The amount of facets to assessing risk are endless. There are several books and write-ups about how to assess risk properly. I’d doubt that if you had read any of them you’d be so foolish as to suggest Risk Based Testing (RBT) for a quick and good solution to deal with time constraint.
Give it a try yourself!
Read James Bach’s really good write-up on Heuristic Risk-Based Testing and remember this is already a very good stab at simplifying the issue! Here’s the link http://www.satisfice.com/articles/hrbt.pdf . If this is a bit much to read then you can just scan across the pages to get an idea.
As you can see there is heaps to think about and assess to get some semi-valid inkling of what risk actually is. What becomes obvious is that it is definitely a non trivial task and involves not only testers but everyone in the project (and sometimes beyond). It will take time, be complex, be open to interpretation, yield false results because of human limitations & traits…
So basically if you haven’t done a really good job at assessing risk in the early phases of your project (and I have yet to see one project that did!) there is NO WAY that you will be doing proper RBT when it comes to the time to cut down testing scope.
Actually saying that you do/will would actually be a lie. What you are doing is a very high level and limited (not to say wrong) risk assessment and running with it. To call that Risk Based Testing is really a stretch. I’d prefer you not sell it as such. It would be feigning a scientific or structured approach and thereby border into the unethical.
Shocked yet?
Ok, so let’s go back to the problem at hand. What you really want is a good way to cut down testing scope and thereby effort.
For some odd reason RBT is seen as minimising the risk to the solution. Risk Based Testing actually INCREASES the risk remaining in the solution! The key idea is just to minimise the INCREASE in risk. Once you have understood that you can actually start to deal with the alternatives.
Since assessing true risk is so complex what can we do that might get us to our goal quicker but with much less effort and still keep the risk increase manageable? This is the point, where I mention the magic words CONTEXT and BRAIN. Use both and be inventive. I’ll give you a few easy ones to get you going.
- Test changes only
- Test only areas that have had high error occurrences in the past (or the converse thereof)
- Test only areas, that are exposed to users
- Test only areas, that you left out in the last test cycle
- Test only new test ideas and scrap the old that possibly were tested before
- Have a vote in your test team what they think should be tested
- The tests that use the most interfaces…
Or any one of dozens of ways of cutting and slicing testing/scope. Of course some of these go a long way to assess risk but on their own they are not RBT yet. You will still get most of where you wanted to be. In this way you will be clearer about what you’re doing and much more genuine.
Scrap the fashionable Risk Based Testing and describe what you will do in your context, why yod be doing so and what it will do for risk and testing scope/plan/timeframe.
Author: Oliver Erlewein
Hello Test World 
Gday Oliver, good blog however I wonder whether when assessing risk mitigation options we need to start at a higher level than the technical. I’ve found that strategically the mitigation action that appears the most obvious is not always the best & as you point out, assessing risk can be a quagmire. By simplifying at this level in terms of likelihood, impact & the ramifications on resources, time, cost & quality, the whole risk process can be streamlined down to the technical level & some of the options here quickly removed from the assessment picture. Sometimes then the best action becomes very obvious.
Geoff, by starting at a higher level… are we heading back to the problem that Oliver actually points out in the post? That you’ll then need to commence engagement of many stakeholders who can answer questions and provide information in relation to that ‘higher level’, therefore commencing the formal long winded process of RBT when it’s too late to do so. Or do you mean something else when you say “start at a higher level than the technical.”?
I’m meaning at strategic testing level as opposed to programme although sometimes it may be necessary to engage at that level. To effectively manage the approach, the authority to make the necessary adjustments without having to consult & seek approval is required. This is something as a Programme Test Manager I usually endeavour to negotiate upfront before taking a programme on. Sometimes I am successful, sometimes not & when not, this is where the additional enagement comes in that can slow the process.
This is meant for Geoff below!
That sounds awfully great but I don’t have a clue what you mean as I don’t understand these things by that name. So what exactly is strategic testing level? What is programme testing level for that matter? How do you actually then manage risk and what measures are at your disposal? Have you looked at all facets when you do that? And what exactly do you agree on up-front?
Fixed the blog now so more levels of response are available.
To clarify:
Strategic test level = I’m meaning the level that has the decision-making authority to change the test approach, strategy, method, resources etc as he/she deems fit in order to achieve the desired result. This is usually but not always a Test Manager.
Programme Test Manager = a senior test management role responsible for multiple testing streams/teams/managers/leads etc. eg. I was one of these for a large Aust telco programme of work, I had 2 Programme Test Managers reporting, each of whom had 4-6 Project Test Managers with teams of 3-10 test analysts each (total 100+ personnel). So I had the authority to set & make key testing decisions in order to meet my responsibilities & commitments to the Programme Director. This included working with the ProgTMs & ProjTMs to assess risks in their streams & agree mitigating actions that we deemed necessary to keep the overall programme on track. I did not require approval from Programme Director to make these changes although I would always advise him of what we were going to do as a matter of courtesy. If he thought something could be iffy, we’d talk through & resolve).
Negotiating up front = before accepting the telco role, I worked with Programme Director to have him agree that I could make these sorts of decisions without necessarily seek his approval.
Hope this helps.
Thanks for that. That puts it more into perspective.
What I suppose is that you had a very limited risk view, which you based your position upon. The perspective was from a testing management perspective. Again, that is a totally fair assumption to make but has nothing to do with the more conventional types of definition of RBT.
And this is basically what I am saying. Yes, you can take any risk assessment at all and shape your testing around that. But if you call that RBT, as many do (and you commendably haven’t done here!,) you’d be feigning a level of risk assessment that just wasn’t done.
I am propagating getting rid of RBT in common test term usage and for testers to actually be aware of what RBT really is with all it’s complexities. And as you pointed out, if you do RBT up-front it is actually a worthwhile (if possibly expensive) thing to do! Just don’t try to sell me RBT when we’re at the bottom of the cliff. For that it is certainly the wrong answer.
Hope that makes some sense. Thanks Geoff!
I think that’s the whole aim Oliver; to ensure that we’re not at the bottom of the cliff before we assess risks. Risks can eventuate at any level & an ideal plan is to have predefined what the risk management processes might be for each level. As you say sounds good in theory however is very rarely done in practice. RBT as you put it I think is far too global a term to really mean anything other than superficial prioritisation. How that prioritisation is arrived at will depend on the amount of effort put into it. I could stick my finger in the wind & say this is P1, that is P2 or I could go through the rigmarole of likelihood, impacts etc. What my original post was about was to avoid doing this at every level, do at the strategic level first & then the lower levels options may become much clearer & the same rigmarole will not be required at every stop.
Risk Management is a fascinating subject all on its own, outside of testing & outside of IT even. Yet the principles remain the same as for most disciplines; the better the effort applied = the better the result.
I should also point out that while I’ve seen and used many risk management tools, I’ve not yet come across one than handles multi-level risks.
Interesting Oliver…
Every action we take as humans is risk based, would you agree? Crossing the road you assess the risk, throwing your children in the air for playtime you assess the risk… whether consciously or not.
So when you say that testers aren’t doing RBT I notice your use of capitals and assume you are referring to a formal process of sorts? So, what is your definition of the formal process of RBT?
Personally I believe that all I ever do is risk based testing, as does any other tester… but I would not call it a formal process so I need to understand what you’re exactly referring to before I agree or perhaps disagree.
Hi Dave,
Completely correct! I see the perpetual assessment of risk in whatever we do. I think that’s probably even the way our brain makes decisions.
When we reference RBT, and what is meant by testers when they give t as an answer to a question, is a formalised, repeatable scientifically accurate process. Not the “gut-feel” type of risk assessment that you point to. Though when said tester gets to actually do RBT they consider little more than what I’d call a gut-feel assessment. My plea is, to up-skill on what formal RBT looks like and then see if you still want to call whatever you are doing RBT.
I don’t define RBT here as there are several flavours and interpretations out there. I suggest read the books, white papers and posts on the topic. They all have something in common though. They all describe a non trivial process. That’s what I’m on about. RBT is not the answer when you’re pressed for time. It is complex and needs serious amounts of effort.
My answer is, choose something simple and stick to it. Gut-feel in my book is 100% ok as a method. Especially if you are or have experienced testers you can consult. Their gut -feel will probably beat the pants off any formalised process anyone can come up with. Just don’t call it RBT because it isn’t.
And whenever you get the urge to tell someone (or write in a document -and I AM guilty of this-) to use RBT, stop and think if that is actually what you mean and mean to do.
Hope that answers that a bit 😉
Cheers Oliver
Yes Oliver, answers it perfectly.
I therefore agree with your post and your position on the subject. Instead of RBT perhaps we should go with GFT… gut feel testing (note the non-use of capitals… let’s keep it informal. ;0)