SoftEd wrote a blog post about UAT and how hard it was (here). I gave a longish reply and thought it might be good to re-iterate my thoughts on User Acceptance Testing (UAT) here on the blog.
I think the primary premise of what UAT should be, that we have here in Wellington/New Zealand, is wrong.
Most of our industry does it wrong. Why?
Because testers influence UAT too much!
UAT = User Acceptance Test
To me that means the actual user (not to be confused with stakeholders or managers) must assuage if the software developed is OK for him/her to use.
That means:
- That the application actually works, as proven by previous test stages (so not the actual mandate for UAT)
- UAT coverage is small and mainly UI – But that’s what most of these actual users actually see (Pareto principle).
- They should be doing what they do day to day – Because the corner cases should have been covered by Functional/System Testing and been proven correct (see next point).
- UAT is NOT supposed to find ANY defects/bugs! – The things that don’t work for them should be change requests. I.e. the primary requirement was at fault and not the software.
- Testers should not be involved in UAT – They are not needed, it’s not a testing phase (confusion in the name!). It is a milestone for requirements and their correctness. It should not be around the actual acceptance of the software, that should be a given by that point.
Because testing always gets pushed to the end (in waterfall projects at least) we look for where we can regain lost time. I think over the decades we have pushed our testing regime into UAT, until it became a test phase.
This is subverting the actual reason for this SDLC phase. As stated above, this phase proves Fit for Purpose (FfP). That answers the question “Is the software usable form a day-to-day business perspective”. If that FfP is influenced by open defects that’d be a catastrophe! As this phase is situated right at the end of the project, this is where fixing defects is the most expensive (bar production). If your project is still fighting defects at this point you should start questioning your project timeline. There is likely a severe issue with your SDLC/management/governance.
Things found in UAT should be functional or systematic changes that need to be made to future releases. I would not see them as fixes. In the context of the project the things UAT should find are changes to requirements or added requirements (stuff that has been missed).
The SoftEd post talks about why UAT is the hardest testing. I think that actually misses the point and enforces the bad habit of seeing UAT as a testing phase, with testing methodologies, testing oversight and control.
I stipulate that all that will achieve is…
- do more system testing on a much higher and therefore worthless level (at that point of the project)
- with people who aren’t even testers
- totally miss the point of highlighting the inadequacies of the software needed for actual implementation and use by the customers/users
What has happened seems to be, that we have actually lost (without replacement) a vital stage of the Go Live process. The software process improvement loop. It has been shoehorned into a testing phase. I don’t want to question that there might actually be another test phase needed that does what we do in UAT today but we should then not lose the actual intention of UAT and the phase associated to it.
Author: Oliver Erlewein
Hello Test World 
Good post.
In the projects I coach testmanagers and testers (mostly banking & insurances) typically some end-users with a very good domain-knowledge are picked as testers and actually do a mixture of refining requirements and system testing.
And developers sometimes even dare to call it agile or customer-on-site but in fact IMO they are not doing their job correctly – there are many bugs found in that late phase – even very obious ones resulting from incorrect system interfaces.
Real UAT is something very different.
And all other “methodologies” are miles away from agile development – especially IT’s waterfall idea to present the final solution at the end of the project – sometimes months and years away from the very first requirements document. And consequently – “Embrace change” could be better coined as “Tell me that I understood everything right, do not say anything different, otherwise we need to go to some steering commitee.”
Well – I’d love to coach real UAT testers and real system testers in (and not outside) IT.
Maud
BTW: UAT is according to IEEE 610 defined as:
Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the
system.
Well, IEEE is vague as can be. It here hinges upon what you see as acceptance criteria. I mention Fit for Purpose above. I don’t think that is stipulated anywhere. I see users using the system and raising what is an issue to them as FfP. Per definition this is outside of the original requirements. As I said, I see this as raising change requests/added functionality (and a decision being made if these need to be introduced before GoLive).
I’d bet that IEEE is thinking of fulfilled requirements, when they use the word acceptance (although that contradicts with the 1st part of the sentence!). I don’t see it like that. That to me is something that should have exhaustively been covered by previous test phases (or what else have they been doing then?).
Interesting article. I agree with what you say. I actually don’t even believe in a UAT phase as it’s too late to be finding problems with the requirements. I prefer to get the business users involved in the creation of the system testing in the first place. That way our system tests actually reflect business usage.
Happy for them to waste time retesting (or as you say revalidating) that the system workflow is as expected in a phase prior to releasing to production, however I’d suggest you’ve got major issues with your BA’s and Testers in the requirements and testing stages if this wasn’t discussed and identified earlier.
Prototyping generally helps to give business users a feel of what they’ll be getting months down the track, and whilst it doesn’t prevent minor CR’s from getting through, it should predominantly mitigate most of them.
Hi Jason,
I here describe the current process and what went wrong in response to SoftEd’s blog post.
If you’d ask me what a better SDLC would look like I’d certainly be with you in less than 1 second. I agree to the issues you see. We’re using tools of the 60ies to develop modern projects. This bad behaviour is enforced by certifications, outdated diplomas and a belief in “we’ve always been doing it this way”. I guess, that’s where Agile really came from (if I understood Alistair Cockburn correctly on Tuesday ;-)). But change is slow… even in IT.
Cheers Oliver
Brilliant arguments Oliver! Totally agree.
As you know I’ve worked in UAT and I can say from experience that it is not fun. IMO, UAT is at its worst when companies use it as a safety net for poor SIT. I once asked a UAT Manager why do we need to spend 6 weeks doing UAT, with a team of testers that are inexperienced in the software being tested, and the reply was: ‘We don’t trust the testing done by SIT’. So instead of working on whatever was wrong with SIT, they implemented a whole new team and strategies to pretty much do SIT all over again, but by a different team of people. Crazy or what?