A little bit of history. Novopay is the second Ministerial Inquiry into an IT project in New Zealand that I am aware of. The first one was the INCIS project from the 90/00′ies run by the Police. The difference between the two is that this report was actually supported by all parties involved, and it is on a project that actually went live.

Anyway, I don’t want to berate MoE or Talent2. I do want to discuss the general issues I see in  many projects and my take on what it means and sometimes how it applies to testers or testing.

For those readers not from NZ, you might need to catch up a bit on the topic by Googling for some press reports on Novopay.

I have read quite a bit into the publicised documentation on the Novopay project pre inquiry. Of course I zeroed in on the testing documentation. What immediately struck me was that there was actually a lot there. Defects were raised and put to stakeholders. I found areas of defects that sounded a lot like the production issues they were having.

I noted that the really interesting defects were coming from an area that was trying to run end of day successions to emulate what would happen later in production. Oddly this testing was stopped by management to prioritise other testing.

I also noted the grim reports from testing and the decisions by stakeholders to ignore them and paint rosy pictures. So I was awaiting the official review with interest.

The report is a resounding slap in the face of the governance and leadership of the whole project. It highlights bad practices that I have seen on so many projects (public and private). It seems to be an industry sickness. So let’s take some of the things said (not necessarily in order):

• This state of affairs and the wider disruptions that were caused were avoidable. It is clear to us that important lessons from the past, in particular those arising from the 1996 education payroll implementation difficulties and the INCIS experience in 2000, should have been learned, but were not.

Well, there there’s the answer to my blog topic question. We don’t!

There are things like the INCIS report and other lessons learned from other projects. Do we use them? NO. I have read the summary to the INCIS report and it annoys me to no end to see how many people in stakeholder positions don’t know this report. INCIS and now this report should be mandatory reading for everyone in IT and those connected to IT projects. Repeating these errors is gross negligence when they have been so clearly defined and in the public domain.

From a testing perspective, we could also see ourselves in a Quality Assurance (QA) role (although I fundamentally don’t agree testers have anything to do with QA but I’ll skip that rant here). If we so do, then we’d be tasked to highlight breakdowns in process(es) too. That would mean the testers/QA staff on this project would have been well served to know the INCIS report and address the points that now both reports have in common with stakeholders.

• It is our overall view that weaknesses in project governance and leadership allowed the service to go live with a number of significant risks which the Ministry and its vendors were over-confident of managing.
• …which invited Ministers English, Parata and Foss to approve the
  continuation of the project following Confidence Point Two, misrepresented its state.
• We reviewed the papers submitted to Ministers and presented to Cabinet, to understand the
  information, risks and advice that were presented and where decisions were made, and found
  that Ministers were not always well served. Reporting to Ministers has been inconsistent, at
  times unduly optimistic and sometimes misrepresented the situation.

So this to me translates to someone knowingly and with intent lying to the approving stakeholders. I don’t know about you but such behaviour to me is unacceptable.  Yes, you might support the little white lie to achieve something short term but this is a breach of ethics on a massive scale. People involved in IT projects usually have some sort of education, certification, belong to industry groups or similar. All of these have ethical charters. Even if some of them are very short they all contain something that forbids misrepresentation and lying.

In my career as a tester I have seen lots of behaviour like this. Some incidents are small, others larger. We constantly get faced with these situations in projects. How do we react? Our ethical standards actually mandate us to expose such behaviour and deal with it openly. Anyone who has been on a project knows how impossible that is. You recognise this happening when the voice in your head is saying “think of your mortgage!”.

Interestingly enough, the report includes a recommendation going forward, that there is a direct feedback from QA/testing to the top stakeholders. This is a good idea but only works if the misrepresentation doesn’t happen on that level too.

One of my pet peeves is that testing reports to project management. This is counter productive as testing is a direct danger to the project go-live. Good testing -that highlights lots of issues- makes the life of a PM and that of a project very difficult. It has the ability to actually break projects. If the “unfiltered” test reports would have reached Cabinet or the Ministers there would have been a good chance the project would have been delayed or worse. This (recommended) action would have invalidated the goals of the project though. That means a PM would have done everything in his power to counteract what testers find. Fixing defects is hard, time consuming and delays a project. Hiding or explaining them away is much easier.

You could thereby deduce that the groundwork of the deception’s was laid by the goals set out to achieve by the project and the actual project structure. It’s easy to set out to do a project on time and in budget. The hard things to do are in scope and quality. These latter are hard to measure, to define, and to get water tight.

A PM should also act in the wider interest than just the project. A project has a specific external context within which it lives. This can be towards it’s company, customers, stakeholders, local conditions, competitors, the economy and others. Ideally, if any of these apply, they should be integrated into the project goals. If they are not, or are treated as implicit, they still should be considered.

• Over the course of the project, Talent2 had missed agreed milestones or deadlines, which eroded trust and confidence in its ability to deliver.

I encounter this on many projects. Either milestones get missed or entry/exit criteria are ignored but everything is still full-steam-ahead. One has to wonder, why these milestones are made in the first place. So the lesson learned here is: No milestone/criteria without consequence(s). I am not talking about blame or money. What is happening is the project context has changed. There needs to be change to re-adapt. The thing I can guarantee with unerring certainty is that not changing the path will not get you where you want to go.

Just imagine as a tester you don’t get the promised environment you have stipulated that is needed for testing. This will have severe implications for quality, effort, validity of tests and much more. Thereby at some point testing will get into a compromised situation. Stakeholders are happy to have resolved the bothersome issue of testing but don’t realise the added, very significant, amount of risk that they have now taken on board.

• Despite the problems, we observed a strong commitment to delivering a successful project and some significant individual efforts.

This was intended as a positive remark but I am unsure it is. A strong commitment is, what gets most projects into trouble. Delivery to a certain date becomes so important that any rationale that advises caution is futile.

[Update] All IT projects require a skilled staff who uses their initiative and commitment to solve (poorly) structured problems. Sometimes though heroes emerge. If these heroes are needed to just get the basic project out the door this points to a dysfunctional project.  The project is suffering from bad governance and leadership, caused by a lack of delegation, planning, overly optimistic timelines, solution complexity, and/or lack of enough specialised know-how in staff. [Update]

• Work commenced on the requirements for the schools payroll project in October 2008. This process was lengthy, and was never actually completed. Even after Go Live, new requirements were being discovered.
• Requirements definition, design, development and testing activity were all occurring in parallel, making it very difficult to maintain a known level of quality.

We are talking big Waterfall type projects. They do prescribe requirements to be finished before anything else begins. This is the 1st sign of danger projects usually display. Hands up, who has been on a project, where this was the case? (Everybody hands down now, please.)

Overlap and change cannot be prevented but can only be tolerated to a certain level. Once we get a statement like the above, where all Waterfall phases are going at once you are kidding yourself if you think you will come out unscathed. This kind of behaviour happens because of unrealistic/overconfident timelines and overconfident vendors/marketing gurus believe she’ll be right (also sometimes a misplaced and forced “all is well” attitude).

Testing in this scenario has become a farce. It is not uncommon that 70% of testing effort has been completely wasted by the time the project ends. In the testing section of the report there is actually a hint of this:

• …poor requirements definition, early testing was wasted.

In clear-text that means, if everyone would have sat around and waited until requirements definition would have ended and then started testing they would have done just as bad a job (i.e. a really stellar job under such chaotic circumstances!).

• There was little direct customer (boards of trustees) or user (principals and school administrators) involvement in the definition of the requirements, and Datacom’s involvement was minimal.
• The execution of the change management plans which the Ministry did have was inadequate, and roles were unclear. The engagement with the payroll service’s customers and users was also insufficient.

Hmmm… so basically nobody bothered to find out what/who the oracles were in order to interview them to find out what this thing should do. Testers can help here! We have learned to find and respect oracles. We act upon their information. Maybe it is time to spread the concept of oracles a bit wider in the SDLC.

• During the service design and development phase, the intended pilot and phased rollout of the service were removed from the project plan.
• Some important areas of functionality were not fully tested prior to Go Live. Some types of testing were not completed to the original scope, on the basis that testing could be completed after Go Live, or that the risks of not doing the testing had been adequately mitigated. Not all System Integration Testing criteria were met.

Also a thing I see time and time again. Scope, especially in testing, is decreased without replacement or risk mitigation. Everybody notices if development effort is cut. Basically functionality will be missing. Cutting testing / QA effort impacts output quality but this is not directly traceable and has a delayed impact. You can actually shift effort and problems from the project(budget) to BAU (budget). From what I can see 100% of projects that try this seem to get away with it.

Then there is the big myth of after Go-Live.

After Go Live the project will be shut down. There will be no testing or anything else going on. It’s a nice way of saying “we won’t do it” without spooking the steering committee.  If you’re lucky there is a project phase 2, that will -begrudgingly- accept this into their scope (don’t fret, phase 2 will never materialise as phase 1 will have failed).

• …meeting of the Project Board, the Novopay Business Owner indicated that a Go Live date of 14 August 2012 was the “absolutely last preferred date”.

“Last preferred”? Is this an order to comply or not? The word preferred is what annoys me. What are the factual reasons for this Go-Live? Preference is akin to whim and that is not enough of a reason. If, on a project like Novopay, I need to Go Live in a critical state I need to be able to assuage my assumptions and risks against why we are doing this. How can I substantiate what risk level something has if I can’t back it up by a tangible goal? This should not be a difficult exercise. Actually it should be easy. Stakeholders have reasons for doing and wanting things. These just need to be communicated. With this transparency, solutions might be viable that were not even thought of before.

• The real Go Live decision was made on 31 May 2012, despite the Confidence Point Two criteria not having been met and schools not being ready. Project governance and leadership allowed a combination of significant risks to be carried into Go Live and overestimated the ability of the Ministry, Talent2 and schools to manage them.
• systems development was continuing through the code freeze right up to Go Live;
• Talent2 missed agreed milestones and deadlines. The Ministry had cause to invoke breaches of the contract for non-delivery from as early as 2010, but did not exercise this option.
• The Ministry was not always willing to take or act on advice, and at times demonstrated misplaced optimism about the state of the project.

Projects are weird beasts. They are stressfuland ruthless and on the other hand, when it comes to making some consequences, everyone backs off and does things like the quoted above. Again this is ignoring change and not reacting appropriately. The logic of reasoning behind such decisions has eluded me so far. I cannot fathom that things like Prince2 and PMP condone such decision making in their relevant processes (and I don’t think they do). Again it is not about blame but dealing with reality in an appropriate manner.

Maybe therein lies the issue. Humanity is averse to dealing with failure. As  a tester we have accepted that failure is something good, especially if it happens early in a process. You can fix, learn and grow by dealing with failure. From a holistic project view I cannot see the same understanding. Failure is demonised and feared. You have to wonder why though. There is the omnipresent statistic that is quoted in next to every keynote at an IT conference: “90% of projects fail”. So if you can’t deal with failure there is a 9 in 10 chance you’ll have a serious issue with the project you’re on. If 9 out of 10 projects fail, why don’t we focus on dealing with failure?

Some people will now refer to their risks and issues registers. Theoretically that is a good idea, but I have not really seen them work. I think it is better to name the beast. If you start from the premise that your project will fail (project failure in my definition is: over budget, reduced scope, insufficient quality or not in time), you can now start thinking about how it will fail and start to actively mitigate.

In comparison I see that risks get raised and then people ignore them until they become issues (and sometimes after that too). By the time the risk becomes an issue the remediation is complex and expensive. Active and early  mitigation might have easily prevented the risk from becoming an issue or have introduced early adaptation to a changing context.

• …there was no overall accountability for Independent Quality Assurance,…

I actually wonder if this accountability needs to be independent. Projects often forget the fourth corner of the balancing square (it’s not a triangle like most literature points out), which is quality. Quality is hard to define at the best of times but everyone knows, when it isn’t achieved. More than accountability is a careful management of quality expectations and the clear communication of these. Testers are not the beholders of quality. Quality is a whole project thing. Everyone involved needs to know these expectations. Trying to retro-fit quality by failures in testing will definitely kill your project. When we get to testing we should be testing that quality is present and highlight where there are omissions only.

• The Novopay project has cost materially more than estimated. Benefits have yet to be fully realised, and in some cases may not be. The complete cost of implementing and establishing Novopay, including the real costs to all stakeholders, substantially exceeds the reported overruns to date. Value for money thinking and expenditure control have been weak.

Feasibility studies are something that you never get to see as a tester (if they even exist, I did hear rumours).  These studies should list in detail, what is expected of a solution and how that will generate a return on the investment your making. Companies do IT projects so they can directly or indirectly make money. Nobody is in this game for just the good of mankind. So all goals and requirements should somehow tie into this feasibility study.

So why are testers not privy to this information? I could actually start calculating the monetary risk or monetary consequences of testing actions or the omission thereof. Why are testers (with their excellent critical thinking abilities)  not invited to scrutinise such documents and calculations?

When the time comes to cut testing effort (see below) such a document would be priceless. You could evaluate what to test and what not and to prepare for the fallout. If Novopay would have done this, I think  they would have focussed on different areas of the application/solution. The huge benefit would have been that the actual financial risk or opportunity could have been given to the steering committee/ministers. That would have made their decision making much easier and more founded.

• By December 2009 Talent2 was advising that the project would not be delivered on time because of delays with testing…

This sentence can be found in reports from many a project. I always balk at such a sentence. What it states is, that testing was delayed. i.e. there is something that happened within testing that caused the time-frame to blow out. There is a second interpretation though, which is more accurate and true but is intentionally hidden in wordplay of this sort. What I am talking about is the interpretation as “testing was delayed”.

So let’s look at a few(!!) reasons why testing can be “delayed”:

1. Late delivery of code
2. Late delivery of a working environment
3. Late delivery and/or changing requirements
4. Low quality code
5. Low quality requirements
6. External factors (e.g. the Christchurch earthquake and the tons of long term issues that it caused)
7. Failure to adapt to change, thereby making existing goals/timelines fictional (this is a project management issue)
8. Bad test preparation (although I usually see this as a consequence of 3 & 5)
9. Bad testing methodology
10. Bad test execution
11. Bad testers
12. …

As you can clearly see there are a lot of reasons that are actually not directly connected to anything testers are doing (1-7). In most projects (like on this one) it is obvious that a combination of points 1 to 7 are actually true. Very rarely things like points 8 to 11 are listed as the issues. And if they are (like on this project) they actually quite easily get mitigated.

So when you comment that “delays in testing” are an issue you’re more likely admitting to a complete project lifecycle/management failure. It’s like blaming the tail-lights on a car for a frontal collision.

•  Novopay should not, however, be regarded as symptomatic of all public sector ICT projects. 

This sentence surprised me. For all the rational and excellent work of the Inquiry in  the report, this sentence does not fit the mold.

It is a political sentence. It is intended to assuage any tendencies to investigate any other project. As much as I can understand that sentiment I do think this is making them guilty of the thing they critcise (“at
times unduly optimistic and sometimes misrepresented the situation“). Yes, surely not all projects are as bad as Novopay but actually  a lot of them are (remember, 90% of projects fail!). This sentence absolves a lot of projects, where it shouldn’t. It also lessens the chances of these projects getting the support they need to become successful. I’d rather have applauded an initiative like they did on the Government security breaches, where a stock-take of all projects over a certain $$$ figure would be reviewed.

Even the projects listed in the report that are successes weren’t in the sense of the definition I gave above (of course there is a lot of speculation here on my side and it depends highly on my definition of success). Don’t get me wrong, I do think they are successes in the wider sense and we can be proud of them but without the Ministerial report outlining what -to them- defines a success, I think they are on shaky ground making that statement.

So in conclusion…

All in all I think this is a great report, if not stellar. Nothing is perfect and being a tester I do pick up on this more than I probably should. The reports, in my opinion,  highlights typical issues in large public & private projects. Everyone should know this report and probably re-read it before starting on a venture like this. While at it, add the INCIS report to that reading list, too.  I really hope these reports will actually make us learn from history, give us a modicum of self criticism and make us embrace a culture of failure in order to succeed.

I know I am no die hard PM or manager but I do know these issues affect me in my testing career every day. They annoy me, they make be uninspired and less effective. I know I am not alone in this. I hope that this report will generate lots of discussion, articles, books,… and that we start dealing openly with the issues at hand.

I will write another post later on the actual recommendations of the report and what I think is a solution.

Author: Oliver Erlewein

All relevant Links:

http://inquiry.novopay.govt.nz/

http://www.novopay.govt.nz/

http://www.minedu.govt.nz/

http://www.talent2.com/home

http://en.wikipedia.org/wiki/INCIS

http://www.minedu.govt.nz/theMinistry/NovopayProject/TestingNovopay.aspx

https://www.google.co.nz/search?q=Novopay&oq=Novopay

I had very high expectations; my goals going in were:
- Complete an entrance examination challenge set down my Miagi-Do Black Belt Instructor, Markus Gaertner
- To facilitate presentation sessions and further develop my facilitation skills
- To specifically meet Michael Bolton and form an impression of him (also to clone him, and bring him back to Sydney)
- To meet and bond with other testers who I had worked with on BBST and Bug Advocacy courses (Illari, Levi, Iain McCowatt, Markus)
- To sample the local beverages, network, and generally hang out causing some mischief
- To take part in challenges, games and entertainment

The presentations were very informative and showed off the depth of experience and testing talent that so many have. Here is a sample of the ones that moved me.

“Have your own methodology”
James Bach: Keynote on “How Do I know I am Context Driven?”
Some resonating points:
- CDT has 3 facets; paradigm, community and approach.
- Our community needs greeters and guides.
- Seek more than shallow-agreement, where important.
But most importantly for me, James spoke about how testers should have their own methodology. I was so happy to hear this, as I  have meet many testers who do not. I have recently announced my own which is about people, learning and mindsets. James’ keynote was an enjoyable topic, loaded with concepts that are critical to a tester in todays context. Every tester should view the slide deck, and the resources on the Satisfice site.

“Safety Language lesson”
Turorial: Michael Bolton’s “What’s the Problem?”
I enjoyed Michael’s approach to tutorials. At first they seem slow and cumbersome, but then things start unravelling at a faster pace. I liked the somewhat open group exercises which included not only having to put the question in to context, but also self-organise and get the group high performing. It was fun doing the triangle challenge at classroom level, as we got a load of different inputs to the problem. I will definitely be attending more of Michael’s classes and presentations.

“Test this”
Presentation: Test Planner by Louise Perold
I thoroughly enjoyed this presentation. I got to facilitate it, and take part a little. I absolutely love test ideas as a basis for test execution. Several years back, myself and a couple of other Kiwis had the idea of testing a brick at a conference workshop. Lou went one better than that and tested Beer! It went down a treat. And I suspect a lot of people went away with a load of new ideas to try as part of their methodology.

As well as classroom lessons, there was a lot of activity outside of presentations. For me, most of my learning and fun came from outside of the classroom. This is what I really got up to at Let’s Test.

“Paul, meet the Paul Holland of New Zealand” – James Bach
This was how James Bach introduced me to Paul Holland. Since that introduction, I think we have been best pals! Paul was a major highlight for me at this conference. As well as his great value humour and playing about, Paul is one heck of a smart and experienced tester. He showed me the ropes of facilitation and he and I formed a team in the test lab on the first night. I really enjoyed seeing his approach to testing, questioning in presentations, and performing his own presentation on metrics. I look forward to working with Paul again at CAST 2013.

“Richard, your Miagi-Do challenge is…” – Markus Gaertner
This was my biggest highlight of Let’s Test. It’s true that what happens outside of the presentations can also have much value. I had heard of Miagi-Do before, but knew little more than that. When I heard part of the examination was a physical testing challenge, something ignited in me. I pressed Markus in the days leading up to Let’s Test for my challenge. One day, Markus came over to me and told me to recruit a team, test a product in the test lab, and report. I loved this idea, and went to work immediately. I was lucky enough to recruit a very high-performing team of four (Erik Bricarp, Guy Mason(@testingqa), Peter H-L, and Ida Waller(@Linix81) ). I was so lucky to have such a dedicated team who questioned, supported each other, and were enthusiastic. The debrief with Markus and the team was emotional for me. The feedback confirmed what I was actually trying to achieve. The secret to a high performing team for me is to drive the correct mindset, empower the members, and don’t forget to have fun. My unwritten goal is to have people smile as they work. Now that is hard to do! I am writing a separate experience report for this challenge, as it has had quite an effect on me.

“Later that night we tied and gagged Bolton”
Well that’s not entirely accurate. But perhaps 30 of us were engaged in the “Mind Reading Challenge” starring Paul Holland and Michael Bolton. We thought we were truly on to something when we had Michael hands behind his back, face down on the table and not allowed to talk. Read his mind now, Paul! And, alas, he did. Don’t know what I’m talking about? Come to a conference and ask. It’s a lot of frustration and fun.

James Bach one-on-one
I enjoy “alone” time with James. He is inspiring and has a wonderful depth of knowledge on pretty much every topic under testing. I decided to give up the presentation about firing 2nd rate testers, and joined James for some quiet time. We ended up continuing the Mind Reader challenge, and I finally made some progress and got it. I still need to practice, but look forward to adding this to my challenge tool kit for others to learn from.

Facilitation
I enjoyed facilitating. Sporting that pretty pink jacket made me feel super important, like Barbie. I felt I was “taking care” of the attendees and presenter. I learned that it was okay to cut off steamy debates, and long threads in the interest of time. I also realised that if a discussion was really that interesting, then it would continue outside of the classroom in person, and on twitter. Paul attended all my sessions, and helped me a lot with his follow-up comments. Kai Pai my man.

“Mine’s bigger than yours”
It was great hanging out with my Nikon brother (same body model), Martin Nilsson (@MartinNilsson8). I enjoyed taking breathtaking photos of the conference and the surrounds. He may have had bigger kit, but mine had deeper penetration. The proof will be in the photos I guess.

Pink Jacket Brigade
The colour went with no other garment, they looked like american baseball jackets, but they were awesome. I enjoyed wearing the jacket, it filled me with a little pride of taking a larger part of Let’s Test. And it was a nice touch to stand up the front at the end for applause.

I’m not one for metrics, but my twitter followers increased by 100 after attending Let’s Test. That’s a great measure of “getting around” at a conference

Conclusion
There is magic at Let’s Test. It may very well be where unicorn tears are formed. I believe that the reason for the success is its location, and the people invited. Initially, I thought ~50% of Swedes attending was a lot. But actually, to put it another way – there were 50% of internationals attending! That is incredible. And it helps we are all stuck together with nothing to do but socialise and talk testing for 3 days.

I completed all my objectives, and added some more items which were unexpected, but very welcomed.

A big thanks go to all those who helped produce this event, especially Henrik and Johan.

What is next for me? (It’s turning into a huge year!)
KWST3
This is next up, on 5/6 July, in Wellington, New Zealand. It will be great to be at home again.

Sydney Tester Meetup
I will be presenting a visual presentation of Let’s Test 2013 at the July Sydney Testers Meetup. It will highlight the fun had, content & resource links, and the information shared. It is free to join and attend. This will probably be published on the Let’s Test channel.

OZWST2
I will be attending this invite-only event in Sydney this year.

Tasting Let’s Test
I will be running the test lab, on Monday 5 August. Details and registration can be found here: http://lets-test.com/. The best time to register is NOW. I am able to offer a decent discount on the ticket price to my network members. Add me on Twitter and Skype, and I can provide this to you.

CAST 2013
I look forward to attending CAST later in August in a facilitation role. It will be great to catch up with those I met at Let’s Test, and meet even more new wonderful talented testers.

Richard Robinson

Firstly, it’s an honour to be included as part of the team here.  What a great bunch of thinking testers to blog with!  Since KWST #2 in 2012 (where we met) I have formed a wonderful friendship with these Kiwi dudes, and have learned a great deal in the process.  Their passion for our craft is infectious, and for that I thank them.

The below article was originally put together by myself and Brian Osman for submission in The Testing Planet.  Unfortunately deadlines got the better of us and we didn’t polish it in time.  I thought it might be a good idea to share this with you all for my first ‘official’ post on HTW.  Being a joint piece helps with my HTW beginners nerves.  ;0)

First we discuss our thoughts on leadership, then what we’re trying to do for our little pieces of the world.  Oh, it’s long… but stick with it.

Brian’s Thoughts:

General McArthur

“A true leader has the confidence to stand alone, the courage to make tough decisions and the compassion to listen to the needs of others.  They do not set out to be a leader, but becomes one by the equality of his actions and the integrity of his intent.” – Gen McArthur

When I came across this quote, I liked the point that leaders sometimes dare to stand alone.  It may not always be obvious but sometimes a stand is what is required and there are many examples of that (we see them daily, just look around you).  Standing alone takes courage.  Doing what is right takes courage.  Helping fill a vacuum takes courage.  A case in point is OZWST - the first Australian Workshop on Software Testing.  David Greenlees, as the founder of this conference, saw a gap in Australia for thinking testers, and lead these testers to come together to DISCUSS and CONFER in a meaningful way.  I don’t think there is another conference in Australia that does just that for software testers.  I suspect the same is true for New Zealand and most countries (except where similar conferences to OZWST modelled on LAWST are held).  And as such David became a more prominent leader in the software testing space in Australia.  This in turn helps to increase influence and allows more people to connect to the leaders as the messages they convey align with their beliefs.  David is now recognised as a Peer Conference and software testing leader in Australia and as such his sphere of influence grows.

But what is leadership?

Leadership is an interesting topic. Are leaders made? Are they born? Is it combination of these or other elements?  Why do some leaders appear more successful than others?

I think leaders evolve.  Some leaders may have more charisma, skill, or talent than others but each of us have the capacity to become better leaders and magnify the talents and skills that we posses.  I also believe that a difference between a good leader and a poor one is the ability of a good leader to allow those of whom they lead to grow without fear.  A poor leader tends to stifle growth and creativity and may lead through fear, not encouragement and understanding.

So in saying that, there have been some notable testing leaders that have influenced my career:

• James Bach – He has challenged me; he has taken me to task and yet, without the world watching he has also corrected, taught, encouraged.  I’m glad to have met James, talked to him, and that he has taken time out to help increase real test leadership down under.

• Michael Bolton – Michael is cool, in the jazz musician type of cool, and his writings are thought provoking.  If anything, I usually tell my students that his blog is one of the top blogs to read on the internet.  Even better to meet him in person.

• Scott Barber – Scott has gotten me involved in a wider community (Software Test Professionals for example). He has encouraged and led from the front.  He’s the type of leader that people will follow.

And of course there are others who have influenced me greatly in different ways (down under would include Oliver Erlewein, Richard Robinson, Aaron Hodder, Andrew Robins, Katrina Edgar, David Greenlees and Anne Marie Charrett to name a few, though there are a lot more.)  When I reflect on these leaders I usually find that I’m drawn to their view of the world (testing wise) and whilst I may not always agree with them, I respect them.  The reason why is mostly because I have liked them as a person and they have influenced my thinking in someway that roughly aligned to my viewpoint; with the first part that equation being the most important.  Without the respect for the person, their influence becomes greatly diminished. It reminds me of something that James Bach said to me…

 “People need to see you as an expert otherwise they will yatter idiotic things at you.”

Which was in response to a prospective employer saying that exploratory testing and risk based testing are mutually exclusive.  James highlights his leadership (to me) by suggesting counsel and at a wider level it shows that the bigger a leader’s sphere of influence, the greater the opportunity to suggest, counsel, correct, and if need be, defend.

David’s Thoughts:

The topic of leadership is certainly a fascinating one.  Actually, I find all subjective topics fascinating.  Oh yes, this is very subjective.  Like art, leadership is in the eye of the beholder.  There are literally thousands of definitions of leadership spread throughout the internet, in books and articles, etc, etc.  Not one of them would be incorrect.  According to the author, that’s what leadership means to them, and being open to those individual definitions allows me, and many others I’m sure, to learn more about the topic.  If a topic is black and white, generally you learn about it once and you’re done.  With subjective topics, just ask someone new for there opinion and you’ll likely learn something new too.  Perhaps another reason why ‘one size fits all’ testing is not for me either!

“Leadership is taking someone to a place they wouldn’t have gone to without you.” – Unknown

That is one of my favourite leadership quotes/definitions.  When I first heard it I thought to myself, “Surely they mean a ‘positive’ place, not just any place.”  However, after more thought I came to the conclusion that it’s not necessarily the case.  I’m sure you can think of a few leaders in our time that have taken people to very dark places indeed.

Another…

“A leader is best when people barely know he exists, when his work is done, his aim fulfilled, they will say: we did it ourselves.” – Lao Tzu

I think combining those two quotes, in that order, gives you my ultimate synopsis of leadership.  The best leaders I have had throughout my career and personal life were not obvious to me at the time.  Only on reflection do I view these people as wonderful leaders.  The work they did for me was ‘behind the scenes’ was truly unselfish.  To this day I couldn’t tell you exactly what steps they took, but I could guess that there was many times where these leaders put me in a position for the sole purpose of my betterment.  It saddens me somewhat, as I’ve lost touch with many of them and therefore can’t go back and thank them as I would like to.  One day perhaps.

An example of some of my leaders specifically in testing includes, but is not limited to:

• Matt Heusser – Matt is my unofficial/informal mentor.  I only say that as we’ve never had the ‘you are my mentor’ conversation, and I don’t believe we need to.  Matt has been key to my learning over the past 12-24 months, and also a big inspiration for my writing.  Matt leads (for lack of a better term) the Miagi-Do School of Software Testing which has taken my learning forward in bounds.  His passion for the betterment of our craft is unquestionable.
• James Bach – James has challenged me on several occasions, and continues to do so regularly.  I won’t deny that he makes me nervous with every encounter.  I have the feeling that no matter what I say it will be challenged, and despite how negative that may sound, I thoroughly enjoy it (after the event).  He reminds me of my Grandmaster in the martial arts (one of the leaders in my personal life).  Sparring with him is painful, extremely painful, but what a feeling it is to come through the other side knowing how much I’ve learned.  Each encounter with James is like that sparring session.  You will have to live with some bruises, but his mission is clear; to make you stronger; to make you better.
• Brian Osman – As you’ve read above, and will do again below, Brian has lead me to start my very own Peer Conference.  He is also relentless in pushing me to begin speaking at conferences.  Watching him do it himself is all the leadership I need.  His courage is also a very noteworthy point.  Brian will stand up for what he believes in no matter what, even if it means leaving a job behind.
• Anne-Marie Charrett – Anne-Marie is a true leader in our craft, to many people, not just me.  Her willingness to develop individuals in her own time simply blows me away.  Her dedication to testing is second to none, and she naturally takes many people on the journey with her, including me.

Brian and David’s Test Leadership Down Under:

They’ve been busy in recent times.  Well, they’ve been busy for longer than that I’m sure, but only recently have they taken a larger and more public step.  Brian and David share a passion for taking the craft of software testing to the next level in New Zealand and Australia (down under as they commonly refer to it), and they’ve started to spread the good word amongst their peers.

Brian and David had both been blogging for some time and had become active members of their local testing communities, and globally online, via forums (how they met), special interest groups, and meet ups.  These were all important avenues for them in achieving their goal, but more was needed.

The Kiwi Workshop on Software Testing (KWST – Brian’s baby) was the first major step.  KWST was born through Brian asking James Bach on Skype one question – “How do we build a community?” James came back and suggested a LAWST style conference that allows test leaders to come together in a way that really allows for discussion, debate, and critical questioning.  This idea intrigued Brian and with quite a bit of effort (and persuasion), KWST was born.  So why KWST?

The reason KWST was created was so:

• Test leadership could come together to discuss meaningful topics
• A community could be created that is open to challenging the craft
• That community could begin to influence the wider community
• That community of critical thinkers would challenge the factory school thinking that seems to permeate middle management

KWST #1 and #2 had completely different dynamics and as such produced different outputs.  The first KWST conference brought people together, enabled a community, coined terminology and generated momentum.  KWST #2 was more about extending the reach of the community, unearthing new leaders, and to be challenged.  Either way, the thoughts, ideas, and building of the community are currently the most important drivers.  Without strong leadership in testing to challenge the status quo there is a possibility that New Zealand could have become a rather large body shop with pockets of resistance.  As it stands now, test leadership is thriving and growing, and no longer accepting the myths perpetrated by factory school thought.

Approximately 6 months after KWST #2 the Australian Workshop on Software Testing (OZWST – David’s baby) was born.  David was inspired by the magic of Peer Conferences and that his mate from across the Tasman Sea was able to lead the way in making it happen.  Brian, with the help of James Bach, had managed to take David to a place he wouldn’t have gone without him.  Brian had lead David via inspiration.  That inspiration was enough to spark David into forming OZWST and running the first successful Australian Peer Conference on software testing.

These Peer Conferences are a form of test leadership.  They take the participants to levels of thought that they would not have gone to otherwise; at least not at that point in time.  A melting pot of subjective topics and themes that encourage and provoke discussion and debate; and learning by default.

Brian and David will continue this journey, with KWST #3 and OZWST #2 already in planning for 2013.  They hope that attendees can take away something from their leadership, and that the software testing community of down under progresses at lightening speed.

Life is learning, and leaders allow you to do just that.

If you have any thoughts, we’ll be looking forward to the in the comments.

Hello Geoff,

We’ve read your article that was in the mid-edition newsletter 21/05/2013. While we have nothing much to comment on Colin Cherry’s part of the article, we would like to challenge some of the things you state in your part.

1) Your selection as to the attendance to KWST #2 was by no means a given. The decision fell in the timeframe between your email from the 14/5/2012 and the sending of the invites on the 27/05/2012. There is a selection process which involves conferring with the organisers of the event as to the make-up of those 20 people. In clear text, you were voted in. By no means do we have a “loyalty card scheme”. We cannot know everyone in the testing industry so we rely heavily on people we know suggesting people we should invite.

2) “Seemingly “brave new world”” – how long would you say (in years) does it take for a brave new world to be established? The context driven testing school was formed by Cem Kaner, Brett Pettichord, James Bach and Brian Marick in 2001. The practices were present years before that. I’d stipulate that is just as tried and proven. The fact that you referred to it as a brave new world only highlights Colin’s point about testers not looking beyond their own back yard.

3) The “brave new world” was Unstructured Testing – Exploratory Testing and Context Driven Testing is far from unstructured. In fact there are many ways to structure Exploratory Testing, so that it is accountable, auditable, reportable, and plannable. The opposite of exploratory testing is scripted testing, and not structured testing.Secondly Exploratory Testing (ET) and Context Driven Testing are not synonyms. Context Driven Testing (CDT) is testing driven by the principles of the context driven school, much like how Agile development is development driven by the principles laid out in the Agile Manifesto.

ET is an approach to testing that focuses on the tester’s skill and judgement to guide their testing as they are in the moment of testing.

So ET and CDT are far from unstructured. But that is actually secondary, as that was not really what the discussion was about at KWST. The discussion we had was about counting test cases in order to to inform and the wider practice of supplying metrics. You were adamant about you “crunching the numbers” without giving any proof or scientific reasoning behind what you were doing or why. This is, at best, called pseudo-science. We noted that this was a common practice in many projects and that does not make it good or lead to successful projects. You, nor anyone else for that matter, could prove any correlation between metrics and the success of a project.

The discussion we had was never about unstructured testing and we would contend that there is no such thing as unstructured testing. It was also not about scripted vs unscripted or any one of those discussions. So we do have some disconnect with what you are writing and our take on what was said.

4) Interestingly enough you then proceed to describe why rigid methodologies fail in most projects. Part of what we do at KWST is to talk about our experiences (Experience Reports or ERs) and challenge one another to try and find different and hopefully better solutions. And we are really pleased to hear that you felt challenged and reflected on what you were doing. But it does appear that you have decided to do the same things you have been doing your whole testing career, never questioning if there is a better way or just a different way. This is the exact thing we try to challenge and improve on at KWST amongst ourselves.

5) You then get to the part where you wonder if you’ve always been a context driven tester. We would contend that your testing, as we understand it from your descriptions, is far from context-driven. As defined by the founders of CDT (who are mentioned above), Context Driven Testing (and the Context Driven School of Testing) is much more than acknowledging the context of a project or organisation. It is a set of guiding principles which are:-

• The value of any practice depends on its context.
• There are good practices in context, but there are no best practices.
• People, working together, are the most important part of any project’s context.
• Projects unfold over time in ways that are often not predictable.
• The product is a solution. If the problem isn’t solved, the product doesn’t work.
• Good software testing is a challenging intellectual process.
• Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

Any tester who is aligned with the above principles allows the context to drive the appropriate response whereas a tester that is context aware is most likely to pay lip service to context (and then continue with their own “tried and true” methods). Knowing context and adapting to context are two different things. A context driven tester will reject best practice as they know that best practice in testing is a fallacy. However, they see that there are many good practices that will morph depending on the project.

Context Driven does not mean taking an ISO or similar standard and watering it down to something one is able to do and can still charge $$$ for without blushing. You stand corrected. You are not (yet) a context driven tester.

6) As for the getting along bit. Different schools of software testing do not all get along because the paradigms behind them are completely incompatible. We can have mutual respect but such respect is earned and built over time and by shared experience. There are many attempts for the differing fractions to communicate which are more or less successful. The discourse is what makes us progress and not the harmony. We admit that there are discussions that may not suit the stomach of many an individual, and can understand that the tone of many of the more aggressive leaders in the field can rub people up the wrong way, including us sometimes. But we are human after-all and we can see from history that discourse is our MO and that it sometimes escalates. It escalates because we are passionate about our profession and want to see it flourish. We want to see our field move forward and be respected, and provide real lasting value to the projects we work on.

It is a wide spread illusion that testing is all known and defined. We’d argue the exact opposite. The whole of IT is still in its infancy and evolving. How dare we be so arrogant to assume we know everything or have a best practice? Even if we have worked in testing 10, 20 or more years doesn’t mean we are right (we could still be successful from a financial perspective though). Humanity believed the world was flat for how long? That sicknesses were caused by demons, that the atom was the smallest particle, that witches existed, that the Earth was the centre of the Universe… and all those that believed in these things were learned, intelligent, successful and totally wrong individuals.

7) We always find it quite hard to follow “If someone else is doing the same however in a manner you don’t like or agree with then unless you are that person’s manager et al, “live and let live”. There are non-combative ways to express an opposing viewpoint and challenge someone. How else do people get exposed to new ideas and improve their thinking? This is what many in our community are attempting to do by attending peer conferences like KWST, OZWST and WeTest, and engage in social media. Secondly, these are issues that affect the profession at large, even if they are occurring in apparent isolation. It sets the expectation of what a software tester does, and the value they provide on a project (and ultimately what they are willing to pay for such services). It affects the market and the demand for certain services.

Secondly, it’s a personal ethics thing. There are certain practices that we feel provide little value at best, distract them from what’s important, and can actually mislead people at worst. Some of us do feel compelled to challenge these practices when we hear about them. The practice of counting test cases is one of them. Which is why we challenged you on that in the hope that you would tell us what you actually did with the data. That is, how you collected the data, how you manipulated that data, and then what you did with the outcome.

We are glad that you reflected on your experience at KWST, but we do feel like our ’camp’ has been misrepresented in your article. Since you have publicly disclosed your experience and thoughts then we would also like to express our view in the form of this email as we are part of this story.

Best Regards

Brian Osman, David Greenlees, Oliver Erlewein

He is from the land of Oz, but other than that he’s a really good guy
He is also the creator of OZWST in Australia and is actively involved in progressing the testing profession. His next challenge is to get the Let’s Test conference started this side of the globe. Watch out 2014 for that one!

The problem, that I am solving here is, how to get unconventional data out of JMeter. When you run JMeter all performance data is neatly saved away in any way you like. But say you want to make a list of the customers and customer numbers you created. JMeter does not deliver such functionality out of the box. Well….there is one way but it’s ugly and tedious at best so I won’t even mention it here.

So I am under the assumption you know your way a little around JMeter and what it does.

1. You will need to install BeanShell on all machines hat this script will execute on. Download bsh-core interpreter from http://www.beanshell.org/download.html
2. Copy the file to the ~/lib directory of JMeter
3. Have some variables ( ${ExampleVar} ) defined that you’d like to write to a file.
4. In your script go to the sampler that you want to write the file and insert a BeanShell PostProcessor
5. In the Script box insert the following script and change to hearts content.

 
import org.apache.jmeter.services.FileServer;

// Get the variable(s) from the JMeter script
tempVar = vars.get("ExampleVar");

// Static elements or calculations
part1 = "Car Speed is: ";
part2 = " km/h";

// Open File(s)
f = new FileOutputStream(FileServer.getFileServer().getBaseDir()+"\\carSpeed.csv", true); 
p = new PrintStream(f); 

// Write data to file 
p.println( part1 + tempVar + part2 );

// Close File(s)
p.close();f.close();

This will store a file directly on the Desktop. Enhance the content as needed. So far I have not noticed any adverse performance issues but I also haven’t tested it in too strenuous situations. I mainly use this when creating test data for performance tests.

It becomes a bit more difficult when you have a distributed test harness. You will need a process, by which you amalgamate all the separate files being created on the clients. So for now I’d suggest not using this when using remote JMeter instances.

Author: Oliver Erlewein

Here is my experience report.

I arrived early, fully aware that technical difficulties were inevitable. I had a set of tasks I needed to get out of the way before people started arriving in 15 minutes. I knew there would be problems, but not quite as many as there turned out to be.

Setup troubleshooting:

1. Getting into the building. I had to negotiate the lifts not going to the level I wanted to go. After some slick social engineering and I was up to my level.
2. Connect my laptop. I guessed that it was going to be a VGA connection. I was wrong, it was HDMI. Both supported by my laptop. I left my Macbook Air at home as it didn’t seem to have any display port. I wanted to use it though, as it is a much smoother ride.
3. Access to internet. Found the network, and arrived at a browser log in page. Requested help from the support staff, but it was a no go. No one knew the username/password. I had to borrow an employee’s computer which was already logged in to the network. It was also a Macbook Air, so I was right at home. There was a thunderbolt to HDMI adaptor which made things work for the Mac.
4. Opened Google Presentation, and got an error loading up the presentation saying it timed out. I got this on my network too, so I wasn’t too worried, clicked around, and got it working.
5. Test YouTube link speed. Added a nice link to the front page to load up some cool relaxation music with beautiful scenes to use as my meet ‘n greet background show.
6. Sound. No way to get sound going through Macbook Air’s thunderbolt port. Came up with a dodgy ‘microphone resting close to Mac speakers’ solution. Worked well enough.
7. Speaker notes. Did not manage to find out how to have speaker notes independent of the main presentation. The Mac just showed what was on the projector. So I  resorted to my print out of notes I had prepared earlier.

Next time I will bring my Mac, and find out how to best get sound through to the sound system.

Despite being nervous and anxious all day, I felt very settled leading up to starting. That was unexpected, but I will definitely go with it. The crowd was about 30 strong. With an intro from Bruce McLeod, I was off and in to the presentation itself.

Some slides contained too many words. I should have known this stuff. Really the only way to do a presentation is to present it all from experience. Maybe that is why James Bach’s great use of Experience Reports in conferences works so well. Its always best when people present from experience. It just makes it feel natural.

I decided to go off script part way through. Instead of referring to my presenter notes, I looked at the slide, and came up with relevant points and stories. I was not getting much enjoyment before from reading content to the group.

Later, I realised that the first part of my topic was largely read, or scripted. I think it sounded boring, which is funny because that is a prime outcome of doing test scripting. Boredom!

Improvements:

My style and content will require a few enhancements (my own judgements on myself).

• Keep posture looking interested. There was a tall table with a tall chair. It was too irresistible to slouch in the chair, or lean over the table swaying my hips.
• Don’t look at the slides when presenting! ugghhhh, this is so 101.
• Cut down the words on the slides to a minimum.
• Zero reading.

Positives:

Some kind people told me these nice things after the presentation:

That I…

• was confident
• dealt with questions well by having a response for each question
• was enthusiastic when talking about my own experiences
• used embedded media well
• got a laugh for the Moonwalking monkey
• added value and excitement through the questions and discussions
• kept the audience alert and energised by having them partake

The Meetup went overtime, and almost all the audience stayed. I take that as a good sign. Afterwards, a lot of people came up to me discuss topics further, ask more questions and say thanks. That was really great.

I think I am on to a good set of ideas. The slide deck was very optimistic as it covered a lot. My wife said it is an entire software testing course. I think I will refine it based on feedback, and slice and dice it to fit other learning modes such as a workshop.

I think that I can get more clarity and understanding if I pose the same topic through a hands-on demonstration with less emphasis on test script negatives, and more on how test ideas offer so much more value to testing.

Here are the slides for the full presentation.

Author: Richard Robinson

The immediate reaction of many a tester (especially if she went through some kind formal training) goes a little like this:

Use Risk Based Testing!

I agree but sort of don’t…

Yes, assessing risk and focusing test cases or test scope or whatever you have planned to do is a good idea.  Assessing risk is commonly seen  as an easy and quick task, at least simpler and quicker than executing a number of test cases.

But this belief doesn’t even hold up to a first tentative inspection.

If you actually delve a bit into the field of assessing risk it becomes a quagmire really quickly. The amount of facets to assessing risk are  endless. There are several books and write-ups about how to assess risk properly. I’d doubt that if you had read any of them you’d be so foolish as to suggest Risk Based Testing (RBT) for a quick and good solution to deal with time constraint.

Give it a try yourself!

Read James Bach’s really good write-up on Heuristic Risk-Based Testing and remember this is already a very good stab at simplifying the issue! Here’s the link http://www.satisfice.com/articles/hrbt.pdf . If this is a bit much to read then you can just scan across the pages to get an idea.

As you can see there is heaps to think about and assess to get some semi-valid inkling of what risk actually is. What becomes obvious is that it is definitely a non trivial task and involves not only testers but everyone in the project (and sometimes beyond). It will take time,  be complex, be open to interpretation, yield false results because of human limitations & traits…

So basically if you haven’t done a really good job at assessing risk in the early phases of your project (and I have yet to see one project that did!) there is NO WAY that you will be doing proper RBT when it comes to the time to cut down testing scope.

Actually saying that you do/will would actually be a lie. What you are  doing is a very high level and limited (not to say wrong) risk assessment and running with it. To call that Risk Based Testing is really a stretch. I’d prefer you not sell it as such. It would be feigning a scientific or structured approach and thereby border into the unethical.

Shocked yet?

Ok, so let’s go back to the problem at hand. What you really want is a good way to cut down testing scope and thereby effort.

For some odd reason RBT is seen as minimising the risk to the solution. Risk Based Testing actually INCREASES the risk remaining in the solution! The key idea is just to minimise the INCREASE in risk. Once you have understood that you can actually start to deal with the alternatives.

Since assessing true risk is so complex what can we do that might get us to our goal quicker but with much less effort and still keep the risk increase manageable? This is the point, where I mention the magic words CONTEXT and BRAIN. Use both and be inventive. I’ll give you a few easy ones to get you going.

1. Test changes only
2. Test only areas that have had high error occurrences in the past (or the converse thereof)
3. Test only areas, that are exposed to users
4. Test only areas, that you left out in the last test cycle
5. Test only new test ideas and scrap the old that possibly were tested before
6. Have a vote in your test team what they think should be tested
7. The tests that use the most interfaces…

Or any one of dozens of ways of cutting and slicing testing/scope. Of course some of these go a long way to assess risk but on their own they are not RBT yet. You will still get most of where you wanted to be.  In this way you will be clearer about what you’re doing and much more genuine.

Scrap the fashionable Risk Based Testing and describe what you will do in your context, why yod be doing so and what it will do for risk and testing scope/plan/timeframe.

Author: Oliver Erlewein

I think the primary premise of what UAT should be, that we have here in Wellington/New Zealand, is wrong.

Most of our industry does it wrong. Why?

Because testers influence UAT too much!

UAT = User Acceptance Test

To me that means the actual user (not to be confused with stakeholders or managers) must assuage if the software developed is OK for him/her to use.

That means:

1. That the application actually works, as proven by previous test stages (so not the actual mandate for UAT)
2. UAT coverage is small and mainly UI – But that’s what most of these actual users actually see (Pareto principle).
3. They should be doing what they do day to day – Because the corner cases should have been covered by Functional/System Testing and been proven correct (see next point).
4. UAT is NOT supposed to find ANY defects/bugs! – The things that don’t work for them should be change requests. I.e. the primary requirement was at fault and not the software.
5. Testers should not be involved in UAT – They are not needed, it’s not a testing phase (confusion in the name!). It is a milestone for requirements and their correctness. It should not be around the actual acceptance of the software, that should be a given by that point.

Because testing always gets pushed to the end (in waterfall projects at least) we look for where we can regain lost time. I think over the decades we have pushed our testing regime into UAT, until it became a test phase.

This is subverting the actual reason for this SDLC phase. As stated above, this phase proves Fit for Purpose (FfP). That answers the question “Is the software usable form a day-to-day business perspective”. If that FfP is influenced by open defects that’d be a catastrophe! As this phase is situated right at the end of the project, this is where fixing defects is the most expensive (bar production). If your project is still fighting defects at this point you should start questioning your project timeline. There is likely a severe issue with your SDLC/management/governance.

Things found in UAT should be functional or systematic changes that need to be made to future releases. I would not see them as fixes. In the context of the project the things UAT should find are changes to requirements or added requirements (stuff that has been missed).

The SoftEd post talks about why UAT is the hardest testing. I think that actually misses the point and enforces the bad habit of seeing UAT as a testing phase, with testing methodologies, testing oversight and control.

I stipulate that all that will achieve is…

1. do more system testing on a much higher and therefore worthless level (at that point of the project)
2. with people who aren’t even testers
3. totally miss the point of highlighting the inadequacies of the software needed for actual implementation and use by the customers/users

What has happened seems to be, that we have actually lost (without replacement) a vital stage of the Go Live process. The software process improvement loop. It has been shoehorned into a testing phase. I don’t want to question that there might actually be another test phase needed that does what we do in UAT today but we should then not lose the actual intention of UAT and the phase associated to it.

Author: Oliver Erlewein

Needless to say the event, instigated by Aaron Hodder, Katrina Edgar and Brian Osman, was quite a success. The topic was “Experiances in Test Automation”. Discussion was lively and there was lots to take home & think about. Read up on the details in these blog posts:

Assurity BLOG

TestSheep NZ

Thanks go to Assurity for helping fund the venue, food and drinks! Also thanks to everyone there for the great participation. The next Workshop will be on the 6th of December. Places are filling up quickly.

Author: Oliver Erlewein