• It is an invite only conference – we are looking for industry thought leaders
• James Bach will again be back as content owner and helping grow the core of professional test leadership in New Zealand
• Some of the brightest, insightful test thinkers down under will be there
• Unlike any other conference held here, this is a CONFERence where ALL participants participate!
• The theme is Ethical challenges faced by testers which is relevant considering the prevalence of dubious practices and certifications in our industry

The twitter hash tag will be KWST2 and we will be tweeting all of the great thoughts and ideas that will flow from this conference. See http://bjosman.wordpress.com/2011/06/28/kwst-kiwi-workshop-on-software-testing/ and http://bjosman.wordpress.com/2011/07/08/kwst-kiwi-workshop-of-software-testing-day-2/ for details of last years event.

Author: Brian

Normally I try and steer away from anything that doesn’t use standard protocols and/or interfaces. Things like Flash, Silverlight and others. Not that there aren’t test tools that handle these things but it’s just that it’s messy to say the least. For open standards like HTML or SOAP there are gazillions of ways to automate.

So I got surprised by having to test an application on -or should I rather say through- Citrix.

In order to understand what the issue is you have to know a little more about Citrix. Citrix is basically a specialised form of Microsofts Remote Desktop Protocol (RDP). Instead of running an application on your desktop it gets executed on a serve(farm) and then just visually transported back to your desktop. The difference to RDP is, that it actually acts like an application. You cannot really see that it is not running locally. The integration to your desktop is seamless. RDP, in comparison, would show you the whole desktop on the remote server.

From a test tool perspective (on the client) you only see pixels and keyboard and mouse interactions with it. There is nothing for a test tool to grab on to really. There are some tools that leverage off the background ICA protocol but that is also difficult to stabilize.

So, whichever way you turn you’re in a bind anyway. In comes Sikuli. I should rather say Sikuli X. This is a beta classed project by MIT, that is testing through the actual front end interface. Usually I am fiercely critical of such test automation but Sikuli really brings some interesting features to the task that mitigates a lot of the inherent risks.

The main issue with such automation is relativity to where objects are on screen. Sikuli leverages off actually finding things on the display but instead of doing a 1:1 search there is some fuzzy-logic thingamajiggy going on, where certain differences are actually allowed. That means the hit-rate is much improved over many other tools.

The editor! The actual scripting editor seamlessly incorporates the concept of using screenshots. The images can be seen in the actual code you write. See the example below. Click on the image to see the full size screenshot.

This in my mind is a novel and easy way of developing. It makes things a lot simpler to visualise. Sikuli is developed in Java and uses Jython as the programming language. This opens up all the powerful language features of Python to you in the scripts. Python is a currently popular laguage so you might be able to leverage off previous know-how.

But let me say this, so that there is no misunderstanding:

1. Although Sikuli is cool and really the best implementation of this testing concept. I would still not use it for large scale test automation. I would not ever use it for any web testing. There things like Selenium and Watir are better suited.
2. The kind of test automation Sikuli does is still fragile, no matter how clever it is. ll your tests should always be treated as fragile and maintenance intensive.
3. If you’re in a real bind for a tool and nothing seems to fit, then this is it.

So now we can get back to the cool stuff! Sikuli can react to things on screen or screen regions. So you can say things like “if this picture turns up on the screen do something”. So you could actually have a program running in the background waiting to detect a login screen. If it has detected it it would automagically log you in. A good example on how to use this tech in a way not really intended here is Sikuli playing Angry Birds.

Go to the Sikuli Blog for more cool stuff.

Now for something more concrete. I’ll do a small MS Word automation. The script is as follows:

I won’t insult anyone’s intelligence by explaining the above script. Just LOOK at it and you will understand. The only thing I need to add is, that the path in the 1st line is split for display purposes, in order for the program to work the 1st two lines must be one. I don’t think there is a quicker way to automate Word without any know-how in VB and the such. So I find it quite useful and easy to do even if you sometimes get issues with rendering times and things overtaking each other prematurely.

Because you’re in Jython and Sikuli is so cool you can also do things like for and while loops. If statements can get really nice with things like if exists(<image to find>) then click(<image to click>). You can also define asynchronous events that track things while you’re executing a script (onAppear, onVanish). So you can catch an error pop-up should it happen.

So my advice is, get Sikuli and have some fun. Add it to your tester-tools-baggie because the day will come when it will help you out of a tight spot. If you have your own Sikuli experiences then please post a comment on what you use it for and how.

Author: Oliver Erlewein

What do I mean by that?

There is such a thing as (and I am having difficulty writing this) Best Practice, when it comes to web page development. These are technological imperatives that can be easily checked by using simple tools. You don’t need to be an HTML guru to use these or to gain more knowledge about your website under test.

What I’m talking about are tools like YSlow (http://developer.yahoo.com/yslow/). A simple tool, that can be applied in seconds to any website. It can be a treasure trove for any tester. You can have it even easier than that. If your site is available online just hack it into http://gtmetrix.com. That gives you the YSlow results and Page Speed results without having to do any installing at all.

Have a look at TradeMe for example. Isn’t that cool?

You can see a heap of information on how the page loads and what issues it has. You can even run comparisons from different locations in the world  (you’d think that the results would be the same….go surprise yourself ).  You can (with a login) also run tests over time and see how the ratings change.

Any issues are highlighted and you can get details on what seems wrong. I’d always suggest understanding the issue before rasing a defect though. There are false positives, where certain issues appear because they are needed for the workings of the page or the technology stack used for your project restricts the outputs.

Personally I expect web designers and web developers to check for such easy to find issues before delivering code to test. Or, in a vendor scenario, I’d expect their test team to have had a look. This should be common knowledge and due process. Thing is, that I have yet to come across someone who does these tests. The consequences from that are blatantly obvious if you run a few tests. It gets worse if we are talking web sites for internal use only. Have a play with websites you know or tested and see how they stack up. You will also be able to easily pick out the sites that actually did their homework.

The main features I generally look for that are easy to deal with and give big wins are:

Number of HTTP requests

I expect a minimum number of resources that have to load to show a web page.

Why is that so important? Each resource requested locks up a thread on your browser (and these are limited to 4-6 threads per site). That means loading of resources is done sequentially. This impacts the load time of the whole web page. Each request for a resource (even from cache) will initiate some kind of communication. The overhead will have an impact on performance.

Example: If you amalgamate two CSS files that overhead has just halved, you are now using one thread instead of two and your compression might be better (more on that below).

Here is the analysis of the IRD page. Go to the Timeline tab. You get a really good representation of how the page loads. Focus on the LHS of each bar. The beige (for lack of a better description) colour denotes the time the request has been waiting to be processed. As the page gets loaded these get longer as resources get locked up. On the YSlow tab it suggests making less resource requests. If you click on “Make less HTTP calls” you’ll see that it lists 25 optimizable requests in total. With this information you could now go back to the developers and ask for a fix.

Expires Headers

As the name indicates these HTTP headers tell the browser and caching proxies how long to cache resources. If this is set to zero or not set at all, each resource will be requested anew with each page load. With the ADSL/cable/fiber speed that private homes have nowadays that should not be an issue one would think but…

It is of interest for the server side. Companies have to pay premium for every bite transmitted out of their datacenter. Not only to the ISP but for infrastructure, rackspace and other costs. If pages can be cached outside of their realm those costs can be minimised and frees up bandwidth and capability for other services (also see topic Content Delivery Networks and ETags).

Additionally this helps save hardware and energy costs. Each request not made to a server saves CPU time and the power, that is needed to generate the response. This means you can do more with less hardware. Certainly the savings per page are minute but looking at economies of scale they do tend to add up to a significant number.

Adding expires headers is usually quite straightforward and as easy as hitting a button in your web server config or changing a line in your web server config file. The question is how long resources should be cached. I usually suggest a standard 24hrs to a maximum of 72hrs. Different resources can/should be cached for different lengths of time.

The thing to note here is that the danger comes when the website changes. In the most extreme case a customer/user could see old content for a while. There are clever ways around this though. Have a Google and you’ll find lots on the topic.

Compress resources

Similar to the expires headers above compression aims at the similar benefits. Gzip compression can be used when browser and server communicate. The web world is quite text heavy and text is easily compressed. Web pages can therefore shrink communication 30 to 70%. As above this has a positive effect on bandwidth and infrastructure requirements.

Keep-Alive Enabled

Whenever the browser loads a resource it has to go through a heap of communication just to establish a link over the network (TCP), HTTP(S) and then to the server. This is quite a significant overhead (hence also the push to reduce resources – see above). In order to circumvent the re-establishing of connections to the same server  Keep-Alives are used. This means a connection is built up and is re-used for multiple requests. As you can imagine this saves a lot of communication and makes things much quicker.

Optimizing JavaScript and CSS

When the browser interprets a web page it can get into situations that are similar to those in databases. Certain things clash and can only proceed sequentially. In the database world that is called locking. For HTTP we’ll call this blocking.

There are two simple rules to get around this as much as possible.

1. Put all CSS to the top
2. Put JavaScript to the bottom

Remove Redirects

The Metservice page also shows a possible issue with redirects. If you look at the first 3 steps of the  loaded web page they are HTTP 301 response codes (all HTTP 3xx codes are redirects). These redirects bounce the browser from one URL to the next. There are good reasons for doing redirects but as you can see the practice in this case costs over a second before the actual page loads.  As a tester I’d always question if these redirects are really needed or could be replaced by a simpler method.

Finally

There are another good two dozen things highlighted in these reports and it’s worthwhile looking at these too. Once you have some practice using the reports they deliver lots of low hanging fruit that can very cheaply increase the speed of your web site under test and if there is one thing all experts can agree on is, that faster is better for business.

An alternative GTMetrix is http://www.webpagetest.org and I’m sure there are a few more out there.

Also of interest is http://www.httparchive.org. They run similar tests on the most common sites on the internet. This can give you an impression where your site ranks and what the most common issues are. You can even add your site so it will get monitored for you. Just beware that this will generate traffic to your site you might not want.

Author: Oliver Erlewein

Bug Investigations

Thousands of words have been written about the investigation part, and it’s usually where the information ends.  You’ve got a crack bug investigation procedure.  You’ve clearly identified your oracles, you’ve mapped your coverage, you know your quality criteria.  You’ve been patrolling the mean streets of your pre-release build, and you’ve noticed something out of the ordinary.  The adrenaline starts pumping, and you’re ready to reach for the red and blues.  We wanna take this perp down.  But hold up, bronco.  Before we grab the pepper spray, let’s talk about what happens after you have a suspect in your sights.  You’re pretty sure you want to make the arrest, but we don’t want to compromise the sentencing later.

Identification

A tester’s credibility is paramount.  You start hauling too many innocent citizens to the station, people are going to stop taking you seriously.  If there is any doubt in your mind, you need to check and double check that ‘bug’.   Are you certain you’ve got the right guy?  Will your justification for arrest hold up in court?  Check an check again.  I recently noticed the pattern used for depicting transparency had changed in a graphical editing tool I was testing.  It seemed like a bug to me, but I wasn’t sure.  I googled if there was a standard for this pattern, and I looked at other similar programs to see how they depicted the pattern.  I asked the developer if the change was intentional, and if so, why.   If the court system is clogged up with illegitimate arrests, then there is a danger that a truly dangerous defendant may get away.

Isolation

Isolation is such an important concept.  You’ve found a bug, and yes, it’s a bug beyond any reasonable doubt.  Now we need to isolate that bug.  What does that mean?  It means narrowing down all the variables involved in recreating that bug to the minimum you can.  For example, if you’re bug begins: “Sometimes the confirmation window…” or “Occasionally the application will…” you need to isolate.  Even if you can reliably recreate the bug, you need to ensure that you’re not including anything irrelevant to the bug.  You also need to ensure you haven’t focussed too narrowly.  If there are five parties to the crime, then we need to make 5 arrests, not six, and not four.  One technique is to list all the variables involved when recreating the bug.

For example, let’s say this recreates a bug: “Open image in image editor.  Resize image. Save and close.  Reopen image.”  What are some of the variables?

• Image type (jpeg, png, bmp)
• Image size in kb
• Image size in pixels
• Location opening from (folder, removable media)
• Resize image bigger, smaller…
• Save over top of existing image
• Save as new image
• Save with a different name
• Location of saving image
• Reopening image using the application UI
• Reopening by double clicking on icon

This is a simplified example, but we should be sure within reason that we have all the relevant details and only the relevant details.

Prosecution

We have the bug in custody.  Now to book him.  The bug report is your prosecution, and may be your only time to make your case that a) this is a bug and b) should be fixed.

Adminy stuff: Your organisation (probably) has a system for how you classify bugs give them severities.  Do follow that.  A bug that is easy to find in the system is more likely to be looked at.  Assigning an appropriate severity aids your credibility.  You may need to justify the severity in the bug report.

The Crime: That is, why is this a bug?  For violating History Consistency Heuristic?  For violating requirement 3.2.5 in the requirements document?

Evidence: Present your evidence.  I love screenshots.  Show pictures, copies of logs, copies of input data, description of the bug, and clear steps to reproduce.

Consequences: What are the consequences of this bug going unfixed?  The consequences may not be self-evident, and never assume they are.  “Because there is no warning that turning on this feature reduces render performance, there is a risk that a user could turn this on inadvertently and go live-to-air and not make frame rate.  This would lead to more support calls, and a loss of reputation.”

Conclusion

I know I for one get “bug lust”.  I see a bug and I want to jump out with the taser, and tag and bag that sucker.  But it does noone any favours making false arrests, or supplying incomplete information.  Slow down, and ensure you take the time to make a good clean arrest.

Author: Aaron Hodder

…(jmeter-server.log on server machine) I could see that the connection back failed. I must add, that I am using Ubuntu in a headless mode to do these things and it looks like a Linux/Debian only issue.

The log output was:

ERROR - jmeter.JMeter: Uncaught exception: org.apache.jorphan.util.JMeterError: Could not return sample
 at org.apache.jmeter.samplers.AsynchSampleSender$Worker.run(AsynchSampleSender.java:148)
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.1.1; nested exception is:
 java.net.ConnectException: Connection refused

After searching the web for quite a bit there was little to go on. Eventually I figured out that the issue is actually on the client side and not the server-side as you would anticipate by the error log. Google didn’t quite help as much as I would have liked it to so I thought I’d just do a blog post o that others can find a clearer answer.

The solution is to go into the /etc/hosts file and find the entry that looks something like

127.0.1.1     <server-name1> <server-name2>

Comment this line out with a # or delete it completely.

Please note though that I don’t know if this has any other implications. The line was added to deal with issues in Debian when using Gnome.

I hope this helps someone else that runs into the same issue and sorry to the non technical people out there for this post.

The tester’s task was to formally document their testing effort using a test case repository tool. To do this they used their memory and experience of using the system to write out in more detail the test steps already performed. The tests were already executed, and bug reported. This was just a documentation exercise to capture the tests, for future use.

The testing performed was done using charters in an exploratory manner. The tester was left to learn about the product feature and build up a group of tests. They came up with many tests, including the following. They knew what each meant, but there was too little detail for them to be reused.

Test Idea: Logon
Tests:
1. Known bad password
2. Invalid password
3. No password

And each of the above were supposed to be split out so anybody could pick up the tests for next time. Ambiguity was to be cleared up, and greater levels of detail added. The intended detail was something like below.

Title: Log in to app with known bad password
Preconditions and Setup
1. User is general user, with general roles and permissions
2. Site is \\testserver\mainbranch\Application3.0\Login.htm
3. OS is Win64b Pro
4. Browser is FF9
Test Steps
1. Go to login page
2. Enter known bad password eg notmypassword12
3. Press Enter or click on Log On button
4. Wait for response
Expected Result
Error message saying “bad password” will appear

This would have to be repeated for each of the three tests they outlined (bad password, invalid password, no password).

But the tester did this:
Title: Log on using various passwords
Preconditions
1. User is general user, with general roles and permissions
2. Site is \\testserver\mainbranch\Application3.0\Login.htm
3. OS is Win64b Pro
4. Browser is FF9
Test Steps
1. Go to login page
2. Enter known bad password eg notmypassword12
3. Press Enter or click on Log On button
4. Wait for response
5. Enter invalid password eg !@#$%^&*
6. Press Enter or click on Log On button
7. Wait for response
8. Enter no password
9. Press Enter or click on Log On button
10. Wait for response
Expected Result
[4] Error message saying “bad password” will appear
[7] Error message saying “invalid password” will appear
[10] Error message saying “please enter password” will appear

The tester made some interesting comments.
Too much admin
The tester decided to mix up multiple tests in the one test case record. The problem in the testers mind was that they felt that there was too much description being replicated. They would have had to copy/paste the preconditions across all tests. They thought that if they combine tests, then there is less administration work.

The system will change anyway
The tester mentioned that there was little point in spending much time on the exercise as the system would change over time, and the tests would need to be checked each time for accuracy and relevance.

I wouldn’t follow this script myself
The tester exclaimed that they would not actually follow the test script. They find test scripts very boring and monotonous to follow. They find that they just read over the script, get some general ideas, and go and perform testing in their own way.

Documenting wastes a lot of my testing time
The tester estimated that they spent more time on documenting previously completed tests, than actually performing the tests themselves.

Illusion of structure
The tester confessed that they knew what they were doing was not adding much value. But they did comment that they thought management like to see tests all in one place, and in a table of some sort. The table gave the test script the illusion of structure.

Is this a disobedient tester or one trying to use their brain?

Author: Richard Robinson

And that got us thinking… what other testing animals make up the testing profession zoo?

• Possum testing – Playing dead in the headlights of process.  aka pathetic compliance
• Leopard testing – “Leopards are agile and stealthy predators”  – Wikipedia.  The stealthy tester is well camouflaged and appears possum-like to the untrained eye.  But quietly, the leopard is an agile hunter and will do valuable testing unseen and below the radar.
• Peacock testing – A deliberate attempt to wow via a magnificent display of extravagant test cases and scripts
• Meerkat testing: Always sticking your head up and being receptive to new information.  First to stand when overhearing a BA talk to a developer and go “wait, what was that?”
• Ostrich testing: Testing with your head in the sand.  If it ain’t in the requirements document, I don’t want to know about it.  It doesn’t exist until it’s written down.
• Penguin Testing: Testing in a tuxedo.  Black Tie testing.  Or Formal Testing.
• Blue Whale Tester: Big ‘fish’ but all blubber.  Has to stay near the surface lest it gets out of its depth.
• Manatee Tester: Noone really knows what they do or what they are.  They just seem to sit there
  

  Noone knows what they actually do

Have you spotted any other testing animals in the wild?

Author: Aaron Hodder

The question in my mind was “Why do we keep on producing such cr*p software that people just don’t like to use?“. I need to caveat that a little as I solely focus on bespoke development for larger organisations here.

There’s lots of rational reasons why this happens. Examples are:

• Overly aggressive timelines
• No well-defined quality criteria
• Poorly understood requirements/no understanding of solution
• Bad SDLC
• Politics
  
• People
• Budget
• …

But there’s one that I actually never thought of listing….. FUN.

I can see how you go FUN?!?!?! Yes, fun. Let me explain…

IT people are not the drones they usually appear like (what managers like calling a resource). All those IT geeks & nerds are actually fiercely creative and clever. If you look at the hobbies and little things they do outside work you’ll realize that they are massively diverse and engaging. I’m talking about the guy that developed a new open source phone interface stack that outperforms anything on the market or the guy that did a full automated regression suite for some game because it was the cool thing to do. All of that with no pay at all.

What motivates them? Two things: challenge and fun

At work we totally ignore fun (not the challenges though). Projects are dull, rigid, predictable; everything but fun. Fun is difficult to plan for. It’s revenue neutral (is it really?). It’s something that the company has just shied away from banning in their work contracts (for legal reasons).

Fun you have usually working with all the diverse and interesting people. Project content wise….

Imagine working at Ferrari and the only important requirement is to get a soul from A-to-B.  The end product certainly wouldn’t be a Ferrari!

I am sure everyone working for Ferrari is really engaged in the product and will have at least a modicum of fun building one of the coolest machines on the planet and watching their team win the F1. Getting a soul from A-to-B is probably quite far down the list of important things (if not further qualified).

Why wouldn’t we try and harness that engagement, the fun and coolness factor? I think agile, scrum & co is a big step in that direction and my personal experience actually reflects that. Does that mean though that these things are out-of-scope for Waterfall/V-Model? I definitely don’t think so. Although I would put my money on the fact that one of the first things that would happen is that the project would go agile & scrum or something along those lines.

Maybe it has to do with trust. Freedom -and therefore fun-is bought by trusting your employees and peers. Missing trust is the cause of so much of the grief and issues on a project level. It is one of the reasons for RFPs, SOWs, contracts, reports, metrics,….

One way to increase fun, coolness and get a great product is to let the SMEs, geeks, nerds and creative heads have more say about how things are done. Even to the extent of redefining requirements and challenging business decisions. It takes a lot of trust to do that and it could go wrong. With the abysmal project success rates on large projects I doubt a case against can really be made.

But look at what you stand to gain!! It could even equate to a new gold rush in IT. Success rates might improve and $$$ wouldn’t become the only motivational tool you have to gain & keep staff (not that $$$ ever are).

So why would I start mulling over such things in a testing blog? The reason is that I look at quality of the produced product.
Let’s stay in the motoring world and imagine you’re a worker at Toyota producing Priuses. Now we all know they produce great cars with a high quality standard. Imagine that worker goes home, where he has a Ferrari 250 GTO in his garage that he is painstakingly restoring. When the restoration is finished which car do you think will have the better end quality? Which one will be “better”?

Now imagine you’d let that employee contribute to designing/engineering the new Prius. Wouldn’t you think it looked more like below? Or any other/better design than the no-design that Prius currently has.

Toyota Fun (my case in point)

The example here is a bit flawed but I think I can make my point. People do good work but they do they excel if they are personally engaged, have fun and pride in the product. That includes quality!

Or take the famous 20% employee-time thingy that Google does (did?). They say all new products come from that 20%. Probably because people are having fun with developing cool stuff. So yes, there are easy ways to harness all this creative energy. Question is then, why it’s only 20% and even more pertinent, why is not everyone following such a successful model?

From a testing perspective I can actually tell (over and above factors like complexity), when a developer/architect/BA/DBA didn’t have fun doing something. It will be riddled with defects and the product “feel” will reflect the work-for-$$$$-attitude. So I/we have a vested interest in seeing developers have fun.

On top of that I think testing something, where a developer had fun and could be creative, is way more fun and satisfying to the tester too. There is nothing as boring as testing a product that has been un-lovingly designed and assembled.

As testers, we can apply this to our own work, too.  Are we having fun while testing?  One of the issues I have with the factory school of software testing is that they are trying to eliminate the fun from testing.  Where is the fun in creating spreadsheets of scripts, and ticking them off?  Where is the personal challenge in declaring you have achieved “100% test coverage” because you have written a single test against each written requirement.  Compare that with the child-like glee in being given a new build, and the personal challenge of trying to find all the ways this program may fail in the field.  Which approach do you think would ultimately be most beneficial for the end user?

So next time you look at your process, product or whatever is in front of you in your work life, ask yourself: “Where is the fun in this?”. If you can’t find any please don’t despair or do something rash! Think about what you need to install fun 2.0 into whatever you are doing. Then try -with baby steps- to move there. You might be surprised about the returns.

Author: Oliver Erlewein

It takes time to build a community - and initially I was doing the *heavy lifting*. I so realised this is natural as it takes time for the community to *come together* and to share the load

Once the community gathers momentum, new voices/leaders emerge – The more members, the more potential leaders. The more leaders, the more momentum generated.

Enthusiasm can become a group multiplier – (paraphrased from General Colin Powell) – When discussions abound and there is energy within the group, the group becomes larger and more involved.

Like ideas/minds attract – Since the group started, I have met approximately a dozen liked minded testers with at least four or five with whom I share a similar philosophy.

Robust discussions take time to generate – it’s a safety thing for most people. They want to feel safe before they contribute

Discussion topics – What is interesting here is that there have been a number of robust conversations (certification, use of tools, performance testing and quality to name a few) that seem to ignite tester’s thoughts, ideas, opinions (or in other words passion). On one discussion thread, what was evident were the voices/leaders in this group not taking comments at face value. As testers, we have the right to question other tester’s comments/experience/statements. This is for a number of reasons:

• First – we want to understand and then possibly learn from you
• Second – we will challenge your assertions. We hope that you can defend your position and by doing so help us question our own models. It does not mean that we will agree with you but such challenges are necessary to build the craft.
• Third – it most cases – it’s not a personal attack
• Fourth – If your assertion is misleading or damaging in anyway to the craft then expect to be challenged (the end result if we don’t is a soulless, pathetic craft devoid of any creativity or thought)

Leaders emerge at unexpected times – often they have something interesting to say – at the very least, its fodder for discussion. I have been surprised at the activity of the quieter members of the group and often I have seen them raise valid and valuable points

There ARE very controversial topics that will get testers attention – It seems to me that certification is very controversial. Lets not shy away from it, lets discuss it

Most members of the group lurk – (which is not in itself a bad thing) but imagine if they actually contributed to the group?

As the groups reputation grows, so does the number of members that join – which leads to more leaders to the group

There is a ton more activity in the middle stages of the year – and a lot less at the bookends

The more vibrant the community, the more robust the conversation – which has been true this year. The group has regularly engaged in some really robust discussions such as performance testing, test tool management and certification.

The more robust the discussions, the more opportunity for *passive* learning by those members who prefer to lurk – There has been the odd comment by a *lurker* saying how much they have gleaned from the online discussions

 In summary, we have a small (184 members) vibrant community. The strongest voices tend to be those testers who align themselves with the context driven school of testing. However, there are a number of testers who don’t align themselves with any school of thought (or haven’t declared themselves as such) who contribute well.

This leads me to the opinion that we are building a better craft OUTSIDE of what I call, the template school of thought (those test managers who see testing as an artifact, standardise driven way to test). If we don’t engage in communities like software testers new zealand, then we will have less leaders participating in worthwhile forums such as KWST.

In my opinion the solution is *simple* – be involved in the craft, participate, engage, debate, discuss and learn!

Author: Brian Osman

*duh* because we want faster response times…. oh and we want to know how to scale our virtual machines…. oh and we want to tune our systems… oh and XXXXX….  there are tons of reasons. Performance testing has it’s testing rigor and we go and “hammer” the system to get at those answers.

One thing I like to do (because it’s fast and cheap) is use a calculator/spreadsheet for performance testing. I take architecture diagrams of present and future systems, infrastructure diagrams, requirements, human oracles and more and put all the numbers together. Then I check if they stack up. Like where the product tries to get 1GB of data across a 10Mbit network link in under a second. I don’t need a test to be able to tell you, that there’s a problem there.

But then it struck me today. There is something similarly simple that I am not doing (and am guessing not many performance testers do)….

Ask yourself, what is the web page that has a response time of 0.000 milliseconds and has a infinitesimally small  throughput footprint?

It’s the page that doesn’t get loaded!

Think of purchasing something online. You run through a dozen screens entering passwords, addresses, delivery types…. on and on it goes. Usually one shop worse than the next. Just as you start thinking that it is actually simpler to drive to the shop and buy the damn thing there, someone comes along and invents the 1-Click purchase. Never mind, what that did to sales of goods but think of what the advantages from the performance perspective are.

1. Fewer web pages, resources and redirects to serve up
2. Less transactions in flight at one time
3. Less database interactions
4. Less infrastructure handshaking and latency

These are just the obvious ones, there’s probably a dozen more. The example here might not even be a good one either but I think you get where I am going.

Is it not time for performance testing to look at a bit more than just response times?  This kind of analysis gets us looking beyond response times of single web pages but looks at complete flows. How much interaction does a whole flow create and can the process/business flow be optimized?

It seems to me that architects and business analysts ignore performance related issues in their designs. It can be false assumptions on what appropriate design is or just bad customer requirements that proliferate this behaviour. Too many unnecessary steps, interactive-polling overload, re-entry or confirmation of trivial data and many more that can be really annoying. Normal performance testing might highlight these issues too but the cost and effort involved might be much higher. This method is quick, easy and cheap.

So next time you front up to a performance testing gig maybe start right at the project beginning and have a look at what can be cut out of process and business flows and screen design/functionality. See if you can’t just go 1-Click. It could even make your pending performance test a lot easier by having less of and a simpler application to test. I must admit I have not yet done this but I can’t see how a simple check like this could not be worthwhile.

…and who knows, there might be a patent hidden in there too

Author: Oliver Erlewein
& thanks Aaron for the good tips for improving this post!