Bug Investigations

Thousands of words have been written about the investigation part, and it’s usually where the information ends.  You’ve got a crack bug investigation procedure.  You’ve clearly identified your oracles, you’ve mapped your coverage, you know your quality criteria.  You’ve been patrolling the mean streets of your pre-release build, and you’ve noticed something out of the ordinary.  The adrenaline starts pumping, and you’re ready to reach for the red and blues.  We wanna take this perp down.  But hold up, bronco.  Before we grab the pepper spray, let’s talk about what happens after you have a suspect in your sights.  You’re pretty sure you want to make the arrest, but we don’t want to compromise the sentencing later.

Identification

A tester’s credibility is paramount.  You start hauling too many innocent citizens to the station, people are going to stop taking you seriously.  If there is any doubt in your mind, you need to check and double check that ‘bug’.   Are you certain you’ve got the right guy?  Will your justification for arrest hold up in court?  Check an check again.  I recently noticed the pattern used for depicting transparency had changed in a graphical editing tool I was testing.  It seemed like a bug to me, but I wasn’t sure.  I googled if there was a standard for this pattern, and I looked at other similar programs to see how they depicted the pattern.  I asked the developer if the change was intentional, and if so, why.   If the court system is clogged up with illegitimate arrests, then there is a danger that a truly dangerous defendant may get away.

Isolation

Isolation is such an important concept.  You’ve found a bug, and yes, it’s a bug beyond any reasonable doubt.  Now we need to isolate that bug.  What does that mean?  It means narrowing down all the variables involved in recreating that bug to the minimum you can.  For example, if you’re bug begins: “Sometimes the confirmation window…” or “Occasionally the application will…” you need to isolate.  Even if you can reliably recreate the bug, you need to ensure that you’re not including anything irrelevant to the bug.  You also need to ensure you haven’t focussed too narrowly.  If there are five parties to the crime, then we need to make 5 arrests, not six, and not four.  One technique is to list all the variables involved when recreating the bug.

For example, let’s say this recreates a bug: “Open image in image editor.  Resize image. Save and close.  Reopen image.”  What are some of the variables?

• Image type (jpeg, png, bmp)
• Image size in kb
• Image size in pixels
• Location opening from (folder, removable media)
• Resize image bigger, smaller…
• Save over top of existing image
• Save as new image
• Save with a different name
• Location of saving image
• Reopening image using the application UI
• Reopening by double clicking on icon

This is a simplified example, but we should be sure within reason that we have all the relevant details and only the relevant details.

Generalisation

You’ve busted the street rat.  But does he represent a much larger crime ring? It’s time to generalise the crime.  For example, you’ve isolated that when you enter a particular string in a field, an SQL error occurs. Now it’s time to find the general case. The specific case may be “Entering O’Connor in the surname field results in SQL error”. To bust this case wide open, we need to generalise.  So we hypothesise, and test.  Hypothesise and test.  In this case, the general case may be “Entering an apostrophe in any input field on this page results in an SQL error”.  Instead of potentially just bringing down a single low level street dealer, we can take down the entire crime ring.

Prosecution

We have the bug in custody.  We may even have an entire crime family roundedup.  Now to book him.  The bug report is your prosecution, and may be your only time to make your case that a) this is a bug and b) should be fixed.

Adminy stuff: Your organisation (probably) has a system for how you classify bugs give them severities.  Do follow that.  A bug that is easy to find in the system is more likely to be looked at.  Assigning an appropriate severity aids your credibility.  You may need to justify the severity in the bug report.

The Crime: That is, why is this a bug?  For violating History Consistency Heuristic?  For violating requirement 3.2.5 in the requirements document?

Evidence: Present your evidence.  I love screenshots.  Show pictures, copies of logs, copies of input data, description of the bug, and clear steps to reproduce.

Consequences: What are the consequences of this bug going unfixed?  The consequences may not be self-evident, and never assume they are.  “Because there is no warning that turning on this feature reduces render performance, there is a risk that a user could turn this on inadvertently and go live-to-air and not make frame rate.  This would lead to more support calls, and a loss of reputation.”

Conclusion

I know I for one get “bug lust”.  I see a bug and I want to jump out with the taser, and tag and bag that sucker.  But it does noone any favours making false arrests, or supplying incomplete information.  Slow down, and ensure you take the time to make a good clean arrest.

Author: Aaron Hodder