Ranger4 DevOps Blog

Quality Metrics for Test and Defect Management

Posted by Paul Hancock on Thu, Sep 18, 2014 @ 10:09 AM

If you're focusing on improving your defect control and management processes, a challenge you may face is determining the right metrics to record, identify and track for areas of improvement. At a high level we talk about:

  • Volume
  • Frequency
  • Time to fix
  • Resource cost associated with fix

But how can we get more granular? What are the consequences of failing to fix a defect? And should all teams be measured in the same way? Agile methodologies suggest tracking of pre production defects is detrimental to software teams and it can be difficult to decide which areas to measure to provide a true picture of software quality.    

Here are some metrics you might like to consider:

1.Defect slippage
2.Defect density
3.Schedule slippage
4.Test case completeness
5.Team's skill metrics and area of expertise
5.Ownership defined in spreadsheet and shared across stakeholders.
6.Metrics of impacted areas of each functionality/feature

When you're considering point 5, your team's skills, you might want to build a skill matrix covering these areas:
• Test case coverage
• Schedule variance
• Effort variance
• Defect density & removal efficiency
• Defect slippage
• Resource utilization

Also address the quality of your defects - oxymoronic, we know, but are your defects raised with:
- Test evidence attached?
- A clear description and expected and actual results?
- Estimated fix and test dates completed for resolution planning?
- The root cause identified correctly?
- Prioritisation to be fixed within defined SLA?
- Their testing priority and business criticality assessed correctly?

In terms of measuring the financial impact of defects to your organisation and the best areas to target for improvement we'd suggest considering the questions below. This is not an exhaustive list but a starting point for considering the impact of defects as well as which areas to target for improvement.  

1. At what stage and how many defects are found across the development lifecycle?

2. What is the average time to resolve a production incident caused by a defect?

3. What is the impact of non availability of systems? 

4. How frequently do new code drops / deployments need to be rolled back or fixed?

5. Ripple effect of system outages from defects can have a detrimental impact on other staff (i.e. staff get pulled in to fix issues in an ad-hoc fashion sometimes).  If this happens within your organisation, how many people are impacted and how much time is wasted? 

Once you have answers to these types of questions you can start to calculate the financial impact of defects thus gaining a better understanding of the best areas to target for improvement. Some of the additional information you'll need to calculate the impact of defects are:

  • What is the complete cost per hour for each resource (£75 for Dev, £50 for Q/A, etc?)
  • For each resource, what percentage of their time is spent on actual project work? (Dev – 75%, Q/A – 60%, Performance, etc)
  • What is the wait time/down time within project teams when a production incident is found?
  • Does your organsiation suffer a financial impact of downtime/poor system performance? For example this could be a loss of revenue from orders through your website 

Once you've put some financials around these areas it should become clearer as to the biggest opportunities for improvement. When considering areas for improvement you should also consider the difficulty and impact of making these changes. The largest opportunity might be difficult to implement or have a large impact for on-going projects. When considering the opportunities for improvement we find it's useful to use a matrix like the one below to decide on which are the best areas for more focus.  

Figure1 resized 600

Once you've decided on the areas for focus you'll probably need senior management commitment and buy in. Think about building a business case which is concise, explains why the changes are being proposed, details the financial return for the business and how long will it take before the business sees the benefit. Without senior management buy in it will be difficult to implement the improvements at the velocity you need.

In summary, focus on metrics, business benefit for your organisation and ensure senior management buy in from the outset.

Topics: test automation