Ranger4 DevOps Blog

Measuring Your DevOps Success

Posted by Helen Beal on Fri, Mar 28, 2014 @ 15:03 PM

A survey from the Vanson Bourne market research agency (with CA) late in 2013 indicated that 39% of those surveyed had adopted some form of DevOps and 27% were planning to do so in the near future. Despite this being such a hot topic in the IT sector, and with a high level of take up, the question we are still most commonly asked is: “Where do we start?”

Our answer is always that an organization’s current position must be baselined first. Having a baseline means you can build a business case, apply targets and goals to your projects and measure your success as you progress through your project ultimately reporting back to the board on how you used the money to save or make more money - and improved your teams’ satisfaction.

DevOps Metrics for Baselining and Measuring Success

There are hard, quantifiable technical and financial metrics we can take, such as:

  • Number and frequency of software releases
  • Volume of defects
  • Time/cost per release
  • MTTR*
  • Number and frequency of outages / performance issues- Revenue/profit impact of outages / performance issues
  • Number and cost of resources**

It’s worth noting that one of the biggest inhibitors to success of DevOps and relating tooling projects is people’s perceptions that they are at risk of losing their jobs as their work becomes automated. Often, particularly in areas like release and deployment management, we find that there are specific individuals who hold all the knowledge around a current process (they wrote all the scripts for example) and who are viewed as heroes when they are the only person who can fix an issue and often do it out of hours and at short notice - but are, in fact, bottlenecks. These individuals are often highly talented, but feel secure in the indispensable role they have created for themselves. Though they will often be happier freed up to do more creative and rewarding work, they are often fearful and this needs to be addressed.

Cultural Metrics

Although cultural metrics are softer, nigh on impossible to apply hard dollar value to, DevOps is about resolving conflict in the workplace, eliminating stress and avoiding burnout - and they are measurable. Happy people are more productive - their health is better, they have more ideas, work more effectively and will put in the extra mile. You can measure across a number of key cultural indicators around feelings about change, failure, going to work, what a typical day’s work entails, in addition to a number of cultural attributes such as:

  • Cross-skilling, knowledge sharing and pairing between teams
  • Working in a fluid but focussed manner
  • Working in multi-disciplinary teams
  • Organizing teams around projects rather than skill-sets
  • Constantly dancing on the edge of failure (in a good way)
  • Position around business demand
  • Extraneous lines of code
  • Attitude to continuous improvement
  • Obsession with metrics
  • Technological experimentation
  • Team autonomy

You can also look at a number of team features such as:

  • Rewards and feelings of success
  • Hierarchical and political obstacles and annoyances
  • Inspiring and fostering creativity

Process Metrics

DevOps is not a process or a tool - but there are a number of processes in the software development lifecycle (SDLC) that affect both traditional development and operations staff to greater or lesser degrees that need to be taken into consideration. All of these process components can be optimised, and all of them can then be improved upon further using appropriate software tooling. An ultimate goal of a typical DevOps project is often to attain true continuous delivery (CD) by linking these processes and tools together to allow fully tested, production ready, committed code to proceed to live without impediment - we often refer to the software infrastructure piece of this as the DevOps toolchain. When baselining current state, it’s useful to measure these component processes and their relative maturity (taking into account use of existing tools and success of implementation). Typically, we look at:

  • Requirements elicitation and management
  • Agile developmen
  • Build
  • Release and deployment
  • Unit testing
  • User Acceptance Testing
  • Quality Assurance
  • Application Performance Monitoring
  • Cloud

How to Influence Metrics

Once you’ve baselined your current position, it’s time to think about your desired future state. Your baselining exercise will probably have highlighted where the key bottlenecks are and areas on which to concentrate. Although we preach, “People, then Process, then Tools”, there are tools, in particular Application Performance Management that can help discover bottlenecks and issues upfront - although it’s imperative you have the right people then using and acting on this data and put the right processes in behind in terms of defect tracking, corrective development, versioning build and deployment.

You might be stumped as to ideas on how to influence cultural change, and whilst making change happen, especially in well-established enterprises can be challenging, it’s by no means impossible. The key is understanding the current position, of course; is there a blame culture? What happens when there is a production outage? How motivated and rewarded do staff feel? And then establish a programme of cultural initiatives to address these.

DevOps Tools for Change

There are a number of tools that can influence the harder and software metrics - for example:

Application Performance Monitoring (APM)

  • Reduces MTTR
  • Makes it easier to create a collaborative approach in teams dealing with issues
  • Identifies root cause fast, eliminating blame games

Application Release Automation (ARA)

  • Enables development to seamlessly pass code to operations who can quickly and consistently deploy into production
  • Enables instant rollback or redeploy when an error is identified by APM in production
  • Reduces fear of failure as rollback/redeploy is so easy

Integration Testing & Virtualization

  • Mimics the production environment so successful test are guaranteed to run
  • Allows testers to ‘shift left’ in test process and collaborate with developers early
  • Fast testing enables fast, confident throughput of change

When to Measure, How to Tweak

When you start delivering your fit-assessed, prioritised DevOps initiatives, the measurement starts immediately and is constant. You’ll be looking for upward trends as well as down. Make sure you share reports regularly with the team - try weekly with the core team and monthly with the extended team. Highlight success and elicit ideas for improvement where areas have proved more challenging. Try things - tweak, monitor, tweak again. But remember: “Any improvement not made at the constraint is an illusion.”

What to Do with Success

Celebrate success! Create rewards and incentive programmes for teams when metrics targets are achieved. Part of the DevOp’s agenda is about improving working conditions - depressurising

Why ask for an external DevOps Maturity Assessment

Whilst no-one’s going to understand your business as well as yourselves, we often meet organisations who are struggling to find the time to save time - they know there are improvements to be made but they are so busy with BAU and firefighting they can’t conceive of stopping and taking stock of their current position. Also, humans being emotional creatures, naturally all working environments have some level of politics or hierarchies going on and sometimes it helps to have an outsider take a pragmatic, neutral view of a situation. So if you’re ready to baseline your current DevOps state and identify the DevOps initiatives that will have real, positive impact on your business but feel you don’t have the time or wherewithal to figure it all out yourselves - please do get in touch.


* The MTTR is the Mean Time to Repair, Resolve or Resolution - each of the definitions means the same and can be used interchangeably. This term is more commonly used when talking about Application Performance Management and the speed at which an outage or performance issue can be fixed, but equally can be used when talking about testing and eliminating defects


Topics: DevOps Cultures, DevOps Maturity