Ranger4 DevOps Blog

Measuring to Learn, Measuring to Improve

Posted by Helen Beal on Thu, Jun 20, 2019 @ 05:06 AM

Having this week delivered one of our DevOps Metrics workshops in Saudi Arabia, while it's fresh in my mind, I thought it worthwhile sharing some detail about this learning experience and how its outcomes drive DevOps adoption and evolution.

Let's start with why the DevOps Metrics Workshop exists. I do harp on about CALMS a lot, but that's because I have found it consistently the quickest and most effective way to express to people the breadth of DevOps concerns. The M is for Measurement and that's important because measurements:

  • Allow us to to learn and improve
  • Generate data which is evidence which builds trust
  • Are the foundations of hypotheses
  • Help inform us what to experiment with next
  • Align us with our purpose, vision and values
  • Form the foundation of setting shared goals

What we find though, as we engage with organisations, is that most are poor performers when it comes to measurement; they either measure nothing or far too much - usually the wrong things and any discoveries are lost in the noise. Our metrics workshop was designed to help organisations at the start of their DevOps journey:

  • Establish an initial iteration of DevOps metrics to drive their improvement
  • Facilitate learning about a wide range of metrics for ways of working including DevOps, agile, lean and ITSM
  • Identify how using certain metrics or pairs or sets of metrics influence particular behaviours

We frequently deliver it as a package of services we refer to as a 'LiftOff' package - an initial increment of discovery, learning and evolution services designed for organisations new to DevOps or early in their journey looking for a fast start and/or to set a solid foundation for their adoption of DevOps ways of working, principles and practices on their way to becoming a high-performing organisation.

As with all of our workshops, this is a highly interactive session (it's a day long for 10 - 20 people), exercise driven with the workshop facilitator's focus on helping the group discover the answers for themselves through what is partly a learning experience. The workshop flow goes something like this:

  • Start with connections stations:
    • The developer:deployment frequency histogram
    • CALMS metrics map
    • Project Aristotle characteristics tally
    • Throughput and stability ratings
  • Uncovering the WHY of measurement
  • Collecting and Connecting Metrics
  • Mapping metrics to your toolchain
  • Mapping metrics to stakeholders
  • Experimentation, emergence and evolution

We are quite clear about the outputs of the workshop:

  • As initial set of key metrics for measurement of DevOps capabilities
  • A metrics monitoring and improvement plan
  • An architecture and build plan for a DevOps dashboard

But we also need to consider the learning and discovery outcomes:

  • A sense of urgency around using metrics at the earliest opportunity in a DevOps evolution to measure to learn and improve
  • An appreciation of the available metrics from a wide range of ways of working: DevOps, agile, lean, ITSM for example
  • An understanding on how metrics drive behaviour for good and bad and how metrics can be in contention and drive balance
  • Why and where we use proxy metrics e.g. unplanned work and rework for quality
  • Acceptance that metrics themselves evolve as an organisation's DevOps fluency builds - for example, deployment frequency becomes less interesting when it's on demand and multiple times a week or day

I hope that's given you a taste for the why, what and how of our DevOps and ways of working metrics workshop - if you'd like to investigate more, please reach out at hello@ranger4.com or complete this form.

Topics: Business metrics, devops purpose, devops assessment, metrics