Ranger4 DevOps Blog

DevOps and Frictionfree Change

Posted by Helen Beal on Tue, Sep 19, 2017 @ 14:09 PM

Change Workshops Banner.jpgWe spend a lot of time talking about change at Ranger4, as you may well imagine, in the context of DevOps. We talk about:

  • Increasing throughput and quality simultaneously as a key DevOps goal
  • Where Change Approval Boards are a key bottleneck (The Theory of Constraints and The First Way: Flow) - particularly when we are Value Stream Mapping
  • What DevOps Target Operating Models look like and how peer-review change works, and what Change Management roles look like

I've been sharing this article, Change Goes Away, a lot in the DevOps Foundation courses we have been delivering in recent months. I love it partly because it's so crisply written but mainly because it's by Rob England, a self-styled IT Skeptic, and who has a primarily IT Operations background. This last part is particularly important since it's often the dev people who want to do away with change, and the IT ops people who want to retain the governance that change management policies and procedures bring - so to have an IT ops person to be such a strong proponent of change changing (!) like this really supports what DevOps is doing here. And gives solid advice on how to manage this potential conflict, together.

Key points made in Rob's article include:

  • Shift Left: Change and quality are tested and governed earlier in the process
  • Distributed Authority: Self-organising teams have autonomy to approve change
  • Loose Coupling: Reducing dependencies allows us to make changes in isolation
  • Automation: Change is automated where possible injecting predictability
  • Small Batch: Doing things little and often reduces risk, impact and makes habits

He also says:

"This doesn't mean we don't have a Change Management team, but they no longer function as gatekeepers for tickets. They watch the flow of change and assure quality."

The annual State of DevOps Reports have, since 2014, highlighted peer reviewed change as a statistically observed attribute of higher performing IT organisations (and there is a correlation between highly performing organisations and highly performing IT organisations). In 2017, the report highlighted the differences in the amount of change that happens manually between high, medium and low performers - it breaks down like this:

  • High: 48%
  • Medium: 67%
  • Low: 59%

You see the 'hump'? Were you expecting a smooth reduction in manual effort as the organisations matured/improved their performance? The authors of the report explain this phenomenon thus:

"At this phase of their DevOps journey, medium performers have automated enough of their work to see real results. But they have also discovered a mountain of technical debt — and that technical debt is holding them back. So the medium performers use the time freed up by increased automation to burn down the technical debt that blocks their further progress. And the ramifications of addressing that technical debt can cause teams to institute more manual controls around changes, adding layers of process that ultimately slow them down."

Most people know that they have technical debt, they feel it through firefighting and constraints every day; but few people measure it. As with quality, that's hard to do - but you can measure unplanned work and rework and these are good proxy metrics to help us get a handle on it and see improvements happen. The authors of The State of DevOps Reports' advice reflects England's observations too:

"While the desire to add more manual controls is understandable, we urge organizations to resist this temptation. Our guidance to teams at this stage is to instead begin shifting the change review process to an earlier phase of the development cycle, and to rely on peer review and automated testing rather than a change review board. Ultimately, this will eliminate the need for a change review board altogether, and the team can move forward with a faster (and more reliable) process for reviewing changes."

If we also think about this in the context of Safety Culture, we have to ask ourselves what things can we do to protect ourselves from failure when change does go wrong. There are several system techniques we can utilise to help here, for example:

  • Testing strategies like canary deployments, feature toggles and blue/green
  • Pre-empting failure using Application Performance Management
  • Rapid root-cause analysis using Application Performance Management

If you want to explore how change works in your organisation and how you can remove it from being a constraint, you may want to consider one of our change workshops.

 

Topics: DevOps Cultures, Change Management