Today I am in Las Vegas at the IBM PartnerWorld Leadership Conference (more on the reasons we are here later on in another post!) whilst my team are back in the UK at IBM's site in London's South Bank running an Open Day which we titled "Integrating Continuous Delivery and Continuous Deployment". We have previously explored some of the definitions of some of the 'continuouses' that are out there, but what's happening today is that around 20 organisations are going to work through some hands on labs showing how Jenkins and UrbanCode work together, and why you would want to do this.
The Application Release Automation (ARA) market has been steadily maturing over recent years and the confusion over what build tools do and what release tools do is abating. As both an IBM and a CloudBees partner at Ranger4, we are well aware that each tool offers significant benefits, but think it's important that our customers understand:
- Where each tools' capabilities start and end
- Under what circumstances it's a good idea to use them together
Additionally, there is a whole host of provisioning tools that we see people using for release management like Chef, Puppet and Ansible - in this post we'll explore why this isn't always the best idea.Here's what Mark Roberts, the IBM Cloud expert delivering today's workshop has to say about why people choose to use UrbanCode:
- Security and governance - UrbanCode allows a high degree of control over what people can do with respect to resources (target machines), process execution (deployments that they can run) and environments in which the deployments will take place (who can deploy to dev, qa, production etc.) Records of who did what are comprehensive - down the change of a value of a single parameter. Most customers that we speak to have a deployment process today but they don't have a layer of security and governance.
- Parameter management - UrbanCode offers excellent control and granularity for parameters used in the deployment process. Whether parameters are specific to a resource, environment or individual component UrbanCode can manage them, store them and reference them from the deployment processes. Most customers have complex parameter management issues and want to get away from name / value pair parameter files to something that is easier to manage, has better security and has an audit trail.
- Plugins - Jenkins has a huge number of plugins but these tend to cover external third party systems such as source code, QA etc. UrbanCode's main focus for the plugins are deployment processes. Many customers want to get away from the complex scripted operations that have evolved over time. Such scripted operations tend to be very environment specific so the management and testing is multiplied by the number of environments and the risk on infrequently used environments is very high.
- Process design - The graphical process designer makes UrbanCode easy to use even if you are new the the technology area that is the target of the deployment. As described above the scripted operations are complex and hard to read and modify. A graphical process is much easier to use.
- Target machines - UrbanCode uses an agent based model for the communication with target machines. This makes the connection extremely secure, reliable and efficient. UrbanCode can operate over significant distances from the server to the agent but in situations where connectivity is poor or there are isolated networks an Agent Relay can be used to hop the communication and content to the required end point. Agent Relays act as a cache for content too so if there are 100 remote servers to update the content to deploy will only be sent over the long distance communication once and the relay will distribute the content to where it is needed.
- Aren't brilliant at orchestrating releases
- Or rolling back changes (failures)
- Don't shine at tracking artifacts
- Weren't designed for tracking changes
- Are not for developing applications
Reason #1: Provisioning tools aren't brilliant at orchestrating releases
Provisioning tools are focused on the end state so you describe the desired state and then the nodes are brought to that state individually. It is extremely declarative, and that works very well for provisioning. But when you're releasing applications, you need to think about to all the things happening in the middle. You need orchestration at every level of your stack to make it work:
- Dependencies between application components: Which versions in which order are being released?
- Dependencies between layers: Can you update front end, back end or database nodes at the same time?
- Dependencies on middleware: At which point do you set up the cache, database and queues?
- Dependencies between nodes: After the first node passes acceptance tests can you then roll out to all the rest of your nodes? Do you need to mitigate downtime by rolling out to nodes gradually?
You can customise provisioning tools to be sequential and efficient, and cleverer, but if it seems like you are building a release management system yourself, maybe you should just use one...
Reason #2: Provisioning tools aren’t great at rolling back changes (failures)
Engineers make promises all the time, but then often break them:
“I won’t sacrifice code quality for development speed. I won’t deploy on Friday evenings. I won’t…”
These things do happen and are often regretted, as deploying new code is hard and often produces unexpected side-effects. It's better if we can deliver value–meaning, new features or bug fixes–in short cycles, which means releasing changes frequently. But every change made comes with the risk of breaking the application in production. When that happens, you really want to be back to the last known good state immediately.
Let’s say, in the best case response to a failure, that you could restore the previous Chef recipe or Ansible playbook and reapply it. This takes time and is prone to human errors, plus when you combine this with the orchestration requirements discussed in my previous point, things get even worse. What you really need is a tool that can detect failures at any point along the rollout and redeploy last working version ondemand.
Release management tools are made to embrace the dynamic nature of application releases, so in case of a failure, everything can automatically be rolled back/redeployed – during or after a release.
Reason #3: Provisioning tools aren’t great at tracking artifacts
Unless something terrible happens, nobody's that interested in the version of MySQL in production, e.g. 5.1.1 and 5.1.3 are fine. This is why Ansible and Chef don’t come with a nice dashboard showing all your artifacts and versions in different environments. However, we are very interested in which build of the application has been tested and is ready for a production release, or alternatively which version just caused a major issue in production and calls for an immediate response.
Applications, versions and environments are the key concepts in release management. As mentioned already you can probably pull and visualize that information yourself, but then you’re already building your own release management software.
Reason #4: Provisioning tools aren’t meant for tracking changes
Tracking changes is a key benefit of the “infrastructure as code” mantra, but as soon as your application grows from being trivial into something larger, it’s important to consider what kind of changes are we talking about. Chef, for example, will tell us which packages and versions are pushed out, but not the actual changes in the application, such as:
- database migrations
- new configuration options
- dependencies between application components
None of these are pleasant to identify when the service is down and you need a fix out ten minutes ago. Unless you’re making changes to mature middleware or libraries, you’ll want a tool that tracks actual changes, not just the metadata.
Reason #5: Provisioning tools aren’t designed for developing applications
It would be nice if we weren't all stressed and tense about go-live events wouldn't it? That means “battle-testing” the application and its roll-out process long before the time when an error or failure starts to negatively affect things. Battle-testing these days is carried out commonly enough: by Operations, after deadlines, while trying to figure out why the application doesn’t even start up.
Instead, it makes way more sense to test the functionality of the changes, as well as releasing them, very early on – right when they are introduced in the development process in fact (shifting left, people!). Developers need to step up when it comes to releasing their own changes.
Provisioning tools really weren’t created with this sort of thing in mind. They’re designed for operations and infrastructure, not developers testing and verifying that application specific database changes, environment properties and configuration are rolled out successfully.
Use Provisioning tools for Provisioning and Release Management tools for Releasing
Provisioning tools are massivelt popular because of the simplicity and ease of use that comes from the focus on the desired end state. But when releasing applications, we also have a lot to consider regarding specific changes and the way they are rolled out, not just the end state of a node.
Application releases are complex, so you should accept this complexity and get an appropriate toolset that helps you with integrating releases into your development process. This way, every push to production can be given a greater vote of confidence.
Release management tools acknowledge that the release must be centered on the application as a whole, and the subsequent experience by your users. These tools focus on your applications in a way that allows them to better orchestrate, test and track releases.
It’s tempting to try and solve every problem with a provisioning hammer on your own and hold on; however, as you will inevitably end up implementing a half-baked, error-prone in-house release management tool, you might as well go with a proper solution to the problem from the beginning. Agreed?