Overcoming deployment anxiety
As a Developer with 15 years of experience, I still sometimes get "deployment anxiety" - when I've backed up the database and tagged a release, but even though the CI pipelines are passing and the staging site is working, I'm holding off on pushing the latest code to be released to production - trying to think of any potential issues that could arise from this deployment and avoid any downtime.
When I thought about this further, the releases that I've felt anxious or nervous about have usually been in at one or both of the following categories:
- The release includes a lot of changes, and maybe a combination of different types of changes such as framework or CMS updates, bug fixes, or new features.
- It's been a long time, maybe weeks or months, since the last production release.
The best way to resolve both of these issues, I think, is to break down the large releases into smaller ones, and to deploy them more frequently.
In the opposite scenario, the releases where the changes are small and it's been a short time since the previous release - ideally minutes or hours - have been the ones where I've been the least nervous.
If a single commit is being released, then I can be confident that if there is a failure, I can either revert it and put things back the way they were or quickly identify the issue and push a fix. This isn't the case for large changes as the potential source of the failure is larger and it will take longer to find and fix.
If a bug fix or a feature needs to be reverted, I'm happy knowing that I can do that easily without also reverting the CMS update that was deployed separately - rather than them all being released together.
There are other advantages too - clients or product owners are generally happier if the new feature or fix that they requested is on production within hours or days rather than weeks or months, and having your latest code deployed to production rather than on a staging branch makes it a lot easier if you need to deploy an urgent fix or security update.
If you're familiar with the DevOps Research and Assessment (DORA) team, three of their key metrics are deployment frequency, lead time for changes, and time to restore service. All of these are improved by small and frequent releases.
In my Deployments with Ansible and Ansistrano talk, I mention that there is a separate rollback role, but I don't think that I've ever used it.
Because I'm deploying small changes often, it's usually much easier to fix forward than it is to rollback, and knowing this makes me a lot less anxious when deploying changes.
- Oliver
