Integrate Functional Testing as Part of Your Deployment
This is a standard best practice. Functional tests should be part of your CI/CD pipeline(s). Failing any of those tests should stop the pipeline(s) from deploying any further, and trigger a rollback as needed.
Integrate Resiliency Testing as Part of Your Deployment
This is a more advanced best practice, but resiliency tests should also be part of your CI/CD pipeline(s). Just as for functional testing, failing any of the resiliency tests should stop the pipeline(s) from deploying any further, and trigger a rollback as needed.
Deploy Using Immutable Infrastructure
The idea behind immutable infrastructure is that you don’t update any component of the infrastructure in place. No OS patching or upgrade, and no software patching. If you need to upgrade or patch any component, for whatever reason, a new version of that component will be deployed and the existing one terminated. How it happens is down to you and may depend upon several factors, including the magnitude of the change, or whether you are allowed some downtime or not. You can adopt a blue-green or a canary deployment strategy to completely or progressively migrate to the new release. Once you’re satisfied with the new release’s behavior, the old environment is decommissioned.
There are many benefits to immutability. By replacing components with every release, the infrastructure is reset to a known state, avoiding configuration drifts. Deployments are also simplified because you don’t need to support updates or upgrades. Deployments are atomic: either they complete successfully, or nothing changes. You build once and deploy (the same code base) to multiple environments, which prevents inconsistent environments. All of this builds more trust in the deployment process and simplifies testing.
Deploy Changes with Automation
You’ve seen it already in the previous sections and chapters, and you will keep seeing this recommendation again and again throughout this book: automate everything (or as much as possible). Deploying a release in a production environment should not be left to humans, or at least as little as possible should be done manually. Automated processes will never miss one step due to distraction, or execute steps in the wrong order. Humans make these mistakes all the time. So, get rid of the humans in the deployment process as much as you can.
To help you with that, you can leverage the AWS Code suite (AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy) or your preferred development toolchain.
Should you remember only one thing from this chapter, it is this: “Everything will eventually fail over time,” (Werner Vogels, CTO of Amazon.com). Failures are a given and it is better to not be under the illusion that they can be prevented forever, however good your design may be.
This is another thing that seems obvious but that is often overlooked; backing up data is paramount, and making sure you can recover with your backup data is even more important. This section will only briefly discuss backups as in Chapter 7, Ensuring Business Continuity, will dive deeper into defining a backup strategy.
So, you want to back up your data, your workload configuration, and everything you need to meet the specific business requirements of your workload. Two requirements, in particular, will define your backup strategy: recovery time objective (RTO) and recovery point objective (RPO). In some cases, you may not even need a backup. Can you reproduce all your important data from other sources to satisfy your RTO and RPO? This is the first thing to validate because you don’t want to go and spend a humongous effort and loads of money on the greatest backup solution ever if you actually don’t need one.
If backups are essential for your workloads, you want to define them, set them up, automatically run them, and forget about them. The latter is a figure of speech. You still want to verify that your backup process works fine.
The major backup solutions out there, which you can find on AWS Marketplace, including AWS’ own solutions, allow you to run your backups on a schedule.
Take AWS Backup as an example. It provides a fully managed, policy-based backup solution. It lets you centralize, enforce, and automate the backup of data for multiple AWS services across your entire AWS organization.
Last but not least, you want to perform periodic recovery of your data to verify backup integrity and processes. This is one of the most critical aspects of backup because backup data that cannot be recovered is useless. So, it’s essential to ensure that your data is recoverable by performing a recovery test.