Jenkins X Progressive Delivery: Canary Deployment Automation
Incremental Delivery for Risk Mitigation
Companies like Netflix and Facebook have long employed incremental delivery strategies to mitigate the risks associated with deployment. This approach involves deploying a new version to a subset of users, allowing for performance and correctness evaluation before rolling out to all users. If certain key indicators are not met, a rollback is performed. We’ll explore how to implement canary releases using Jenkins X, making it easier to adopt this approach in your applications.
What is a Canary Release?
A canary release involves sending a small portion of traffic to the new version of an application and verifying its correctness before publishing it to other users. This approach allows for a controlled rollout of new features or updates, reducing the risk of widespread disruption.
How Does it Work?
Facebook, for instance, first provides the new version to internal staff and a small number of users before rolling it out to all other users. You don’t need to be Facebook to implement canary releases, though! Martin Fowler’s website provides more information on publishing canary releases.
Jenkins X and Canary Releases
If you already have an application in Jenkins X, you can use the jx promote command to promote it to the “production” environment. However, with the introduction of canary releases, you can now automatically and progressively introduce the new version to a certain percentage of users. If a failure occurs, the application will automatically be rolled back without human intervention.
Note: This new feature is still in its early stages, and the steps outlined below may not be required in the future as they will be automated by Jenkins X.
Installing Required Plug-ins
To implement canary releases, you’ll need to install three Jenkins X plug-ins:
- Istio: A service mesh that allows you to manage traffic to your services.
- Prometheus: The most popular monitoring system for Kubernetes.
- Flagger: A project that uses Istio and Prometheus to automate canary publishing and rollback.
You can install these plug-ins using the following commands:
jx create addon istio
jx create addon prometheus
jx create addon flagger
This will start Istio in the jx-production namespace and collect data.
Configuring Istio Ingress
To use Istio ingress, you’ll need to get the Istio ingress IP and point a wildcard domain (e.g., *.example.com) to it. This will allow you to use hostname routing for multiple services.
Configuring Your Application
To enable canary releases, you’ll need to add a canary.yaml file to your Helm chart directory. You’ll also need to append the following configuration to your values.yaml file:
canary:
enabled: true
traffic:
- weight: 10
percentage: 10
- weight: 20
percentage: 20
- weight: 30
percentage: 30
- weight: 40
percentage: 40
- weight: 100
percentage: 100
This configuration will send 10% of traffic to the new version for the first minute, 20% for the next two minutes, and so on.
Automated Rollback
If the canary release fails due to certain conditions (e.g., request duration longer than 500ms or more than 1% error response), Flagger will notice the failure and roll back the release if it occurs five times.
Visualizing Canaries
To visualize canary releases, you can use a Grafana Flagger panel. You can access it locally using Kubernetes port forwarding and selecting the canary-analysis panel.
Additional Information
Please note that Istio’s default behavior may prevent you from accessing external cluster Pods. You can learn how to control Istio ingress traffic to resolve this issue.
If the canary release fails and is automatically rolled back, the Jenkins X GitOps warehouse will still use the new version instead of the old version. This is expected to be fixed in upcoming versions.