Automating Kubernetes Canary Deployments with GitOps

In high-velocity development environments, pushing code directly to production can lead to regressions that affect all users. Canary deployments mitigate this risk by routing a small percentage of traffic (e.g., 5%) to the new version, verifying its health using live metrics, and gradually scaling it up to 100%. In this post, we'll set up automated canary deployments using Argo Rollouts and Prometheus.

Introducing Argo Rollouts

Argo Rollouts is a Kubernetes controller and set of CRDs that provides advanced deployment capabilities such as blue-green, canary, and canary analysis. It replaces the default Kubernetes Deployment object and integrates with ingress controllers and service meshes to shape traffic dynamically.

Step 1: The Rollout Manifest

Instead of a standard Deployment, we define a Rollout resource. We specify a canary strategy with steps that increment the traffic routing percentage and pause for analysis.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  replicas: 5
  strategy:
    canary:
      analysis:
        templates:
          - templateName: prometheus-error-rate
        args:
          - name: service-name
            value: payment-service
      steps:
        - setWeight: 10
          pause: { duration: 5m }
        - setWeight: 25
          pause: { duration: 10m }
        - setWeight: 50
          pause: { duration: 15m }

Step 2: Configuring Automated Analysis

An Analysis Template defines how to query metrics from a monitoring system (like Prometheus) to validate the new canary version. If the error rate exceeds a specified threshold during the analysis window, Argo Rollouts will automatically abort the rollout and restore the stable version.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: prometheus-error-rate
spec:
  metrics:
  - name: success-rate
    interval: 1m
    successCondition: result[0] >= 0.995
    failureLimit: 2
    provider:
      prometheus:
        address: http://prometheus.monitoring.svc.cluster.local:9090
        query: |
          sum(rate(http_requests_total{status=~"2.*|3.*", service="{{args.service-name}}"}[2m])) 
          / 
          sum(rate(http_requests_total{service="{{args.service-name}}"}[2m]))

How GitOps Ties It Together

With GitOps, the source of truth is your Git repository. When a developer merges a change:

1. GitHub Actions builds the new Docker image and updates the tag in the Git repository manifest.
2. ArgoCD detects the drift, syncs the state, and applies the new Rollout resource to the cluster.
3. The Argo Rollouts controller starts routing 10% of traffic to the new replica set and initiates the Prometheus Analysis run.
4. If success rate drops below 99.5% twice, Argo Rollouts aborts the deploy, sets traffic back to 100% stable, and alerts the SRE team via Webhooks.