As part of our ongoing efforts to streamline processes within Engineering, we are implementing a centralised approach to Kubernetes deployments. This centralised process is called the Service Contract Pipeline, which will serve as our standardised deployment method.

Why do this?

As a technology company grows, the ability to deliver quickly and effectively becomes an ongoing challenge. With more teams being created to deliver a variety of services and features, their individual processes can easily start to deviate over time. Despite the fact that they are often working towards the same goal, they end up doing things in different ways. This growing complexity becomes difficult to manage and can only scale to a certain extent.

By implementing centralised tooling, we can eliminate unnecessary duplication of effort across teams and ensure that everyone is working according to the same standards and best practices. Standardising the deployment of new features and fixes for the services we provide is a primary focus of the centralised Service Contract Pipeline. Here are some of the key benefits:

  1. Enhanced security for our deployment process and the ability to enforce standards using quality gates.
  2. Simplified integrations with other tools for better observability, communication, and reporting.
  3. Improved developer experience allowing teams to focus on feature development.
  4. Reduced support burden on our SRE team through automated processes and a self-service interface.
  5. The ability to abstract from specific deployment tools (such as Helm or Argo CD), ensuring that our application code remains decoupled from them.

By implementing the Service Contract Pipeline, we aim to create a more efficient and standardised deployment process to help us overcome the challenges that come with growth and complexity.

This solution is just one component of a comprehensive toolkit being developed by SRE to enhance standardisation. These tools will provide a primary supported path for our software delivery.

How will this work?

We currently manage our deployment workflow using a collection of repositories, broadly split by organisational areas, that contain several configuration files. These configurations are nested on multiple levels, and when combined generate deployment configurations, per service, per cluster. The multiple layers of configuration can be confusing, especially when shared across multiple deployments.

To simplify our deployment workflow, we have introduced a new single interface called a Service Contract, which aims to provide a more straightforward and efficient method for managing deployments.

What is a service contract?

It's a YAML configuration file that includes important metadata about a deployment. In simple terms, it specifies the service to deploy (as defined in our Backstage service catalogue), the target Kubernetes cluster, and the Helm configuration for the deployment.

Here is a simple example of a service contract:

apiVersion: deployment-config/v4
service: example-backend-service
environment: testing-aws-eu-west-1
namespace:
  name: example-backend-service
slack: '#owning_team_channel'
appVersion: 1.2.3
helmChart:
  name: generic-application-chart
  version: 1.0.0
  values:
    deployments:
      example-backend-service:
        containers:
          - name: hello-world
            repository: internal-registry.io/sre/hello-world
            tag: 1.2.3

We currently support a simple yet flexible helmChart setting that allows deploying helm charts we manage internally. We provide a generic application chart that allows for the deployment of most native Kubernetes resources, and some custom resources that are enabled across our clusters.

We also support some Argo CD features in the service contract to:

  • Configure the Argo CD Application that will manage this deployment
  • Configure an Argo CD ApplicationSet to enable automated preview deployments for testing

What does the pipeline do?

The pipeline is part of our continuous deployment (CD) solution and is implemented using Argo Workflows, a Kubernetes-native workflow engine that enables the creation of powerful and scalable pipelines.

The pipeline offers several key features that enhance our deployment process:

  • Syntactical validation and business rule checks: It ensures that the service contract is properly structured and complies with the required rules and standards.
  • Generation of Kubernetes manifests: Based on the information provided in the service contract, the pipeline generates the necessary manifests that define how the deployment should be configured in the Kubernetes cluster.
  • Committing manifest changes to Git: The pipeline automatically saves the changes made to the manifests in Git. This allows for easy version control and ensures that the deployment changes are tracked and recorded.
  • Integration with Argo CD: Once the manifest changes are committed, the pipeline facilitates the promotion of these changes to the target cluster using Argo CD
  • Deployment progress updates via Slack: Throughout the deployment process, the pipeline provides real-time updates on the progress and status of the deployment through Slack. This helps keep the team informed and enables efficient communication during the deployment phase.

These features work together to ensure that deployments are validated, generated, and promoted in a controlled and transparent manner, providing a reliable and efficient deployment process for our services.

How to trigger deployments using the pipeline?

Submitting service contracts to the new pipeline is automated. All you need to do is create a merge request with the desired changes, and the deployment is triggered once merged into the main branch.

Next steps

The SRE Pipeline team is actively working on introducing new features to enhance the Service Contract Pipeline. These upcoming additions include:

  • Integration with security and testing tools: We are integrating reports from SonarQube code quality scanning and CI unit tests to improve the overall visibility of our SSDLC process.
  • Evaluation of infrastructure deployment tools: As part of our development plans, we aim to extend the Service Contract pipeline to handle cloud infrastructure deployments in addition to Kubernetes resources. We will be assessing Crossplane in the near future.