The 7 essential stages of a CI/CD pipeline
Make your software pipelines rock solid with these essential phases.
CI/CD pipelines are a great way to get your application from code into production. A good pipeline ensures that your software is is automatically built, tested and deployed.
But if you’re designing your own CI/CD pipeline, what sort of structure should it have?
A pipeline is often divided into stages, with each stage performing a set of tasks.
In this article, we’ll take a look at the essential stages of a good CI/CD pipeline, so that you can improve your DevOps workflow.
What is the goal of a CI/CD pipeline?
The main goal of a good, end-to-end CI/CD pipeline is to reduce the risks involved in deploying software.
How does it do it?
By providing a quick feedback loop on the quality of the software, and by making the deployment process easily-repeatable.
This, in turn, allows you to deploy more frequently, which is one of the key metrics of DevOps.
So deployment ceases to be a huge once-in-a-lifetime event, and instead becomes as regular as going to the kitchen to make a coffee.
What’s a feedback loop?
When the pipeline runs unit tests on your code, and it fails a test? That’s just one example of feedback. You know that your code isn’t ready for production.
From Continuous Delivery by Jez Humble and David Farley:
[A pipeline] creates a powerful feedback loop: Since it’s so simple to deploy your application to testing environments, your team gets rapid feedback on both the code and the deployment process.
Automated deployment to production also reduces risk, as they also explain:
Since the deployment process (whether to a development machine or for final release) is automated, it gets run and therefore tested regularly, lowering the risk of a release and transferring knowledge of the deployment process to the development team.
That’s really it. So when you’re thinking of all of the different stages that you add to your pipeline, just remember that the primary goal is always about reducing risk.
An excellent pipeline should take care of the following:
-
Compiling and testing the code (Continuous Integration)
-
Producing an artifact from the code, ready to be deployed (Continuous Delivery)
-
Deploying the application to a server automatically (Continuous Deployment)
Anything extra in the pipeline is “nice to have”, and depends on what your software needs to do. But the main goal is always about getting that software tested, and into production.
The 7 essential stages of a CI/CD pipeline
So now we’ve seen what a pipeline is for, let’s take a look at some of the most common stages in a CI/CD pipeline. We put these stages into pipelines in order to support the goals above.
Now let’s dive into what happens in each of these stages.
The trigger
The best pipelines are triggered automatically when new code is committed to the repository.
You can either configure your CI/CD tool to poll for changes to a Git repository, or you can set up a Webhook to notify your CI/CD tool whenever a developer makes a push.
The main reason for doing this is to make running the pipeline automatic, and friction-free.
If you leave the pipeline to be run manually, people will sometimes forget to run it, or choose not to. It’s far better to just make this an automated step.
There is huge confidence that comes from knowing that your code is going to be tested on every single change.
Code checkout
In this first stage, the CI server will check out the code from the source code repository, such as GitHub or Bitbucket.
The CI/CD tool usually receives information from a poll, or a webhook, which says which specific commit triggered the pipeline.
The pipeline then checks out the source code at a given commit point, and starts the process.
Compile the code
If you’re developing in a compiled language like Java, the first thing you’ll probably need to do is compile your program.
This means that your CI tool needs to have access to whatever build tools you need to compile your app. For example, if it’s Java, you’ll use something like Maven or Gradle.
Ideally this stage should run in a clean, fresh environment. This is one of the great use cases for Docker containers – being able to create fresh build environments easily and repeatably.
Run unit tests
The next key element of your CI/CD pipeline is unit testing. This is the stage where you configure your CI/CD tool to execute the tests that are in your codebase.
You might use Maven or Gradle to do this in Java, or Jest in JavaScript.
The aim at this point is not only to verify that all the unit tests pass, but that the tests are being maintained and enhanced as the code base grows.
If the application is growing rapidly, but the number of tests stays the same, this isn’t great, because it could mean that there are large parts of the code base which are untested.
According to Martin Fowler, the bulk of your tests should be unit tests. Unit tests offer the biggest “bang for your buck”. They are easy to write, cheap to run and have the lowest cost to maintain.
Package the code
Once all of the tests are passing, you can now move on to packaging the code. Exactly how you package your application depends on your programming language and target environment.
If you’re using Java, you might build a JAR file. If you’re using Docker containers, you might build a Docker image.
Whatever packaging format you choose, a good practice is that you should build the binary only once. Don’t build a different binary for each environment, because this will cause your pipeline to become very complex.
Run acceptance tests
Now comes the time to perform acceptance testing on your application.
Acceptance tests are a way of ensuring that your software does what it is meant to do, and that it meets the original requirements.
But manual acceptance testing is very tedious and time-consuming. So there are a growing number of ways that you can perform automated acceptance testing.
What is automated acceptance testing?
As defined by Børge Haugset and Geir Kjetil Hanssen:
The basic idea of automated acceptance testing is to document requirements and desired outcome, in a format that can be automatically and repeatedly tested – very much based on the same philosophy as for unit testing.
Acceptance tests generally cover the functional parts of the application. For example, on a shopping website, an acceptance test might ensure that a user can add a product to their basket.
A lot of acceptance tests can be automated. You can use tools like Cucumber and its Gherkin syntax for describing acceptance test criteria, or Selenium, which can simulate user actions, such as opening web sites or clicking on links.
The point of this stage is to reduce or eliminate the time spent manually testing.
If your current process requires building a binary, and then sending it over to a test team to be manually tested, then you will know that this takes a lot of time. BBy automating as many tests as possible, you can reduce the time it takes to get your software into the hands of real users.
Delivery or Deployment
Finally, when the application has been tested, it can move into the delivery or deployment stage.
At this stage, you have an artifact ready to be deployed (continuous delivery). Or, you can continue to CI/CD heaven and automatically deploy your software (continuous deployment).
Most pipelines don’t make it to this final stage, and that’s a shame. There is enormous value in being able to automatically deploy your software into production.
To achieve continuous deployment, you need a production environment to deploy into. If you’re deploying to a public cloud, you can use the cloud provider’s API to deploy your application. Or if you’re using Kubernetes, you might use a Helm chart to deploy your app.
And finally, this is the end of your pipeline!
The next time a developer commits code into the repository, it will be run all over again.
How to make sure that your CI/CD pipeline is a success
Once you’ve got your pipeline stages set up, how do you ensure it’s a success? How do you challenge any of the blockers or push-back that you might get?
It’s difficult implementing a full end-to-end pipeline, especially if you’re used to working in the traditional way of manually packaging software and passing it between different teams. You might get a lot of pushback.
So here are three things I make sure to do, to ensure the pipeline is a success:
-
Invest in writing tests: Some people are sceptical about CI/CD, because they think there’s a chance that your software will go into production with bugs. This is of course a risk. You can minimise it by ensuring you have a proper suite of unit tests and acceptance tests.
-
Make it easy to execute: Even the most well-designed pipeline will struggle to gain traction if it’s difficult to run, or perhaps if it can only be run by people with the right permissions. Set up a trigger to ensure that your pipeline runs automatically on every single commit.
-
Keep credentials safe: Some teams fear that a CI/CD pipeline means sharing “the keys to the kingdom”, by giving out test or production server credentials. Sure, there are some risks, and you don’t want your pipeline to be compromised! So read the documentation for your CI/CD tool thoroughly, observe any security settings they recommend, and don’t leave passwords anywhere where they could be found.
For most problems, there is usually a solution. Many of the world’s biggest software companies are using CI/CD to release dozens, hundreds, and sometimes thousands of times per day. If they can do it, you can too!