CI/CD pipelines are a great way to get your application code from source into production. They ensure that your software is automatically tested and deployed.
But if you’re designing your own CI/CD pipeline, what sort of structure should it have?
A pipeline is often divided into stages, with each stage performing a set of tasks.
In this article, I’ll take a look at the essential stages of a good CI/CD pipeline, that will improve your DevOps workflow.
What is the goal of a CI/CD pipeline?
The main goal of a good, end-to-end CI/CD pipeline is to reduce the risk involved in deploying software.
How does it do this?
By providing fast feedback loops on the software, and by making deployment a repeatable process. This in turn allows you to deploy more frequently, which is one of the key metrics of DevOps.
So deployment ceases to be a huge once-in-a-lifetime event, and instead becomes as regular as going to the kitchen to make a coffee.
What kind of feedback?
When the pipeline runs unit tests on your code, and it fails a test? That’s just one example of feedback. You know that your code isn’t ready for production.
From Continuous Delivery by Jez Humble and David Farley:
[A pipeline] creates a powerful feedback loop: Since it’s so simple to deploy your application to testing environments, your team gets rapid feedback on both the code and the deployment process.
Automated deployment to production alsp reduces risk, as they also explain:
Since the deployment process (whether to a development machine or for final release) is automated, it gets run and therefore tested regularly, lowering the risk of a release and transferring knowledge of the deployment process to the development team.
That’s really it. So when you’re thinking of all of the different stages that you add to your pipeline, just remember that the primary goal is always about reducing risk.
An excellent pipeline should take care of the following:
Compiling and testing the code (Continuous Integration)
Producing an artifact from the code, ready to be deployed (Continuous Delivery)
Deploying the application to a server automatically (Continuous Deployment)
Anything extra in the pipeline is “nice to have”, and depends on what your software needs to do. But the main goal is always about getting that software tested, and into production.
The 7 essential stages of a CI/CD pipeline
So now we’ve seen what a pipeline is for, let’s take a look at some of the most common stages in a CI/CD pipeline. We put these stages into pipelines in order to support the goals above.
Feel free to use this blueprint as a starting point for your own deployment pipelines.
1. The trigger
The best pipelines are triggered automatically when new code is committed to the repository.
You can either configure your CI/CD tool to poll for changes to a Git repository, or you can set up a Webhook to notify your CI/CD tool whenever a developer makes a push.
The main reason for doing this is to make running the pipeline automatic, and friction-free.
If you leave the pipeline to be run manually, people will sometimes forget to run it, or choose not to. It’s far better to just make this an automated step.
There is huge confidence that comes from knowing that your code is going to be tested on every single change.
2. Code checkout
In this first stage, the CI server will check out the code from the source code repository, such as GitHub or Bitbucket.
The CI/CD tool usually receives information from a poll, or a webhook, which says which specific commit triggered the pipeline.
The pipeline then checks out the source code at a given commit point, and starts the process.
3. Compile the code
If you’re developing in a compiled language like Java, the first thing you’ll probably need to do is compile your program.
This means that your CI tool needs to have access to whatever build tools you need to compile your app. For example, if it’s Java, you’ll use something like Maven or Gradle.
Ideally this stage should run in a clean, fresh environment. This is one of the great use cases for Docker containers – being able to create fresh build environments easily and repeatably.
4. Run unit tests
The next key element of your CI/CD pipeline is unit testing. This is the stage where you configure your CI/CD tool to execute the tests that are in your codebase.
The aim at this point is not only to verify that all the unit tests pass, but that the tests are being maintained and enhanced as the code base grows.
If the application is growing rapidly, but the number of tests stays the same, this isn’t great, because it could mean that there are large parts of the code base which are untested.
According to Martin Fowler, the bulk of your tests should be unit tests. Unit tests offer the biggest “bang for your buck”. They are easy to write, cheap to run and have the lowest cost to maintain.
5. Package the code
Once all of the tests are passing, you can now move on to packaging the code. Exactly how you package your application depends on your programming language and target environment.
If you’re using Java, you might build a JAR file. If you’re using Docker containers, you might build a Docker image.
Whatever packaging format you choose, a good practice is that you should build the binary only once. Don’t build a different binary for each environment, because this will cause your pipeline to become very complex.
6. Run acceptance tests
Now comes the time to perform acceptance testing on your application.
Acceptance tests are a way of ensuring that your software does what it is meant to do, and that it meets the original requirements.
But manual acceptance testing is very tedious and time-consuming. So there are a growing number of ways that you can perform automated acceptance testing.
A lot of acceptance tests can be automated. You can use tools like Cucumber and its Gherkin syntax for describing acceptance test criteria, or Selenium, which can simulate user actions, such as opening web sites or clicking on links.
The point of this stage is to reduce or eliminate the time spent manually testing.
If your current process requires building a binary, and then sending it over to a test team to be manually tested, then you will know that this takes a lot of time. BBy automating as many tests as possible, you can reduce the time it takes to get your software into the hands of real users.
7. Delivery or Deployment
Finally, when the application has been tested, it can move into the delivery or deployment stage.
At this stage, you have an artifact ready to be deployed (continuous delivery). Or, you can continue to CI/CD heaven and automatically deploy your software (continuous deployment).
Most pipelines don’t make it to this final stage, and that’s a shame. There is enormous value in being able to automatically deploy your software into production.
To achieve continuous deployment, you need a production environment to deploy into. If you’re deploying to a public cloud, you can use the cloud provider’s API to deploy your application. Or if you’re using Kubernetes, you might use a Helm chart to deploy your app.
And finally, this is the end of your pipeline!
The next time a developer commits code into the repository, it will be run all over again.
How to make sure that your CI/CD pipeline is a success
Once you’ve got your pipeline stages set up, how do you ensure it’s a success? How do you challenge any of the blockers or push-back that you might get?
It’s difficult implementing a full end-to-end pipeline, especially if you’re used to working in the traditional way of manually packaging software and passing it between different teams. You might get a lot of pushback.
So here are three things I make sure to do, to ensure the pipeline is a success:
Invest in writing tests: Some people are sceptical about CI/CD, because they think there’s a chance that your software will go into production with bugs. This is of course a risk. You can minimise it by ensuring you have a proper suite of unit tests and acceptance tests.
Make it easy to execute: Even the most well-designed pipeline will struggle to gain traction if it’s difficult to run, or perhaps if it can only be run by people with the right permissions. Set up a trigger to ensure that your pipeline runs automatically on every single commit.
Keep credentials safe: Some teams fear that a CI/CD pipeline means sharing “the keys to the kingdom”, by giving out test or production server credentials. But most CI/CD tools have a special place to hold credentials securely, so as long as you stick to the docs for your CI/CD tool, you will be fine. You can store credentials as an admin user, and ensure that regular users cannot retrieve them.
For most problems, there is usually a solution. Many of the world’s biggest software companies are using CI/CD to release dozens, hundreds, and sometimes thousands of times per day. If they can do it, you can too!
Got any thoughts on what you've just read? Sign in with your GitHub account and leave a comment.
(All comments get added as Issues in our GitHub repo here. We don't use your GitHub account to do anything else.)