Build a Java container CI/CD pipeline with Concourse

I recently had the pleasure of using Concourse for the first time.

(Yes I know I’m late to the party.)

Well, I have to say that Concourse is quite nice! It incorporates a lot of good practices, and is a refreshing change from the headache of Jenkinsfiles.

But I did struggle to build and deploy a Java application with Concourse. I struggled quite a lot, in fact.

What I learned about Concourse

Firstly, I had a few hitches in deploying Concourse.

This was partially down to the security restrictions of the platform I’m using, OpenShift (Kubernetes). Note that there isn’t “cloud-native” option for Concourse (a full Kubernetes-based architecture) yet.

(But I managed to get it deployed with Helm, and if you want to find out how, then read my raw notes here.)

Then, I had to understand the Concourse concepts and “way of doing things”. By its own admission, Concourse is puritanical.

Unlike stages in Jenkins, each Job in Concourse is stateless and Concourse relies on external Resources like Git repos, Docker registries and S3 buckets to store and track state.

Finally, I had some annoying issues with configuring variables, for things like usernames and passwords.

Thankfully, I think I understand things a bit better now, but it was tough going for a while.

Anyway, the result of my learnings was a Java demo app with a Concourse CI pipeline.

I wanted to share what I’d done here, so that you can benefit from my learnings and hopefully go off and build great things!

So here’s the steps I followed to build a Docker image for a Java application, with Concourse CI.

What we’re going to build

We’re going to build a simple CI/CD pipeline which compiles a Java application from source, runs unit tests, and then builds a container image.

Here’s a summary of what you’ll get if you follow this article:

• A pipeline in Concourse, consisting of two jobs, and a Resource, which fetches the source code from GitHub

• A Job which runs unit tests for the application, with Maven

• A second Job, which uses Google’s Jib to build a small Docker image and push it to a container image registry.

In my case, I’m using Quay.io as my registry, but you could push to Docker Hub instead, or your own private registry if you like.

So get your application ready. Because next, we’re going to start creating the pipeline.

The steps

Building a Docker image with Maven and Jib

There are lots of ways you can build a Docker image for your app. Dockerfile, Eclipse JKube, OpenShift S2I….

But I decided to use Jib here. Jib is a juicy little tool from Google engineering, which builds a Docker (OCI) image for your Java app.

I add the Jib Maven Plugin to my project’s pom.xml, inside the build/plugins section.

I also need to tell Jib which registry it should push the container image, once it has been built.

And passwords: what about password to the registry?! Well, we also need to tell Jib that too.

We’ll use placeholders for the username and password to authenticate to the registry, so we don’t hardcode them in the Maven POM:

Add a configuration block - this goes within the plugin element:

<build>
    <plugins>
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>jib-maven-plugin</artifactId>
            <version>2.7.1</version>
            <configuration>
                <to>
                    <image>quay.io/tdonohue/hello-java</image>
                    <auth>
                        <username>${env.REGISTRY_USERNAME}</username>
                        <password>${env.REGISTRY_PASSWORD}</password>
                    </auth>
                </to>
            </configuration>
        </plugin>
    </plugins>
</build>

That’s Jib all set up! If you want to test it out first, you can run it locally with:

mvn compile jib:build -Djib.to.image=quay.io/youruser/yourimage:latest \
  -Djib.to.auth.username=$REGISTRY_USERNAME \
  -Djib.to.auth.password=$REGISTRY_PASSWORD

…and Jib will build and push the image.

But we need to hook this into Concourse. So next, we’ll define the pipeline and tasks.

Creating the Tasks

We need to tell Concourse exactly what commands it should execute as part of the pipeline.

These commands go into Tasks, and several Tasks make up one Job. Several Jobs make up a pipeline.

The Testing task

I’m going to create the first task, which will unit test the code.

Create a file called unit-test.yml:

---
# Task definition

platform: linux
container_limits: {}

# Use the 'maven:3.6.3' image from Docker Hub
image_resource:
  type: registry-image
  source:
    repository: maven
    tag: 3.6.3

# Cache the Maven repository directory
caches:
  - path: $HOME/.m2/repository

# We will provide an input called 'src' to this Task
inputs:
  - name: git
outputs:
  - name: build

# What do we want this task to do?
# Tell Concourse the commands it should execute, in this Task
run:
  path: /bin/sh
  args:
  - -c
  - |
    mvn -f git/pom.xml test

    # Make this output available to the 'test' Job
    cp -R git/* build

I’m only running tests here (mvn -f src/pom.xml test), so that I can run this task separately from the Docker build.

We’ll see why later.

By the way, welcome to YAML-land! Yes this is a lot of YAML already, and we’re about to write some more.

A separate Task will package the app into a Docker image. So let’s create that Task next.

The Docker build task

The next task is to build the Docker image with Jib.

This will run Maven again, but this time it will skip tests and just build the image.

We’re effectively forcing this separation between a “testing” phase and a “packaging” phase. In Maven, these tasks are normally rolled together, but I want them separate so I can visually distinguish where the pipeline might fail.

Create a file called maven-package.yml:

---
# A task to build a Docker image for a Java app with Jib

# Task definition

platform: linux
container_limits: {}

# Use the 'maven:3.6.3' image from Docker Hub
image_resource:
  type: registry-image
  source:
    repository: maven
    tag: 3.6.3

# Cache Maven artifacts so we don't download the internet again
# Caches are scoped to a particular task name inside of a pipeline's job.
caches:
  - path: $HOME/.m2/repository

# We will provide an input called 'git' to this Task
# This causes Concourse to git checkout a specific commit.
# It also allows us to execute this step independently, for a specific Git commit.
# However, the parent pipeline only runs this task when the previous 'unit' job has passed.
inputs:
  - name: git

outputs:
  - name: image # Concourse creates this directory for us. This will hold the OCI image .tar

# What do we want this task to do?
# Tell Concourse the commands it should execute, in this Task
# Build an OCI image as a .tar file. Skip unit tests, because we've already run them in the previous Job
#     mvn compile jib:buildTar ... -DskipTests
run:
  path: /bin/sh
  args:
  - -c
  - |
    mvn -B -f git/pom.xml compile jib:buildTar -DskipTests -Djib.outputPaths.tar=$(pwd)/image/hello-java.tar

What does this task do?

• Receive the source code from Git

• Compile the code

• Produce a Docker (OCI) image as a .tar file, which can be used as a Concourse Resource for another action.

Finally, we need to put it all together and define a pipeline! So let’s do that next.

Creating a pipeline

The final step is to create a Pipeline in Concourse. I define this in a pipeline.yml file in my application source code repo.

(You could also define this in a ‘common’ repository somewhere, if you want to reuse it across several applications.)

So, create a file pipeline.yml:

---
# Declare the resources that Concourse will continually check and use.
resources:
- name: git
  icon: github
  type: git
  source:
    uri: https://github.com/monodot/hello-java
    branch: master
- name: image
  type: registry-image
  icon: docker
  source:
    repository: ((image-name))
    username: ((registry-username))
    password: ((registry-password))
    tag: latest

jobs:

# This test job will fetch the source code from GitHub, and execute the test task.
- name: unit
  plan:
  - get: git
    version: latest # Just process the latest Git commit, not every commit (this is the default setting)
    trigger: true # Trigger this job whenever Concourse detects a new Resource version (i.e. a new Git commit)
  - task: run-unit-tests
    file: git/ci/concourse/tasks/unit-test.yml

# This build job will fetch the source code,
# then execute a 'build' task which is defined separately in the repository.
- name: package
  plan:
  - get: git # Concourse is stateless and needs to be given an external input to this Job.
    passed:
      - unit # The 'unit' job must have passed for this job to execute
    trigger: true
  - task: maven-package
    file: git/ci/concourse/tasks/maven-package.yml
  # Take the image built in the previous task, and push it to our Docker registry Resource
  - put: image
    params: {image: image/hello-java.tar}

That’s the whole pipeline! It defines:

• 2 Resources, one for our source code in GitHub, and one which represents the Docker image in a registry

• 1 Job, which runs the unit tests

• 1 Job, which is triggered when the unit tests complete, and builds the Docker image

• Finally, a task in the second job takes the output image from Jib and pushes it to the Docker registry

Fire it up and get it going

Now we’re going to log in to Concourse, create the pipeline and set it going.

Make sure you have the fly tool installed, then:

fly -t my-server login -k -c ${CONCOURSE_URL} -u ${CONCOURSE_USER} -p ${CONCOURSE_PASSWORD}

And to install the pipeline:

fly -t my-server set-pipeline -c ci/concourse/pipeline.yml -p hello-java \
    -v registry-username=${REGISTRY_USERNAME} \
    -v registry-password=${REGISTRY_PASSWORD} \
    -v image-name=yourregistry.example.com/youruser/hello-java

Now you can run the job from the Concourse CI console, or trigger it manually like this:

fly -t my-server trigger-job -j hello-java/unit --watch

The verdict

Concourse is nice, but I’m not sure whether I would choose it for a new deployment, though.

Some reading which really helped me understand better:

• How to pass job outputs to a different job in Concourse (Stack Overflow)

If you’re using Kubernetes, then Tekton takes the same principles (e.g. building in containers) but it is completely cloud-native, and is also a supported option in OpenShift.

And there’s a wealth of other options available, like GitLab CI, which is very popular.

Whatever you choose, good luck and happy CI/CD-ing.