Have you ever heard about the secret menu at McDonald’s?
It’s a set of items that don’t appear on the regular McDonald’s menu, but you can ask for them, and they’ll serve them to you.
I’ve never been brave enough to ask the server for one of these off-menu items.
And I can’t quite see myself asking for The McGangBang just yet.
But I like the idea that you can take McDonald’s basic primitives – the Big Mac, the McChicken Sandwich, and so on – and remix them to create more interesting products.
This is kind of what Custom Resources do in Kubernetes.
They allow you to extend the menu (the Kubernetes API), and define your own objects, which users can create, manage and delete inside the cluster.
Then, a controller, which watches for these custom objects, can perform actions, like creating a Deployment and a Service to install an app. (Or putting together two McChicken Sandwiches)
Want to make your own custom burger store inside Kubernetes?
In this article we’ll dive into creating a basic Custom Resource Definition!
Creating a Custom Resource in Kubernetes
(Well that got a bit weird, didn’t it?)
A Custom Resource Definition (CRD) defines the name and structure of a custom object type that you want to allow in your Kubernetes cluster.
A CRD is a way of defining what the object will be called, and what properties it can have: like size, colour or location, for example.
Are Custom Resources a way of storing application data in the cluster?
It might seem like this is a way of storing application data in the cluster, or using it like a database.
Although this is possible, it’s not a reason that I’d recommend that you’d use Custom Resources.
Custom Resources are really about adding your own custom capabilities to your Kubernetes cluster.
Generally, we would use Custom Resources to be able to make something happen in the cluster, like deploy a complex application.
I tend to think of Custom Resources as like instructions to give to a factory, which will then deploy or assemble something in the cluster for you.
For the example, I’m going to keep with the fast food theme. (Can you tell that I’m hungry as I’m writing this?)
We’re going to create a new Custom Resource Definition called the BurgerStore.
As well as running Tutorial Works, I’ve been growing a hugely successful burger franchise, and we’re now expanding into offering our franchisees an ordering app, which will be running on Kubernetes.
We’ll provide the infrastructure (the Kubernetes cluster), and all the franchisees need to do is create a BurgerStore custom resource to get going.
In the case of my CRD, I want it to allow my franchisees to define:
The address of the burger store
Which currency it takes payments in
How many instances of the app should be deployed
Given my requirements, I’ve created this example custom resource definition for BurgerStore:
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: burgerstores.tutorialworks.com spec: group: tutorialworks.com names: kind: BurgerStore plural: burgerstores listKind: BurgerStoreList singular: burgerstore scope: Namespaced versions: - name: v1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: owner: type: string description: The name of the owner of this Burger Store franchise address: type: string description: The physical address of the Burger Store currency: type: string description: The currency which this Burger Store accepts payments in replicas: type: integer description: The number of instances of the Burger Store app to deploy additionalPrinterColumns: - name: Owner jsonPath: .spec.owner type: string - name: Currency jsonPath: .spec.currency type: string - name: Replicas jsonPath: .spec.replicas type: integer - name: Age jsonPath: .metadata.creationTimestamp type: date
You’ll need cluster-admin privileges to be able to do this next command, because creating Custom Resource Definitions is something which happens at cluster-level.
Let’s create the Custom Resource Definition using
# kubectl apply -f burgerstore-crd.yml
You’ll get this in response:
That’s it. Kubernetes has created the new custom resource. The custom resource won’t do anything yet, but this is the first step done.
To check that the Kubernetes API now knows about “burgerstores”, we can use the command
kubectl api-resources. This is a handy command that “Prints the supported API resources”.
If we pipe it to
grep we’ll see that
burgerstores is in the list:
# kubectl api-resources | grep burger burgerstores tutorialworks.com/v1 true BurgerStore
Great! That’s the first part done.
Now that Kubernetes knows about the concept and structure of a BurgerStore, it will let me create multiple BurgerStore objects.
But, I’m a cluster-admin. I need to allow regular (non-admin) users to be able to create and delete BurgerStore objects. We’ll do that next.
Allowing regular users to create CR instances
Once you’ve created a CRD, it’s generally available in the Kubernetes API, cluster-wide, although you can limit exactly who can create these objects.
I only give my franchisees unprivileged access to the Kubernetes cluster, so I need to allow these regular users to create BurgerStore instances.
If you’re using Kubernetes, you can create a new ClusterRole and grant the permissions to it:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: burgerstores-admin-edit rules: - apiGroups: ["tutorialworks.com"] resources: ["burgerstores"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"]
And for the view permission:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: burgerstores-view rules: - apiGroups: ["tutorialworks.com"] resources: ["burgerstores"] verbs: ["get", "list", "watch"]
Or if you’re using OpenShift, you can grant permissions to the built-in
admin ClusterRoles, by using these special labels:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole items: - metadata: name: aggregate-burgerstores-admin-edit labels: rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rules: - apiGroups: ["tutorialworks.com"] resources: ["burgerstores"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete", "deletecollection"] - metadata: name: aggregate-burgerstores-view labels: # Add these permissions to the "view" default role. rbac.authorization.k8s.io/aggregate-to-view: "true" rbac.authorization.k8s.io/aggregate-to-cluster-reader: "true" rules: - apiGroups: ["tutorialworks.com"] resources: ["burgerstores"] verbs: ["get", "list", "watch"]
This rather wordy configuration tells OpenShift to give full access to admins (through the admin ClusterRole), and read-only access to limited users who have the view role.
Apply the config above to your cluster to make this happen.
In the next update to this article, we’ll build a website, so I can see and manage all of my burger store franchises, using the Kubernetes API! 🍔