Troubleshooting in Kubernetes! It can be a bit of a nightmare.
When you deploy an app and it’s not working, where do you even start?
In this article I’ll talk about how to debug an app that’s deployed, and seems to be running, but for some reason you can’t access it.
I’ll show you the three things that I look at, when investigating.
But first, coffee:
That’s better. Caffeine acquired. Now we can begin.
How to start troubleshooting
Here’s a typical application architecture on Kubernetes. I’ve highlighted the three main areas that I would check:
Because Kubernetes is such a complex platform, there are quite a few places to look, to see what’s happening.
But I would start by checking these three things:
Check your Pods
Check the Service
Check the Route or Ingress
Let’s look at each of these in turn.
1. Check your app
Firstly, let’s look at at the application itself.
As I am just your humble guide to Kubernetes troubleshooting, I can’t possibly list every scenario which might cause an issue in your application. But let’s look at some common ways to troubleshoot.
You can start by checking whether the pod is in a
kubectl get pods <pod name>
Then, the next useful point for troubleshooting is by looking at the logs of your Pod:
kubectl logs <pod name>
Look for any warning or error level logs. 👀 The logs might show why a part of the application has failed to start.
Is the app missing a config file? Perhaps you need to supply some custom configuration to the app, in a ConfigMap or Secret
Is the app trying to connect to another service, which doesn’t exist? e.g. an incorrect database URL, or third-party API URL?
Is the app trying to connect to another service, but it’s using the wrong username and password?
If everything looks OK, try accessing the app from inside the Pod itself.
I open a terminal in the app Pod, and then access the app using
curl. For example, if your app runs on port 8080, then try this from inside the Pod:
Do you get a good result?
If not, is there anything obvious in the error message?
Do you see anything in the logs?
curldoesn’t exist in the Pod, can you try something else, like
What usually causes errors with apps on Kubernetes?
After debugging more than enough of my own bad Kubernetes problems, I think that apps deployed-but-not-working on Kubernetes are often down to one of a few things:
App is listening on the wrong ports
App is listening on the wrong interface (it should listen on
Wrong configuration of the app
App can’t read or write a file on disk
That doesn’t cover every possible error, but it covers a lot of them!
If your app seems to be running OK, the next step is to look at the Service.
2. Test the Service
The next step outwards from your application Pods is the Service.
The Service is the load-balancing object in Kubernetes. It’s important because it makes your app accessible within the cluster.
But a Service can easily be misconfigured. From a simple typo, to using the wrong ports, I’ve been there, done that.
Start by getting the service name for your app:
kubectl get svc
Once you’ve figured out which service it is, try a few things:
Can you access the Service from another Pod? Open a terminal inside another Pod, and try something like:
Can you reach the app?
Check the configuration of the Service (
kubectl describe svc ...). Does everything look OK? Is it pointing to the correct ports on your Pod?
If you can access the Service within the cluster, but you can’t access the app externally, then it’s probably an issue with the Route or Ingress.
What usually causes errors with Services on Kubernetes?
The reasons this might not work:
The Service is pointing to the wrong ports. The Service needs to know which port to forward requests to. If the Service
portfield doesn’t match your application’s port, it’ll fail.
The Service selector is pointing to the wrong Pods. The Service looks for Pods which match the labels it has in its
selectorfield. If the
selectordoesn’t match the labels you’ve given in your
DeploymentConfig, it won’t find your Pods!
Did you pass this step?
Then the final step in this little troubleshooting guide is to check the traffic getting into the cluster.
3. Check the route or ingress
If your app is facing the outside world, you’ll probably be accessing it via a Route or an Ingress.
So if your app seems to be working inside the cluster, the final step is to check if you can access it from your desktop.
A web browser and
curl are the essential tools for this step.
First, try visiting the app in a web browser if it’s a web app. Or, if it’s an API, then try to
curl your app’s endpoint.
If your request times out, or you just can’t get to the app at all, then you might have a problem.
You can get some information about the Route/Ingress object, either using
kubectl describe ... or using your Kubernetes/OpenShift dashboard:
kubectl describe ingress ...
Or in OpenShift:
oc describe route ...
What usually causes errors with Routes and Ingresses on Kubernetes?
What are the potential causes for not being able to access an app from outside the cluster?
The Route is HTTPS-only, and you’re accessing it via
http://. Try adding
https://to the front.
The Ingress or Route are pointing to the wrong Service, perhaps
Something’s up with the Router in your Kubernetes cluster! Call your cluster admin…
In summary then: troubleshooting your app on Kubernetes can seem a bit overwhelming. But the best way is to break it down into parts, and check each part in turn.
Check that your app is healthy and servicing requests
Check that your service is accessible from other pods
Check your Ingress or Route
Good luck, you will figure it out. And when you do, you will have learned a ton in the process.