Kubernetes Troubleshooting: Where’s My App (ffs)

Kubernetes

Troubleshooting in Kubernetes! It can be a bit of a nightmare.

When you deploy an app and it’s not working, where do you even start?

In this article I’ll talk about how to debug an app that’s deployed, and seems to be running, but for some reason you can’t access it.

I’ll show you the three things that I look at, when investigating.

But first, coffee:

Feed me
Just hook it to my vein pls

That’s better. Caffeine acquired. Now we can begin.

How to start troubleshooting

Here’s a typical application architecture on Kubernetes. I’ve highlighted the three main areas that I would check:

Troubleshooting apps in Kubernetes
Troubleshooting applications in Kubernetes: where to begin?!

Because Kubernetes is such a complex platform, there are quite a few places to look, to see what’s happening.

But I would start by checking these three things:

  1. Check your Pods

  2. Check the Service

  3. Check the Route or Ingress

Let’s look at each of these in turn.

1. Check your app

Firstly, let’s look at at the application itself.

As I am just your humble guide to Kubernetes troubleshooting, I can’t possibly list every scenario which might cause an issue in your application. But let’s look at some common ways to troubleshoot.

Troubleshooting tips

You can start by checking whether the pod is in a Running state:

kubectl get pods <pod name>

Then, the next useful point for troubleshooting is by looking at the logs of your Pod:

kubectl logs <pod name>

Look for any warning or error level logs. 👀 The logs might show why a part of the application has failed to start.

  • Is the app missing a config file? Perhaps you need to supply some custom configuration to the app, in a ConfigMap or Secret

  • Is the app trying to connect to another service, which doesn’t exist? e.g. an incorrect database URL, or third-party API URL?

  • Is the app trying to connect to another service, but it’s using the wrong username and password?

If everything looks OK, try accessing the app from inside the Pod itself.

I open a terminal in the app Pod, and then access the app using curl. For example, if your app runs on port 8080, then try this from inside the Pod:

curl localhost:8080
  • Do you get a good result?

  • If not, is there anything obvious in the error message?

  • Do you see anything in the logs?

  • If curl doesn’t exist in the Pod, can you try something else, like wget?

What usually causes errors with apps on Kubernetes?

After debugging more than enough of my own bad Kubernetes problems, I think that apps deployed-but-not-working on Kubernetes are often down to one of a few things:

  • App is listening on the wrong ports

  • App is listening on the wrong interface (it should listen on 0.0.0.0, not localhost)

  • Wrong configuration of the app

  • App can’t read or write a file on disk

That doesn’t cover every possible error, but it covers a lot of them!

If your app seems to be running OK, the next step is to look at the Service.

2. Test the Service

The next step outwards from your application Pods is the Service.

The Service is the load-balancing object in Kubernetes. It’s important because it makes your app accessible within the cluster.

But a Service can easily be misconfigured. From a simple typo, to using the wrong ports, I’ve been there, done that.

Troubleshooting tips

Start by getting the service name for your app:

kubectl get svc

Once you’ve figured out which service it is, try a few things:

  • Can you access the Service from another Pod? Open a terminal inside another Pod, and try something like:

    curl http://myappservice:8080
    

    Can you reach the app?

  • Check the configuration of the Service (kubectl describe svc ...). Does everything look OK? Is it pointing to the correct ports on your Pod?

If you can access the Service within the cluster, but you can’t access the app externally, then it’s probably an issue with the Route or Ingress.

What usually causes errors with Services on Kubernetes?

The reasons this might not work:

  • The Service is pointing to the wrong ports. The Service needs to know which port to forward requests to. If the Service port field doesn’t match your application’s port, it’ll fail.

  • The Service selector is pointing to the wrong Pods. The Service looks for Pods which match the labels it has in its selector field. If the selector doesn’t match the labels you’ve given in your Deployment or DeploymentConfig, it won’t find your Pods!

Did you pass this step?

Then the final step in this little troubleshooting guide is to check the traffic getting into the cluster.

3. Check the route or ingress

If your app is facing the outside world, you’ll probably be accessing it via a Route or an Ingress.

So if your app seems to be working inside the cluster, the final step is to check if you can access it from your desktop.

A web browser and curl are the essential tools for this step.

First, try visiting the app in a web browser if it’s a web app. Or, if it’s an API, then try to curl your app’s endpoint.

If your request times out, or you just can’t get to the app at all, then you might have a problem.

You can get some information about the Route/Ingress object, either using kubectl describe ... or using your Kubernetes/OpenShift dashboard:

kubectl describe ingress ...

Or in OpenShift:

oc describe route ...

What usually causes errors with Routes and Ingresses on Kubernetes?

What are the potential causes for not being able to access an app from outside the cluster?

  • The Route is HTTPS-only, and you’re accessing it via http://. Try adding https:// to the front.

  • The Ingress or Route are pointing to the wrong Service, perhaps

  • Something’s up with the Router in your Kubernetes cluster! Call your cluster admin…

TL;DR

In summary then: troubleshooting your app on Kubernetes can seem a bit overwhelming. But the best way is to break it down into parts, and check each part in turn.

  • Check that your app is healthy and servicing requests

  • Check that your service is accessible from other pods

  • Check your Ingress or Route

Good luck, you will figure it out. And when you do, you will have learned a ton in the process.