Debugging Docker Containers

Since my post about docker and SSL made the top 7 last week, I've been thinking about things which I may possibly know but take for granted, things which may be useful to other people. There are several things I could discuss like the inner workings of containers, the more practical side of using docker, how docker elbows its way into your iptables etc, but I think for this week's post I'm going to stick to something simple, though powerful and super easy: debugging docker containers.

One thing I've noticed while observing new users of docker over the years is that for all intents and purposes, containers appear to be these little black boxes connected together via all manner of tubes and pipes, with nothing to indicate what's going on other than a couple of lights and a lack of fire.

I imagine it looks something a little bit like this.

Now, the first you can do to find out what's going on is to check the logs of a container. Most folks know this, but you can do that using the docker logs command, like this.

$ docker logs website

[WEBSITE] Connecting to API...
[WEBSITE] Metal Tube connected to API
[WEBSITE] Loading blog posts via metal tube from API
[WEBSITE] Blog posts loaded
[WEBSITE] Listening on port 80

Sometimes, things take a bit longer. You kinda just have to wait for it, but that's fine, we can watch and wait using the -f or follow flag.

$ docker logs -f api

[API] Connecting to database...
[API] Connection established via glass pipe
[API] Request received from website, loading data... 
[API] Data loaded [ 4%]
[API] Data loaded [11%]
[API] Data loaded [19%]
[API] Data loaded [24%]

This is great for logs that are piped to the standard output (stdout) but you know, sometimes people containerise the strangest of things. I've seen all sorts, including containers within containers.

Sometimes an application will write to a log file, so docker logs won't really help you. Maybe you have a database that isn't really doing anything. Perhaps it's running really slowly. The container is running...

$ docker ps

CONTAINER ID    IMAGE         STATUS          NAMES
1e98dd08aee8    project/web   Up 10 minutes   website
b76207f35778    project/api   Up 10 minutes   api
35f9ba3ed4c8    project/db    Up 10 minutes   database

...but for some reason when we check the container logs, there is nothing apparent going on, and we have to delve deeper.

$ docker logs database

[DATABASE] Loading...
[DATABASE] Listening on port 21000

We need to find the hidden logs. To find them, we must enter the haunted labyrinth running container, which we can do easily by using the docker exec command. The way it works is that you specify a command and a container to run it in, and docker executes that command within the container for you. If we run an interactive shell then we can have a poke around the container. Psst, here's a link so you can read more about docker exec.

$ docker exec -ti database bash

root@35f9ba3ed4c8:/#

We're in. You'll notice that we used the -ti flags. According to the docker docs, -t or --tty allocates a pseudo-TTY and -i or --interactive keeps stdin open even if not attached. All we need to know is that it hooks us up to the container so we can have ourselves a little look around, and find those pesky logs.

This is a really powerful trick, and I've used it so many times over the years. Sometimes you need to look at an application that is running, or find some logs, or perhaps you need to *gulp* edit some code while an application is running without going through the standard build process.

Once you've attached to the container and spawned a shell, you're free to look around, and that's so useful. You can easily read logs.

$ docker exec -ti database bash

root@35f9ba3ed4c8:/# tail -f /var/log/database.log

[MyDB][0.0001] started spooling database contraption
[MyDB][0.0004] reading tables from an ancient .dat file
[MyDB][0.0012] unable to read data, tables.dat corrupted...

Bingo.

Sometimes, you might not be able to spawn a bash shell, especially if the container is based on a slimmed down image that's been on a juice detox, but you tend to always be able to rely on sh being available.

$ docker exec -ti database sh

# ps -p $$
  PID TTY          TIME CMD
   56 pts/0    00:00:00 sh

There are a few other useful tricks as well. You might not use these often but every now and then they'll come in handy. First up, docker top. This is like the top we all know well, except it runs within a container. Containers usually only have one process running, but that's not always the case.

$ docker top database

PID                 USER                TIME                COMMAND
2157                999                 0:35                mydb --bind_ip_all

Then you have the docker stats command. You can either run it against a specific container or view all running containers. It's useful when you want to monitor containers under load, or if you think you may have a memory leak somewhere. I've had to shorten the output a little bit, but you get the idea.

$ docker stats api

ID      NAME  CPU %   MEM USAGE / LIMIT     MEM %   NET I/O       BLOCK I/O
b76207  api   0.31%   44.52MiB / 15.64GiB   0.28%   1.75kB / 0B   1.64MB / 14.2MB

Now, when we run docker ps we see some information about a container: the image it's running, its status, when it was created etc, however containers actually have a lot more information behind the scenes such as where their filesystems are stored, what IP address they have assigned, and we can view all of this via the docker inspect command.

The actual output is much larger, so I've truncated it to give you a rough idea of what it looks like. This is perhaps 10% of the information actually available.

$ docker inspect website

[
    {
        "Id": "1e98dd08aee8b1c6e79eccf0551f4af88299a52fec567a8b27ad4711fe2ac287",
        "Created": "2020-02-19T15:25:13.582120171Z",
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false
        },
        "Image": "sha256:6d5319f761c08cb3053c676a274ce6500f31c98fb1c1fcab8ab736e39968a2fe",
        "LogPath": "/var/lib/docker/containers/1e1933e26402e3b062051c63c3fbc6d327641059213192ab6b94d1afb36484ca/1e1933e26402e3b062051c63c3fbc6d327641059213192ab6b94d1afb36484ca-json.log",
        "Name": "/website",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux"
    }
]

If you know what you're looking for, it's easier to extract that information than it is to sift through the entire document. You can do that using the --format argument. For example, getting the IP address of a container.

$ docker inspect --format '{{.NetworkSettings.IPAddress}}' api

172.17.0.3

The last trick I want to show you is less about docker, but it's still super useful. Sometimes you may have running services in containers but you're not able to connect to them. Is it a firewall issue? A networking issue? Is it an application issue? Well, whenever we have issues the best thing to do is a binary search of possible causes, that is to say, half things, and then half them again. If you can connect two processes over a network then you can rule out a network issue, for example.

We can do this using netcat, or nc. It is perhaps one of the most versatile and powerful tools that you can put in your digital toolbox. The concept is simple, on one end you create a listener, and on the other end you create a connector.

Let's create a bridge network and two containers to demonstrate: one alpine, one ubuntu. Sometimes nc is installed (alpine) and sometimes we have to install it (ubuntu) so that's what we'll do.

$ docker network create metal-tube

7507bfb685510ae7e93f808cdc2392bedf25b2912cc77abc3801d6c575795d30

--------------------------------------------------------------------

$ docker run --rm -ti --network metal-tube --name jeff alpine

jeff#

--------------------------------------------------------------------

$ docker run --rm -ti --network metal-tube --name alan ubuntu

alan# apt update -qq && apt install netcat -qqy

alan#

Okay, we're ready. So we have two containers on the same network, which we've called metal-tube. We've given them the names jeff and alan. I've changed the terminal prompts to jeff# and alan# to make this easier to read.

I've previously talked about user defined bridge networks in my post Automatic SSL with Let's Encrypt & Nginx, but the main thing to know is that they have built in DNS servers that allow you to resolve containers by their names. Don't believe me? Let's ping Alan from Jeff.

jeff# ping alan

PING alan (172.21.0.3): 56 data bytes
64 bytes from 172.21.0.3: seq=0 ttl=64 time=0.247 ms

So now, let's see if we can connect from Alan to Jeff. For this, we first setup a listener on Alan with netcat using the -l listen flag, the -p local port flag, and, so that we know what's going on, the -vv very verbose flag.

alan# nc -vvlp 9000

listening on [any] 9000 ...

Now this will hang until it receives data. Over on Jeff, we can attempt to send some data via netcat. The way we'll do this is to echo some data into the netcat process via a | pipe. We'll be using the -vv very verbose flag again.

jeff# echo "hello" | nc -vv alan 9000

alan (172.21.0.3:9000) open
sent 6, rcvd 0

Great, it seems to have worked. If it didn't, we may have have seen something like nc: alan (172.21.0.3:9000): Connection refused. Run the command again and you'll see. Once a connection to a netcat listener is closed, the listener process exits. But let's look at Alan now.

alan# nc -vvlp 9000

listening on [any] 9000 ...
connect to [172.21.0.3] from jeff.metal-tube [172.21.0.2] 39993
hello
 sent 0, rcvd 6

It worked, therefore we know that TCP connections between these two containers are working fine. Now, locally, they're going to work fine, but sometimes you may have services on different physical servers, and this is often a great way to test connectivity between machines. You don't always need to setup a netcat listener either, you can use netcat on it's own to connect to a service to check that it's up and accepting connections. That's super useful too.

There are a number of other tricks you can use to debug containers, such as overriding entry points, mapping volumes to keep an eye on the filesystem, and more, but the above are the ones I use 95% of the time. If you have some cool tricks, why not share them in the comments below?