This post demonstrates how to externally audit or debug (docker) container. Please note that docker exec is the current recommended approach.
For purposes of this post I will be using demo container - simple HTTP service built from the scratch as a base (empty image). This means that demo does not contain any other tools except the application binary.
$ docker run --rm --name demo -d kirecek/hello-world
$ docker exec -it demo /bin/sh
...executable file not found in $PATH": unknown`
This is a very common error when devs try to debug applications inside containers. The following step, after the error, is usually to install missing packages and continue with debugging.
But there is another way!
Containers are only isolated group of processes running on a single host, leveraging the linux kernel namespaces. You can not see and reach parent processes in isolated ones, however, vice versa you have full control. Let’s take a closer look at our demo container.
$ pid=$(docker inspect -f '{{.State.Pid}}' demo)
$ sudo ls -l /proc/$pid/ns
lrwxrwxrwx 1 root root 0 Nov 6 19:27 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Nov 6 18:48 ipc -> 'ipc:[4026533099]'
lrwxrwxrwx 1 root root 0 Nov 6 18:48 mnt -> 'mnt:[4026533097]'
lrwxrwxrwx 1 root root 0 Nov 6 18:32 net -> 'net:[4026533102]'
lrwxrwxrwx 1 root root 0 Nov 6 18:48 pid -> 'pid:[4026533100]'
lrwxrwxrwx 1 root root 0 Nov 6 19:52 pid_for_children -> 'pid:[4026533100]'
lrwxrwxrwx 1 root root 0 Nov 6 19:27 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Nov 6 18:48 uts -> 'uts:[4026533098]'
$ sudo ls -l /proc/self/ns
117524 lrwxrwxrwx 1 root root 0 Nov 6 19:59 cgroup -> 'cgroup:[4026531835]'
117519 lrwxrwxrwx 1 root root 0 Nov 6 19:59 ipc -> 'ipc:[4026531839]'
117523 lrwxrwxrwx 1 root root 0 Nov 6 19:59 mnt -> 'mnt:[4026531840]'
117517 lrwxrwxrwx 1 root root 0 Nov 6 19:59 net -> 'net:[4026532008]'
117520 lrwxrwxrwx 1 root root 0 Nov 6 19:59 pid -> 'pid:[4026531836]'
117521 lrwxrwxrwx 1 root root 0 Nov 6 19:59 pid_for_children -> 'pid:[4026531836]'
117522 lrwxrwxrwx 1 root root 0 Nov 6 19:59 user -> 'user:[4026531837]'
117518 lrwxrwxrwx 1 root root 0 Nov 6 19:59 uts -> 'uts:[4026531838]'
As you can see, the demo container is completely isolated using namespaces. This is achieved using unshare(1). So if we can run a program with namespaces unshared from a parent, can we also re-use those namespaces for another program?
The answer to that question is yes. unix-utils package provides nsenter - simple wrapper around setns(2) that allows running a new process in the context of an existing process. Simply said, it can enter existing namespaces or spawn a process into a new set of namespaces.
# -t = target process to get namespaces from
# --uts = UTS namespace (hostname, etc)
$ sudo nsenter -t $pid --uts ls -l /proc/self/ns
lrwxrwxrwx 1 root root 0 Nov 6 20:11 cgroup -> 'cgroup:[4026531835]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 ipc -> 'ipc:[4026531839]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 mnt -> 'mnt:[4026531840]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 net -> 'net:[4026532008]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 pid -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 pid_for_children -> 'pid:[4026531836]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 user -> 'user:[4026531837]'
lrwxrwxrwx 1 root root 0 Nov 6 20:11 uts -> 'uts:[4026533098]'
$ sudo nsenter -t $pid -u hostname
7d061f9e15c7
$ docker ps -q # container ID is also a hostname by default
7d061f9e15c7
Note the UTS namespace is the same as the one in /proc/self/ns.
I bet you use curl, netcat or other network tools to test communication. Remember, we don’t have any tools in the container.
$ docker exec -t demo curl localhost:8080
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"curl\": executable file not found in $PATH": unknown
$ curl localhost:8080 # port is not bound so it won't work
curl: (7) Failed to connect to localhost port 8080: Connection refused
Let’s use hosts ones!
$ sudo nsenter -t $pid --net curl localhost:8080
Hello, World!
$ sudo nsenter -t $pid -n netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::8080 :::* LISTEN 2804/app
Network namespaces can be managed with ip too. However, output of ip netns list is empty. The reason for that is that unshare does not create bind mount under /var/run/netns. Well, we can do that ourselves:
$ sudo mkdir -p /var/run/netns/
$ sudo ln -sfT /proc/$pid/ns/net /var/run/netns/demo
$ sudo ip netns list
demo
$ sudo ip netns exec demo curl localhost:8080
Hello, World!
That’s it for now. See nsenter --help and explore the tool :-)