A Docker image with Kubernetes manifests for investigation and troubleshooting your cluster.

Purpose

The DOKS team provides this image for use as-is and for transparency as the image used when a request to "deploy a debug pod" is made to our customers, which may occur when deeper investigation is needed with direct access to a cluster.

Usage

Prefer the DaemonSet when you need debug pods on many nodes, or the ephemeral debug-node helper when you need access to a single node. Avoid leaving a long-lived doks-debug Deployment in the cluster.

DaemonSet

kubectl apply -f k8s/daemonset.yaml

This DaemonSet manifest will:

1. Ensure a pod with our Docker image is running indefinitely on every node.
2. Use hostPID, hostIPC, and hostNetwork.
3. Mount the entire host filesystem to /host in the containers.
4. Mount the containerd socket at /run/containerd/containerd.sock from the host into the container.
5. Tolerate all taints (operator: Exists) so pods can land on cordoned or specially tainted nodes.

In order to make use of these workloads, you can exec into a pod of choice by name:

kubectl -n kube-system exec -it my-pod-name bash

If you know the specific node name that you're interested in, you can exec into the debug pod on that node with:

NODE_NAME="my-node-name"
POD_NAME=$(kubectl -n kube-system get pods --field-selector spec.nodeName=${NODE_NAME} -ojsonpath='{.items[0].metadata.name}')
kubectl -n kube-system exec -it ${POD_NAME} bash

Clean up when finished:

kubectl delete -f k8s/daemonset.yaml

Ephemeral single-node access (debug-node)

For short-lived access to one node, use script/debug-node. It creates a doks-debug Deployment pinned with a nodeSelector, execs into the host via chroot /host, and deletes the Deployment when you exit.

./script/debug-node <node-name>

The Deployment manifest does not include a default catch-all toleration. A long-lived Deployment with tolerations: [{operator: Exists}] can reschedule onto draining nodes and block scale-down or upgrades. If you need to reach a tainted or cordoned node for a brief session, pass --tolerate-all:

./script/debug-node --tolerate-all <node-name>

Requires kubectl, curl (if the local manifest is unavailable), and yq.

Once you're in, you have access to the set of tools listed in the Dockerfile. This includes:

• vim - is a greatly improved version of the good old UNIX editor Vi.
• screen - is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.
• curl - is a command-line tool for transferring data specified with URL syntax.
• jq - is a lightweight and flexible command-line JSON processor.
• dnsutils - includes various client programs related to DNS that are derived from the BIND source tree, specifically dig, nslookup, and nsupdate.
• iputils-ping - includes the ping tool that sends ICMP ECHO_REQUEST packets to a host in order to test if the host is reachable via the network.
• tcpdump - a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture.
• traceroute - tracks the route packets taken from an IP network on their way to a given host.
• net-tools - includes the important tools for controlling the network subsystem of the Linux kernel, specifically arp, ifconfig, and netstat.
• netcat - is a multi-tool for interacting with TCP and UDP; it can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6.
• iproute2 - is a collection of utilities for controlling TCP / IP networking and traffic control in Linux.
• strace - is a diagnostic, debugging and instructional userspace utility with a traditional command-line interface for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state.
• dstat - is a versatile replacement for vmstat, iostat, netstat and ifstat. Dstat overcomes some of their limitations and adds some extra features, more counters and flexibility. Dstat is handy for monitoring systems during performance tuning tests, benchmarks or troubleshooting.
• htop - is interactive process viewer for Unix systems.
• atop - is an advanced interactive monitor for Linux-systems to view the load on system-level and process-level.
• wget - for retrieving files using HTTP, HTTPS, FTP and FTPS.
• crictl - A CLI for CRI endpoints. Configured to use /run/containerd/containerd.sock as a default endpoint.
• amd-smi - AMD GPU system management CLI (ROCm 7.14 standalone amdrocm-amdsmi). Use on AMD GPU nodes from the container shell: (e.g. amd-smi version, amd-smi list).

Tips and Tricks

chroot + systemctl

chroot /host /bin/bash
systemctl status kubelet
journalctl -xe
journalctl -u kubelet

Contributing

At DigitalOcean we value and love our community! If you have any issues or would like to contribute, feel free to open an issue or PR and cc any of the maintainers.