A Kubernetes controller that serves DNS A/AAAA records, which resolve to the external IPs of all ready nodes in user-defined areas.
I have a self-hosted Kubernetes cluster that consists of nodes at various locations. Some of those nodes have a public IPv4, some also have IPv6 connectivity; others are NAT'ed by their home routers and cannot be reached externally. Several nodes are running on bare-metal hardware, others are set up as VMs running in the cloud.
Most load balancing solutions cannot provide load balancing and failover for such a cluster.
However, I also run an authoritative DNS service in my cluster, and DNS can be used for this purpose like so:
-
Kubernetes API server knows which nodes are alive and which are not;
-
nodes can be classified into multiple "areas" (home, external, cloud and so on). Annotations are a good way to set the area names for each node;
-
external IPs for each node are also set in the annotations. Since I don't have a K8s cloud provider - my Kubelets cannot set EXTERNAL-IP by themselves, and some of my nodes can also be NAT'ed, so I have to provide external IPs manually;
-
the controller provides a builtin DNS server, which is authoritative for its own zone;
-
the controller also reconciles Node objects, checks node liveness and updates the DNS records accordingly;
-
my external DNS (CoreDNS) forwards requests to the controller's DNS zone, so that, for example, a request to home.lb.pootis.network should land at the controller's DNS server, and it will return all known IPs of nodes that are currently alive in the "home" area;
-
a service record (
git.pootis.network) can be then defined as a CNAME tohome.lb.pootis.network.
The controller watches Node objects in the cluster.
Each node may carry two annotations (you should set your own annotation keys, this is just an example):
-
k8s.pootis.network/node-areas(example:home, external). A comma-separated list of area names the node belongs to. Each area name should be a valid DNS label. -
k8s.pootis.network/node-ips(example:1.2.3.4, 2001:db8::1). A comma-separated list of external IPv4/IPv6 addresses which should be advertised for this node.
The annotation keys can be changed with the --areas-annotation and --ips-annotation flags.
The controller can additionally track individual workloads by using annotated Services; see Service records below.
For zone lb.pootis.network. and an area home, querying home.lb.pootis.network. returns A/AAAA records for all ready nodes that list home in their areas annotation.
The zone apex serves SOA and NS records.
DNS records can also use an optional IP family prefix that restricts the answer to a single IP family:
v4.<name>resolves only to IPv4 (A) addresses;v6.<name>resolves only to IPv6 (AAAA) addresses.
For example, if home.lb.pootis.network. resolves to both an IPv4 and an IPv6 address, then:
v4.home.lb.pootis.network.returns only the IPv4 addresses;v6.home.lb.pootis.network.returns only the IPv6 addresses;
The prefix also works for Service records, for example, v4.git.apps.lb.pootis.network..
The controller is capable of advertising a DNS record that tracks a workload (Deployment, StatefulSet and such) as it moves between nodes.
To opt-in, annotate a Service with k8s.pootis.network/dns: "true" (the key is configurable via --service-annotation).
The controller will then serve a record at <service>.<namespace>.<zone>, and that record resolves to the external IPs
(from the node IPs annotation) that belong to every node that currently hosts a ready endpoint of that Service,
also taking node readiness into account.
This works with any Service that has endpoints, even a headless Service, regardless of the workload behind it.
Example:
- a single-replica Deployment is gated behind a Service
gitin namespaceapps; - a client queries
git.apps.lb.pootis.network.; - the DNS server returns the external IP of the node (or nodes, if there were multiple replicas) that currently run the pod, and the DNS response becomes empty (NODATA) if there are no ready replicas anywhere.
The Service controller mirrors kube-proxy's terminating-endpoints behavior: if a Service has no ready endpoints, the controller tries to falls back to endpoints that are still serving while terminating (serving-terminating state), and only then it will bail with NODATA.
The controller requires read access to Node, Service, and EndpointSlice objects by the means of a ClusterRole:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: k8s-node-dns
rules:
- apiGroups:
- ""
resources:
- nodes
- services
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- get
- list
- watchA Helm chart is available in GHCR OCI:
helm install k8s-node-dns oci://ghcr.io/ds8088/charts/k8s-node-dnschart/values.yaml contains all available configuration options.
config.zone, config.soaNS, and config.soaEmail options are mandatory for Helm chart deployment.
k8s-node-dns --zone lb.pootis.network --soa-email admin@pootis.network --soa-ns ns1.lb.pootis.network:1.2.3.4Go 1.26+ is required.
go build ./...To run unit tests:
go test -race -v ./...An integration test is also included, which runs the tool against a real Kubernetes API server.
To enable it, install setup-envtest and run:
export KUBEBUILDER_ASSETS=$(setup-envtest use -p path 1.34.1)
RUN_INTEGRATION_TESTS=1 go test ./...