feat: add an L4 LoadBalancer mode for non-VM services

README.md

@@ -70,6 +70,49 @@ The nftables ruleset placed in table `ip cozy_proxy` consists of:
   `allowICMP: "true"` annotation is set, the pod IP is added to
   `icmp_allowed_pods` and ICMP toward it is accepted before the drop rule.
 
+### Rule scoping
+
+The two rewrites are deliberately not scoped alike. Every controller instance
+watches every Service, but programs:
+
+| Object | Scope | Why |
+|---|---|---|
+| `svc_pod` (`ingress_dnat`), `allowed_ports`, `icmp_allowed_pods` | the node hosting the backend pod | A non-hosting node would rewrite the destination before the packet has even left it. The hosting node then records a conntrack tuple of `(client -> podIP)`, while the reply leaves the pod and gets `saddr` rewritten to the service IP by `egress_snat` at priority `raw`, before conntrack. `(svcIP -> client)` matches nothing, so the reply is not `established` and `port_filter` drops it. |
+| `pod_svc` (`egress_snat`) | every node | An intra-cluster client is source-NATed by the CNI to its own node address, which the overlay knows how to reach directly. The backend's reply is then tunnelled straight to that node and never traverses the hosting node's netfilter hooks, so the client's node is the only place left where the pod IP can still be turned back into the service IP. Without the entry there, the reply arrives with the wrong source and the client answers with a RST — a cross-node connection that hangs while the same-node one works. |
+
+`NODE_NAME` carries the node identity; the chart injects it from
+`spec.nodeName`. When it is unset the ingress scope check is disabled and
+everything is programmed everywhere, so the binary still runs under a
+deployment that does not inject it.
+
+## L4 LoadBalancer mode (experimental)
+
+Besides VMs, cozy-proxy can take over the public IP of ordinary `LoadBalancer`
+services — several ports, several backends, port translation — so that it is
+reachable from every pod the way it is from the Internet. A service opts in with
+both:
+
+```yaml
+metadata:
+  labels:
+    networking.cozystack.io/lb-proxy: cozy-proxy
+  annotations:
+    service.cilium.io/type: ClusterIP   # Cilium releases the VIP, keeps the ClusterIP
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Local
+```
+
+The node from which MetalLB (L2) announces the VIP DNATs new connections to its
+local ready backends, masquerading only sources that are node IPs; everything
+else addressed to the VIP is dropped. The mode has its own table, `ip
+cozy_proxy_l4`, and never touches the VM mode's. It is off by default: run with
+`--enable-l4-loadbalancer` (chart value `l4LoadBalancer.enabled`). Phase 1
+handles TCP, IPv4 and `externalTrafficPolicy: Local`.
+
+See [the RFC](docs/rfc/l4-loadbalancer-mode.md) for the model, the datapath,
+the security implications and the rollout plan.
+
 ## Installation
 
 Install controller using Helm-chart:

charts/cozy-proxy/templates/daemonset.yaml

@@ -21,6 +21,27 @@ spec:
         - name: cozy-proxy
           image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
           imagePullPolicy: {{ .Values.image.pullPolicy }}
+          {{- $keepL4Table := and (not .Values.l4LoadBalancer.enabled) (not .Values.l4LoadBalancer.removeTableWhenDisabled) }}
+          {{- if or .Values.l4LoadBalancer.enabled (not .Values.vmMode.enabled) $keepL4Table }}
+          args:
+            {{- if .Values.l4LoadBalancer.enabled }}
+            - --enable-l4-loadbalancer
+            {{- end }}
+            {{- if $keepL4Table }}
+            - --remove-l4-table-when-disabled=false
+            {{- end }}
+            {{- if not .Values.vmMode.enabled }}
+            - --enable-vm-mode=false
+            {{- end }}
+          {{- end }}
+          env:
+            # Scopes the ingress rules to the backends hosted on this node; see
+            # "Rule scoping" in the README for why a non-hosting node must not
+            # program them.
+            - name: NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
           securityContext:
             privileged: true
             capabilities:

charts/cozy-proxy/templates/role.yaml

@@ -9,4 +9,15 @@ rules:
   - apiGroups: [""]
     resources: ["services", "endpoints"]
     verbs: ["get", "list", "watch"]
+  {{- if .Values.l4LoadBalancer.enabled }}
+  - apiGroups: [""]
+    resources: ["nodes"]
+    verbs: ["list", "watch"]
+  - apiGroups: ["discovery.k8s.io"]
+    resources: ["endpointslices"]
+    verbs: ["list", "watch"]
+  - apiGroups: ["metallb.io"]
+    resources: ["servicel2statuses"]
+    verbs: ["list", "watch"]
+  {{- end }}
 {{- end }}

charts/cozy-proxy/values.yaml

@@ -10,3 +10,21 @@ daemonset:
 
 rbac:
   create: true
+
+# L4 LoadBalancer mode: takes over from Cilium the public IP of services
+# labelled networking.cozystack.io/lb-proxy=cozy-proxy that also carry
+# service.cilium.io/type=ClusterIP. See docs/rfc/l4-loadbalancer-mode.md.
+l4LoadBalancer:
+  enabled: false
+  # With the mode disabled, remove the table an earlier run programmed on the
+  # next start: this is how the mode is rolled back. Set it to false on a
+  # release that runs next to a second release running the L4 mode, or each of
+  # its starts deletes that release's table.
+  removeTableWhenDisabled: true
+
+# VM mode, for services labelled service.kubernetes.io/service-proxy-name=cozy-proxy.
+# Turn it off only to run the L4 mode as a second release next to a cozy-proxy
+# that keeps the VM mode, for instance to try the L4 mode on a live cluster.
+# That cozy-proxy then needs l4LoadBalancer.removeTableWhenDisabled=false.
+vmMode:
+  enabled: true