Kubernetes Integration

[ArshSSandhu]

Kubernetes Integration

Summary

Add Kubernetes as a supported workload provider in Manager alongside Proxmox, Docker, and Dummy nodes.

The goal is to allow Manager to communicate directly with a local or remote Kubernetes API and provision and manage workloads through the existing node-provider abstraction.

This will also make local development and testing easier because Kubernetes can run through Docker Desktop or kind, while Proxmox normally requires physical hardware.

This issue will serve as the parent tracking issue for the complete Kubernetes integration. The implementation should be divided into smaller child issues and focused pull requests.

Background

The Docker Node API work in #375 and #394 added Docker as another provider behind the existing Node API abstraction.

The Kubernetes integration should follow the same general design:

Manager
   |
   +-- ProxmoxApi
   |
   +-- DockerApi
   |
   +-- DummyApi
   |
   +-- KubernetesApi

Provider selection should remain centralized in Node.api() or another single provider-dispatch location.

Provider-specific checks such as:

if (nodeType === 'kubernetes')

should not be spread throughout the jobs, routers, and business logic.

Proposed Kubernetes resource mapping

This mapping should be confirmed before implementation.

Manager concept | Kubernetes concept -- | -- Manager Node | Kubernetes cluster/context Manager Container | Kubernetes Deployment Running instance | Pod created by the Deployment Provider container ID | Stable namespaced Deployment identifier Start | Scale Deployment to one replica Stop | Scale Deployment to zero replicas Reconfigure | Patch Deployment pod template Delete | Delete Deployment Live status | Deployment availability and replica status

Manager should not store a Pod name or Pod UID as the permanent provider ID because Pods are temporary and may be replaced during restarts, failures, or rollouts.

A possible provider ID format is:

namespace/deployment-name

Example:

default/example-container

Proposed Manager labels

Kubernetes resources created by Manager should include labels that allow Manager to identify and list only the resources it owns.

Example:

org.mieweb.opensource-server.managed=true
org.mieweb.opensource-server.node-id=<manager-node-id>
org.mieweb.opensource-server.container-id=<manager-container-id>

The exact label names can be finalized during implementation.

Proposed configuration

A Kubernetes-backed Manager node represents access to a Kubernetes cluster and context rather than a single Kubernetes worker node.

The initial implementation may require:

• Kubeconfig path

• Optional kubeconfig context

• Default namespace

• TLS verification

• Kubernetes API connection validation

The kubeconfig file should be mounted or otherwise available to the Manager process.

Sensitive kubeconfig contents, bearer tokens, certificates, and private keys must not be returned through node serialization or written to logs.

Support for in-cluster authentication may be added later or included if it fits naturally into the initial design.

Main requirements

Kubernetes API client

• Communicate with Kubernetes through its native API.

• Use the official Kubernetes Node.js client where practical.

• Do not shell out to kubectl.

• Support local and remote Kubernetes clusters.

• Validate authentication and connectivity.

Workload creation

Manager should be able to create a Kubernetes Deployment using the existing container creation flow.

The Deployment should support:

• OCI container image

• Hostname or Kubernetes-safe workload name

• CPU request or limit

• Memory request or limit

• Environment variables

• Entrypoint or command

• One desired replica

• Manager ownership labels

• Configured namespace

Kubernetes and its container runtime should handle image pulling and image-layer caching.

Lifecycle operations

The integration should support:

• Create

• Start

• Stop

• Live status

• Reconfigure

• Delete

Proposed lifecycle behavior:

Start      -> scale Deployment to 1
Stop       -> scale Deployment to 0
Status     -> inspect Deployment replicas and conditions
Reconfigure -> patch Deployment and wait for rollout
Delete     -> delete Deployment

Status handling

A Kubernetes workload should be considered running when its Deployment has the required available replica.

Possible mappings:

available replica present -> running
desired replicas = 0      -> offline
rollout in progress       -> creating or updating
Deployment missing        -> missing
API unavailable           -> unknown
Deployment failed         -> failed

The final mapping should remain compatible with Manager’s existing status values where possible.

Networking

The current provisioning flow assumes that every provider returns a MAC address and IP address.

That assumption does not work cleanly for Kubernetes:

• Kubernetes Pods do not provide a stable Manager-level MAC address.

• Pod IP addresses are ephemeral.

• A Pod IP may change after a restart or rollout.

The shared provider abstraction should be updated so network information is based on provider capabilities rather than being mandatory.

For the initial Kubernetes implementation:

• MAC address may be null.

• The current Pod IP may be stored or displayed as informational data.

• Pod IP must be treated as ephemeral.

• Workload creation must not fail only because a MAC address is unavailable.

Stable networking through Services, Ingress, LoadBalancer, or NodePort may be handled in a later issue.

Provider abstraction work

The current jobs and provider APIs still contain Proxmox-specific terminology and assumptions such as:

• VMID allocation

• LXC method names

• Storage resolution

• Proxmox task IDs

• Mandatory MAC and IP discovery

• Docker-specific task parsing

• Direct nodeType checks

The Kubernetes integration may require introducing a more provider-neutral workload lifecycle interface.

Possible interface direction:

createWorkload()
updateWorkload()
startWorkload()
stopWorkload()
deleteWorkload()
getWorkloadStatus()
getWorkloadNetwork()
listWorkloads()
waitForOperation()

Compatibility wrappers may temporarily preserve existing LXC-named methods while the shared flow is migrated.

The exact abstraction should be designed before or during the first child issue.

Local development workflow

The integration should be testable without Proxmox physical hardware.

Supported development options may include:

• Docker Desktop Kubernetes

• kind

• Another local Kubernetes cluster

Expected local test flow:

1. Start a local Kubernetes cluster.

2. Mount or configure a kubeconfig for Manager.

3. Add a Kubernetes node in Manager.

4. Validate Kubernetes API connectivity.

5. Create an nginx:latest workload.

6. Confirm a Deployment and Ready Pod are created.

7. Confirm Manager reports the workload as running.

8. Stop it and confirm the Deployment scales to zero.

9. Start it and confirm a new Pod becomes Ready.

10. Change an environment variable or resource request.

11. Confirm Kubernetes performs a successful rollout.

12. Delete the workload.

13. Confirm the Deployment and owned Pods are removed.

Security requirements

• TLS verification should remain enabled by default.

• Kubeconfig credentials must not be logged.

• Kubernetes tokens, certificates, and private keys must not be exposed through API responses.

• Manager should not require cluster-admin.

• Document a least-privilege RBAC configuration.

• Prefer namespace-scoped permissions.

• Only request the resource types and operations needed by Manager.

• Do not read or list Kubernetes Secrets unless a future feature explicitly requires it.

Proposed child issues

1. Refactor Node API for provider-neutral workload lifecycle

• Identify Proxmox- and Docker-specific assumptions.

• Define the provider-neutral lifecycle contract.

• Make network metadata optional and provider-aware.

• Reduce provider-specific branching outside dispatch.

• Add abstraction-level tests.

2. Add Kubernetes node configuration and authentication

• Add the kubernetes node type.

• Add migrations for Kubernetes configuration.

• Add kubeconfig path, context, and namespace support.

• Add backend validation.

• Add Kubernetes connection testing.

• Update OpenAPI.

• Update TypeScript types.

• Update node form and node list UI.

3. Implement Kubernetes workload provisioning

• Add KubernetesApi.

• Connect through the Kubernetes API.

• Create labeled Deployments.

• Apply image, CPU, memory, environment variables, and entrypoint.

• Store a stable Deployment-based provider ID.

• Wait for Deployment readiness.

4. Implement Kubernetes workload lifecycle

• Live status

• Start

• Stop

• Delete

• Deployment discovery

• Error handling

• Timeout handling

5. Implement Kubernetes reconfiguration, networking, tests, and documentation

• Patch Deployment pod templates.

• Wait for rollout completion.

• Handle ephemeral Pod IP addresses.

• Support nullable MAC addresses.

• Add mocked unit tests.

• Document local-cluster testing.

• Document Kubernetes RBAC.

• Complete end-to-end testing.

Child issue scope may be adjusted after maintainers confirm the architecture.

Non-goals for the initial integration

The first Kubernetes integration does not need to support:

• Creating Kubernetes clusters

• Deleting Kubernetes clusters

• Managing control-plane components

• Arbitrary YAML manifest application

• Helm charts

• StatefulSets

• PersistentVolumeClaims

• Multi-container Pods

• Horizontal Pod Autoscaling

• Multiple replicas beyond the start/stop mapping

• Ingress management

• LoadBalancer management

• NodePort management

• Pod exec

• Pod log streaming

• Kubernetes dashboard functionality

• Importing unmanaged Kubernetes workloads

• Full kubectl replacement

These can be implemented through later issues.

Acceptance criteria

• The proposed Deployment-based resource mapping is reviewed and approved.

• The Kubernetes integration is divided into focused child issues.

• Manager supports kubernetes as a node type.

• Manager can authenticate with a local Kubernetes cluster.

• Manager communicates through the Kubernetes API without invoking kubectl.

• Manager can create a labeled single-replica Deployment.

• The permanent provider ID refers to the Deployment rather than a Pod.

• Start works by scaling the Deployment to one replica.

• Stop works by scaling the Deployment to zero replicas.

• Live status reflects actual Kubernetes Deployment state.

• Reconfiguration updates the Deployment and waits for rollout.

• Delete removes the Deployment and its owned Pods.

• Kubernetes workloads are not required to provide a MAC address.

• Pod IP addresses are treated as ephemeral.

• Provider-specific Kubernetes branching is not scattered throughout shared code.

• Existing Proxmox, Docker, and Dummy functionality continues to work.

• OpenAPI, UI, tests, local setup documentation, and RBAC documentation are updated.

Open design questions

Before beginning implementation, maintainers should confirm:

1. Should a Manager Container map to a Kubernetes Deployment?

2. Should the provider ID use namespace/deployment-name?

3. Should kubeconfig configuration be stored as a mounted file path?

4. Should Kubernetes namespaces be configured per node or per container?

5. Should Pod IP be stored in the existing ipv4Address field or only returned dynamically?

6. Should the first implementation create a Kubernetes Service, or should service exposure remain out of scope?

7. How much of the existing LXC-named API should be refactored before adding KubernetesApi?

8. Should the work be delivered through the proposed five child issues and PRs?

Related work

• Docker-type Node API #375 — Docker-type Node API

• feat(nodes): add Docker API backend #394 — Add Docker API backend

[ArshSSandhu]

@horner @runleveldev

I created this parent issue to plan the complete Kubernetes integration before beginning implementation.

I reviewed the existing Docker Node API work and proposed mapping each Manager container to a Kubernetes Deployment rather than an individual Pod, since Pods are temporary and can be replaced by Kubernetes.

I also proposed dividing the implementation into smaller child issues and focused PRs so the work is easier to review, test, and adjust.

Could you please review the proposed architecture, scope, child-issue breakdown, and the open design questions at the bottom? Once we agree on the overall direction, I will create the child issues and begin with the provider-abstraction and local Kubernetes setup work.