Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ jobs:

- name: Test standalone deployment helpers
run: |
python3 -m pip install 'tomli>=2,<3'
make deploy-standalone-test
make build-helper-test
adx-integration-contracts:
Expand Down
26 changes: 26 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,32 @@ distill-fs release's packaged manifest.

Use [`deploy/README.md`](./deploy/README.md) as the deployment entry point. AKernel supports standalone, existing Kubernetes clusters via Helm, and Terraform-based cloud provisioning.

The direct Helm chart enables the cgroup v2 parent memory policy with
`total_reserve="16GiB"` for the ordinary KVM profile. Standalone and both cloud
Terraform profiles keep it disabled by default. Cloud opt-in uses
`sandbox_memory_limit_enabled`; guided profiles expose
`SANDBOX_MEMORY_LIMIT_ENABLED` and `SANDBOX_MEMORY_TOTAL_RESERVE`. Qualify cgroup
v2 and effective memory (host MemTotal clipped by visible ancestor limits)
above the reserve with room for sandbox requests before enabling the policy.
The bundled standalone ADX profile retains its `auto` resource source, which
does not guarantee scheduling capacity follows the sandbox parent ceiling.
Operators enabling the policy must select and qualify a compatible
`resource_source`, including memory capacity, allocation/release accounting and
schedulable CPU. The disabled default does not fix resource-source limitations.
PVM also keeps the policy disabled until explicitly selected with
`AKERNEL_PVM_MEMORY_LIMIT_ENABLED=true` in standalone or
`node.config.sandboxd.pvmMemoryLimitEnabled=true` in Helm. Size the reserve for
the selected L1 and qualify the finite-cap and scheduling combination on the
matching PVM host before rollout; ordinary runsc tests do not qualify PVM.
Validate PVM memory switches against parsed TOML. PVM chart rendering requires
Helm 3.17 or newer, and standalone uses Python 3.11 or Python 3.10 with
`tomli`. Reject a standalone rewrite unless the parsed configuration matches
the requested policy and preserves the other fields.
Adding or lowering a parent cap requires an empty sandbox subtree; an existing
stricter cap is retained. This aggregate sandbox cap does not bound all node
components or guarantee avoidance of host OOM.
See `deploy/README.md#aggregate-sandbox-memory-budget` for migration and scope.

The [standalone guide](./deploy/standalone/README.md) owns source startup, supported platforms, runtime selection, and SDK setup. Preserve the source launcher's native-daemon architecture selection, clean sandboxd gitlink check, occupied-name preflight, profile locking, unique source image, and node/gateway health checks. Status and stop must verify recorded container identities, images, and data mounts; stop also requires an empty sandbox inventory. Source management is independent of cloud profiles and must preserve unrelated containers.

The all-in-one image and node launchers declare lowercase `container=oci`
Expand Down
5 changes: 5 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ INSTALL_MONITOR ?=
INSTALL_DRAGONFLY ?=
NODE_IMAGE_PULL_MODE ?=
CHUNK_DB_SIZE ?=
SANDBOX_MEMORY_LIMIT_ENABLED ?=
SANDBOX_MEMORY_TOTAL_RESERVE ?=
JUICEFS_ENABLED ?= false
JUICEFS_STORAGE_CLASS ?=
JUICEFS_CACHE_MIB ?= 4096
Expand All @@ -52,6 +54,7 @@ help:
@echo " make config INSTALL_DRAGONFLY=true Enable optional P2P image distribution"
@echo " make config NODE_IMAGE_PULL_MODE=direct Use direct registry pulls on new ACK workers"
@echo " make config ENABLE_RUNC=true Build and register the optional runc runtime"
@echo " make config SANDBOX_MEMORY_LIMIT_ENABLED=true SANDBOX_MEMORY_TOTAL_RESERVE=16GiB Opt into a qualified cgroup v2 sandbox budget"
@echo " make config JUICEFS_ENABLED=true JUICEFS_STORAGE_CLASS=... Deploy JuiceFS/PostgreSQL/Ceph"
@echo " make config CHECKPOINT_STORAGE_HOST_PATH=/srv/shared Use an existing AFS/JuiceFS mount"
@echo " make config SCHEDULE_PLACEMENT_POLICY=binpack Use compact scheduling"
Expand Down Expand Up @@ -105,6 +108,8 @@ config:
if [[ -n "$(INSTALL_DRAGONFLY)" ]]; then args+=(--install-dragonfly "$(INSTALL_DRAGONFLY)"); fi; \
if [[ -n "$(NODE_IMAGE_PULL_MODE)" ]]; then args+=(--node-image-pull-mode "$(NODE_IMAGE_PULL_MODE)"); fi; \
if [[ -n "$(CHUNK_DB_SIZE)" ]]; then args+=(--chunk-db-size "$(CHUNK_DB_SIZE)"); fi; \
if [[ -n "$(SANDBOX_MEMORY_LIMIT_ENABLED)" ]]; then args+=(--sandbox-memory-limit-enabled "$(SANDBOX_MEMORY_LIMIT_ENABLED)"); fi; \
if [[ -n "$(SANDBOX_MEMORY_TOTAL_RESERVE)" ]]; then args+=(--sandbox-memory-total-reserve "$(SANDBOX_MEMORY_TOTAL_RESERVE)"); fi; \
args+=(--juicefs-enabled "$(JUICEFS_ENABLED)"); \
args+=(--juicefs-cache-mib "$(JUICEFS_CACHE_MIB)" --juicefs-cache-large-write "$(JUICEFS_CACHE_LARGE_WRITE)"); \
if [[ -n "$(JUICEFS_CACHE_HOST_PATH)" ]]; then args+=(--juicefs-cache-host-path "$(JUICEFS_CACHE_HOST_PATH)"); fi; \
Expand Down
30 changes: 30 additions & 0 deletions builder/scripts/test_afs_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,36 @@ def assert_adx_valid(self, target):
if binary:
subprocess.run([binary, "--config", str(target), "validate"], check=True)

@unittest.skipUnless(os.environ.get("AKERNEL_TEST_ADXCTL"), "requires packaged ADX CLI")
def test_packaged_cli_preserves_standalone_auto_capacity_after_preparation(self):
binary = os.environ["AKERNEL_TEST_ADXCTL"]
source = ROOT / "builder/config/adx-standalone.yaml"

def resource_source(path):
deployment = yaml.safe_load(subprocess.run(
[binary, "--config", str(path), "config", "dump"],
check=True, capture_output=True, text=True,
).stdout)
workers = [service for service in deployment["services"]
if service["role"] == "adxlet"]
self.assertEqual(len(workers), 1)
return workers[0]["config"]["resource_source"]

baseline = resource_source(source)
self.assertEqual(baseline["kind"], "auto")
for pvm in (False, True):
for trace, mount in ((False, None), (True, None),
(False, Path("/mnt/akernel-shared"))):
with self.subTest(pvm=pvm, trace=trace, mount=mount), tempfile.TemporaryDirectory() as directory:
target = Path(directory) / "prepared.yaml"
with patch.dict(os.environ, {"ENABLE_TRACE": str(trace).lower()}):
# Only mount inspection is simulated; capacity comes
# from the real packaged CLI's resolved deployments.
with patch.object(CONFIG.subprocess, "run") as run:
run.return_value.stdout = "afs-dfs fuse rw\n"
CONFIG.configure(source, target, mount, "test/checkpoints", pvm=pvm)
self.assertEqual(resource_source(target), baseline)

def test_trace_preserves_explicit_services_and_only_changes_present_roles(self):
for profile in ("standalone-managed-redis", "node", "coordinator", "ingress-api"):
source = ROOT / f"src/agent-dx/build/config/examples/deployment-{profile}.yaml"
Expand Down
85 changes: 85 additions & 0 deletions deploy/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,91 @@ when a shared filesystem is independently required.
For one deployment serving both native and E2B SDKs with version-independent
application logs, see [the dual SDK/logging profile](docs/dual-sdk-and-application-logs.md).

## Aggregate sandbox memory budget

The direct Helm chart enables the cgroup v2 aggregate sandbox memory policy by
default for the ordinary KVM backend profile:

```toml
[plugin.resource.memory_limit]
enabled = true
total_reserve = "16GiB"
```

Effective node memory is the minimum of host `MemTotal` and visible finite
ancestor cgroup limits. Sandboxd subtracts the reserve, rounds the result down
to a page boundary and preserves any existing stricter `/akernel/memory.max`.
The sandboxd resource socket is clipped to that effective parent ceiling,
without subtracting the reserve a second time. Helm and cloud Adxlet configs
consume that socket. For example, effective memory of 64 GiB and this reserve
produce a candidate of 48 GiB; an existing 40 GiB cap remains 40 GiB.

Standalone keeps the policy disabled by default, with `total_reserve="16GiB"`
available for explicit configuration. Its bundled ADX profile retains the
`auto` resource source, including PVM, trace and shared-storage variants.
`auto` does not guarantee public scheduling capacity follows the sandbox parent
ceiling. Before enabling the policy, operators must configure a compatible ADX
`resource_source` and qualify memory capacity against the parent budget,
allocation and release after sandbox create/delete, and schedulable CPU. This
also applies to custom ADX configurations. The disabled default does not repair
or qualify resource-source behavior; CPU discovery is unchanged by this policy.

Enabling the policy requires cgroup v2 and effective memory greater than the
reserve, with enough remaining capacity for the requested sandboxes.
Docker/OrbStack VM limits and visible ancestor limits participate in this
calculation. Use a matching node image built with the pinned sandboxd's
`total_reserve` support. On a smaller node, explicitly choose a smaller positive
`total_reserve` or keep `enabled=false` in the sandboxd TOML before starting.
Standalone reads its source config template; Helm uses
`node.config.sandboxd.config` (`core.node.config.sandboxd.config` in the umbrella
chart). Cloud profiles expose the enabled flag and reserve described below,
alongside their existing complete-config override. Values use integer bytes or
`B`, `KiB`, `MiB`, `GiB`, `TiB`; `16G`, percentages and `auto` are not accepted.
Verify schedulable CPU from the selected resource provider as well as memory
before rollout, including when using Helm's sandboxd source.

The PVM profile keeps this policy disabled by default because enabled parent
enforcement has not been qualified on its matching host. Selecting PVM overrides
the TOML `enabled` value through standalone
`AKERNEL_PVM_MEMORY_LIMIT_ENABLED=false` or Helm
`node.config.sandboxd.pvmMemoryLimitEnabled=false` (`core.` prefix in the umbrella
chart). Explicitly set that switch to `true` and choose a suitable positive
`total_reserve` in the source TOML or complete Helm config to opt in for host
qualification. Both switches accept only `true` or `false` and affect only PVM.
PVM CI retains the disabled policy and does not claim parent-limit coverage.

Both cloud Terraform profiles keep the policy disabled by default. Their
`sandbox_memory_limit_enabled` input is an explicit opt-in, with
`sandbox_memory_total_reserve="16GiB"`. Guided profiles expose the same inputs:

```bash
make config SANDBOX_MEMORY_LIMIT_ENABLED=true SANDBOX_MEMORY_TOTAL_RESERVE=16GiB
```

Select and qualify a cgroup v2 node image before enabling it: the current standard Aliyun
`node_pool_image_type="AliyunLinux3"` default uses cgroup v1, and Huawei's
selected node image also needs cgroup-mode verification. These options do not
change the node operating system or migrate existing nodes to cgroup v2.

This reserve is deployment policy, not a measured component budget or a
guarantee against host OOM. Distill-fs retains its independent 64 GiB limit and
is not automatically deducted. Control components, other Pods and host kernel
memory remain outside the sandbox parent; their limits, reclaim behavior and
peak demand need separate consideration. In particular, kernel memory excluded
by a host's `cgroup.memory=nokmem` setting is not covered by the parent limit.

For existing nodes, adding a finite cap to an unlimited parent or lowering a
cap requires an empty sandbox subtree and residual usage below the new limit.
An equal or stricter existing cap can be reused with live sandboxes; a larger
candidate never automatically raises it. Review the rollout before upgrading:
ConfigMap checksums trigger Pod replacement, and the current chart permits
200 unavailable nodes. Use a canary or a separately constrained rollout after
draining, rather than enabling the policy across live nodes together. Custom
TOML containing the removed reserve fields must be migrated even when the
policy is disabled. See the pinned sandboxd
[memory-limit guide](../src/sandboxd/doc/node-memory-limit.md) for migration and
the complete startup contract.

## Guided deployment

The image installs the complete gVisor bundle pinned by sandboxd's runtime manifest: runsc, the containerd shim, and adjacent `gvisor-bin/` helpers. Its SHA-512 verifies the archive, not a standalone runsc binary. Keep these executables together when building custom images; replacing only runsc can leave mismatched checkpoint/restore helpers.
Expand Down
38 changes: 38 additions & 0 deletions deploy/akernel/charts/core/scripts/pvm-memory-limit.tpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{{- $config := fromToml .config -}}
{{- if hasKey $config "Error" -}}
{{- fail "PVM sandboxd.config must be valid TOML with no duplicate fields" -}}
{{- end -}}
{{- $table := $config -}}
{{- range $key := list "plugin" "resource" "memory_limit" -}}
{{- /* sandboxd also accepts uppercase aliases for its struct fields. */ -}}
{{- if hasKey $table (upper $key) -}}
{{- fail "PVM memory policy requires lowercase plugin.resource.memory_limit keys" -}}
{{- end -}}
{{- if hasKey $table $key -}}
{{- $table = get $table $key -}}
{{- if not (kindIs "map" $table) -}}
{{- fail (printf "PVM memory policy requires %s to be a TOML table" $key) -}}
{{- end -}}
{{- else -}}
{{- $table = dict -}}
{{- end -}}
{{- end -}}
{{- if hasKey $table "enabled" -}}
{{- if not (kindIs "bool" (get $table "enabled")) -}}
{{- fail "PVM memory policy requires a boolean enabled field" -}}
{{- end -}}
{{- if ne (get $table "enabled") .enabled -}}
{{- $_ := set $table "enabled" .enabled -}}
{{- $rendered := toToml $config -}}
{{- if or (not $rendered) (hasKey (fromToml $rendered) "Error") -}}
{{- fail "Could not render the PVM sandboxd configuration as TOML" -}}
{{- end -}}
{{- $rendered -}}
{{- else -}}
{{- .config -}}
{{- end -}}
{{- else if .enabled -}}
{{- fail "PVM memory policy requires a boolean enabled field when opting in" -}}
{{- else -}}
{{- .config -}}
{{- end -}}
11 changes: 11 additions & 0 deletions deploy/akernel/charts/core/templates/node/configmap.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ data:
{{- if not (has $firecrackerBackend (list "kvm" "pvm")) }}
{{- fail "node.config.sandboxd.firecrackerBackend must be kvm or pvm" }}
{{- end }}
{{- $pvmMemoryLimitEnabled := toString .Values.node.config.sandboxd.pvmMemoryLimitEnabled }}
{{- if not (has $pvmMemoryLimitEnabled (list "true" "false")) }}
{{- fail "node.config.sandboxd.pvmMemoryLimitEnabled must be true or false" }}
{{- end }}
{{- if eq $firecrackerBackend "pvm" }}
{{- if not (and (contains "[plugin.runtime.firecracker]" $sandboxdConfig) (contains "firecracker=\"/usr/local/bin/firecracker\"" $sandboxdConfig)) }}
{{- fail "PVM selection requires the standard Firecracker config section and binary entry" }}
Expand All @@ -83,6 +87,13 @@ data:
{{- end }}
{{- $sandboxdConfig = replace $chunkDbMarker (printf "chunk_db_size=%s" ($chunkDbSize | quote)) $sandboxdConfig }}
{{- end }}
{{- if eq $firecrackerBackend "pvm" }}
{{- if not (semverCompare ">=3.17.0" .Capabilities.HelmVersion.Version) }}
{{- fail "PVM configuration requires Helm 3.17 or newer for TOML validation" }}
{{- end }}
{{- /* Parse after consuming template markers. Bind fromToml only on the PVM path. */}}
{{- $sandboxdConfig = tpl (.Files.Get "scripts/pvm-memory-limit.tpl") (dict "Template" .Template "config" $sandboxdConfig "enabled" (eq $pvmMemoryLimitEnabled "true")) }}
{{- end }}
{{ $sandboxdConfig | indent 4 }}

registry.json: |
Expand Down
Loading