bug(vm): capability probe cannot read /proc/self/status on rootless RHEL HPC host #4014

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 1, 2026
2. elezar commented on Oct 1, 2026

   

   elezar

   on Oct 1, 2026

   Member

   More actions

   📋 triage-agent

   Triage Assessment

   Classification: needs-investigation

   Summary

   The report supplies a concrete reproduction, matching component versions, and an actionable guest startup error. It warrants investigation as a VM guest runtime-qualification failure. It is not established as a duplicate of #3579 or fixed by #3580; the underlying cause of the permission denial remains unconfirmed.

   Investigation

   • Drew's bug: CPU-only microVM startup inherits host GPU filesystem requirements #3579, fixed by fix(vm): scope GPU filesystem enrichment to assigned workloads #3580, concerned CPU-only VMs on an NVIDIA ARM64 host: host-side GPU discovery promoted /proc from read-only to read-write, and the gateway rejected the live-policy update with filesystem read_only path '/proc' cannot be removed on a live sandbox.
   • fix(vm): scope GPU filesystem enrichment to assigned workloads #3580 merged on September 23. Its merge commit 069ae6bd966c4ced02b5cff0a26aee0a7f23a82f is an ancestor of the v0.1.2 tag, released September 28. This report already targets that release, so upgrading merely to pick up fix(vm): scope GPU filesystem enrichment to assigned workloads #3580 is not a resolution.
   • In both v0.1.2 and the inspected current source, crates/openshell-sandbox/src/main.rs::qualify_runtime reads /proc/self/status to check capability sets. run_boundary performs qualification before starting the sandbox boundary. The failing read precedes the explicit nondumpable setting and subsequent Landlock/seccomp qualification probes; it is a different failure path from supervisor policy enrichment.
   • Guest init mounts procfs and the guest launcher transitions to the non-root, zero-capability identity. Successful privileged initialization does not prove that procfs is accessible after this transition. The owning investigation spans the VM guest init/embedded runtime and sandbox identity/qualification path.
   • The host's SELinux, NFS, namespace limit, and sysctls are useful context, but the reported read occurs inside the embedded VM guest. There is insufficient evidence to attribute it to any particular host setting or prescribe changing those settings. The later workqueue message also does not establish the cause of the first failure.
   • Source and release comparison were completed; a faithful reproduction on the reported RHEL/KVM host was not available. No matching fixed report was found in the searches performed.

   Impact Signals

   • Affected users/scope: At least the reported x86_64 RHEL HPC configuration; every attempted VM sandbox fails before workload launch. Broader scope is unknown.
   • Regression: Unknown; no previously working version/configuration is identified.
   • Workaround: None demonstrated on this host. Docker/Podman are unavailable, and additional VM memory has not resolved the failure.
   • Evidence quality: Strong localization from the reported console error and corresponding source path; insufficient guest diagnostics to establish the root cause or a supported remediation.

   Evidence Needed for Investigation

   Please provide redacted gateway VM-driver configuration and the full guest console from boot through the first error, including any init drop-ins or runtime/rootfs overrides. A diagnostic run on the failing guest should capture its kernel version, actual /proc mount type/options, ownership/modes of /proc, /proc/1, and /proc/1/status, and whether the status read succeeds before versus after the same UID/GID/capability transition. Capture guest fs.suid_dumpable and the process's PR_GET_DUMPABLE value alongside that transition. Guest init suppresses procfs mount errors, so a diagnostic build should expose mount outcomes. These are guest diagnostics; host /proc output alone cannot distinguish the failure. Please also indicate whether the same matching v0.1.2 artifacts have ever worked on this host. Do not include tokens or bootstrap credentials.

   Related HPC / Apptainer Question

   #1393 is closed as not planned. #2255 remains open, and the maintainer's follow-up explicitly excludes rootless mode from its first release. Those records do not establish an Apptainer delivery date or guarantee that the OCI proposal would support this locked-down host. The MicroVM runtime is a separate path; this issue concerns its failing guest startup. Any updated Apptainer roadmap commitment requires a maintainer answer.

   Human Decision Required

   Decide whether to pursue deeper investigation. If yes, apply state:accepted, associate the issue with a roadmap item, or both, and decide whether the work remains human-owned. To queue unattended investigation/planning, also apply agent:plan-requested; a direct request to create-spike or build-from-issue can authorize that phase instead. state:validated records completion of this factual triage, not a confirmed root cause or product acceptance.
3. elezar

   added

   state:validated

   spike

   area:compute

   area:sandboxSandbox runtime and isolation workSandbox runtime and isolation work

   topic:compatibilityCompatibility-related workCompatibility-related work

   os:linuxIssue affects Linux hostsIssue affects Linux hosts

   arch:amd64Affects amd64 architectureAffects amd64 architecture

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Oct 1, 2026
4. github-actions commented on Oct 16, 2026

   

   github-actionsBot

   on Oct 16, 2026 – with GitHub Actions

   More actions

   This issue has had no activity for 14 days and is now marked stale. It may be closed in 14 days if there is no further activity. Comment or remove the state:stale label to keep it open.
5. github-actions

   added

   state:staleInactive item at risk of automatic closure.Inactive item at risk of automatic closure.

   on Oct 16, 2026
6. bombazook commented on Oct 25, 2026

   

   bombazook

   on Oct 25, 2026

   More actions

   Same failure on a different host profile, which may help narrow it down: it reproduces here without SELinux, NFS, rootless mode, or x86_64.

   Environment

   • OpenShell 0.1.5 (CLI, gateway and openshell-driver-vm from the 0.1.5-1 arm64 .deb)
   • Armbian 26.8.3 (Debian trixie), kernel 6.1.115-vendor-rk35xx, aarch64 (Rockchip RK3588: 4× Cortex-A55 + 4× Cortex-A76), 16 GB RAM
   • Gateway runs as root (system unit), compute_driver = "vm", TLS on, /dev/kvm present
   • State on local ext4 (NVMe), no NFS
   • Host LSMs: capability,apparmor,bpf; no SELinux, no Yama
   • fs.suid_dumpable = 0, user.max_user_namespaces = 63711

   Gateway config ([openshell.drivers.vm])

   state_dir = "/var/lib/openshell/vm"
   default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
   bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest"
   enable_bind_mounts = false

   Repro

   openshell sandbox create --name os4014-1792955449 --from nvcr.io/nvidia/base/ubuntu:24.04 --detach -- sleep 300

   [9.1s] Preparing rootfs
   [11.4s] Preparing overlay disk
   [12.2s] Waiting for VM supervisor
   Error:   × sandbox entered error phase while provisioning: ProcessExited: VM process exited with status 0
   

   These are the conditions afterwards. The phase is Error, and the provisioning phase is preparation.

   Condition	Status	Reason	Message
   WorkloadIdentity	True	DriverResolved	Resolved workload UID:GID is 1000:1000
   ConfigurationReady	False	ConfigurationPending	Waiting for effective configuration validation before workload activation
   Ready	False	ProcessExited	VM process exited with status 0

   Guest console (rootfs-console.log)

   [0.005s] setting up writable overlay root
   [0.094s] using prepared image rootfs lowerdir
   [0.192s] reconciled sandbox account (1000:1000)
   [0.202s] prepared /sandbox ownership (1000:1000)
   [0.211s] hostname=os4014-1792955449
   [0.234s] OPENSHELL_SANDBOX_ID=<redacted>
   [0.236s] starting OpenShell VM sandbox
   Error:   x read /proc/self/status
     `-> Permission denied (os error 13)
   

   Sandbox logs

   [sandbox] [INFO ] [openshell_supervisor] Starting sandbox supervision command=["sleep", "300"]
   [gateway] [INFO ] [openshell_server::compute] Sandbox phase changed
   [gateway] [WARN ] [openshell_server::compute] VM process exited with status 0
   

   The same happens with our own image, a Debian-based image with a sandbox user, and with the community base image.

   The failing read is the first one in qualify_runtime (crates/openshell-sandbox/src/main.rs). This host differs from the original report in SELinux, NFS, rootless mode and architecture. That points to the guest side, such as the procfs mount or the guest kernel config of the VM runtime on Linux, rather than to anything specific to the RHEL host.
7. vpzed commented on Oct 26, 2026

   

   vpzed

   on Oct 26, 2026

   More actions

   User Story

   I am evaluating OpenShell to run AI agents with separate-kernel MicroVM isolation and integrated policy controls. On my test deployment, the guest fails before its main workload starts, which blocks that evaluation.

   The diagnostic commands were executed by an AI assistant on systems I provided, under my approval. I reviewed the configuration, mechanism, and results, but I am not claiming that I manually typed every command.

   I need the released MicroVM driver to start a non-root workload without relaxing the gateway's shipped file-creation permissions or exposing host directories.

   Problem Statement

   Ubuntu 26.04.1 LTS x86_64, kernel 7.0.0-38, with the OpenShell 0.1.5 Debian package, a non-root gateway running as UID/GID 2001:2001 and a (bind) mount-free MicroVM workload configured as 1000:1000 fails runtime qualification under the package's shipped UMask=0077 setting.

   Similar to other reports here, the guest console reports read /proc/self/status: Permission denied (os error 13) after reconciling the guest account to 1000:1000. The sandbox enters Error, with configuration pending and no main-process start recorded.

   A controlled fresh-state sequence using service umasks 0077 → 0022 → 0077 produced failure → success → failure, with the same released binary, pinned image, explicit policy, guest identity, and VM sizing. Each case used its own database, VM cache, and writable overlay. All three reported WorkloadIdentity=True, reason DriverResolved, with UID:GID 1000:1000.

   Observation	A1: 0077	B: 0022	A2: 0077
   Create exit	1	0	1
   Phase before stop	Error	Ready	Error
   Configuration	Pending	Accepted	Pending
   Main workload started	No	Yes	No
   Actual non-root /proc/self/status read	Not reached	Succeeded as 1000:1000	Not reached
   Stable disk /upper mode	0700	0755	0700
   Stable disk /upper UID:GID	2001:2001	2001:2001	2001:2001

   The directory metadata was inspected read-only after confirmed sandbox stop. These observations strongly implicate umask-dependent guest-root traversal permissions, but do not prove that /upper is the only affected object. The error naming /proc/self/status does not by itself establish a procfs-specific defect.

   Impact / Why This Matters

   The failure prevents the intended non-root MicroVM workflow from reaching application execution, so I cannot proceed to evaluate the remaining policy controls.

   Using 0022 in an isolated diagnostic case allowed startup, but I have not accepted it as a deployment workaround: the safety of all other generated files and credentials under that broader setting has not been evaluated. Running the workload as root or changing its identity merely to avoid the failure would also change the workflow being evaluated.

   Acceptance Criteria

   • With the released package's restrictive UMask=0077, a mount-free MicroVM on the described host configuration reaches Ready and accepts its configuration with gateway UID/GID 2001:2001 and guest UID/GID 1000:1000.
   • The resolved identity remains 1000:1000, and the guest successfully executes id and actually reads /proc/self/status as that non-root identity.
   • This works with freshly created VM/database/cache state, without requiring a root workload, global umask relaxation, host-directory sharing, or manual changes to retained guest disks.
   • Any correction preserves intended host-private and guest-sensitive permissions rather than broadly relaxing generated state.

   Reproduction Steps

   Use an isolated Linux test deployment of the released 0.1.5 Debian package, with KVM accessible to its non-root user-service account. In the tested setup that account had UID/GID 2001:2001, while the guest was explicitly configured as 1000:1000. A host account already numbered 1000 is not an equivalent ownership comparison.

   Prerequisites: the packaged user service and local TLS are already configured, the CLI has an authenticated registration for this gateway, and no sandbox is running. Keep the package-managed TLS location unchanged across cases. Save existing nonsecret gateway configuration/drop-ins privately before changing them. Do not overwrite an existing installation's configuration or run this against valuable retained workloads. No OpenShell source checkout, custom image build, Docker/Podman installation, or project-specific helper is required for the failing operation.

   1. Prepare one fresh case

   Run the cases sequentially as a1/0077, b/0022, and a2/0077, stopping and preserving each case before preparing the next. The shell umask stays 077 for private administrative files; the service override below is the variable being tested.

   set -eu
   umask 077
   CASE=a1
   MASK=0077
   GATEWAY=ftest  # Replace with your existing authenticated CLI registration.
   ROOT="$HOME/openshell-umask-repro"
   CASE_DIR="$ROOT/$CASE"
   IMAGE='nvcr.io/nvidia/base/ubuntu@sha256:fdfe3e69644d4c52ccdf2195413d1037e17b0b2bc8ff000806c548cafdc686b6'
   
   test "$(id -u)" = 2001
   test "$(id -g)" = 2001
   test "$(systemctl --user show openshell-gateway -p ActiveState --value)" = inactive
   test "$(systemctl --user show openshell-gateway -p MainPID --value)" = 0
   mkdir -p "$ROOT"
   test ! -e "$CASE_DIR" && test ! -L "$CASE_DIR"
   mkdir -m 700 "$CASE_DIR"
   mkdir -m 700 "$CASE_DIR/state" "$CASE_DIR/evidence"
   
   cat > "$CASE_DIR/gateway.toml" <<EOF
   [openshell]
   version = 2
   [openshell.gateway]
   compute_driver = "vm"
   bind_address = "127.0.0.1:17670"
   [openshell.drivers.vm]
   state_dir = "$CASE_DIR/vm"
   driver_dir = "/usr/libexec/openshell"
   default_image = "$IMAGE"
   bootstrap_image = "$IMAGE"
   sandbox_uid = 1000
   sandbox_gid = 1000
   vcpus = 2
   mem_mib = 2048
   overlay_disk_mib = 4096
   EOF

   Write this explicit policy to $CASE_DIR/policy.yaml. It is the same custom keyless policy in every case, not the upstream default policy:

   version: 1
   filesystem_policy:
     include_workdir: false
     read_only:
       - /bin
       - /usr
       - /lib
       - /lib64
       - /etc
       - /proc
       - /dev/urandom
     read_write:
       - /sandbox
       - /tmp
       - /dev/null
   landlock:
     compatibility: hard_requirement
   process:
     run_as_user: "1000"
     run_as_group: "1000"
   network_policies: {}
   network_middlewares: {}

   2. Select the case and start the user service

   After reviewing/backing up the existing nonsecret configuration, select the new case:

   cp "$CASE_DIR/gateway.toml" "$HOME/.config/openshell/gateway.toml"
   mkdir -p "$HOME/.config/systemd/user/openshell-gateway.service.d"
   cat > "$HOME/.config/systemd/user/openshell-gateway.service.d/comparison.conf" <<EOF
   [Service]
   UMask=$MASK
   Environment=XDG_STATE_HOME=$CASE_DIR/state
   EOF
   openshell-gateway config preflight --path "$HOME/.config/openshell/gateway.toml"
   systemctl --user daemon-reload
   systemctl --user start openshell-gateway

   Verify the actual gateway and driver /proc/<pid>/status show the intended umask and UID/GID, and that the new database is under $CASE_DIR/state/openshell/gateway/. Use bounded read-only readiness checks if the listener is still starting; do not repeatedly restart the service. Before creation, require authenticated status and a complete empty inventory:

   openshell -g "$GATEWAY" status
   openshell -g "$GATEWAY" sandbox list --all-workspaces -o json

   Stop if the inventory is nonempty or the active paths/identity/umask do not match the new case. Check other unit drop-ins/environment overrides rather than assuming this one is effective.

   3. Make one create attempt and inspect the result

   if openshell -g "$GATEWAY" sandbox create \
     --name "review-$CASE" --from "$IMAGE" \
     --policy "$CASE_DIR/policy.yaml" \
     --no-auto-providers --restart-policy never --approval-mode manual \
     --detach --no-tty -o json -- /bin/sleep 600 \
     > "$CASE_DIR/evidence/create.stdout" \
     2> "$CASE_DIR/evidence/create.stderr"; then
     CREATE_EXIT=0
   else
     CREATE_EXIT=$?
   fi
   printf '%s\n' "$CREATE_EXIT" > "$CASE_DIR/evidence/create.exit"
   openshell -g "$GATEWAY" sandbox get "review-$CASE" -o json \
     > "$CASE_DIR/evidence/get.json"

   The executed comparison bounded each create client at 210 seconds; none timed out. A timeout or client failure is not permission to retry: inspect the exact sandbox first and preserve evidence. Examine WorkloadIdentity, readiness, configuration admission, and main-process start separately. Preserve a small relevant guest-console excerpt from this case's sandbox UUID directory; do not publish tokens, runtime descriptors, keys, or full private logs.

   For any Ready case, including an unexpectedly successful A case:

   openshell -g "$GATEWAY" sandbox provider list "review-$CASE" -o json
   openshell -g "$GATEWAY" sandbox exec "review-$CASE" \
     --timeout 15 --no-tty --no-login-shell -- \
     /bin/sh -c 'set -eu; /usr/bin/id; /bin/cat /proc/self/status >/dev/null; printf "proc-status-read-ok\n"'

   4. Preserve and stop before the next case

   Capture the case's bounded diagnostics before stopping. Then:

   openshell -g "$GATEWAY" sandbox stop "review-$CASE"
   openshell -g "$GATEWAY" sandbox get "review-$CASE" -o json

   Require Stopped and no VM/supervisor writers before inspecting disk metadata. Optional supporting evidence, using the exact UUID returned by sandbox get and the already-present VM prerequisite debugfs:

   # Set SANDBOX_ID to this case's API-returned UUID, not its sandbox name.
   debugfs -R 'stat /upper' "$CASE_DIR/vm/sandboxes/$SANDBOX_ID/overlay.ext4"

   Do not use debugfs -w, change disk permissions, delete the sandbox, or reuse its database/cache. Preserve post-stop diagnostics, then stop the case gateway and verify it and its descendants are gone:

   systemctl --user stop openshell-gateway
   systemctl --user show openshell-gateway \
     -p ActiveState -p SubState -p MainPID -p UnitFileState

   Repeat from step 1 with CASE=b, MASK=0022, then CASE=a2, MASK=0077, always using unused paths. Keep image, policy, guest identity and sizing unchanged. Leave the service stopped after the final case, retain evidence, and record which configuration remains selected. This is a diagnostic comparison, not installation guidance for a permissive umask.

   Suggested UX (if applicable)

   No new CLI or configuration interface is requested. The existing documented MicroVM identity configuration should work under the packaged service permissions. This report does not prescribe a particular internal fix.

   Environment

   • OpenShell CLI/gateway/VM driver: 0.1.5, released openshell_0.1.5-1_amd64.deb (latest stable at draft review).[5]
   • Package SHA256: 6af7859a87ab0b3bf96aef2216db191b597308ac00df9bfe5d0e5e9912f8e2ab.
   • Installed VM-driver SHA256: c93fc9a2d9612817ce205ea1c3a74f49df6cda75ced0bd72af93c848488a4874.
   • Host: Ubuntu 26.04.1, x86_64, kernel 7.0.0-38-generic; systemd 259.9; e2fsprogs 1.47.2-3ubuntu4.
   • Deployment: native packaged user-service gateway, non-root UID/GID 2001:2001, KVM accessible; non-GPU libkrun MicroVM path.
   • Gateway user had KVM group membership and linger enabled for the SSH-operated test deployment. No account/group changes between cases.
   • Guest: explicit 1000:1000 in driver configuration and policy. No shared host-directory mounts requested; the live successful VM processes had no --vm-mount arguments.
   • Each case: fresh database, VM state/cache and writable overlay, with the same package-managed local TLS trust.
   • No custom image, Podman/Docker image helper, provider attachment, source patch, or runtime build was required for this reproduction.
   • B's console separately warned that pids.max was unavailable. This comparison does not establish resource-limit enforcement, egress enforcement, artifact export, or comprehensive isolation.

   Logs

   Minimal excerpt from each failing A case, with ANSI escapes removed and variable timestamps omitted:

   reconciled sandbox account (1000:1000)
   Error: x read /proc/self/status
     `-> Permission denied (os error 13)
   

   Failing A cases' authenticated conditions:

   WorkloadIdentity: True / DriverResolved
     Resolved workload UID:GID is 1000:1000
   ConfigurationReady: False / ConfigurationPending
   Ready: False / ProcessExited
     VM process exited with status 0
   main_process_started_at_ms: 0
   

   Successful B exec output (exit 0):

   uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu)
   proc-status-read-ok
   

   All three sandboxes were explicitly stopped and verified Stopped; the gateway was then stopped, with its prior service cgroup removed and no visible runtime descendants. Evidence and disks were preserved. The process inspection was unprivileged, not an exhaustive privileged host audit.