Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 18 additions & 6 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,12 +297,24 @@ sandbox and can negate OpenShell workspace isolation and filesystem-policy
controls. Driver-owned supervisor, token, and TLS bind mounts stay reserved.

Network features follow the driver/substrate split. Drivers own only the outer
fence and protected channel. The sandbox owns seccomp notification, local DNS,
socket virtualization, process observation, and binary identity. The supervisor
owns DNS eligibility, policy authorization, destination filtering, upstream
dials, relay behavior, credential rewriting, and OCSF decisions. No supported
path requires nftables, a workload network namespace, proxy environment
variables, added capabilities, or an unconfined AppArmor profile.
fence and protected channel. The native Linux adapter owns seccomp notification,
local DNS, socket virtualization, process observation, and binary identity. The
Kubernetes gVisor adapter instead combines the sentry and zero-rule workload
`NetworkPolicy` with a workload-local explicit proxy whose streams cross the
protected channel. The supervisor owns DNS eligibility, policy authorization,
destination filtering, upstream dials, relay behavior, credential rewriting,
and OCSF decisions. Native mode does not require proxy environment variables;
gVisor mode injects loopback HTTP proxy variables and applies endpoint-only
policy without added capabilities or an unconfined AppArmor profile.

The BlueField VM prototype keeps the same split while allowing one accelerated
guest VF behind a separate hardware outer fence. The compute driver owns VF and
representor assignment; the isolation backend proves that DPU steering starts
default-deny and that supervisor-authorized, process-attributed flows can be
installed and revoked for the current sandbox generation. Ordinary VM evidence
continues to require a NIC-less guest. The prototype currently defines only the
fence and confirmation evidence; it does not attach hardware or enable a runtime
adapter.

The Kubernetes deployment packaging has two ownership boundaries. The gateway
chart owns the gateway workload, configuration, Services, PKI, and
Expand Down
37 changes: 31 additions & 6 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ Each sandbox has three trust levels:
| Component | Role |
|---|---|
| Supervisor | Owns gateway credentials, admitted policy, L7 proxying, SSH, and gateway relays. It never executes inside the agent workload. |
| Sandbox | Runs as the same non-root identity as the agent, installs the workload seccomp listener, applies the Landlock baseline, owns child processes, and mediates the protected supervisor channel. |
| Agent child | Inherits the sandbox network listener and runs with zero capabilities, `no_new_privs`, Landlock, and the final syscall filter. |
| Sandbox | Runs as the same non-root identity as the agent, owns child processes, mediates the protected supervisor channel, and instantiates the driver-selected native Linux or gVisor adapter. |
| Agent child | Runs with zero capabilities behind the selected adapter. Native Linux children inherit seccomp mediation, `no_new_privs`, Landlock, and the final syscall filter; gVisor children rely on the sentry, OCI mounts, and the outer network fence. |

The runtime grants neither trusted component nor agent child any Linux
capability inside the workload. Drivers resolve one exact non-root UID, GID,
Expand Down Expand Up @@ -65,9 +65,13 @@ replacement from granting authority.
run untrusted code yet.
3. `openshell-supervisor` loads policy and runtime settings from the gateway,
attaches to the sandbox, and verifies the driver's generation and evidence.
4. The sandbox installs its seccomp notification broker and Landlock baseline,
then reports measured confirmation. The supervisor must accept that evidence
before it sends the launch permit.
4. The sandbox starts the selected runtime adapter, validates its mechanism-
specific audit evidence, and reports backend-neutral
enforcement properties. The supervisor must accept those properties and
their immutable session and resource binding before it sends the launch
permit. Other isolation backends may establish the same properties with
different mechanisms and retain their detailed evidence in backend-owned
audit data.
5. The sandbox starts the canonical process through its single workload
launcher. The supervisor starts SSH and registers its gateway session.
6. Exec, signaling, PTY, DNS, TCP, and loopback-forwarding operations cross the
Expand Down Expand Up @@ -96,9 +100,30 @@ OpenShell uses overlapping controls rather than a single sandbox primitive:
| Filesystem policy | Landlock restricts the paths the agent can read or write. |
| Process policy | Sandbox and children run as one immutable non-root identity with zero capabilities. |
| Seccomp notification | Virtualizes supported INET sockets and sends DNS/TCP decisions to the supervisor without nftables or proxy environment variables. |
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, or Kubernetes NetworkPolicy prevents any missed or unsupported kernel path from escaping. |
| Driver outer fence | Docker `network_mode=none`, a NIC-less VM, Kubernetes NetworkPolicy, or a default-deny BlueField DPU assignment prevents any missed or unsupported kernel path from escaping. |
| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. |

The BlueField VM prototype permits one workload-visible VF without weakening
the ordinary VM fence. Its distinct fence binds the VF, representor, DPU,
assignment generation, and policy generation to the sandbox generation. The
DPU starts default-deny with no unmanaged guest uplinks. Before an accelerated
flow can carry traffic, the backend must prove an attributed supervisor
authorization round trip, revocation, and fail-closed behavior when its control
channel is lost. These measurements remain backend-owned audit evidence and
project into the same backend-neutral properties as other adapters. This layer
defines and tests that confirmation contract; hardware discovery, VF attachment,
and flow programming remain follow-up work and no driver selects it yet.

The Kubernetes gVisor adapter uses the same authenticated lifecycle and process
backend with different enforcement mechanisms. The gVisor sentry and OCI mounts
provide the workload boundary. A zero-rule Kubernetes egress `NetworkPolicy`
blocks direct connections, while a workload-local HTTP/CONNECT listener reverse-
tunnels streams to the existing supervisor proxy. The supervisor loads policy
in endpoint-only mode because the tunnel authenticates the sandbox generation,
not an individual executable. This adapter intentionally does not claim native
Landlock path policy, nested child seccomp, transparent TCP, or per-binary
network attribution.

The supervisor may enrich baseline filesystem allowances for runtime-required
paths, such as proxy support files or GPU device paths when a GPU is present.
These internal allowances must stay sandbox-scoped and avoid exposing host
Expand Down
6 changes: 4 additions & 2 deletions crates/openshell-driver-docker/src/isolation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@ use std::path::PathBuf;
use openshell_isolation_interface::contract::{DriverFenceEvidence, ResolvedWorkloadIdentity};
use openshell_sandbox_backend::GPU_RESOURCE_CLAIM;
use openshell_sandbox_backend::boundary_protocol::{
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeDescriptor,
SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
BoundaryConfig, BoundaryListener, GatewayVerificationKey, SandboxRuntimeAdapter,
SandboxRuntimeDescriptor, SandboxTlsClientConfig, SandboxTlsServerConfig, SandboxTransport,
};

/// Driver-owned inputs that bind one Docker container to one boundary.
Expand Down Expand Up @@ -78,6 +78,7 @@ impl DockerBoundarySpec {
resource_claims: resource_claims.clone(),
resource_claim_files: BTreeMap::new(),
workload_identity: self.workload_identity.clone(),
adapter: SandboxRuntimeAdapter::default(),
driver_fence: driver_fence.clone(),
child_env: self.child_env,
},
Expand All @@ -86,6 +87,7 @@ impl DockerBoundarySpec {
generation: self.generation,
session_id: self.session_id,
workload_identity: self.workload_identity,
adapter: SandboxRuntimeAdapter::default(),
transport: SandboxTransport::Unix {
socket_path: self.control_socket,
},
Expand Down
10 changes: 10 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,16 @@ default seccomp profile. The sandbox installs a nested seccomp user-notification
filter without requesting a capability in the Pod spec. Startup fails closed
when the runtime blocks the required seccomp or Landlock operations.

An effective `pod.runtime_class_name` of `gvisor` selects the gVisor adapter in
the same sandbox backend. The workload Pod keeps the non-root, drop-all-
capabilities posture but omits the Kubernetes seccomp profile and custom sysctl
that GKE Sandbox does not support. The sandbox qualifies the gVisor sentry,
starts an explicit proxy on `127.0.0.1:3128`, and reverse-tunnels proxy streams
to the supervisor over the authenticated boundary protocol. The empty-egress
`NetworkPolicy` blocks direct workload connections. This mode applies endpoint-
only network policy and does not apply Landlock path rules, the nested child
seccomp filter, transparent TCP, or per-binary network attribution.

The supervisor Pod has a direct, non-controller owner reference to the Sandbox
resource. This links its garbage-collection lifecycle to the sandbox without
competing with the Agent Sandbox controller for workload-Pod ownership.
Expand Down
145 changes: 125 additions & 20 deletions crates/openshell-driver-kubernetes/src/driver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1962,6 +1962,7 @@ impl KubernetesComputeDriver {
return Err(fail("pair labels changed"));
}
let spec = pod.spec.as_ref().ok_or_else(|| fail("missing Pod spec"))?;
let gvisor = spec.runtime_class_name.as_deref() == Some("gvisor");
if spec.host_network == Some(true)
|| spec.host_pid == Some(true)
|| spec.host_ipc == Some(true)
Expand Down Expand Up @@ -2013,10 +2014,12 @@ impl KubernetesComputeDriver {
.supplemental_groups
.as_deref()
.is_some_and(|groups| !groups.is_empty())
|| security
.seccomp_profile
.as_ref()
.is_none_or(|profile| profile.type_ != "RuntimeDefault")
|| (!gvisor
&& security
.seccomp_profile
.as_ref()
.is_none_or(|profile| profile.type_ != "RuntimeDefault"))
|| (gvisor && security.seccomp_profile.is_some())
{
return Err(fail("numeric identity, groups, or seccomp profile changed"));
}
Expand Down Expand Up @@ -2046,9 +2049,17 @@ impl KubernetesComputeDriver {
&& sysctl.get("value").and_then(serde_json::Value::as_str) == Some("0")
})
});
if !unprivileged_port_sysctl {
if !gvisor && !unprivileged_port_sysctl {
return Err(fail("safe unprivileged-port sysctl changed"));
}
if gvisor
&& pod_json
.pointer("/spec/securityContext/sysctls")
.and_then(serde_json::Value::as_array)
.is_some_and(|sysctls| !sysctls.is_empty())
{
return Err(fail("gVisor workload must not request custom sysctls"));
}
let check_container = |container: &k8s_openapi::api::core::v1::Container,
name: &str|
-> Result<(), KubernetesDriverError> {
Expand Down Expand Up @@ -2265,6 +2276,16 @@ impl KubernetesComputeDriver {
agent_gid,
&names.sandbox_secret,
)?;
let runtime_adapter = if workload_pod
.spec
.as_ref()
.and_then(|spec| spec.runtime_class_name.as_deref())
== Some("gvisor")
{
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
} else {
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
};
let workload_pod_uid =
workload_pod.metadata.uid.clone().ok_or_else(|| {
KubernetesDriverError::Message("workload Pod has no UID".to_string())
Expand Down Expand Up @@ -2308,6 +2329,22 @@ impl KubernetesComputeDriver {
child_env.extend(spec.environment.clone());
}
child_env.retain(|name, _| !name.starts_with("OPENSHELL_"));
if matches!(
runtime_adapter,
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
) {
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
}
child_env.insert(
"NO_PROXY".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
child_env.insert(
"no_proxy".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
}
let host_gateway_ip = self.config.host_gateway_ip.parse().ok();
let session_id = launch_authentication.supervisor.session_id;
let tls = generate_sandbox_tls_material(session_id)
Expand Down Expand Up @@ -2368,6 +2405,7 @@ impl KubernetesComputeDriver {
},
host_gateway_ip,
workload_identity,
adapter: runtime_adapter,
child_env,
}
.provision();
Expand Down Expand Up @@ -2470,7 +2508,7 @@ impl KubernetesComputeDriver {
supervisor_uid: &str,
agent_uid: u32,
agent_gid: u32,
child_env: std::collections::HashMap<String, String>,
mut child_env: std::collections::HashMap<String, String>,
launch_authentication: &openshell_core::jwt::SandboxLaunchAuthentication,
) -> Result<(), KubernetesDriverError> {
let namespace_uid = Api::<Namespace>::all(self.client.clone())
Expand Down Expand Up @@ -2525,6 +2563,32 @@ impl KubernetesComputeDriver {
agent_gid,
&names.sandbox_secret,
)?;
let runtime_adapter = if workload_pod
.spec
.as_ref()
.and_then(|spec| spec.runtime_class_name.as_deref())
== Some("gvisor")
{
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
} else {
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::NativeLinux
};
if matches!(
runtime_adapter,
openshell_sandbox_backend::boundary_protocol::SandboxRuntimeAdapter::Gvisor
) {
for name in ["HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"] {
child_env.insert(name.to_string(), "http://127.0.0.1:3128".to_string());
}
child_env.insert(
"NO_PROXY".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
child_env.insert(
"no_proxy".to_string(),
"localhost,127.0.0.1,::1".to_string(),
);
}
let workload_pod_uid =
workload_pod.metadata.uid.clone().ok_or_else(|| {
KubernetesDriverError::Message("workload Pod has no UID".to_string())
Expand Down Expand Up @@ -2606,6 +2670,7 @@ impl KubernetesComputeDriver {
},
host_gateway_ip: self.config.host_gateway_ip.parse().ok(),
workload_identity,
adapter: runtime_adapter,
child_env,
}
.provision();
Expand Down Expand Up @@ -5215,6 +5280,10 @@ fn apply_supervisor_sandbox_runtime_boundary(
else {
return;
};
let gvisor = spec
.get("runtimeClassName")
.and_then(serde_json::Value::as_str)
== Some("gvisor");
spec.insert("hostNetwork".to_string(), serde_json::json!(false));
spec.insert("hostPID".to_string(), serde_json::json!(false));
spec.insert("hostIPC".to_string(), serde_json::json!(false));
Expand All @@ -5238,20 +5307,22 @@ fn apply_supervisor_sandbox_runtime_boundary(
]
}),
);
spec.insert(
"securityContext".to_string(),
serde_json::json!({
"runAsUser": params.sandbox_uid,
"runAsGroup": params.sandbox_gid,
"runAsNonRoot": true,
"fsGroup": params.sandbox_gid,
"fsGroupChangePolicy": "OnRootMismatch",
"supplementalGroups": [],
"supplementalGroupsPolicy": "Strict",
"seccompProfile": {"type": "RuntimeDefault"},
"sysctls": [{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}]
}),
);
let mut pod_security = serde_json::json!({
"runAsUser": params.sandbox_uid,
"runAsGroup": params.sandbox_gid,
"runAsNonRoot": true,
"fsGroup": params.sandbox_gid,
"fsGroupChangePolicy": "OnRootMismatch",
"supplementalGroups": [],
"supplementalGroupsPolicy": "Strict"
});
if !gvisor {
pod_security["seccompProfile"] = serde_json::json!({"type": "RuntimeDefault"});
pod_security["sysctls"] = serde_json::json!([
{"name": "net.ipv4.ip_unprivileged_port_start", "value": "0"}
]);
}
spec.insert("securityContext".to_string(), pod_security);
let volumes = spec
.entry("volumes")
.or_insert_with(|| serde_json::json!([]))
Expand Down Expand Up @@ -8504,6 +8575,40 @@ mod tests {
);
}

#[test]
fn gvisor_runtime_omits_incompatible_kernel_security_context() {
let pod_template = {
let params = SandboxPodParams {
default_runtime_class_name: "gvisor",
..SandboxPodParams::default()
};
sandbox_template_to_k8s(
&SandboxTemplate::default(),
false,
&std::collections::HashMap::new(),
true,
&params,
)
};

assert_eq!(
pod_template["spec"]["runtimeClassName"],
serde_json::json!("gvisor")
);
assert!(
pod_template["spec"]["securityContext"]["seccompProfile"].is_null(),
"GKE Sandbox rejects Kubernetes seccomp profiles"
);
assert!(
pod_template["spec"]["securityContext"]["sysctls"].is_null(),
"GKE Sandbox rejects custom sysctls"
);
assert_eq!(
pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["drop"],
serde_json::json!(["ALL"])
);
}

#[test]
fn template_runtime_class_name_overrides_config_default() {
let template = SandboxTemplate {
Expand Down
Loading
Loading