diff --git a/dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md b/dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md new file mode 100644 index 00000000..8140cb4d --- /dev/null +++ b/dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md @@ -0,0 +1,16 @@ +### Added + +- **Per-phase network policy for `docker-*` runs** — build, solve and verify + each get `none`, `allowlist` (egress sidecar) or `public`. Defaults: build + public, solve and verify none. Set from task.toml (`[environment]`, + `[agent]`, `[verifier]` `network_mode` / `allowed_hosts`, Harbor's + `allow_internet`), or `dokimi tasks run --network PHASE=MODE[:hosts]`. + The effective plan and each phase's source land in `metrics.network`; a + skill's `metadata.network` is recorded as `metrics.network.skill`, not + granted. + +### Changed + +- **A task whose `network_mode` is not `none`, `allowlist` or `public`, or that + lists `allowed_hosts` without `allowlist`, now errors the run** instead of + running offline. diff --git a/dokimi/docs/coding-harnesses.md b/dokimi/docs/coding-harnesses.md index 6fff8971..d2428656 100644 --- a/dokimi/docs/coding-harnesses.md +++ b/dokimi/docs/coding-harnesses.md @@ -302,16 +302,19 @@ The result is in `metrics.isolation_probe` and `isolation_probe.json`: data, and `scripts/check_smoke_runs.py` fails the run on it. No `egress.jsonl` is written for an `unknown` cell: a missing file is not "no requests". The file is also absent when no sandbox ran - (`docker-oracle`, the `-host` variants, a build failure), so use - `metrics.egress.status` to tell unknown from a cell that ran no sandbox. + (`docker-oracle` and the `-host` variants with no allowlisted phase, a + build failure), so use `metrics.egress.status` to tell unknown from a + cell that ran no sandbox. The file and `metrics.egress` cover the + solve's sidecar only: an allowlisted verifier's decisions are + `metrics.network.verify_egress`, also saved in `network.json`. `docker-probe` reads `violated` on a passing cell, because the probe requests denied hosts (`example.com`) by design; judge a probe by `isolation_probe.json`, not by `egress.status`. -- **`network_mode = "public"`.** It applies to the verifier only. The - container joins the default bridge after the agent exits. Every - uxlfoundation/skills task declares it, and none of their solutions or tests - uses the network, so a task whose solve needs a host must list it under - `environment.egress`. +- **`network_mode = "public"`.** An `[environment]`-level `public` applies + to the oracle and the verifier, not to an agent: every uxlfoundation/skills + task declares it, and none of their solutions or tests uses the network. A + task whose agent needs a host lists it under `environment.egress`. See + [Network policy](#network-policy) for the other sources. - **Environment.** HOME is its own tmpfs, `/run/abhome`; the image's `/tmp` is left alone. The environment is: - the provider-credential allowlist; @@ -520,6 +523,43 @@ A `command:` harness (including a `--command-template` that wraps `harbor`) keeps its own rule: a `reward.txt` / `reward.json` under the run dir, else a `reward: x` on its output, else exit code 0 is a pass. +### Network policy + +Every `docker-*` run has a network per phase, recorded in +`metrics.network` with the source of each phase's setting, so two runs can +be checked for the same network before their verdicts are compared: + +| Phase | Default | What runs | +|---|---|---| +| `build` | `public` (host network, host proxy as build args) | `docker build` | +| `solve` | `none` (an in-container agent still reaches its LLM through the sidecar) | the oracle, or the agent | +| `verify` | `none` | `tests/test.sh` | + +The modes are Harbor's: `none` (Harbor's `no-network`), `public` (the +default bridge plus the host's proxy variables), and `allowlist` (the listed +`host[:port]` entries only, through an egress sidecar; the verifier's +decisions land in `metrics.network.verify_egress`). Later sources win: + +1. The defaults above. +2. The task: `[environment]` `network_mode` and `allowed_hosts` (or Harbor's + deprecated `allow_internet`), then `[agent]` and `[verifier]` overrides, + and `environment.egress` for an agent's solve. An unknown mode refuses + the cell, and its `network.json` holds the reason. +3. The operator: `dokimi tasks run --network PHASE=MODE[:host,...]`, + repeatable, with `all` for every phase. For example + `--network verify=none` grades a public task offline, and + `--network solve=allowlist:pypi.org,files.pythonhosted.org` lets every + arm install packages. + +A skill that needs the network at solve time declares it in its `SKILL.md` +frontmatter as `metadata.network` (`allowlist:host,...` or `public`), with +the reason for people in `compatibility`, the Agent Skills spec's field for +environment needs. The run does not widen the skill arm for it: its baseline +would then run on another network. `metrics.network.skill.missing` lists the +declared hosts this run did not grant; pass them with `--network solve=...` +so every arm gets them. A value that does not parse is recorded as it is, with +`error` and `missing: null`, and the arm still runs. + ## BenchFlow Cells Skills are not bind-mounted: BenchFlow copies the `--skills-dir` root into the diff --git a/dokimi/docs/contributing-terminal-bench-task.md b/dokimi/docs/contributing-terminal-bench-task.md index 03dadb7c..19c814a8 100644 --- a/dokimi/docs/contributing-terminal-bench-task.md +++ b/dokimi/docs/contributing-terminal-bench-task.md @@ -115,9 +115,9 @@ Field by field, with the consumer that actually reads it: | `environment.build_timeout_sec` | `docker-*` harnesses | Image-build budget; used instead of the default when set (oneAPI/pip-heavy images need > 10 min). | | `environment.docker_image` | Harbor | Image tag Harbor expects to exist (`harbor run` does **not** build it for you). The in-repo `docker-*` harnesses build and tag `dokimi/:latest` themselves. | | `environment.cpus`, `memory_mb` | `docker-*` harnesses | Container limits (`--cpus`, `--memory`); default to 4 CPUs / 2048 MB when absent. A malformed value falls back to the default rather than failing the run. | -| `environment.network_mode` | `docker-*` harnesses | `"public"` gives the **verifier** NAT'd internet (bridge), and `docker-oracle`'s solve too. The `docker-claude` agent never gets it: it solves on an internal network whose only way out is the egress proxy (see `environment.egress`). Anything else, including absent, means `--network none`. The image *build* always gets host network. Prefer offline. | +| `environment.network_mode` | `docker-*` harnesses, Harbor | `"no-network"` (absent means the same), `"allowlist"` with `allowed_hosts`, or `"public"` (NAT'd internet, bridge). It applies to the **verifier** and `docker-oracle`'s solve, not to an agent: an agent solves on an internal network whose only way out is the egress proxy (see `environment.egress`), unless `[agent] network_mode` says otherwise. `[agent]` and `[verifier]` take the same keys for their phase. Any other value, or `allowed_hosts` without `"allowlist"`, errors the run. The image *build* gets host network unless the operator passes `--network build=none`. See [Network policy](./coding-harnesses.md#network-policy). Prefer offline. | | `environment.egress` | `docker-claude` | Hosts the agent's solve may reach through the egress proxy, e.g. `["pypi.org", "files.pythonhosted.org"]`. A bare host means `host:443`; `host:port` is accepted. No wildcards (`*.pythonhosted.org`), commas or leading/trailing dots: list each host. A plain-HTTP host is reached directly, so it must be in the operator's `NO_PROXY`. | -| `environment.allow_internet` | Harbor | Harbor's equivalent policy flag; set it `false` and keep it consistent with `network_mode`. | +| `environment.allow_internet` | `docker-*` harnesses, Harbor | Harbor's deprecated flag: `false` is `"no-network"`, `true` is `"public"`, read only when `network_mode` is absent. Keep it consistent with `network_mode`. | | `environment.storage_mb`, `gpus` | Harbor | Declared resource envelope. | Unknown tables and keys are ignored, so a task may carry extra fields the diff --git a/dokimi/dokimi/commands/tasks.py b/dokimi/dokimi/commands/tasks.py index 69895485..bb082c52 100644 --- a/dokimi/dokimi/commands/tasks.py +++ b/dokimi/dokimi/commands/tasks.py @@ -189,6 +189,7 @@ def cmd_tasks_run(args: argparse.Namespace) -> None: command_template=args.command_template, dry_run=args.dry_run, repeats=args.repeats, + network=tuple(args.network), ) try: runner.validate_harnesses() @@ -269,6 +270,14 @@ def register(sub, positive_int) -> None: "{model}, {output_dir}, {prompt}, {prompt_quoted}." ), ) + run_p.add_argument( + "--network", action="append", default=[], metavar="PHASE=MODE", + help=( + "Network for a docker-* phase (build, solve, verify, or all): none, public, or " + "allowlist:host[:port],... Repeatable; overrides the task's own network_mode. " + "Default build=public solve=none verify=none" + ), + ) run_p.add_argument( "--dry-run", action="store_true", help="Build artifact without invoking agents" ) diff --git a/dokimi/dokimi/harnesses/container_agent.py b/dokimi/dokimi/harnesses/container_agent.py index 8b5d9b15..f0c32480 100644 --- a/dokimi/dokimi/harnesses/container_agent.py +++ b/dokimi/dokimi/harnesses/container_agent.py @@ -139,14 +139,17 @@ class ContainerSandbox: @classmethod def open(cls, run_id: str, docker: Docker, log, *, source_env: dict[str, str], task_egress: Optional[list[str] | str] = None, - with_agent: bool = True, runner: process.Runner = process.run) -> "ContainerSandbox": + with_agent: bool = True, include_llm: bool = True, + runner: process.Runner = process.run) -> "ContainerSandbox": + """The network and sidecar for one run; ``include_llm`` off (the oracle) allows only *task_egress*.""" if with_agent: preflight(source_env) binary = claude_binary() if with_agent else None extra_hosts = [task_egress] if isinstance(task_egress, str) else list(task_egress or []) # Every entry, not only the task's: a malformed one (a Vertex region with a comma) # fails here with its name, before the sidecar splits it into extra allowed hosts. - allow = [normalize_entry(h) for h in llm_hosts(source_env) + [str(h) for h in extra_hosts]] + llm = llm_hosts(source_env) if include_llm else [] + allow = [normalize_entry(h) for h in llm + [str(h) for h in extra_hosts]] files = {v: Path(source_env[v]) for v in _FILE_VARS if source_env.get(v) and Path(source_env[v]).is_file()} sandbox = cls(network=f"dokimi-net-{run_id}", proxy_name=f"dokimi-egress-{run_id}", @@ -183,6 +186,21 @@ def _start_proxy(self, docker: Docker, log) -> None: time.sleep(0.2) raise RuntimeError(f"egress proxy {self.proxy_name} did not start; see run.log") + def verifier_sidecar(self, docker: Docker, log, allow: list[str]) -> "ContainerSandbox": + """A fresh sidecar on this network for the verifier's allowlist, after :meth:`stop_proxy`. + + Its own name and decision log; no LLM hosts. The caller stops it before :meth:`close`. + """ + side = ContainerSandbox(network=self.network, proxy_name=f"{self.proxy_name}-verify", + allow=sorted(allow), binary=None, label=self.label, runner=self.runner) + docker(["rm", "-f", side.proxy_name], log, check=False) + try: + side._start_proxy(docker, log) + except BaseException: # started, it holds the upstream proxy env and the network: close() can't see it + docker(["rm", "-f", side.proxy_name], log, check=False) + raise + return side + @property def proxy_url(self) -> str: return f"http://egress:{PROXY_PORT}" diff --git a/dokimi/dokimi/harnesses/docker_solver.py b/dokimi/dokimi/harnesses/docker_solver.py index 87b36cf0..0d841639 100644 --- a/dokimi/dokimi/harnesses/docker_solver.py +++ b/dokimi/dokimi/harnesses/docker_solver.py @@ -74,6 +74,8 @@ from .command import _parse_reward_text from .container_agent import AGENT_BIN, DOCKER_STEP_TIMEOUT, FILES_DIR, ContainerSandbox, Docker from .models import HarnessResult, OperationRecord +from .network_policy import (DEFAULTS, NetworkPlan, PhasePolicy, build_args, enter_phase, needs_sandbox, + proxy_env_flags, resolve, resolve_recorded, skill_needs, start_args, redact as _redact) from .skill_treatment import Treatment, build_plugin, check_skill, skill_listing, treatment_notice from .solver_stream import record_solver, run_solver from .tasks import ( # noqa: F401 (timeouts re-exported) @@ -81,7 +83,6 @@ _DEFAULT_VERIFY_TIMEOUT, FORMAT_TASK_MD, TaskSpec, - _is_public, _task_build_timeout, _task_verify_timeout, ) @@ -89,7 +90,6 @@ # Recompile commands embedded in instruction.md, e.g. `g++ -O3 ... -o /app/bench`. # C tasks (missing-restrict) use gcc, C++ tasks use g++. _RECOMPILE_RE = re.compile(r"(g\+\+|gcc) [^`\n]*-o /app/[^ `\n]*") -_PROXY_VARS = ("http_proxy", "https_proxy", "no_proxy", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY") _DEFAULT_SOLVE_TIMEOUT = 600.0 _CONTAINER_PLUGIN = "/opt/abskill" # The only tree a docker-* agent edits: the host agent's snapshot and copy-back, @@ -124,6 +124,7 @@ class DockerSolverHarness: kind: str = "docker" env: dict[str, str] = field(default_factory=dict) agent_location: str = "container" + network: tuple[str, ...] = () # --network PHASE=MODE overrides (see :mod:`.network_policy`) runner: process.Runner = field(default=process.run, compare=False, repr=False) skill_notice: bool = True @@ -138,6 +139,7 @@ def run( ) -> HarnessResult: _refuse_unrunnable(task, self.mode, self.agent_location) output_dir = Path(output_dir) if output_dir else None + plan = self._plan(task, record=None if dry_run else output_dir) if output_dir: output_dir.mkdir(parents=True, exist_ok=True) model = model or self.model_default @@ -169,6 +171,7 @@ def run( operations=operations, metrics={ "dry_run": dry_run, "mode": self.mode, "unresolved_env": unresolved, + "network": self._network_metrics(plan), "error": "task needs host env not forwarded: " + "; ".join(f"{t}.env {', '.join(n)}" for t, n in unresolved.items()), }, @@ -185,7 +188,7 @@ def run( passed=False, output_dir=str(output_dir) if output_dir else None, operations=operations, - metrics={"dry_run": True, "mode": self.mode}, + metrics={"dry_run": True, "mode": self.mode, "network": self._network_metrics(plan)}, ) if output_dir is None: @@ -203,11 +206,12 @@ def run( start = time.time() returncode = 0 extra_metrics: dict[str, Any] = {} + network = self._network_metrics(plan) reward: Optional[float] = None sandbox: Optional[ContainerSandbox] = None try: try: - self._build_image(task, image, log) + self._build_image(task, image, log, plan.build) except subprocess.TimeoutExpired as exc: # The image never got built, so no agent ran and there is no # verdict: raise so the cell is errored like a build that fails @@ -216,15 +220,12 @@ def run( raise RuntimeError( f"docker build timed out after {exc.timeout}s; see run.log" ) from exc - if self.mode in ("claude", "probe") and self.agent_location == "container": - sandbox = ContainerSandbox.open(container, self._docker, log, source_env=operator_source_env(), - task_egress=task.environment.get("egress"), - with_agent=self.mode == "claude", runner=self.runner) + sandbox = self._open_sandbox(plan, container, log) try: - self._start_container(task, image, container, log, sandbox) + bridged = self._start_container(task, image, container, log, sandbox, plan) try: if self.mode == "oracle": - self._solve_oracle(task, container, log, timeout=solve_timeout) + self._solve_oracle(task, container, log, plan.solve, timeout=solve_timeout) elif self.mode == "claude": usage = self._solve_claude( task, container, output_dir, model, solve_timeout, @@ -235,12 +236,9 @@ def run( sandbox) else: raise ValueError(f"unknown mode {self.mode!r}") - if sandbox is not None: - sandbox.stop_proxy(self._docker, log) # the verifier gets no route out either - if _is_public(task): # "public" is the verifier's, after the agent exits - self._docker(["network", "connect", "bridge", container], log) if self.mode != "probe": # the probe's reward is its own verdict - reward = self._verify(task, container, output_dir, log) + reward = self._verify_phase(task, plan.verify, container, output_dir, log, + sandbox, bridged, network) finally: self._docker(["rm", "-f", container], log, check=False) finally: @@ -252,8 +250,7 @@ def run( returncode = 124 reward = 0.0 except Exception as exc: - # Nothing recorded yet (a build failure): let the runner mark it errored. - # After a solve (a failed copy back), keep its cost and isolation record. + # A build failure: the runner errors the cell. After a solve, keep its cost and isolation record. if not extra_metrics: raise log.write(f"\nERROR: {type(exc).__name__}: {exc}\n") @@ -262,9 +259,10 @@ def run( reward = None finally: log.close() + (output_dir / "network.json").write_text(json.dumps(network, indent=1), encoding="utf-8") elapsed = time.time() - start - metrics: dict[str, Any] = {"dry_run": False, "mode": self.mode} + metrics: dict[str, Any] = {"dry_run": False, "mode": self.mode, "network": network} passed = reward is not None and reward >= 1.0 if usage is not None: metrics.update(usage.as_metrics_dict()) @@ -291,8 +289,8 @@ def _docker(self, args: list[str], log, **kwargs: Any) -> process.Completed: # ── pipeline steps ────────────────────────────────────────────────────── - def _build_image(self, task: TaskSpec, image: str, log) -> None: - """Build the task image; pass host proxy through (corporate networks). + def _build_image(self, task: TaskSpec, image: str, log, net: Optional[PhasePolicy] = None) -> None: + """Build the task image under *net*: ``public`` passes the host proxy through. Task-bundled skills (``environment/skills``) are agent capabilities, not image content: a ``COPY . /x`` in the Dockerfile would otherwise bake @@ -301,11 +299,7 @@ def _build_image(self, task: TaskSpec, image: str, log) -> None: same rule BenchFlow applies to its no-skill runs. """ env_dir = task.path / "environment" - build_args: list[str] = ["--network", "host"] - for var in _PROXY_VARS: - val = os.environ.get(var) - if val: - build_args += ["--build-arg", f"{var}={val}"] + flags = build_args(net or DEFAULTS["build"]) # Honor the task's own build_timeout_sec (some images need >10 min); # _DEFAULT_BUILD_TIMEOUT is only the fallback when the task omits it. timeout = _task_build_timeout(task) @@ -323,20 +317,18 @@ def _build_image(self, task: TaskSpec, image: str, log) -> None: # a missing source fails the build in every arm. (context / "skills").mkdir() self._docker( - ["build", "-q", *build_args, "-t", image, + ["build", "-q", *flags, "-t", image, "-f", str(context / "Dockerfile"), str(context)], log, timeout=timeout, ) def _start_container(self, task: TaskSpec, image: str, container: str, log, - sandbox: Optional[ContainerSandbox] = None) -> None: - # --network none unless the task declares environment.network_mode = - # "public"; then bridge (NAT'd internet), not host, so host-local - # services stay out of reach. Build always gets host network. An - # in-container agent brings its own internal network + egress proxy, - # and a "public" task joins bridge only for its verifier. - net_args = (sandbox.run_args() if sandbox is not None - else ["--network", "bridge" if _is_public(task) else "none"]) + sandbox: Optional[ContainerSandbox] = None, + plan: Optional[NetworkPlan] = None) -> bool: + """Start the task container on the solve phase's network; returns whether it is on bridge.""" + plan = plan or self._plan(task) + inside = self._solve_in_container() + net_args = sandbox.run_args() if sandbox is not None else start_args(plan, solve_in_container=inside) cpus_raw = task.environment.get("cpus") cpus = str(cpus_raw) if cpus_raw is not None else "4" @@ -360,8 +352,10 @@ def _start_container(self, task: TaskSpec, image: str, container: str, log, "--cpus", cpus, "--memory", memory, image, "sleep", str(lifetime)], log, ) + return enter_phase(plan.solve if inside else DEFAULTS["solve"], container, self._docker, log, + bridged="bridge" in net_args) - def _solve_oracle(self, task: TaskSpec, container: str, log, + def _solve_oracle(self, task: TaskSpec, container: str, log, net: Optional[PhasePolicy] = None, timeout: float = _DEFAULT_SOLVE_TIMEOUT) -> None: """Apply the oracle's solve.sh as root (some images set a non-root USER), within the agent's ``timeout``: an oracle that runs out of time is a failed cell, like an agent. @@ -375,7 +369,7 @@ def _solve_oracle(self, task: TaskSpec, container: str, log, """ _copy_dir_into(task.solution_dir, container, ("/solution", "/oracle"), log, self._docker) self._docker( - ["exec", "-u", "root", *_exec_env(task, "oracle"), container, "bash", "-lc", + ["exec", "-u", "root", *_exec_env(task, "oracle", net), container, "bash", "-lc", "chmod +x /solution/solve.sh && /solution/solve.sh"], log, check=False, timeout=timeout, ) @@ -393,7 +387,7 @@ def _solve_claude( # A rerun must not find its own previous tree or verdict during the solve. _rmtree(output_dir / "work") (output_dir / "reward.txt").unlink(missing_ok=True) - if sandbox is not None: + if sandbox is not None and self.agent_location == "container": # a host arm stays on the host return self._solve_claude_in_container(task, container, output_dir, model, timeout, operations, log, extra_metrics, sandbox) workdir = _require_agent_workdir(task, container, log, self._docker) @@ -575,7 +569,7 @@ def _solve_probe(self, task: TaskSpec, container: str, output_dir: Path, log, untested = {k: "untested" for k in LEAK_CLASSES} # Where docker-claude is refused, its probe would vouch for an arm that never runs. workdir = _require_agent_workdir(task, container, log, self._docker) - if sandbox is not None: + if sandbox is not None and self.agent_location == "container": # not for a verify allowlist sandbox.workdir = workdir sandbox.prepare(container, self._docker, log) skill_dir = self._resolve_skill_dir() # docker-probe:: forbidden, never offered @@ -625,7 +619,8 @@ def _recompile(self, task: TaskSpec, container: str, log) -> list[str]: failed.append(cc) return failed - def _verify(self, task: TaskSpec, container: str, output_dir: Path, log) -> Optional[float]: + def _verify(self, task: TaskSpec, container: str, output_dir: Path, log, + net: Optional[PhasePolicy] = None) -> Optional[float]: """Run the verifier in-container; pull /logs/verifier/reward.txt back out. Mounted at both ``/tests`` (Harbor) and ``/verifier`` (BenchFlow): a @@ -637,7 +632,7 @@ def _verify(self, task: TaskSpec, container: str, output_dir: Path, log) -> Opti # proxy credentials then (the sidecar kept them from the agent; see :func:`_exec_env`). agent_code = self.mode == "claude" and self.agent_location == "container" proc = self._docker( - ["exec", "-u", "root", *_exec_env(task, "verifier", credentials=not agent_code), + ["exec", "-u", "root", *_exec_env(task, "verifier", net, credentials=not agent_code), container, "bash", "-lc", "mkdir -p /logs/verifier && chmod +x /tests/test.sh && /tests/test.sh"], log, check=False, timeout=_task_verify_timeout(task), @@ -662,6 +657,41 @@ def _verify(self, task: TaskSpec, container: str, output_dir: Path, log) -> Opti # No reward file written: fall back to verifier exit code. return 1.0 if proc.returncode == 0 else 0.0 + def _plan(self, task: TaskSpec, record: Optional[Path] = None) -> NetworkPlan: + return resolve_recorded(task, self.network, agent=self.mode != "oracle", record=record) + + def _solve_in_container(self) -> bool: + return self.mode == "oracle" or self.agent_location == "container" + + def _network_metrics(self, plan: NetworkPlan) -> dict[str, Any]: # a host agent's solve is not ours + skill = skill_needs(self._resolve_skill_dir(), plan) if self.mode == "claude" else None + return {**plan.as_metrics(), "solve_enforced": self._solve_in_container(), "skill": skill} + + def _open_sandbox(self, plan: NetworkPlan, container: str, log) -> Optional[ContainerSandbox]: + """The internal network and sidecar: always for an in-container agent, else when a phase needs one.""" + agent = self.mode != "oracle" and self.agent_location == "container" + if not (agent or needs_sandbox(plan, solve_in_container=self._solve_in_container())): + return None + return ContainerSandbox.open(container, self._docker, log, source_env=operator_source_env(), + task_egress=list(plan.solve.hosts), with_agent=agent and self.mode == "claude", + include_llm=agent, runner=self.runner) + + def _verify_phase(self, task: TaskSpec, net: PhasePolicy, container: str, output_dir: Path, log, + sandbox: Optional[ContainerSandbox], bridged: bool, network: dict[str, Any]) -> Optional[float]: + """Stop the solve's sidecar, give the container the verifier's network, verify.""" + side = None + if sandbox is not None: + sandbox.stop_proxy(self._docker, log) # nothing the solve was allowed carries over + enter_phase(net, container, self._docker, log, bridged=bridged) + if sandbox is not None and net.mode == "allowlist": + side = sandbox.verifier_sidecar(self._docker, log, list(net.hosts)) + try: + return self._verify(task, container, output_dir, log, net) + finally: + if side is not None: + side.stop_proxy(self._docker, log) + network["verify_egress"] = side.egress_summary() + # ── helpers ───────────────────────────────────────────────────────────── def _resolve_skill_dir(self) -> Optional[Path]: @@ -798,32 +828,26 @@ def _copy_dir_into(src: Path, container: str, dests: tuple[str, ...], log, docke docker(["cp", f"{src}/.", f"{container}:{dest}"], log) -def _exec_env(task: TaskSpec, table: str, *, credentials: bool = True) -> list[str]: +def _exec_env(task: TaskSpec, table: str, net: Optional[PhasePolicy] = None, *, + credentials: bool = True) -> list[str]: """``docker exec -e`` flags for the oracle or verifier step. - A public-network task gets the host's proxy variables: behind a corporate - proxy a verifier that installs ``uv`` or pytest at verify time (52 of 101 - SkillsBench ``test.sh`` install both, 90 install something) otherwise fails - with reward 0 (measured on ``parallel-tfidf-search``). They go on the exec, - not on ``docker run``, so they reach only the steps the task's - ``network_mode`` is about. With ``credentials=False`` (the verifier after - an in-container agent, which runs the agent's code) their ``user:pw@`` is - dropped: the egress sidecar holds the credentials so the agent never sees - them, and a planted ``conftest.py`` must not read them at verify time. + First the proxy variables of the step's network policy (:func:`.network_policy.proxy_env_flags`), + on the exec, not on ``docker run``, so they reach only that step: a public verifier behind + a corporate proxy that installs pytest at verify time (52 of 101 SkillsBench ``test.sh``) + otherwise fails with reward 0. With ``credentials=False`` (the verifier after an + in-container agent runs the agent's code) their ``user:pw@`` is dropped. Without *net*, + the task's own plan. Then the task's ``[verifier.env]`` / ``[oracle.env]``: only literal values are forwarded; BenchFlow's ``${VAR}`` host references are left to BenchFlow, so a secret is never read from our environment on a task file's say-so. """ - flags: list[str] = [] - if str(task.environment.get("network_mode") or "none") == "public": - for var in _PROXY_VARS: - val = os.environ.get(var) - if val: - if not credentials: - val = _USERINFO_RE.sub(r"\1", val) - flags += ["-e", f"{var}={val}"] + if net is None: + plan = resolve(task) + net = plan.verify if table == "verifier" else plan.solve + flags = proxy_env_flags(net, credentials=credentials) source = task.verifier if table == "verifier" else task.oracle env = source.get("env") if not isinstance(env, dict): @@ -857,30 +881,6 @@ def ignore(directory: str, names: list[str]) -> list[str]: return ignore -# userinfo after a scheme or right after "VAR=" (a scheme-less proxy); greedy, -# so a raw "@" or "=" in the password (base64 "==") is covered up to the last -# "@" before the host. -_CREDENTIALS_RE = re.compile(r"(://|=)[^/\s]+@") -# The same userinfo in a bare proxy value (no "VAR="), to drop it. -_USERINFO_RE = re.compile(r"(^|://)[^/\s]+@") - - -def _redact(arg: str) -> str: - """Hide ``user:password@`` in proxy URLs written to run.log. - - The pattern needs ``://`` or ``VAR=`` before the userinfo; a step that - echoes a scheme-less proxy on its own (``u:pw@proxy:912``) is caught by - the host's own proxy userinfo, matched literally. - """ - out = _CREDENTIALS_RE.sub(r"\1***@", arg) - for var in _PROXY_VARS: - found = _USERINFO_RE.search(os.environ.get(var, "")) - userinfo = found.group(0)[len(found.group(1)):-1] if found else "" - if ":" in userinfo: # a password: a bare user name is too short to match safely - out = out.replace(userinfo + "@", "***@") - return out - - def _describe_timeout(exc: subprocess.TimeoutExpired) -> str: """``TimeoutExpired`` for run.log: its ``str()`` holds the raw argv.""" cmd = exc.cmd if isinstance(exc.cmd, (list, tuple)) else [str(exc.cmd)] diff --git a/dokimi/dokimi/harnesses/network_policy.py b/dokimi/dokimi/harnesses/network_policy.py new file mode 100644 index 00000000..327be653 --- /dev/null +++ b/dokimi/dokimi/harnesses/network_policy.py @@ -0,0 +1,305 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The network a ``docker-*`` task run gets, per phase: build, solve, verify. + +Three modes, the same three Harbor's ``task.toml`` names (``network_mode`` = +``no-network`` / ``allowlist`` / ``public``, with ``allowed_hosts``): + +``none`` + No route out. Off unless something says otherwise: the solve (an + in-container agent still reaches its LLM through the egress sidecar, and + nothing else) and the verifier. +``allowlist`` + Only the listed ``host[:port]`` entries (port 443 when omitted), through the + :mod:`.egress_proxy` sidecar. Every request is logged; the verifier's + decisions are kept as ``metrics.network.verify_egress``. +``public`` + The default bridge (NAT'd internet, not the host network) plus the host's + proxy variables. The image build gets it by default: ``apt``/``pip`` in a + Dockerfile needs it, and nothing under test runs then. + +Where a phase's policy comes from, later wins: + +1. :data:`DEFAULTS`. +2. The task: ``[environment]`` ``network_mode`` / ``allowed_hosts`` (or Harbor's + deprecated ``allow_internet``) for the oracle and the verifier; + ``[agent]`` and ``[verifier]`` overrides for their phase; dokimi's own + ``environment.egress`` hosts for an agent's solve. An ``[environment]``-level + ``public`` is not extended to an agent: Harbor defaults to public and catalogs + copy it, so honouring it would hand every agent the internet. An agent gets + the internet only from an explicit ``[agent]`` policy or the operator. +3. The operator: ``dokimi tasks run --network PHASE=MODE[:host,...]``. + +The effective plan, with each phase's source, is recorded in every run's +``metrics.network``, so two runs can be checked for the same network before +their verdicts are compared. +""" + +from __future__ import annotations + +import json +import os +import re +from dataclasses import dataclass, replace +from pathlib import Path +from typing import Any, Callable, Iterable, Optional, Sequence + +from dokimi.utils import parse_frontmatter + +from .egress_proxy import normalize_entry + +PHASES = ("build", "solve", "verify") +MODES = ("none", "allowlist", "public") +_MODE_ALIASES = {"no-network": "none", "off": "none", "open": "public", "on": "public"} +_PHASE_ALIASES = {"setup": "build", "agent": "solve", "oracle": "solve", "verifier": "verify", + "all": "build,solve,verify"} +PROXY_VARS = ("http_proxy", "https_proxy", "no_proxy", "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY") +SIDECAR_URL = "http://egress:3128" # container_agent.PROXY_PORT on the internal network's alias +_KEEP_SCHEME = r"\1" + + +@dataclass(frozen=True) +class PhasePolicy: + mode: str + hosts: tuple[str, ...] = () + source: str = "default" + + def label(self) -> str: + return f"allowlist:{','.join(self.hosts)}" if self.mode == "allowlist" else self.mode + + def as_dict(self) -> dict[str, Any]: + return {"mode": self.mode, "hosts": list(self.hosts), "source": self.source} + + +DEFAULTS = {"build": PhasePolicy("public"), "solve": PhasePolicy("none"), "verify": PhasePolicy("none")} + + +@dataclass(frozen=True) +class NetworkPlan: + build: PhasePolicy + solve: PhasePolicy + verify: PhasePolicy + + def as_metrics(self) -> dict[str, Any]: + return {p: getattr(self, p).as_dict() for p in PHASES} + + def summary(self) -> str: + return " ".join(f"{p}={getattr(self, p).label()}" for p in PHASES) + + +def policy(mode: Any, hosts: Iterable[Any] = (), source: str = "default") -> PhasePolicy: + """A validated policy; an unknown mode, or hosts on a mode without a list, is a ``ValueError``.""" + name = _MODE_ALIASES.get(str(mode).strip().lower(), str(mode).strip().lower()) + if name not in MODES: + raise ValueError(f"network mode {mode!r} ({source}): expected one of {', '.join(MODES)}") + entries = tuple(sorted({normalize_entry(str(h)) for h in hosts})) + if entries and name != "allowlist": + raise ValueError(f"network hosts ({source}) are only valid with mode allowlist, not {name}") + return PhasePolicy(name, entries, source) + + +def parse_override(spec: str) -> dict[str, PhasePolicy]: + """``PHASE[,PHASE]=MODE[:host,...]`` (``all`` for every phase) -> one policy per phase.""" + phases_raw, sep, value = spec.partition("=") + if not sep or not value: + raise ValueError(f"--network {spec!r}: expected PHASE=MODE, e.g. verify=none or " + "solve=allowlist:pypi.org,files.pythonhosted.org") + mode, _, hosts = value.partition(":") + phases = [p.strip() for raw in phases_raw.split(",") + for p in _PHASE_ALIASES.get(raw.strip().lower(), raw.strip().lower()).split(",")] + unknown = [p for p in phases if p not in PHASES] + if unknown: + raise ValueError(f"--network {spec!r}: unknown phase {', '.join(unknown)}; " + f"expected {', '.join(PHASES)} or all") + pol = policy(mode, [h for h in hosts.split(",") if h.strip()], "cli") + if pol.mode == "allowlist" and not pol.hosts: + raise ValueError(f"--network {spec!r}: allowlist needs hosts, e.g. allowlist:pypi.org") + if pol.mode == "allowlist" and "build" in phases: + raise ValueError(f"--network {spec!r}: the image build takes none or public (docker build " + "cannot route through the egress sidecar)") + return {p: pol for p in phases} + + +def _declared(table: Any, where: str) -> Optional[PhasePolicy]: + """A table's ``network_mode`` (+ ``allowed_hosts``), or Harbor's ``allow_internet`` bool.""" + if not isinstance(table, dict): + return None + hosts = table.get("allowed_hosts") or [] + hosts = [hosts] if isinstance(hosts, str) else list(hosts) + if table.get("network_mode") is not None: + return policy(table["network_mode"], hosts, f"task [{where}] network_mode") + if isinstance(table.get("allow_internet"), bool): + return policy("public" if table["allow_internet"] else "none", (), f"task [{where}] allow_internet") + return None + + +def _with_hosts(base: PhasePolicy, hosts: Sequence[str], source: str) -> PhasePolicy: + """*base* widened by *hosts*: ``none`` becomes an allowlist, ``public`` stays public.""" + if not hosts or base.mode == "public": + return base + merged = policy("allowlist", [*base.hosts, *hosts], source) + return replace(merged, source=source if base.source == "default" else f"{base.source} + {source}") + + +def resolve(task: Any, overrides: Sequence[str] = (), *, agent: bool = False) -> NetworkPlan: + """The plan for one run of *task*; *agent* when an LLM agent (not the oracle) solves it.""" + plan = dict(DEFAULTS) + baseline = _declared(task.environment, "environment") + if baseline is not None: + plan["verify"] = baseline + if not agent: + plan["solve"] = baseline + elif baseline.mode == "allowlist": + plan["solve"] = baseline # explicit hosts are safe to extend; "public" is not + egress = task.environment.get("egress") + if agent and egress: + extra = [egress] if isinstance(egress, str) else [str(h) for h in egress] + plan["solve"] = _with_hosts(plan["solve"], extra, "task environment.egress") + for phase, table in (("solve", task.agent), ("verify", task.verifier)): + explicit = _declared(table, "agent" if phase == "solve" else "verifier") + if explicit is not None: + plan[phase] = explicit + for spec in overrides: + plan.update(parse_override(spec)) + return NetworkPlan(**plan) + + +def resolve_recorded(task: Any, overrides: Sequence[str], *, agent: bool, record: Optional[Path]) -> NetworkPlan: + """:func:`resolve`; a refused plan also leaves its reason in *record* ``/network.json``, the cell's record.""" + try: + return resolve(task, overrides, agent=agent) + except ValueError as exc: + if record is not None: + record.mkdir(parents=True, exist_ok=True) + (record / "network.json").write_text(json.dumps({"error": str(exc)}, indent=1), encoding="utf-8") + raise + + +def skill_needs(skill_dir: Optional[Path], plan: NetworkPlan) -> Optional[dict[str, Any]]: + """What a skill declares it needs at solve time, and what this run's plan leaves out. + + A skill declares hosts in its ``SKILL.md`` frontmatter as ``metadata.network`` + (``allowlist:host,...`` or ``public``; the Agent Skills spec's ``metadata`` is + a free string map, and its ``compatibility`` text is where the reason goes for + people). The plan is not widened for the skill arm alone: its baseline would + then run on a different network, and the delta would not be the skill's. The + operator widens every arm with ``--network solve=...``; ``missing`` says + which declared hosts the run did not grant. + """ + path = skill_dir / "SKILL.md" if skill_dir is not None else None + if path is None or not path.is_file(): + return None + meta, _ = parse_frontmatter(path.read_text(encoding="utf-8")) + declared = (meta.get("metadata") or {}).get("network") if isinstance(meta.get("metadata"), dict) else None + if not declared: + return None + mode, _, hosts = str(declared).partition(":") + try: + want = policy(mode, [h for h in hosts.split(",") if h.strip()], "skill metadata.network") + except ValueError as exc: # only recorded: a value we cannot read must not error the arm + return {"declared": str(declared), "missing": None, "error": str(exc)} + if plan.solve.mode == "public": + missing: list[str] = [] + elif want.mode == "public": + missing = ["*"] + else: + missing = [h for h in want.hosts if h not in plan.solve.hosts] + return {"declared": want.label(), "missing": missing} + + +# ── applying a plan ───────────────────────────────────────────────────────── + + +def proxy_env_flags(pol: PhasePolicy, *, credentials: bool = True) -> list[str]: + """``docker exec -e`` flags for a step under *pol*. + + ``public``: the host's proxy variables (behind a corporate proxy a verifier + that installs pytest at verify time fails without them), with their + ``user:pw@`` dropped when ``credentials`` is off (the verifier after an + in-container agent runs the agent's code). ``allowlist``: the sidecar, which + holds the upstream credentials itself. ``none``: nothing. + """ + if pol.mode == "allowlist": + return [a for v in PROXY_VARS for a in + ("-e", f"{v}={'localhost,127.0.0.1,::1' if v.lower() == 'no_proxy' else SIDECAR_URL}")] + if pol.mode != "public": + return [] + flags: list[str] = [] + for var in PROXY_VARS: + val = os.environ.get(var) + if val: + flags += ["-e", f"{var}={val if credentials else _USERINFO_RE.sub(_KEEP_SCHEME, val)}"] + return flags + + +def build_args(pol: PhasePolicy) -> list[str]: + """``docker build`` network flags: host network and proxy build-args, or none.""" + if pol.mode == "none": + return ["--network", "none"] + flags = ["--network", "host"] + for var in PROXY_VARS: + if os.environ.get(var): + flags += ["--build-arg", f"{var}={os.environ[var]}"] + return flags + + +def container_phases(plan: NetworkPlan, *, solve_in_container: bool) -> list[PhasePolicy]: + return ([plan.solve] if solve_in_container else []) + [plan.verify] + + +def needs_sandbox(plan: NetworkPlan, *, solve_in_container: bool) -> bool: + """An allowlist phase needs the per-run internal network, for its sidecar.""" + return any(p.mode == "allowlist" for p in container_phases(plan, solve_in_container=solve_in_container)) + + +def start_args(plan: NetworkPlan, *, solve_in_container: bool) -> list[str]: + """``docker run`` network without a sidecar: bridge when a phase is public, else none. + + A phase that is ``none`` then leaves the bridge (:func:`enter_phase`) before + anything but the container's ``sleep`` runs: no network at all, stricter than + an internal network, which still reaches services on the Docker host's gateway. + """ + public = any(p.mode == "public" for p in container_phases(plan, solve_in_container=solve_in_container)) + return ["--network", "bridge" if public else "none"] + + +Docker = Callable[..., Any] + + +def enter_phase(pol: PhasePolicy, container: str, docker: Docker, log, *, bridged: bool) -> bool: + """Join or leave the default bridge for *pol*; returns whether the container is on it now. + + The sidecar (``allowlist``) is the caller's: it must be started after this, + on the container's internal network. + """ + want = pol.mode == "public" + if want and not bridged: + docker(["network", "connect", "bridge", container], log) + elif bridged and not want: + docker(["network", "disconnect", "-f", "bridge", container], log) + return want + + +# userinfo after a scheme or right after "VAR=" (a scheme-less proxy); greedy, +# so a raw "@" or "=" in the password (base64 "==") is covered up to the last +# "@" before the host. +_CREDENTIALS_RE = re.compile(r"(://|=)[^/\s]+@") +# The same userinfo in a bare proxy value (no "VAR="), to drop it. +_USERINFO_RE = re.compile(r"(^|://)[^/\s]+@") + + +def redact(arg: str) -> str: + """Hide ``user:password@`` in proxy URLs written to run.log. + + The pattern needs ``://`` or ``VAR=`` before the userinfo; a step that + echoes a scheme-less proxy on its own (``u:pw@proxy:912``) is caught by + the host's own proxy userinfo, matched literally. + """ + out = _CREDENTIALS_RE.sub(r"\1***@", arg) + for var in PROXY_VARS: + found = _USERINFO_RE.search(os.environ.get(var, "")) + userinfo = found.group(0)[len(found.group(1)):-1] if found else "" + if ":" in userinfo: # a password: a bare user name is too short to match safely + out = out.replace(userinfo + "@", "***@") + return out diff --git a/dokimi/dokimi/harnesses/runner.py b/dokimi/dokimi/harnesses/runner.py index f6a039e7..8f805aa1 100644 --- a/dokimi/dokimi/harnesses/runner.py +++ b/dokimi/dokimi/harnesses/runner.py @@ -6,7 +6,7 @@ from __future__ import annotations import math -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -15,7 +15,9 @@ from dokimi.eval import binary_stats from dokimi.eval.stats import adjust_pvalues +from .docker_solver import DockerSolverHarness from .models import HarnessResult, OperationRecord, TaskRunResult +from .network_policy import parse_override from .registry import create_harness, find_with_skill_arm from .tasks import TaskSpec @@ -37,14 +39,25 @@ class TaskSuiteRunner: command_template: str | None = None dry_run: bool = False repeats: int = 1 + network: tuple[str, ...] = () # --network PHASE=MODE overrides, docker-* harnesses only # Starts the processes of every harness except the `docker-*` ones (docker_solver spawns its own). runner: process.Runner = field(default=process.run, repr=False) def validate_harnesses(self) -> None: """Build every harness id once; a bad id (e.g. a missing skill path) raises ``ValueError`` before any cell has run and been paid for.""" + for spec in self.network: + parse_override(spec) for harness_id in self.harnesses: - create_harness(harness_id, command_template=self.command_template, runner=self.runner) + harness = self._create(harness_id) + if self.network and not isinstance(harness, DockerSolverHarness): + raise ValueError(f"--network applies to docker-* harnesses only, not {harness_id}") + + def _create(self, harness_id: str) -> Any: + harness = create_harness(harness_id, command_template=self.command_template, runner=self.runner) + if self.network and isinstance(harness, DockerSolverHarness): + harness = replace(harness, network=tuple(self.network)) + return harness def run(self) -> dict[str, Any]: self.validate_harnesses() @@ -56,7 +69,7 @@ def run(self) -> dict[str, Any]: # say so once instead of recording N identical errored cells. checked: set[type] = set() for harness_id in self.harnesses: - harness = create_harness(harness_id, command_template=self.command_template, runner=self.runner) + harness = self._create(harness_id) preflight = getattr(harness, "preflight", None) if callable(preflight) and type(harness) not in checked: checked.add(type(harness)) @@ -64,7 +77,7 @@ def run(self) -> dict[str, Any]: for task in self.tasks: for harness_id in self.harnesses: for rep in range(1, repeats + 1): - harness = create_harness(harness_id, command_template=self.command_template, runner=self.runner) + harness = self._create(harness_id) output_dir = self.output_root / task.name / _safe_id(harness_id) if repeats > 1: output_dir = output_dir / f"run{rep}" diff --git a/dokimi/scripts/growth_baseline.json b/dokimi/scripts/growth_baseline.json index 674cad96..21fd4b51 100644 --- a/dokimi/scripts/growth_baseline.json +++ b/dokimi/scripts/growth_baseline.json @@ -47,7 +47,7 @@ "dokimi/harnesses/benchflow.py::BenchflowHarness.run": {"complexity": 19}, "dokimi/harnesses/benchflow.py::skill_loads": {"complexity": 12}, "dokimi/harnesses/command.py::CommandHarness.run": {"complexity": 15, "rationale": "pre-existing when the gate was introduced"}, - "dokimi/harnesses/docker_solver.py::DockerSolverHarness.run": {"complexity": 20}, + "dokimi/harnesses/docker_solver.py::DockerSolverHarness.run": {"complexity": 17}, "dokimi/harnesses/exploration.py::summarize_exploration": {"complexity": 15}, "dokimi/harnesses/registry.py::_create": {"complexity": 17, "rationale": "pre-existing when the gate was introduced"}, "dokimi/harnesses/runner.py::summarize_task_results": {"complexity": 14, "rationale": "pre-existing when the gate was introduced"}, diff --git a/dokimi/tests/test_docker_task_md.py b/dokimi/tests/test_docker_task_md.py index a13134de..2fcf4f0d 100644 --- a/dokimi/tests/test_docker_task_md.py +++ b/dokimi/tests/test_docker_task_md.py @@ -773,7 +773,8 @@ def test_probe_ignores_unresolved_env_it_never_uses(tmp_path, monkeypatch): calls = cli.args res = DockerSolverHarness(harness_id="docker-probe", mode="probe", runner=cli).run( task, output_dir=tmp_path / "out", dry_run=True) - assert res.metrics == {"dry_run": True, "mode": "probe"} and calls == [] + assert {k: v for k, v in res.metrics.items() if k != "network"} == {"dry_run": True, "mode": "probe"} + assert res.metrics["network"]["solve"]["mode"] == "none" and calls == [] # public is not the agent's claude = DockerSolverHarness(harness_id="docker-claude", mode="claude", runner=cli).run( task, output_dir=tmp_path / "out", dry_run=True) assert claude.metrics["unresolved_env"] == {"verifier": ["SECRET"]} diff --git a/dokimi/tests/test_network_policy.py b/dokimi/tests/test_network_policy.py new file mode 100644 index 00000000..a305fba4 --- /dev/null +++ b/dokimi/tests/test_network_policy.py @@ -0,0 +1,302 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""The per-phase network policy of docker-* runs: where it comes from, and what each phase gets. + +Docker is never run: :class:`DockerCLI` records the argv of every docker step, +so the tests read which network the container is on when the oracle and the +verifier execute. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import replace +from pathlib import Path + +import pytest + +from dokimi.harnesses import TaskSuiteRunner, load_task +from dokimi.harnesses.network_policy import parse_override, resolve +from dokimi.harnesses.registry import create_harness +from dokimi.process import reply +from docker_cli_fake import DockerCLI + + +@pytest.fixture(autouse=True) +def _hermetic(monkeypatch): + for name in list(os.environ): + if name.lower() in ("http_proxy", "https_proxy", "no_proxy", "all_proxy"): + monkeypatch.delenv(name) + + +def _task(tmp_path: Path, toml: str = "") -> Path: + task = tmp_path / "net-task" + for d in ("tests", "environment", "solution"): + (task / d).mkdir(parents=True, exist_ok=True) + (task / "instruction.md").write_text("x", encoding="utf-8") + (task / "task.toml").write_text("[metadata]\ntags = []\n" + toml, encoding="utf-8") + return task + + +def _modes(plan) -> dict[str, str]: + return {p: getattr(plan, p).label() for p in ("build", "solve", "verify")} + + +# ── resolution ────────────────────────────────────────────────────────────── + + +def test_a_task_that_says_nothing_gets_no_network_after_the_build(tmp_path): + plan = resolve(load_task(_task(tmp_path))) + assert _modes(plan) == {"build": "public", "solve": "none", "verify": "none"} + assert plan.verify.source == "default" + + +def test_environment_public_reaches_the_oracle_and_verifier_but_not_an_agent(tmp_path): + task = load_task(_task(tmp_path, '[environment]\nnetwork_mode = "public"\n')) + assert _modes(resolve(task)) == {"build": "public", "solve": "public", "verify": "public"} + agent = resolve(task, agent=True) + assert agent.solve.mode == "none" and agent.verify.mode == "public" + assert agent.verify.source == "task [environment] network_mode" + + +def test_an_environment_allowlist_reaches_an_agent_with_its_hosts(tmp_path): + toml = '[environment]\nnetwork_mode = "allowlist"\nallowed_hosts = ["pypi.org"]\n' + solve = resolve(load_task(_task(tmp_path, toml)), agent=True).solve + assert (solve.label(), solve.source) == ("allowlist:pypi.org:443", "task [environment] network_mode") + merged = resolve(load_task(_task(tmp_path / "e", toml + 'egress = ["huggingface.co"]\n')), agent=True).solve + assert merged.label() == "allowlist:huggingface.co:443,pypi.org:443" + +def test_harbor_phase_overrides_and_allowlists_are_honoured(tmp_path): + task = load_task(_task(tmp_path, ( + '[environment]\nnetwork_mode = "no-network"\n' + '[agent]\nnetwork_mode = "allowlist"\nallowed_hosts = ["pypi.org", "files.pythonhosted.org"]\n' + '[verifier]\nnetwork_mode = "public"\n'))) + plan = resolve(task, agent=True) + assert plan.solve.hosts == ("files.pythonhosted.org:443", "pypi.org:443") + assert plan.solve.source == "task [agent] network_mode" + assert plan.verify.mode == "public" + + +def test_harbor_allow_internet_false_is_none_and_true_is_public(tmp_path): + off = load_task(_task(tmp_path / "a", "[environment]\nallow_internet = false\n")) + on = load_task(_task(tmp_path / "b", "[environment]\nallow_internet = true\n")) + assert resolve(off).verify.mode == "none" + assert resolve(on).verify.label() == "public" + + +def test_dokimi_egress_widens_the_agent_solve_into_an_allowlist(tmp_path): + task = load_task(_task(tmp_path, '[environment]\negress = ["huggingface.co"]\n')) + plan = resolve(task, agent=True) + assert plan.solve.label() == "allowlist:huggingface.co:443" + assert resolve(task).solve.mode == "none" # the oracle is not the agent + + +def test_cli_overrides_win_over_the_task(tmp_path): + task = load_task(_task(tmp_path, '[environment]\nnetwork_mode = "public"\n')) + plan = resolve(task, ["verify=none", "solve=allowlist:httpbin.org:80"]) + assert _modes(plan) == {"build": "public", "solve": "allowlist:httpbin.org:80", "verify": "none"} + assert plan.verify.source == "cli" + assert _modes(resolve(task, ["all=none"])) == {"build": "none", "solve": "none", "verify": "none"} + + +@pytest.mark.parametrize("spec, why", [ + ("verify", "expected PHASE=MODE"), + ("grade=none", "unknown phase grade"), + ("verify=bridge", "expected one of none, allowlist, public"), + ("verify=allowlist", "allowlist needs hosts"), + ("build=allowlist:pypi.org", "the image build takes none or public"), + ("verify=public:pypi.org", "only valid with mode allowlist"), + ("verify=allowlist:https://pypi.org/simple", "expected host"), +]) +def test_bad_overrides_are_refused_with_the_reason(spec, why): + with pytest.raises(ValueError, match=why): + parse_override(spec) + + +def test_an_unknown_task_network_mode_refuses_the_cell(tmp_path): + task = load_task(_task(tmp_path, '[environment]\nnetwork_mode = "host"\n')) + with pytest.raises(ValueError, match="network mode 'host'"): + create_harness("docker-oracle").run(task, output_dir=tmp_path / "out", dry_run=True) + with pytest.raises(ValueError, match="network mode 'host'"): + create_harness("docker-oracle", runner=DockerCLI()).run(task, output_dir=tmp_path / "run") + assert "network mode 'host'" in json.loads((tmp_path / "run" / "network.json").read_text())["error"] + + +# ── what each phase gets ──────────────────────────────────────────────────── + + +def _oracle(tmp_path: Path, *overrides: str, toml: str = "") -> tuple[DockerCLI, dict]: + cli = DockerCLI(reward="1.0") + harness = create_harness("docker-oracle", runner=cli) + harness = replace(harness, network=overrides) + result = harness.run(load_task(_task(tmp_path, toml)), output_dir=tmp_path / "out") + assert result.reward == 1.0 + return cli, result.metrics + + +def _index(args: list[list[str]], pred) -> int: + return next(i for i, a in enumerate(args) if pred(a)) + + +def _exec_of(script: str): + return lambda a: a[0] == "exec" and script in a[-1] + + +def test_default_oracle_run_is_offline_from_start_to_verdict(tmp_path): + cli, metrics = _oracle(tmp_path) + run = next(a for a in cli.args if a[0] == "run") + assert run[run.index("--network") + 1] == "none" + assert not any(a[0] == "network" for a in cli.args) + build = next(a for a in cli.args if a[0] == "build") + assert build[build.index("--network") + 1] == "host" + assert metrics["network"]["verify"] == {"mode": "none", "hosts": [], "source": "default"} + + +def test_a_public_verifier_joins_bridge_only_after_an_offline_solve(tmp_path, monkeypatch): + monkeypatch.setenv("HTTPS_PROXY", "http://user:pw@proxy.corp:912") + cli, metrics = _oracle(tmp_path, "verify=public") + args = cli.args + run = next(a for a in args if a[0] == "run") + assert run[run.index("--network") + 1] == "bridge" + leave = _index(args, lambda a: a[:2] == ["network", "disconnect"]) + oracle = _index(args, _exec_of("solve.sh")) + join = _index(args, lambda a: a[:2] == ["network", "connect"]) + verify = _index(args, _exec_of("test.sh")) + assert leave < oracle < join < verify + assert "HTTPS_PROXY=http://user:pw@proxy.corp:912" in args[verify] + assert not any("PROXY" in x for x in args[oracle]) + assert metrics["network"]["verify"]["source"] == "cli" + + +def test_an_allowlisted_verifier_goes_through_its_own_sidecar(tmp_path): + cli, metrics = _oracle(tmp_path, "verify=allowlist:pypi.org,files.pythonhosted.org") + args = cli.args + assert any(a[:3] == ["network", "create", "--internal"] for a in args) + side = next(a for a in args if a[0] == "create" and a[a.index("--name") + 1].endswith("-verify")) + assert side[side.index("--allow") + 1] == "files.pythonhosted.org:443,pypi.org:443" + verify = args[_index(args, _exec_of("test.sh"))] + assert "HTTPS_PROXY=http://egress:3128" in verify + oracle = args[_index(args, _exec_of("solve.sh"))] + assert not any("PROXY" in x for x in oracle) + assert metrics["network"]["verify_egress"]["status"] == "ok" + saved = json.loads((tmp_path / "out" / "network.json").read_text(encoding="utf-8")) + assert saved["verify_egress"] == metrics["network"]["verify_egress"] # the cell record has it too + after = args[_index(args, _exec_of("test.sh")):] + assert ["rm", "-f", side[side.index("--name") + 1]] in after # stopped once the verifier is done + + +def test_an_allowlisted_oracle_solves_through_the_sidecar(tmp_path): + cli, metrics = _oracle(tmp_path, "solve=allowlist:pypi.org") + args = cli.args + side = next(a for a in args if a[0] == "create" and "--allow" in a) + assert side[side.index("--allow") + 1] == "pypi.org:443" + assert "HTTPS_PROXY=http://egress:3128" in args[_index(args, _exec_of("solve.sh"))] + assert metrics["network"]["solve_enforced"] is True + + +def test_a_public_oracle_solve_gets_the_host_proxy(tmp_path, monkeypatch): + monkeypatch.setenv("HTTPS_PROXY", "http://user:pw@proxy.corp:912") + cli, _ = _oracle(tmp_path, "solve=public") + assert "HTTPS_PROXY=http://user:pw@proxy.corp:912" in cli.args[_index(cli.args, _exec_of("solve.sh"))] + + +def test_a_verify_sidecar_that_never_gets_ready_is_removed(tmp_path, monkeypatch): + monkeypatch.setattr("dokimi.harnesses.container_agent.time.sleep", lambda _s: None) + + def never_ready(call): + return reply("") if call.argv[1] == "logs" and call.argv[-1].endswith("-verify") else None + + cli = DockerCLI(answer=never_ready) + harness = replace(create_harness("docker-oracle", runner=cli), network=("verify=allowlist:pypi.org",)) + result = harness.run(load_task(_task(tmp_path)), output_dir=tmp_path / "out") + assert "-verify did not start" in result.metrics["error"] + started = _index(cli.args, lambda a: a[0] == "start" and a[-1].endswith("-verify")) + assert any(a[:2] == ["rm", "-f"] and a[-1].endswith("-verify") for a in cli.args[started:]) + + +def test_build_none_builds_offline(tmp_path): + cli, _ = _oracle(tmp_path, "build=none") + build = next(a for a in cli.args if a[0] == "build") + assert build[build.index("--network") + 1] == "none" and "--build-arg" not in build + + +def test_a_task_public_oracle_keeps_its_network_unless_the_operator_closes_it(tmp_path): + toml = '[environment]\nnetwork_mode = "public"\n' + cli, _ = _oracle(tmp_path / "a", toml=toml) + assert not any(a[:2] == ["network", "disconnect"] for a in cli.args) + cli, metrics = _oracle(tmp_path / "b", "all=none", toml=toml) + run = next(a for a in cli.args if a[0] == "run") + assert run[run.index("--network") + 1] == "none" + assert metrics["network"]["verify"]["source"] == "cli" + + +@pytest.mark.parametrize("arm", ["docker-claude-host", "docker-probe-host"]) +def test_a_verify_allowlist_leaves_a_host_arm_on_the_host(tmp_path, arm): + toml = '[verifier]\nnetwork_mode = "allowlist"\nallowed_hosts = ["pypi.org"]\n' + cli = DockerCLI(reward="1.0", solver='{"type": "result", "result": "ok"}') + result = create_harness(arm, runner=cli).run(load_task(_task(tmp_path, toml)), output_dir=tmp_path / "out") + assert result.metrics["isolation"]["agent_location"] == "host" + assert result.metrics["network"]["solve_enforced"] is False # the host agent's solve is not ours + assert not any(a[0] == "exec" and "-c" in a and "--env-file" in a for a in cli.args) # no probe inside + assert all(c.argv[0] != "docker" for c in cli.solver_calls()) + +def test_the_suite_runner_refuses_network_for_a_harness_it_cannot_apply_to(tmp_path): + task = load_task(_task(tmp_path)) + with pytest.raises(ValueError, match="docker-\\* harnesses only"): + TaskSuiteRunner(tasks=[task], harnesses=["codex"], network=("verify=none",)).validate_harnesses() + with pytest.raises(ValueError, match="unknown phase"): + TaskSuiteRunner(tasks=[task], harnesses=["docker-oracle"], network=("x=none",)).validate_harnesses() + runner = TaskSuiteRunner(tasks=[task], harnesses=["docker-oracle"], network=("verify=public",), + output_root=tmp_path / "r", dry_run=True) + row = runner.run()["results"][0] + assert row["metrics"]["network"]["verify"]["mode"] == "public" + + +def _skill(tmp_path: Path, network: str) -> Path: + skill = tmp_path / "docs-fetch" + skill.mkdir(parents=True) + (skill / "SKILL.md").write_text( + "---\nname: docs-fetch\ndescription: Fetch the upstream docs.\n" + "compatibility: needs network access to pypi.org\n" + f"metadata:\n network: \"{network}\"\n---\nbody\n", encoding="utf-8") + return skill + + +def test_a_skill_declares_its_hosts_and_the_run_says_which_it_did_not_grant(tmp_path): + from dokimi.harnesses.network_policy import skill_needs + + task = load_task(_task(tmp_path)) + skill = _skill(tmp_path, "allowlist:pypi.org,files.pythonhosted.org") + closed = skill_needs(skill, resolve(task, agent=True)) + assert closed == {"declared": "allowlist:files.pythonhosted.org:443,pypi.org:443", + "missing": ["files.pythonhosted.org:443", "pypi.org:443"]} + granted = skill_needs(skill, resolve(task, ["solve=allowlist:pypi.org,files.pythonhosted.org"], agent=True)) + assert granted is not None and granted["missing"] == [] + assert skill_needs(_skill(tmp_path / "p", "public"), resolve(task, agent=True))["missing"] == ["*"] + harness = create_harness(f"docker-claude-skill:{skill}") + metrics = harness.run(task, output_dir=tmp_path / "out", dry_run=True).metrics + assert metrics["network"]["skill"]["missing"] == closed["missing"] # recorded, not widened + assert metrics["network"]["solve"]["mode"] == "none" + + +@pytest.mark.parametrize("value", ["pypi.org", "true", "allowlist:bad host!"]) +def test_an_unreadable_skill_network_is_recorded_and_the_arm_still_runs(tmp_path, value): + skill = _skill(tmp_path, value) + harness = create_harness(f"docker-claude-skill:{skill}") + metrics = harness.run(load_task(_task(tmp_path)), output_dir=tmp_path / "out", dry_run=True).metrics + got = metrics["network"]["skill"] + assert got["declared"] == value and got["missing"] is None and got["error"] + + +def test_an_errored_cell_still_records_its_network(tmp_path, monkeypatch): + harness = replace(create_harness("docker-oracle", runner=DockerCLI()), network=("verify=public",)) + + def broken(*_a, **_k): + raise RuntimeError("docker build failed (exit 1); see run.log") + + monkeypatch.setattr(type(harness), "_build_image", broken) + with pytest.raises(RuntimeError, match="docker build failed"): + harness.run(load_task(_task(tmp_path)), output_dir=tmp_path / "out") + assert json.loads((tmp_path / "out" / "network.json").read_text())["verify"]["mode"] == "public"