fix(installer): harden the Windows install→first-experiment path — checksum-driven downloads, reliable cluster-create, writable ingest & training volumes, local-chart support (#611)

[shujaatTracebloc]

What & why

Hardens the Windows install → first-dataset-ingest → first-experiment path end to end, driven by real field failures on a Windows machine behind a filtering corporate/AV network. Follow-on to the resilient multi-transport download landed in #607/#608. Each fix below is a place a real user got stuck; together they take the flow from "dies at step 2" to "installs, provisions, ingests, and runs an experiment" on a hostPath (Windows/WSL2) install.

Closes #611. Contributes to #578.

Fixes

1. Checksum-driven tool download (k3d / kubectl / helm) — install-k8s.ps1

#607's multi-transport download validated by size + magic bytes only, so a binary truncated mid-transfer to between the size floor and the real size still passed and dead-ended at a no-retry checksum step. Now the checksum is the completeness test:

• Get-VerifiedDownload -Sha256 — after a transport lands a size/magic-valid file, its SHA-256 must match or the next transport (curl.exe → BITS) is tried. A truncated/altered copy self-heals.
• The checksum-list files (checksums.txt, *.sha256) are fetched first and gated on a hash-anchored -MatchPattern (a 64-hex hash adjacent to the asset / at the start of the body), so a proxy error page echoing the URL can't fail the gate open (Bugbot).
• helm gains checksum verification on the PS path for the first time (parity with bash).
• Field symptom fixed: System tool checksum verification failed on a correct-but-truncated k3d download.

2. Reliable cluster-create — install-k8s.ps1

Wait-ProcessWithDeadline polled HasExited but never called WaitForExit(), so with Start-Process -RedirectStandardOutput the k3d process's ExitCode came back null and $null -ne 0 misread a successful create as a failure. Now it calls WaitForExit() before returning (reliable exit code for every caller), plus a defense-in-depth fallback that trusts k3d's created successfully marker — checking both stdout and stderr (k3d logs success via logrus to stderr) (Bugbot).

• Field symptom fixed: Failed to create compute environment even though the cluster was up.

3. Writable hostPath volumes for ingest and training/inference — chart (jobs-manager-deployment.yaml)

Two permission failures, same root cause. On hostPath installs kubelet doesn't apply fsGroup (kubernetes/kubernetes#138411), so a hostPath dir is created root-owned and the non-root pods that mount it can't write:

• Ingest: tb data ingest failed at the copy step — mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied.
• Training / inference: a spawned pod failed with PermissionError [Errno 13]: '/data/logs/<run>' while creating its per-run log dir (os.makedirs).

mysql-data already had a privileged init-chown; the shared and logs volumes had none. Now, on hostPath only, jobs-manager runs a privileged init-writable-data (root, caps CHOWN+FOWNER+FSETID) that makes both /data/shared and /data/logs chmod 3777 — world-writable (multiple non-root writers whose UIDs the chart doesn't control: jobs-manager, the spawned training/inference pods, the CLI ingest-staging pod) + setgid (GID-1000 inheritance; FSETID is required or the kernel strips setgid after the chown) + sticky (one writer can't unlink another's files). The spawned pods mount the same hostPath dirs, so fixing them once reaches them too. Training and inference share one pod spec (client-runtime job.yaml), so /data/logs covers both; every other write path there is an emptyDir or read-only. Each dir is fixed independently and best-effort, so a chown that can't complete (NFS root_squash) doesn't wedge the edge. No fsGroup is set — it's a no-op on hostPath and would strip the spawned pods' group-0 access on CSI; CSI is untouched (matches develop) and its pods keep their documented posture (docs/SECURITY.md §5.3).

4. Windows local-chart support (TRACEBLOC_CHART_PATH) — install-k8s.ps1

The Windows installer could only install the published chart, so branch-only chart fixes (like #3 above, or #585's global.imageRegistry) were untestable from a Windows install. Added TRACEBLOC_CHART_PATH — parity with the bash installer's _resolve_chart_ref — so a local chart directory can be installed (skipping helm repo add); the published repo remains the default.

Tests

• Pester source-guards + unit tests for every installer item above (checksum-driven retry, hash-anchored gates, WaitForExit, both-stream fallback, local-chart ref). Full suite green (451).
• helm-unittest for the writable-volume init: present on hostPath / absent on CSI, no fsGroup on either path, and asserts the init has caps CHOWN+FOWNER+FSETID, runs the per-dir best-effort loop with chmod 3777, and mounts both /data/shared and /data/logs (mounting both is required or the chown is a no-op). Chart bumped 1.9.15 → 1.9.20 across the chart-affecting fixes (chart-content guard).
• Windows-facing scripts are manifested; manifest.sha256 regenerated. bats/macOS suites unaffected (Linux/macOS paths untouched except the already-merged Installer tool acquisition must survive a proxy/AV-truncated binary download (k3d/kubectl/helm) #607 shared-bash bits, which a full cross-OS audit confirmed safe).

Scope note

This PR spans the installer scripts and the chart because they're the two halves of the same Windows install→ingest→experiment flow being validated together (kept in one PR by request).

The chart change is confined to the hostPath path: the privileged init-writable-data container is gated to hostPath and sets no fsGroup, so managed (EKS/AKS/OC) installs are genuinely untouched — CSI matches develop, and its spawned pods keep their documented posture (docs/SECURITY.md §5.3). (An earlier revision set fsGroup: 1000 on jobs-manager; @saadqbal and Bugbot correctly flagged that it never reaches the spawned writers and its relabel would strip their group-0 access on CSI, so it was removed.)

CI note

If pii-gate / pii-check is red, it is failing closed because the org PII_DENYLIST secret is unavailable — an org-wide infra issue affecting every PR, not this PR's content (tracked separately; the pii-gate-override label doesn't currently bypass it — tracebloc/.github#165). All other checks and Cursor Bugbot are green.

🤖 Generated with Claude Code

[saadqbal]

Careful, well-tested PR — the checksum-as-completeness-test and the both-streams k3d exit fallback are the right calls. A few non-blocking things to weigh, mostly on the chart side (left inline).

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 15e0f37. Configure here.}