fix(ingest): create dataset dirs group-writable/setgid so teardown can reclaim them (client-runtime#172)

[divyasinghds]

What

Make the ingestor write its destination tree group-writable + setgid so the data delete teardown pod can reclaim datasets on hostPath installs. Companion (part 2 of 2) to tracebloc/client-runtime#172.

Why

On a hostPath dataset the kubelet ignores the teardown pod's fsGroup (it only applies to CSI/ownership-managing volumes), so files the ingest Job writes stay owned by the writing uid/group and data delete (uid/gid 65532) can't remove them — they leak (remainder of client#259). Group deletion needs the containing directories to be group-owned (65532) and group-writable.

Change

• _ensure_dest_dir — centralises the four os.makedirs(cfg.DEST_PATH) sites and sets the created dir setgid + group-writable (0o2775) so new entries inherit the group and the group can delete them. Only a dir we create is chmod'd — a pre-existing dir is left untouched, so we don't override an operator's mode or mask a genuine permission error (an unwritable dest still surfaces downstream, as its test asserts).
• cli/run.main: os.umask(0o002) — every directory/file the run writes under DEST_PATH is group-writable, not just the DEST_PATH root (deletion needs write on the containing dir, so the whole tree must be group-writable).

Pairs with

tracebloc/client-runtime#172 runs the ingest Job with gid 65532 (+ supplemental group), so created files are group-owned by the teardown group. This PR makes the dirs group-writable so that group can actually delete. Both are needed.

Tests

• _ensure_dest_dir: new dir is setgid + group-writable; a pre-existing dir's mode is left untouched.
• The pre-existing "unwritable dest raises" test stays green (pre-existing dir not re-chmod'd).
• 66 file-transfer/config tests pass locally.

Refs tracebloc/client-runtime#172

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes default filesystem permissions for all ingest writes under DEST_PATH; misconfiguration could affect teardown or operator-owned dirs, though existing dirs are left untouched and chmod is best-effort.

Overview
Fixes hostPath datasets where data delete (uid/gid 65532) could not reclaim ingest output because kubelet ignores teardown fsGroup on hostPath.

file_transfer adds _ensure_dest_dir and uses it everywhere DEST_PATH was created via plain makedirs. Newly created destination directories get setgid + group-writable (0o2775) so the teardown group can delete the tree; pre-existing dirs are not re-chmod’d, and chmod failures are logged only.

cli/run.main sets os.umask(0o002) at startup so files and nested dirs written under DEST_PATH stay group-writable for the same teardown story.

Package version bumps to 0.8.4; tests cover new-dir permissions and unchanged modes on existing dirs.

^{Reviewed by Cursor Bugbot for commit 6f19f7f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[shujaatTracebloc]

LGTM ✅

Reviewed the logic and it's sound: _ensure_dest_dir centralizes the four os.makedirs(DEST_PATH) sites and only chmods a directory it creates (setgid + group-writable 0o2775), leaving pre-existing dirs untouched — so it neither overrides an operator's mode nor masks a genuine permission error (the "unwritable dest raises" test stays green). The os.umask(0o002) in run.main extends group-writability to the whole tree, which is what the hostPath teardown (gid 65532, fsGroup ignored) actually needs to reclaim files. Pairs correctly with client-runtime#172. Tests cover both the fresh-dir and pre-existing-dir cases. Bugbot clean.

I pushed two commits to unblock CI (no logic change):

• Bumped __version__ 0.8.3 → 0.8.4 so the change reaches a released ingestor image (image builds only on a vX.Y.Z tag).
• Reformatted the touched files with the CI-pinned black 26.3.1 to satisfy quality/format.

Note: pii-gate / pii-check is red because it's failing closed on a missing org secret (PII_DENYLIST empty/unavailable, backend#1409) — not a content issue and not this PR's fault. It blocks every PR in the repo until an org admin sets the secret (or a pii-gate-override label is applied). That needs to be cleared before merge.