Repository navigation
fix(backup): refuse to copy another stack's sets off-host - #1328
Merged
Merged
Conversation
backups/volumes/ is one directory for every stack on a host; only the far side is per STACK. On 2026-11-08 a hand-run `make backup ARGS=--copy-only` on trinity without STACK=sensitive copied the sensitive sets into oracle's observability directory, and the far-side prune deleted five real observability sets. They were restored the same morning. copy_offhost now refuses, before any ssh, every set whose MANIFEST names a stack other than STACK, so the prune after it never runs. --self-test plants a foreign set and checks the refusal; it fails with the guard off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The shared copy_offhost stack check blocks all legitimate NAS backups because the caller defaults to observability while NAS manifests declare media.
1 open issue
What changed in this PR
This PR adds a stack-identity check to off-host volume backups to prevent sets from being copied into another stack’s destination and triggering destructive retention.
Changes:
- Reject mismatched or missing MANIFEST stack identities before remote access.
- Add a regression fixture for foreign-stack refusal.
- Record the backup incident, recovery, and safeguard in the changelog.
| File | Description |
|---|---|
scripts/backup-volumes.sh |
Adds the pre-copy stack check and regression assertions. |
docs/changelog.md |
Records the misplaced backups, recovery, and new guard. |
Review effort: Balanced
backup-nas.sh sources backup-volumes.sh with STACK at its observability default, but every NAS MANIFEST names `stack media`, so the new check in copy_offhost refused every NAS set. The check now compares against SET_STACK: STACK on a direct run (the environment is not read), and what a sourcing caller sets otherwise. backup-nas.sh sets SET_STACK=media. The self-test now copies backup-nas.sh into its fixture repo and checks that a media set passes the check to the far side, and that SET_STACK in the environment does not loosen a direct run. With SET_STACK=media removed, the NAS fixture fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed
backup-volumes.shnow refuses to copy a set off-host when its MANIFEST names a stack other than the one it copies for. The refusal happens before any ssh, so the far-side prune never runs. That stack isSET_STACK:STACKon a direct run, andmediaforbackup-nas.sh, which sharescopy_offhostand writesstack media.--self-testgains fixtures for both, anddocs/changelog.mdrecords what happened.Why
On 2026-11-08 a hand-run
make backup ARGS=--copy-onlyontrinitywas missingSTACK=sensitive, so it ran asobservability.backups/volumes/is shared by every stack on a host; only the far side is per stack. The run copiedtrinity's seven sensitive sets intooracle'sbackups/volumes/observability, and retention there deleted the five real observability sets. They were restored the same morning fromprometheus(changelog, 2026-11-08). Nothing in the script stopped it.Blast radius
copy_offhoston every caller:backup-volumes.shontrinity(sensitive) andprometheus(observability),backup-library.sh(setsSTACK=sensitive, writessensitive), andbackup-nas.sh(now setsSET_STACK=media, writesmedia).backup-wiki.sh,backup-offsite.shandcarry-household-copy.shsource the file but never callcopy_offhost. If the check were wrong, it would refuse a legitimate copy. The job would fail loudly, and the existing copy-failure alerts would fire. It cannot delete anything. Every retained set on both hosts names its stack, and the field has existed since the script's first commit.secrets/*.sops.yamlVerification
scripts/backup-volumes.sh --self-test: 20/20 PASS. The new fixtures check that a foreign set is refused before ssh, thatSET_STACKin the environment doesn't loosen a direct run, and thatbackup-nas.sh'smediasets pass the check and reach the (.invalid) far side.With the call to
refuse_foreign_setsdisabled, the refusal assertions FAIL. WithSET_STACK=mediaremoved frombackup-nas.sh, the NAS assertions FAIL: the NAS set is refused. Copilot caught that on the first revision.make validateon the first revision: all checks passed, 10 skipped. Five were host-specific (not the deployment checkout: unit syntax, installed timers, stack timers), and five were lint tools not installed here (ruff, ansible-lint). On the second revision, only the self-test, lint andcheck_docs.pywere re-run locally; CI runs the rest.python3 scripts/check_docs.py: OK../scripts/lint.sh: passed (shellcheck PASS), with yamllint, ruff and ansible-lint skipped as not installed.make validatepassesDeployed to the lab and confirmed working (converge picks it up on
trinity;prometheuson its next converge)Docs updated (
docs/changelog.md)🤖 Generated with Claude Code