Skip to content

fix(backup): refuse to copy another stack's sets off-host - #1328

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/backup-copy-stack-check
Nov 8, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/backup-copy-stack-check

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

What changed

backup-volumes.sh now refuses to copy a set off-host when its MANIFEST names a stack other than the one it copies for. The refusal happens before any ssh, so the far-side prune never runs. That stack is SET_STACK: STACK on a direct run, and media for backup-nas.sh, which shares copy_offhost and writes stack media. --self-test gains fixtures for both, and docs/changelog.md records what happened.

Why

On 2026-11-08 a hand-run make backup ARGS=--copy-only on trinity was missing STACK=sensitive, so it ran as observability. backups/volumes/ is shared by every stack on a host; only the far side is per stack. The run copied trinity's seven sensitive sets into oracle's backups/volumes/observability, and retention there deleted the five real observability sets. They were restored the same morning from prometheus (changelog, 2026-11-08). Nothing in the script stopped it.

Blast radius

copy_offhost on every caller: backup-volumes.sh on trinity (sensitive) and prometheus (observability), backup-library.sh (sets STACK=sensitive, writes sensitive), and backup-nas.sh (now sets SET_STACK=media, writes media). backup-wiki.sh, backup-offsite.sh and carry-household-copy.sh source the file but never call copy_offhost. If the check were wrong, it would refuse a legitimate copy. The job would fail loudly, and the existing copy-failure alerts would fire. It cannot delete anything. Every retained set on both hosts names its stack, and the field has existed since the script's first commit.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • scripts/backup-volumes.sh --self-test: 20/20 PASS. The new fixtures check that a foreign set is refused before ssh, that SET_STACK in the environment doesn't loosen a direct run, and that backup-nas.sh's media sets pass the check and reach the (.invalid) far side.

  • With the call to refuse_foreign_sets disabled, the refusal assertions FAIL. With SET_STACK=media removed from backup-nas.sh, the NAS assertions FAIL: the NAS set is refused. Copilot caught that on the first revision.

  • make validate on the first revision: all checks passed, 10 skipped. Five were host-specific (not the deployment checkout: unit syntax, installed timers, stack timers), and five were lint tools not installed here (ruff, ansible-lint). On the second revision, only the self-test, lint and check_docs.py were re-run locally; CI runs the rest.

  • python3 scripts/check_docs.py: OK. ./scripts/lint.sh: passed (shellcheck PASS), with yamllint, ruff and ansible-lint skipped as not installed.

  • make validate passes

  • Deployed to the lab and confirmed working (converge picks it up on trinity; prometheus on its next converge)

  • Docs updated (docs/changelog.md)

🤖 Generated with Claude Code

backups/volumes/ is one directory for every stack on a host; only the far
side is per STACK. On 2026-11-08 a hand-run `make backup ARGS=--copy-only`
on trinity without STACK=sensitive copied the sensitive sets into oracle's
observability directory, and the far-side prune deleted five real
observability sets. They were restored the same morning.

copy_offhost now refuses, before any ssh, every set whose MANIFEST names a
stack other than STACK, so the prune after it never runs. --self-test
plants a foreign set and checks the refusal; it fails with the guard off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 8, 2026 22:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shared copy_offhost stack check blocks all legitimate NAS backups because the caller defaults to observability while NAS manifests declare media.

1 open issue
What changed in this PR

This PR adds a stack-identity check to off-host volume backups to prevent sets from being copied into another stack’s destination and triggering destructive retention.

Changes:

  • Reject mismatched or missing MANIFEST stack identities before remote access.
  • Add a regression fixture for foreign-stack refusal.
  • Record the backup incident, recovery, and safeguard in the changelog.
File Description
scripts/​backup-volumes.sh Adds the pre-copy stack check and regression assertions.
docs/​changelog.md Records the misplaced backups, recovery, and new guard.

Review effort: Balanced

Comment thread scripts/backup-volumes.sh Resolved
backup-nas.sh sources backup-volumes.sh with STACK at its observability
default, but every NAS MANIFEST names `stack media`, so the new check in
copy_offhost refused every NAS set. The check now compares against
SET_STACK: STACK on a direct run (the environment is not read), and what
a sourcing caller sets otherwise. backup-nas.sh sets SET_STACK=media.

The self-test now copies backup-nas.sh into its fixture repo and checks
that a media set passes the check to the far side, and that SET_STACK
in the environment does not loosen a direct run. With SET_STACK=media
removed, the NAS fixture fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt enabled auto-merge (squash) November 8, 2026 22:40
@Gerrrt
Gerrrt merged commit 4ca18d4 into main Nov 8, 2026
9 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/backup-copy-stack-check branch November 8, 2026 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants