Repository navigation
docs(ups): re-read the shutdown runbook against the estate, and subscribe trinity (#574) - #1330
Merged
Merged
Conversation
…ribe trinity (#574) Three things moved after steps 0 to 4 were built on 2026-09-23, and step 5 would have proved the wrong sequence: - the rack move put trinity, gandalf, the console and the KVM on the PDU; - Saruman runs thirteen guests, four in ordered stages, so a hung guest per stage could take about 15 minutes against the card's 8; - the UPS's outlet groups were found, with their shedding settings unread. ADR-0107 fills the trinity row ADR-0049 left open: a third NUT secondary on 10.0.99.1:3493 behind a pass/block pair on igc0.99. The four ordered guests get down=90, and 6.5 sizes the final low-battery duration from the remaining worst case (about 9.5 minutes) as well as the measured run. The runbook gains step 4b, the 3.6 timeout change, an outlet-group read under Before you start, trinity in step 5's loops, order and power-on, and the step 7 rows that follow. Nothing was changed on a host. Refs #574 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved battery-threshold timing, runtime-based shedding, and rebuilds that discard down=90 can cut power before guest shutdown completes.
5 open issues
Rebuild paths omit the 90-second guest shutdown timeout · New Runtime shedding can leave too little time for shutdown · New Low-battery threshold is too short for the revised shutdown bound · New Winterfell listener is enabled before its firewall rules are installed · New HOSTSYNC cannot prevent early monitor logout during guest teardown · New
What changed in this PR
This documentation-only PR updates the UPS shutdown plan for the post-rack-move estate and the remaining #574 maintenance window. No host configuration is changed.
Changes:
- Adds ADR-0107 to make
trinitya third NUT secondary and capSaruman’s ordered guest shutdown timeouts. - Expands the runbook with firewall rules, outlet-shedding checks, timing verification, recovery steps, and worst-case battery budgeting.
- Updates supporting documentation and the README’s ADR count.
| File | Description |
|---|---|
README.md |
Increases the ADR count to 107. |
docs/security.md |
Adds the planned trinity shutdown policy. |
docs/runbooks/shut-down-on-the-ups.md |
Updates configuration, shutdown proof, recovery, and battery-budget procedures. |
docs/roadmap.md |
Expands the remaining UPS maintenance-window gate. |
docs/hardware.md |
Records the pending subscription and outlet-shedding checks. |
docs/changelog.md |
Records the runbook revision and new decision. |
docs/adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md |
Defines the third secondary, guest timeout caps, and battery-budget policy. |
docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md |
Adds a pointer to ADR-0107 without rewriting the original decision. |
Review effort: Balanced
…e card before the window (#574) Review on #1330 found five gaps: - the lab-domain rebuild paths (build-the-lab-domain.md and tofu's startup_orders output) set startup without down=90, so a rebuild undid it; - the shedding check allowed a runtime shed below the low-battery duration, which cuts the PDU mid-halt. Group 1 now sheds on nothing; - the 9.5-minute bound exceeded the 8 minutes left on the card until 6.5. New 3.8 raises it to 16 minutes before the window; - 1.4 added the Winterfell listener, so a fresh build opened 3493 to all of VLAN 99 before 4b.1's rules. It is now added only in 4b.2; - raising HOSTSYNC cannot help when Saruman's upsmon logs out at the start of its halt. New 3.7 sets SHUTDOWNEXIT no and orders nut-monitor Before=pve-guests.service, so the logout tracks the guests, and sets HOSTSYNC 600. Troubleshooting and step 7 follow. ADR-0107 and the changelog entry say the same. Refs #574 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


What changed
Docs only. The UPS shutdown runbook was re-read against the estate as it stands after the rack move, so the window for steps 5 to 7 proves the right sequence. New ADR-0107 makes
trinitya third NUT secondary and capsSaruman's ordered guests atdown=90. Nothing was changed on a host.Why
Refs #574. Steps 0 to 4 were built on 2026-09-23, and three things have moved since:
trinityis on the PDU since the 2026-11-07 rack move (Move the estate into the 32U rack #919), along withgandalf, the console and the KVM. ADR-0049 lefttrinity's row "to be decided when it is placed", and nothing told it to halt.Sarumanruns 13 guests, not 2. Four are in ordered stages. With a hung guest per stage, the default timeouts add up to about 15 min, against the card's 8-min low-battery duration (read 2026-10-09 on No host shuts down on the UPS's signal, and nothing records what powers smaug #574).What the runbook gains:
igc0.99first, thenLISTEN 10.0.99.1 3493, a block proved fromprometheuswith 53 as the control, thennut-clientontrinity.startup:lines and setsdown=90on the four ordered guests. That leaves a worst case of about 9½ min.trinity's LUKS passphrase.trinityin the timing loop, the expected order, the alert table, and power-on, which follows the move runbook's §7.ADR-0049 is accepted and stays as written. It gains only a pointer note, the same pattern ADR-0106 used on 0007, 0014 and 0017.
hardware.md,security.md, the roadmap and the changelog say the same thing. The README's ADR count goes from 106 to 107.Blast radius
None until the window. This changes text only. When it is applied, it adds one pass/block pair on Winterfell (
igc0.99,10.0.99.40→10.0.99.1:3493), a listener on10.0.99.1, and shorter guest stop timeouts onSaruman.secrets/*.sops.yamlVerification
python scripts/check_docs.py:docs OK../scripts/lint.sh: passed, with 11 tools skipped and not run, because there's no Docker daemon or pipx on this Windows machine. The skipped tools include markdownlint-cli2, shellcheck and editorconfig-checker, and CI runs them.Grepped the runbook for leftover two-guest, three-host and
alexander/phoenix-only text.make validatepasses (not run: see the skipped tools above)Deployed to the lab and confirmed working (the window does this)
Docs updated
🤖 Generated with Claude Code