Skip to content

docs(ups): re-read the shutdown runbook against the estate, and subscribe trinity (#574) - #1330

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/ups-runbook-prep-574
Nov 9, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/ups-runbook-prep-574

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Nov 8, 2026

Copy link
Copy Markdown
Owner

What changed

Docs only. The UPS shutdown runbook was re-read against the estate as it stands after the rack move, so the window for steps 5 to 7 proves the right sequence. New ADR-0107 makes trinity a third NUT secondary and caps Saruman's ordered guests at down=90. Nothing was changed on a host.

Why

Refs #574. Steps 0 to 4 were built on 2026-09-23, and three things have moved since:

  • trinity is on the PDU since the 2026-11-07 rack move (Move the estate into the 32U rack #919), along with gandalf, the console and the KVM. ADR-0049 left trinity's row "to be decided when it is placed", and nothing told it to halt.
  • Saruman runs 13 guests, not 2. Four are in ordered stages. With a hung guest per stage, the default timeouts add up to about 15 min, against the card's 8-min low-battery duration (read 2026-10-09 on No host shuts down on the UPS's signal, and nothing records what powers smaug #574).
  • The UPS's outlet groups were read for the first time, but their shedding settings were not. Load shedding on group 1 would cut the whole PDU before any host is told to halt.

What the runbook gains:

  • Step 4b: rules on igc0.99 first, then LISTEN 10.0.99.1 3493, a block proved from prometheus with 53 as the control, then nut-client on trinity.
  • Step 3.6: reads the startup: lines and sets down=90 on the four ordered guests. That leaves a worst case of about 9½ min.
  • Before you start: two new boxes, the outlet-group and load-shedding read, and trinity's LUKS passphrase.
  • Step 5: trinity in the timing loop, the expected order, the alert table, and power-on, which follows the move runbook's §7.
  • §6.5: the final low-battery duration is now the larger of the measured run and the worst-case bound, plus half again. Today's worked example gives 15 min.
  • Step 7 and troubleshooting: rows and an entry for the new pieces.

ADR-0049 is accepted and stays as written. It gains only a pointer note, the same pattern ADR-0106 used on 0007, 0014 and 0017. hardware.md, security.md, the roadmap and the changelog say the same thing. The README's ADR count goes from 106 to 107.

Blast radius

None until the window. This changes text only. When it is applied, it adds one pass/block pair on Winterfell (igc0.99, 10.0.99.40 → 10.0.99.1:3493), a listener on 10.0.99.1, and shorter guest stop timeouts on Saruman.

  • No change to network segmentation or firewall rules (this PR documents a pair to be added, nothing is applied)
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • python scripts/check_docs.py: docs OK.

  • ./scripts/lint.sh: passed, with 11 tools skipped and not run, because there's no Docker daemon or pipx on this Windows machine. The skipped tools include markdownlint-cli2, shellcheck and editorconfig-checker, and CI runs them.

  • Grepped the runbook for leftover two-guest, three-host and alexander/phoenix-only text.

  • make validate passes (not run: see the skipped tools above)

  • Deployed to the lab and confirmed working (the window does this)

  • Docs updated

🤖 Generated with Claude Code

…ribe trinity (#574)

Three things moved after steps 0 to 4 were built on 2026-09-23, and step 5
would have proved the wrong sequence:

- the rack move put trinity, gandalf, the console and the KVM on the PDU;
- Saruman runs thirteen guests, four in ordered stages, so a hung guest per
  stage could take about 15 minutes against the card's 8;
- the UPS's outlet groups were found, with their shedding settings unread.

ADR-0107 fills the trinity row ADR-0049 left open: a third NUT secondary on
10.0.99.1:3493 behind a pass/block pair on igc0.99. The four ordered guests
get down=90, and 6.5 sizes the final low-battery duration from the
remaining worst case (about 9.5 minutes) as well as the measured run.

The runbook gains step 4b, the 3.6 timeout change, an outlet-group read
under Before you start, trinity in step 5's loops, order and power-on, and
the step 7 rows that follow. Nothing was changed on a host.

Refs #574

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 8, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved battery-threshold timing, runtime-based shedding, and rebuilds that discard down=90 can cut power before guest shutdown completes.

5 open issues
What changed in this PR

This documentation-only PR updates the UPS shutdown plan for the post-rack-move estate and the remaining #574 maintenance window. No host configuration is changed.

Changes:

  • Adds ADR-0107 to make trinity a third NUT secondary and cap Saruman’s ordered guest shutdown timeouts.
  • Expands the runbook with firewall rules, outlet-shedding checks, timing verification, recovery steps, and worst-case battery budgeting.
  • Updates supporting documentation and the README’s ADR count.
File Description
README.md Increases the ADR count to 107.
docs/​security.md Adds the planned trinity shutdown policy.
docs/​runbooks/​shut-down-on-the-ups.md Updates configuration, shutdown proof, recovery, and battery-budget procedures.
docs/​roadmap.md Expands the remaining UPS maintenance-window gate.
docs/​hardware.md Records the pending subscription and outlet-shedding checks.
docs/​changelog.md Records the runbook revision and new decision.
docs/​adr/​0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md Defines the third secondary, guest timeout caps, and battery-budget policy.
docs/​adr/​0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md Adds a pointer to ADR-0107 without rewriting the original decision.

Review effort: Balanced

Comment thread docs/runbooks/shut-down-on-the-ups.md Resolved
Comment thread docs/runbooks/shut-down-on-the-ups.md Resolved
Comment thread docs/runbooks/shut-down-on-the-ups.md Resolved
Comment thread docs/runbooks/shut-down-on-the-ups.md Resolved
…e card before the window (#574)

Review on #1330 found five gaps:

- the lab-domain rebuild paths (build-the-lab-domain.md and tofu's
  startup_orders output) set startup without down=90, so a rebuild undid it;
- the shedding check allowed a runtime shed below the low-battery duration,
  which cuts the PDU mid-halt. Group 1 now sheds on nothing;
- the 9.5-minute bound exceeded the 8 minutes left on the card until 6.5.
  New 3.8 raises it to 16 minutes before the window;
- 1.4 added the Winterfell listener, so a fresh build opened 3493 to all of
  VLAN 99 before 4b.1's rules. It is now added only in 4b.2;
- raising HOSTSYNC cannot help when Saruman's upsmon logs out at the start
  of its halt. New 3.7 sets SHUTDOWNEXIT no and orders nut-monitor
  Before=pve-guests.service, so the logout tracks the guests, and sets
  HOSTSYNC 600. Troubleshooting and step 7 follow.

ADR-0107 and the changelog entry say the same.

Refs #574

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 2eed7ff into main Nov 9, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ups-runbook-prep-574 branch November 9, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants