docs(ups): re-read the shutdown runbook against the estate, and subscribe trinity (#574)

README.md

@@ -134,7 +134,7 @@ documents for different readers.
   at all. Every pinned digest is also scanned weekly for fixable HIGH and
   CRITICAL CVEs, and each image with findings has an open issue until a scan
   finds it clean.
-- **Documented decisions and runbooks.** 106 ADRs covering what was chosen
+- **Documented decisions and runbooks.** 107 ADRs covering what was chosen
   and what was rejected — including the costs accepted knowingly; 51
   runbooks for the operations that are easy to get wrong at 1am, one of which
   is the handover page a successor reads first. Every critical alert links to
@@ -229,7 +229,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology
 │   ├── architecture.md  network.md  hardware.md
 │   ├── observability.md  security.md  roadmap.md  changelog.md
 │   ├── diagrams/             # the network diagram (SVG) and its predecessors
-│   ├── adr/                  # 106 ADRs — architecture decision records
+│   ├── adr/                  # 107 ADRs — architecture decision records
 │   ├── exercises/            # purple-team exercise records — see its README
 │   └── runbooks/             # 51 runbooks; successor-handover.md is the front door
 └── Makefile                  # make help

docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md

@@ -54,6 +54,15 @@ power source [#413](https://github.com/Gerrrt/HomeLab/issues/413) owed
 > ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
 > alias and command below that names `.110` now names `.20`, and the decision
 > is unchanged. The text below is left as written, per ADR-0001.
+>
+> **The `trinity` row, and the guest stop.** The last row of *Who shuts down*
+> below, `trinity` *to be decided when it is placed*, is decided by
+> [ADR-0107](0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md):
+> it is on the PDU since the 2026-11-07 rack move and becomes a third
+> secondary, on `10.0.99.1:3493`. The same ADR bounds `Saruman`'s ordered
+> guests at `down=90`, and sizes the final low-battery duration from the worst
+> case as well as the measured run, because *The low-battery point is the
+> card's* below was written for two guests and `Saruman` now runs thirteen.
 
 ## Context
 

docs/adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md

@@ -0,0 +1,144 @@
+# ADR-0107: Subscribe `trinity` to the UPS, and bound `Saruman`'s guest stop
+
+**Status:** Accepted · 2026-11 · decides the `trinity` row
+[ADR-0049](0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)
+left open, and amends its low-battery reasoning for the guest set `Saruman`
+now carries
+
+## Context
+
+ADR-0049's *Who shuts down* table ends with a row it could not fill:
+*"`trinity`, when built — to be recorded when it is placed — to be decided
+when it is placed."* It has been placed. The move into the 32U rack on
+2026-11-06 and 2026-11-07
+([#919](https://github.com/Gerrrt/HomeLab/issues/919),
+[`move-into-the-32u-rack.md`](../runbooks/move-into-the-32u-rack.md)) put
+`trinity` on PDU outlet 7. The same move put `gandalf` on outlet 8, the
+console's brick on 9 and the KVM on 10, and `hardware.md` records it. The PDU
+is UPS outlet group 1. So `trinity` is on the pack, and nothing tells it to
+stop. A cut that outlasts the pack stops it hard.
+
+`trinity` is the host where that costs most. It holds the household's own
+data: Immich's library on a USB disk strapped to its tray, Paperless,
+Vaultwarden, Memos and Actual, each with a database. Both disks are LUKS. It
+is also AdGuard, the one forwarder behind `morpheus`'s resolver
+([ADR-0055](0055-forward-to-adguard-alone.md)).
+
+The second change is on `Saruman`. ADR-0049 was written for a hypervisor
+with two guests, and the runbook's step 3.6 read both as `order=any`. Read on
+2026-10-09 ([#574](https://github.com/Gerrrt/HomeLab/issues/574)),
+it runs thirteen, and four of the six lab-domain guests are ordered
+(`order=1..4,up=120`). Proxmox stops ordered guests one stage at a time and
+allows each stage its `down` timeout, 180 s by default, before it moves on. The
+last real shutdown, on 2026-10-06, stopped ten guests in 2 min 10 s:
+32 s for the unordered group, then about 1 min 35 s for the four ordered
+stages. That is the typical case. The worst case is a hung guest in every
+stage: 180 s for the unordered group plus 4 × 180 s, about **15 minutes**, against
+the 8 minutes of low-battery duration the card held when step 0 read it.
+`ramuh` has hung after live snapshots before, so this is not hypothetical.
+
+## Decision
+
+**`trinity` becomes a third NUT secondary, on `morpheus`'s Winterfell
+address, behind a pass/block pair in the same shape as the other two. The
+four ordered guests on `Saruman` get `down=90`, and `Saruman`'s monitor
+stays logged in until they are down. The card's low-battery duration is
+raised to cover the worst case that remains before the window, not after
+the one measured run, and the PDU's outlet group sheds on nothing.**
+
+### `trinity` subscribes
+
+| Host | Powered by | Acts on the signal | How | Rule it costs |
+| --- | --- | --- | --- | --- |
+| `trinity` | PDU outlet 7 | **Yes, first** | `nut-client` secondary against `10.0.99.1:3493`; `shutdown -h` stops Docker, which stops every container | A pass from `10.0.99.40` above a block from Winterfell, both to `10.0.99.1:3493`, on `igc0.99` |
+| `gandalf` | PDU outlet 8 | No | The range is disposable ([ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) and off between sessions; nothing on it is the only copy of anything | None |
+| The console and the KVM | PDU outlets 9 and 10 | No | No state to lose | None |
+| The MokerLink at U14 | UPS outlet group 3 | No | Same as the TP-Link it replaced in ADR-0049's table | None |
+
+Every other row of ADR-0049's table stands.
+
+**It crosses nothing.** `trinity` and `morpheus` are both on Winterfell, so
+the subscription reaches the firewall's own address on its own segment, as
+`Saruman`'s and `smaug`'s do on theirs. `upsd` gains a third `LISTEN` line,
+`10.0.99.1 3493`. It still binds no Hicks address and never the WAN.
+
+**The pair narrows the listener, as ADR-0049's pairs do.** Without it, every
+host on Winterfell could reach `10.0.99.1:3493` through the segment's egress
+rule: `prometheus`, `oracle`, and the card itself. The pass from `10.0.99.40`
+goes above a block from the Winterfell net, and both go above the egress
+rule, beside *Block HTTPS to pfSense*. Hicks does not reach `10.0.99.1:3493`
+either way: it reaches Winterfell on a named list
+([ADR-0031](0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)),
+and 3493 is not on it.
+
+**The same secondary login as the other two.** ADR-0049's argument holds: a
+secondary can read status and cannot set `FSD`, so a second login would buy
+separate revocation and nothing else. The primary's login stays on the
+firewall.
+
+### `Saruman`'s guest stop is bounded
+
+Each of the four ordered guests gets `down=90` in its `startup:` line,
+with its `order` and `up` kept as they are. A hung stage then costs 90 s, not
+180. The worst case becomes 180 s (the unordered group, left at the default)
+plus 4 × 90 s, plus the host's own halt: about **9½ minutes**. That is still
+more than 8. Every rebuild path sets the same string:
+[`build-the-lab-domain.md`](../runbooks/build-the-lab-domain.md) and the
+`startup_orders` output in `tofu/guests.tf` both carry `down=90`. A
+rebuild that dropped it would quietly put each stage back to 180 s.
+
+### `Saruman`'s monitor stays logged in until its guests are down
+
+`morpheus` halts once its secondaries have logged out, with `HOSTSYNC` as
+the ceiling on that wait. NUT's default is for a secondary's `upsmon` to exit
+as soon as it has issued `SHUTDOWNCMD`. Systemd can also stop `nut-monitor`
+before `pve-guests` has finished. Either way `Saruman` logs out at the start
+of its halt, and the firewall then halts while guests are still stopping,
+however large `HOSTSYNC` is. So `Saruman`'s `upsmon.conf` gets
+`SHUTDOWNEXIT no` (NUT 2.8.1 and later), and a systemd drop-in orders
+`nut-monitor` `Before=pve-guests.service`, which stops it after the guests.
+`HOSTSYNC` then caps a wait that tracks the guests, and it goes to 600 s,
+the bound above rounded up.
+
+### The low-battery duration covers the worst case, from now on
+
+ADR-0049 left the card's threshold to be read, then set from the measured
+sequence. A run that went well, like 2026-10-06's, is shorter than what a
+real cut can produce. The worst case to `morpheus` down is now about
+**10½ minutes**: 9½ to `Saruman` down, `FINALDELAY 30`, and the firewall's
+own halt. The duration is that plus half again, **16 minutes**. It is set
+**before** the window (runbook §3.8), not after it: the 8 minutes on the
+card today would run the pack out partway through a real cut. §6.5 then
+keeps the larger of that rule and the measured run plus half again. At the
+23 % load read on 2026-10-09, the card claims about 44 minutes of runtime,
+so shutdown would begin a little under half an hour into a cut. That still
+rides out every shorter cut, which is the property the pack was bought for.
+
+For the same reason, **group 1, the PDU, sheds on nothing.** Any shed on time
+on battery or on runtime remaining either cuts the PDU before the signal or
+cuts it partway through the halt the duration was sized for. The runbook
+turns every such setting off before step 5.
+
+## Consequences
+
+- **A third pass/block pair joins the ruleset**, on `igc0.99`. ADR-0013's
+  table gains it. Position is again the failure mode, and the proof is the
+  same as ADR-0049's: the block reads blocked from `prometheus`, with 53 open
+  as the control.
+- **The house loses DNS filtering, and its forwarder, at the start of the
+  sequence instead of at the end.** `trinity` halts with the first group, and
+  `morpheus`'s resolver forwards to AdGuard alone, so names outside the
+  estate stop resolving while `Saruman`'s guests are still stopping. Nothing
+  in a halt needs them. The domain guests resolve through their own domain
+  controllers, and `morpheus`'s own records are local. Ordering `trinity`
+  after `Saruman` would need `upssched` on a secondary, and that is not worth
+  it for minutes of a cut that has already lasted half an hour.
+- **`trinity` comes up second, after `morpheus`.** That is the order the move
+  runbook's §7 already uses. If the TPM does not unseal, the passphrase
+  prompt is on KVM port 7.
+- **A hung guest is stopped hard after 90 s, not 180.** On a domain
+  controller that is a dirty stop, but a bounded one. The alternative is
+  the whole hypervisor losing power, which is worse.
+- **This is a decision, not a build.** Nothing here is true on the hosts
+  until the runbook's step 4b and the 3.6 timeouts are applied and step 5
+  proves the order with three secondaries.

docs/changelog.md

@@ -19,6 +19,26 @@ docstring gives: it is a record, not a claim about now.
 
 ## 2026-11-08
 
+- **[#574](https://github.com/Gerrrt/HomeLab/issues/574): the UPS shutdown
+  runbook re-read against the estate before its window, and `trinity`
+  decided.** Three things had moved since steps 0 to 4 were built on
+  2026-09-23. The rack move put `trinity`, `gandalf`, the console and the KVM
+  on the PDU. `Saruman` runs thirteen guests, four in ordered stages, so a
+  hung guest per stage could take about 15 minutes against the card's 8. The
+  UPS's outlet groups were found, with their shedding settings unread.
+  [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)
+  fills the `trinity` row ADR-0049 left open: it becomes a third secondary on
+  `10.0.99.1:3493` behind a pass/block pair on `igc0.99`, and the four ordered
+  guests get `down=90`, in the rebuild paths too. Review found that raising
+  `HOSTSYNC` could never have kept the firewall up while the guests stopped,
+  because `Saruman`'s monitor logs out at the start of its halt. So the
+  runbook now keeps it logged in (`SHUTDOWNEXIT no` and a `nut-monitor`
+  drop-in ordered before `pve-guests`) and sets `HOSTSYNC` to 600. That
+  leaves a worst case of about 10½ minutes to the firewall down. The card
+  goes to 16 minutes before the window rather than after it, and group 1
+  sheds on nothing. Nothing was changed on a host. Step 4b, the 3.6 to 3.8
+  changes and the outlet-group read are the first things the window does.
+
 - **A copy run under the wrong `STACK` pruned five observability sets on
   `oracle`, and `backup-volumes.sh` now refuses it.** `ScheduledJobStale`
   fired for `backup-sensitive` and `verify-backups-sensitive` on `trinity`.

docs/hardware.md

@@ -60,6 +60,10 @@ that turns that estimate into a number is
 move, `trinity`, `gandalf` and the console are on the PDU too, and `smaug` is on
 it directly rather than by the long cord to the media room it had until
 2026-11-06.
+[ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)
+makes `trinity` a third host that halts on the signal. `gandalf`, the console
+and the KVM do not. That subscription is the runbook's step 4b, and it is
+not built yet.
 
 Off-rack: only the eero Pro 6E units, distributed through the house.
 Both laptops ride a mains cut out on their own cells, so each cell is a
@@ -168,10 +172,13 @@ spare.
 group 1, both laptops' chargers on group 2, and the MokerLink on group 3. The
 Smart-UPS X's card can switch and sequence each group separately, so the PDU
 — and with it every host but the laptops — is one group. **Each group's
-turn-off and turn-on settings are not read yet**: check them against the
-shutdown in
+turn-off, turn-on and load-shedding settings are not read yet**: check them
+against the shutdown in
 [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md)
 and the power-up order in the move runbook's §7, and write them here.
+[`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) reads them
+under *Before you start*, because shedding on group 1 would cut the PDU
+before any host is told to halt.
 
 **Two more decisions of 2026-10-16, both on the purchase table.** The UPS's
 AP9641 card gets an APC AP9335T temperature probe, for rack ambient at the

docs/roadmap.md

@@ -53,9 +53,14 @@ Closes when it is empty.
   UPS's signal.** Decided by
   [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md).
   Steps 0–4 were built on 2026-09-23: NUT on `morpheus`, its rules, and
-  `Saruman` and `smaug` subscribed. The sequence is armed and not proved. The
-  gate is a rack visit, which proves the order with `upsmon -c fsd` and pulls
-  the mains once to replace the card's 47-minute claim with a number.
+  `Saruman` and `smaug` subscribed. The sequence is armed and not proved.
+  Since the rack move,
+  [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)
+  adds `trinity` as a third subscriber and caps `Saruman`'s ordered guests at
+  `down=90`, and neither is applied yet. The gate is a rack visit. It reads
+  the outlet groups, applies step 4b and steps 3.6 to 3.8, proves the order
+  with `upsmon -c fsd`, and pulls the mains once to replace the card's
+  47-minute claim with a number.
   → [runbook](runbooks/shut-down-on-the-ups.md)
 - **[#1226](https://github.com/Gerrrt/HomeLab/issues/1226) The HomePods in
   Home Assistant, for control.** Decided by

docs/runbooks/build-the-lab-domain.md

@@ -423,10 +423,10 @@ skips step 4.
    same commands by guest name; run them as written here:
 
    ```bash
-   qm set 150 --startup order=1,up=120
-   qm set 151 --startup order=2,up=120
-   qm set 152 --startup order=3,up=120
-   qm set 153 --startup order=4,up=120
+   qm set 150 --startup order=1,up=120,down=90
+   qm set 151 --startup order=2,up=120,down=90
+   qm set 152 --startup order=3,up=120,down=90
+   qm set 153 --startup order=4,up=120,down=90
    ```
 
    Check `qm config` for each one: the MAC, the `smbios1` UUID, `startup` on
@@ -536,7 +536,7 @@ for spec in "150 bahamut 4096 60 1" \
     --scsi0 "large_data:$4,discard=on,iothread=1,ssd=1" \
     --net0 virtio,bridge=vmbr0,firewall=0 \
     --agent enabled=1 \
-    --onboot 1 --startup "order=$5,up=120" \
+    --onboot 1 --startup "order=$5,up=120,down=90" \
     --ide2 local:iso/windows-server-2025-eval.iso,media=cdrom \
     --ide0 local:iso/virtio-win.iso,media=cdrom \
     --boot order='scsi0;ide2'
@@ -602,7 +602,11 @@ Six of those flags are worth knowing rather than copying.
 - **`--balloon 0`.** Ballooning on a 128 GB host buys nothing here, and it makes
   the guests' own memory metrics — which #266 will read — move for reasons that
   are not the workload.
-- **`--startup order=N,up=120` on the servers only.** The IOPS argument above.
+- **`--startup order=N,up=120,down=90` on the servers only.** The IOPS
+  argument above. `down=90` caps each stage of the host's shutdown at 90 s,
+  and the UPS's low-battery budget counts on that
+  ([ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)).
+  A rebuild that drops it puts each stage back to 180 s.
   The endpoints get `--onboot 0` because ADR-0029 runs them per session; a
   machine that comes back by itself is a machine that is not on demand.
 - **`iothread=1` with `virtio-scsi-single`.** They pair, and `iothread` does