diff --git a/README.md b/README.md index ad140991..922044d1 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ documents for different readers. at all. Every pinned digest is also scanned weekly for fixable HIGH and CRITICAL CVEs, and each image with findings has an open issue until a scan finds it clean. -- **Documented decisions and runbooks.** 106 ADRs covering what was chosen +- **Documented decisions and runbooks.** 107 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 51 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. Every critical alert links to @@ -229,7 +229,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md changelog.md │ ├── diagrams/ # the network diagram (SVG) and its predecessors -│ ├── adr/ # 106 ADRs — architecture decision records +│ ├── adr/ # 107 ADRs — architecture decision records │ ├── exercises/ # purple-team exercise records — see its README │ └── runbooks/ # 51 runbooks; successor-handover.md is the front door └── Makefile # make help diff --git a/docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md b/docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md index e7d6d1a2..4345e90b 100644 --- a/docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md +++ b/docs/adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md @@ -54,6 +54,15 @@ power source [#413](https://github.com/Gerrrt/HomeLab/issues/413) owed > ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export, > alias and command below that names `.110` now names `.20`, and the decision > is unchanged. The text below is left as written, per ADR-0001. +> +> **The `trinity` row, and the guest stop.** The last row of *Who shuts down* +> below, `trinity` *to be decided when it is placed*, is decided by +> [ADR-0107](0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md): +> it is on the PDU since the 2026-11-07 rack move and becomes a third +> secondary, on `10.0.99.1:3493`. The same ADR bounds `Saruman`'s ordered +> guests at `down=90`, and sizes the final low-battery duration from the worst +> case as well as the measured run, because *The low-battery point is the +> card's* below was written for two guests and `Saruman` now runs thirteen. ## Context diff --git a/docs/adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md b/docs/adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md new file mode 100644 index 00000000..eb45cebd --- /dev/null +++ b/docs/adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md @@ -0,0 +1,144 @@ +# ADR-0107: Subscribe `trinity` to the UPS, and bound `Saruman`'s guest stop + +**Status:** Accepted · 2026-11 · decides the `trinity` row +[ADR-0049](0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) +left open, and amends its low-battery reasoning for the guest set `Saruman` +now carries + +## Context + +ADR-0049's *Who shuts down* table ends with a row it could not fill: +*"`trinity`, when built — to be recorded when it is placed — to be decided +when it is placed."* It has been placed. The move into the 32U rack on +2026-11-06 and 2026-11-07 +([#919](https://github.com/Gerrrt/HomeLab/issues/919), +[`move-into-the-32u-rack.md`](../runbooks/move-into-the-32u-rack.md)) put +`trinity` on PDU outlet 7. The same move put `gandalf` on outlet 8, the +console's brick on 9 and the KVM on 10, and `hardware.md` records it. The PDU +is UPS outlet group 1. So `trinity` is on the pack, and nothing tells it to +stop. A cut that outlasts the pack stops it hard. + +`trinity` is the host where that costs most. It holds the household's own +data: Immich's library on a USB disk strapped to its tray, Paperless, +Vaultwarden, Memos and Actual, each with a database. Both disks are LUKS. It +is also AdGuard, the one forwarder behind `morpheus`'s resolver +([ADR-0055](0055-forward-to-adguard-alone.md)). + +The second change is on `Saruman`. ADR-0049 was written for a hypervisor +with two guests, and the runbook's step 3.6 read both as `order=any`. Read on +2026-10-09 ([#574](https://github.com/Gerrrt/HomeLab/issues/574)), +it runs thirteen, and four of the six lab-domain guests are ordered +(`order=1..4,up=120`). Proxmox stops ordered guests one stage at a time and +allows each stage its `down` timeout, 180 s by default, before it moves on. The +last real shutdown, on 2026-10-06, stopped ten guests in 2 min 10 s: +32 s for the unordered group, then about 1 min 35 s for the four ordered +stages. That is the typical case. The worst case is a hung guest in every +stage: 180 s for the unordered group plus 4 × 180 s, about **15 minutes**, against +the 8 minutes of low-battery duration the card held when step 0 read it. +`ramuh` has hung after live snapshots before, so this is not hypothetical. + +## Decision + +**`trinity` becomes a third NUT secondary, on `morpheus`'s Winterfell +address, behind a pass/block pair in the same shape as the other two. The +four ordered guests on `Saruman` get `down=90`, and `Saruman`'s monitor +stays logged in until they are down. The card's low-battery duration is +raised to cover the worst case that remains before the window, not after +the one measured run, and the PDU's outlet group sheds on nothing.** + +### `trinity` subscribes + +| Host | Powered by | Acts on the signal | How | Rule it costs | +| --- | --- | --- | --- | --- | +| `trinity` | PDU outlet 7 | **Yes, first** | `nut-client` secondary against `10.0.99.1:3493`; `shutdown -h` stops Docker, which stops every container | A pass from `10.0.99.40` above a block from Winterfell, both to `10.0.99.1:3493`, on `igc0.99` | +| `gandalf` | PDU outlet 8 | No | The range is disposable ([ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) and off between sessions; nothing on it is the only copy of anything | None | +| The console and the KVM | PDU outlets 9 and 10 | No | No state to lose | None | +| The MokerLink at U14 | UPS outlet group 3 | No | Same as the TP-Link it replaced in ADR-0049's table | None | + +Every other row of ADR-0049's table stands. + +**It crosses nothing.** `trinity` and `morpheus` are both on Winterfell, so +the subscription reaches the firewall's own address on its own segment, as +`Saruman`'s and `smaug`'s do on theirs. `upsd` gains a third `LISTEN` line, +`10.0.99.1 3493`. It still binds no Hicks address and never the WAN. + +**The pair narrows the listener, as ADR-0049's pairs do.** Without it, every +host on Winterfell could reach `10.0.99.1:3493` through the segment's egress +rule: `prometheus`, `oracle`, and the card itself. The pass from `10.0.99.40` +goes above a block from the Winterfell net, and both go above the egress +rule, beside *Block HTTPS to pfSense*. Hicks does not reach `10.0.99.1:3493` +either way: it reaches Winterfell on a named list +([ADR-0031](0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)), +and 3493 is not on it. + +**The same secondary login as the other two.** ADR-0049's argument holds: a +secondary can read status and cannot set `FSD`, so a second login would buy +separate revocation and nothing else. The primary's login stays on the +firewall. + +### `Saruman`'s guest stop is bounded + +Each of the four ordered guests gets `down=90` in its `startup:` line, +with its `order` and `up` kept as they are. A hung stage then costs 90 s, not +180. The worst case becomes 180 s (the unordered group, left at the default) +plus 4 × 90 s, plus the host's own halt: about **9½ minutes**. That is still +more than 8. Every rebuild path sets the same string: +[`build-the-lab-domain.md`](../runbooks/build-the-lab-domain.md) and the +`startup_orders` output in `tofu/guests.tf` both carry `down=90`. A +rebuild that dropped it would quietly put each stage back to 180 s. + +### `Saruman`'s monitor stays logged in until its guests are down + +`morpheus` halts once its secondaries have logged out, with `HOSTSYNC` as +the ceiling on that wait. NUT's default is for a secondary's `upsmon` to exit +as soon as it has issued `SHUTDOWNCMD`. Systemd can also stop `nut-monitor` +before `pve-guests` has finished. Either way `Saruman` logs out at the start +of its halt, and the firewall then halts while guests are still stopping, +however large `HOSTSYNC` is. So `Saruman`'s `upsmon.conf` gets +`SHUTDOWNEXIT no` (NUT 2.8.1 and later), and a systemd drop-in orders +`nut-monitor` `Before=pve-guests.service`, which stops it after the guests. +`HOSTSYNC` then caps a wait that tracks the guests, and it goes to 600 s, +the bound above rounded up. + +### The low-battery duration covers the worst case, from now on + +ADR-0049 left the card's threshold to be read, then set from the measured +sequence. A run that went well, like 2026-10-06's, is shorter than what a +real cut can produce. The worst case to `morpheus` down is now about +**10½ minutes**: 9½ to `Saruman` down, `FINALDELAY 30`, and the firewall's +own halt. The duration is that plus half again, **16 minutes**. It is set +**before** the window (runbook §3.8), not after it: the 8 minutes on the +card today would run the pack out partway through a real cut. §6.5 then +keeps the larger of that rule and the measured run plus half again. At the +23 % load read on 2026-10-09, the card claims about 44 minutes of runtime, +so shutdown would begin a little under half an hour into a cut. That still +rides out every shorter cut, which is the property the pack was bought for. + +For the same reason, **group 1, the PDU, sheds on nothing.** Any shed on time +on battery or on runtime remaining either cuts the PDU before the signal or +cuts it partway through the halt the duration was sized for. The runbook +turns every such setting off before step 5. + +## Consequences + +- **A third pass/block pair joins the ruleset**, on `igc0.99`. ADR-0013's + table gains it. Position is again the failure mode, and the proof is the + same as ADR-0049's: the block reads blocked from `prometheus`, with 53 open + as the control. +- **The house loses DNS filtering, and its forwarder, at the start of the + sequence instead of at the end.** `trinity` halts with the first group, and + `morpheus`'s resolver forwards to AdGuard alone, so names outside the + estate stop resolving while `Saruman`'s guests are still stopping. Nothing + in a halt needs them. The domain guests resolve through their own domain + controllers, and `morpheus`'s own records are local. Ordering `trinity` + after `Saruman` would need `upssched` on a secondary, and that is not worth + it for minutes of a cut that has already lasted half an hour. +- **`trinity` comes up second, after `morpheus`.** That is the order the move + runbook's §7 already uses. If the TPM does not unseal, the passphrase + prompt is on KVM port 7. +- **A hung guest is stopped hard after 90 s, not 180.** On a domain + controller that is a dirty stop, but a bounded one. The alternative is + the whole hypervisor losing power, which is worse. +- **This is a decision, not a build.** Nothing here is true on the hosts + until the runbook's step 4b and the 3.6 timeouts are applied and step 5 + proves the order with three secondaries. diff --git a/docs/changelog.md b/docs/changelog.md index 816ab6ce..d71bba6b 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,26 @@ docstring gives: it is a record, not a claim about now. ## 2026-11-08 +- **[#574](https://github.com/Gerrrt/HomeLab/issues/574): the UPS shutdown + runbook re-read against the estate before its window, and `trinity` + decided.** Three things had moved since steps 0 to 4 were built on + 2026-09-23. The rack move put `trinity`, `gandalf`, the console and the KVM + on the PDU. `Saruman` runs thirteen guests, four in ordered stages, so a + hung guest per stage could take about 15 minutes against the card's 8. The + UPS's outlet groups were found, with their shedding settings unread. + [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) + fills the `trinity` row ADR-0049 left open: it becomes a third secondary on + `10.0.99.1:3493` behind a pass/block pair on `igc0.99`, and the four ordered + guests get `down=90`, in the rebuild paths too. Review found that raising + `HOSTSYNC` could never have kept the firewall up while the guests stopped, + because `Saruman`'s monitor logs out at the start of its halt. So the + runbook now keeps it logged in (`SHUTDOWNEXIT no` and a `nut-monitor` + drop-in ordered before `pve-guests`) and sets `HOSTSYNC` to 600. That + leaves a worst case of about 10½ minutes to the firewall down. The card + goes to 16 minutes before the window rather than after it, and group 1 + sheds on nothing. Nothing was changed on a host. Step 4b, the 3.6 to 3.8 + changes and the outlet-group read are the first things the window does. + - **A copy run under the wrong `STACK` pruned five observability sets on `oracle`, and `backup-volumes.sh` now refuses it.** `ScheduledJobStale` fired for `backup-sensitive` and `verify-backups-sensitive` on `trinity`. diff --git a/docs/hardware.md b/docs/hardware.md index b2633ec5..3a2bcc43 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -60,6 +60,10 @@ that turns that estimate into a number is move, `trinity`, `gandalf` and the console are on the PDU too, and `smaug` is on it directly rather than by the long cord to the media room it had until 2026-11-06. +[ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) +makes `trinity` a third host that halts on the signal. `gandalf`, the console +and the KVM do not. That subscription is the runbook's step 4b, and it is +not built yet. Off-rack: only the eero Pro 6E units, distributed through the house. Both laptops ride a mains cut out on their own cells, so each cell is a @@ -168,10 +172,13 @@ spare. group 1, both laptops' chargers on group 2, and the MokerLink on group 3. The Smart-UPS X's card can switch and sequence each group separately, so the PDU — and with it every host but the laptops — is one group. **Each group's -turn-off and turn-on settings are not read yet**: check them against the -shutdown in +turn-off, turn-on and load-shedding settings are not read yet**: check them +against the shutdown in [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) and the power-up order in the move runbook's §7, and write them here. +[`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) reads them +under *Before you start*, because shedding on group 1 would cut the PDU +before any host is told to halt. **Two more decisions of 2026-10-16, both on the purchase table.** The UPS's AP9641 card gets an APC AP9335T temperature probe, for rack ambient at the diff --git a/docs/roadmap.md b/docs/roadmap.md index 2b6180e2..5df81cef 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -53,9 +53,14 @@ Closes when it is empty. UPS's signal.** Decided by [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md). Steps 0–4 were built on 2026-09-23: NUT on `morpheus`, its rules, and - `Saruman` and `smaug` subscribed. The sequence is armed and not proved. The - gate is a rack visit, which proves the order with `upsmon -c fsd` and pulls - the mains once to replace the card's 47-minute claim with a number. + `Saruman` and `smaug` subscribed. The sequence is armed and not proved. + Since the rack move, + [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) + adds `trinity` as a third subscriber and caps `Saruman`'s ordered guests at + `down=90`, and neither is applied yet. The gate is a rack visit. It reads + the outlet groups, applies step 4b and steps 3.6 to 3.8, proves the order + with `upsmon -c fsd`, and pulls the mains once to replace the card's + 47-minute claim with a number. → [runbook](runbooks/shut-down-on-the-ups.md) - **[#1226](https://github.com/Gerrrt/HomeLab/issues/1226) The HomePods in Home Assistant, for control.** Decided by diff --git a/docs/runbooks/build-the-lab-domain.md b/docs/runbooks/build-the-lab-domain.md index e328f905..0ea62a2a 100644 --- a/docs/runbooks/build-the-lab-domain.md +++ b/docs/runbooks/build-the-lab-domain.md @@ -423,10 +423,10 @@ skips step 4. same commands by guest name; run them as written here: ```bash - qm set 150 --startup order=1,up=120 - qm set 151 --startup order=2,up=120 - qm set 152 --startup order=3,up=120 - qm set 153 --startup order=4,up=120 + qm set 150 --startup order=1,up=120,down=90 + qm set 151 --startup order=2,up=120,down=90 + qm set 152 --startup order=3,up=120,down=90 + qm set 153 --startup order=4,up=120,down=90 ``` Check `qm config` for each one: the MAC, the `smbios1` UUID, `startup` on @@ -536,7 +536,7 @@ for spec in "150 bahamut 4096 60 1" \ --scsi0 "large_data:$4,discard=on,iothread=1,ssd=1" \ --net0 virtio,bridge=vmbr0,firewall=0 \ --agent enabled=1 \ - --onboot 1 --startup "order=$5,up=120" \ + --onboot 1 --startup "order=$5,up=120,down=90" \ --ide2 local:iso/windows-server-2025-eval.iso,media=cdrom \ --ide0 local:iso/virtio-win.iso,media=cdrom \ --boot order='scsi0;ide2' @@ -602,7 +602,11 @@ Six of those flags are worth knowing rather than copying. - **`--balloon 0`.** Ballooning on a 128 GB host buys nothing here, and it makes the guests' own memory metrics — which #266 will read — move for reasons that are not the workload. -- **`--startup order=N,up=120` on the servers only.** The IOPS argument above. +- **`--startup order=N,up=120,down=90` on the servers only.** The IOPS + argument above. `down=90` caps each stage of the host's shutdown at 90 s, + and the UPS's low-battery budget counts on that + ([ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)). + A rebuild that drops it puts each stage back to 180 s. The endpoints get `--onboot 0` because ADR-0029 runs them per session; a machine that comes back by itself is a machine that is not on demand. - **`iothread=1` with `virtio-scsi-single`.** They pair, and `iothread` does diff --git a/docs/runbooks/shut-down-on-the-ups.md b/docs/runbooks/shut-down-on-the-ups.md index 963561e0..a310a0d5 100644 --- a/docs/runbooks/shut-down-on-the-ups.md +++ b/docs/runbooks/shut-down-on-the-ups.md @@ -40,28 +40,69 @@ follow the steps.** > - Step 0 expected a factory 2 minutes. The card already read 8, and the > factory value was never observed. > - Step 3 used `sudo`. `Saruman` logs in as root and has none. +> +> **Re-read against the estate, 2026-11-08; still armed and unproved.** Three +> things changed after steps 0 to 4 were built, and step 5 would have proved +> the wrong sequence: +> +> - **The rack move put `trinity` on the PDU** (2026-11-07, +> [#919](https://github.com/Gerrrt/HomeLab/issues/919)), with `gandalf`, +> the console and the KVM. +> [ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) +> makes `trinity` a third secondary. That is the new +> [step 4b](#step-4b--subscribe-trinity), and **it is not built yet**. +> - **`Saruman` runs thirteen guests, not two**, four of them in ordered +> stages. Step 3.6 now caps those four at `down=90`. 3.7 keeps its monitor +> logged in until they are down and raises `HOSTSYNC` to 600, and 3.8 +> raises the card to 16 minutes before the window. **None of this is +> applied yet** either. +> - **The UPS's outlet groups were read for the first time** (PDU on +> group 1). Their turn-off and load-shedding settings were not, and a +> group-1 setting that sheds the PDU early would cut every host +> mid-sequence. That is a new box under *Before you start*. +> +> §6.5 now keeps the low-battery duration at or above the worst case plus +> half again, whatever the measured run says. ## What this does | Host | What it ends up doing | | --- | --- | | `morpheus` | Runs the NUT server, talks to the UPS card, halts **last** | -| `Saruman` | Subscribes, halts its two guests then itself, **first** | +| `Saruman` | Subscribes, halts its guests then itself, **first** | | `smaug` | Subscribes, exports `erebor` and halts, **first** | -| `neo` | Nothing. It loses power with the pack | +| `trinity` | Subscribes, stops its containers and halts, **first** (step 4b) | +| `neo`, `gandalf`, the console, the KVM, the MokerLink | Nothing. They lose power with the pack | | `prometheus`, `oracle` | Nothing. They ride the cut on their own cells | ## Before you start -Tick all seven. Steps 1 to 6 assume every one of these is done. +Tick all nine. Steps 1 to 6 assume every one of these is done. - [ ] **A maintenance window outside anyone's working hours.** Step 5 halts - `Saruman`, both its guests, `smaug` and `morpheus`. While `morpheus` is - down the house has no DNS and no DHCP. -- [ ] **Physical or out-of-band access to all three hosts**, because step 5 + `Saruman` and every guest on it, `smaug`, `trinity` and `morpheus`. + While `morpheus` is down the house has no DNS and no DHCP. From the + moment `trinity` halts, it has no DNS filter and no Home Assistant, + Immich, Paperless or Vaultwarden either. +- [ ] **Physical or out-of-band access to all four hosts**, because step 5 leaves them all powered off: the KVM in U23 (port 8) for `morpheus`, - the iLO at `10.0.30.10` from a Mac on Hicks for `Saruman`, and the - power button on the front of `smaug`, on its side at rack U5–U9. + port 7 for `trinity`, the iLO at `10.0.30.10` from a Mac on Hicks for + `Saruman`, and the power button on the front of `smaug`, on its side at + rack U5–U9. +- [ ] **`trinity`'s LUKS passphrase**, in case its TPM does not unseal when + it comes back in 5.4. The prompt appears on KVM port 7. +- [ ] **The UPS's outlet-group settings, read at `https://10.0.99.10`.** For + each of the three groups, note the turn-off delay, the turn-on delay + and any load shedding: shed on time on battery, or shed on runtime + remaining. **Group 1 is the PDU and every host but the laptops. Turn + off every time-on-battery and runtime-remaining shed on group 1**, at + any value, before step 5. Above the low-battery duration, a shed cuts + the PDU before any secondary is told to halt. Below it, a shed cuts the + PDU partway through a halt the duration was sized to fit. In both + cases the hosts stop uncleanly. Group 1 goes off only when the pack + runs out. Read the page back after saving, then write the values into + `hardware.md`'s outlet-group paragraph. The labels vary by firmware; + look for them under the outlet-group configuration pages. - [ ] **The UPS card's SNMPv3 passphrases**, rendered on the **main checkout** and never in a worktree (the command is below this list). They are `SNMP_AUTHPASS_APC` and `SNMP_PRIVPASS_APC` in the rendered file, and @@ -258,6 +299,12 @@ LISTEN 10.0.30.1 3493 LISTEN 10.0.40.1 3493 ``` +**Two lines only, even on a fresh build.** `trinity`'s listener, +`LISTEN 10.0.99.1 3493` +([ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)), +is added in 4b.2, after 4b.1's rules are in. If you add it here, 1.5's save +opens 3493 to every host on Winterfell until step 4b. + **Additional configuration lines for upsd.users**, with your invented username and password in place of `` and ``: @@ -300,7 +347,8 @@ FINALDELAY 30 ``` and three `sockstat` lines showing `upsd` bound to `127.0.0.1:3493`, -`10.0.30.1:3493` and `10.0.40.1:3493`. +`10.0.30.1:3493` and `10.0.40.1:3493`. After step 4b there is a fourth +`LISTEN` line and binding, `10.0.99.1`. After 3.7, `HOSTSYNC` reads 600. The command hides every password line on purpose — `authPassword` and `privPassword` among them, which is why the pattern matches a capital `P` as @@ -394,7 +442,8 @@ already in the list in 1.3 — check it survived the save, in the readback from Two per interface: a **pass** for the one host that should reach the listener, and a **block** for everything else on that segment, in that order, both above -the existing *Allow internet* rule. +the existing *Allow internet* rule. Winterfell's pair, for `trinity`, is in +step 4b. ### 2.1 Add the ImaginationLAN pass @@ -561,18 +610,102 @@ over IPv6 loopback. In the Proxmox web interface, **Datacenter → Options → HA Settings** and each guest's **Options → Start/Shutdown order**. Note the shutdown timeout. -`Saruman` runs two guests, `alexander` (VMID 140) and `phoenix` (VMID 170). -`shutdown -h` stops both before the host goes down. You need to know the -timeout because step 5 measures how long that actually takes, and -`HOSTSYNC 120` from 1.4 has to be larger than the answer. +`shutdown -h` stops every running guest before the host goes down. Guests +with no `order` stop first, together. Then each ordered stage stops in +reverse order, one stage at a time. Proxmox gives every stage its `down` +timeout, 180 s when unset, and then stops the guest hard. + +**Read 2026-09-23: two guests, both `order=any`**, so the worst case was +about three minutes, inside the card's 8-minute low-battery window. + +**Read 2026-10-09: thirteen guests** +([#574](https://github.com/Gerrrt/HomeLab/issues/574)). Four of the six +lab-domain guests are ordered, at `order=1..4,up=120`. The last real +shutdown, on 2026-10-06, stopped ten of them in 2 min 10 s: 32 s for the +unordered group, then about 1 min 35 s for the four ordered stages. If a +guest hangs in each stage, the default timeouts add up to about 15 minutes, +which is well past 8. `ramuh` has hung after live snapshots before. + +**So cap the four ordered stages at 90 s**, per +[ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md). +On `Saruman`, list the guests that have a `startup` line: + +```bash +grep -H '^startup:' /etc/pve/qemu-server/*.conf /etc/pve/lxc/*.conf 2>/dev/null +``` + +**You should see** four lines with `order=1` to `order=4` and `up=120`. If +a guest shows a different `order` or `up`, use what it shows, not what is +written here. For each of the four, keep its `order` and `up` and add +`down=90`: + +```bash +qm set --startup order=,up=120,down=90 +``` + +Re-run the `grep`. **Each of the four lines should now end `down=90`.** + +That makes the worst case to `Saruman` down 180 s for the unordered group +plus 4 × 90 s, plus the host's own halt: about **9½ minutes**. Rebuilds +keep it: [`build-the-lab-domain.md`](build-the-lab-domain.md) and +`tofu output startup_orders` both set `down=90` since 2026-11-08. After any +rebuild of a lab-domain server, re-run the `grep` above anyway. + +### 3.7 Keep `Saruman`'s monitor logged in until its guests are down + +`morpheus` halts once every secondary has logged out, or after `HOSTSYNC`, +whichever is first. By default `Saruman`'s `upsmon` exits as soon as it has +run its `SHUTDOWNCMD`, and `nut-monitor` can then stop before `pve-guests` +has stopped the guests. Either way the secondary logs out at the start of +the halt, not the end, and the firewall halts while the guests are still +stopping. **Raising `HOSTSYNC` alone does not fix that**: it is only a +ceiling on a wait that has already ended. + +So keep the monitor running until the guests are down, then let `HOSTSYNC` +cap the wait. On `Saruman`, add one line to `/etc/nut/upsmon.conf`: + +```text +SHUTDOWNEXIT no +``` + +Then order `nut-monitor` so that systemd stops it after `pve-guests`: + +```bash +mkdir -p /etc/systemd/system/nut-monitor.service.d && printf '[Unit]\nBefore=pve-guests.service\n' > /etc/systemd/system/nut-monitor.service.d/after-guests.conf && systemctl daemon-reload && systemctl restart nut-monitor +``` + +`Before=` is the right word, even though it reads backwards. Systemd stops +units in the reverse of their start order, so a unit that starts before +`pve-guests` stops after it. -**Read 2026-09-23: both guests are `order=any`**, so neither has a shutdown -timeout of its own. Proxmox then allows 180 s per guest and stops guests of -the same order in parallel, so the worst case is about three minutes plus the -host's own shutdown. That fits inside the card's 8-minute low-battery window, -which is the bound that matters. `HOSTSYNC` only bounds how long the firewall -waits for its monitors to log out, and `Saruman`'s monitor logs out as soon -as shutdown stops it. Step 5 measures what actually happens. +**Check both took:** + +```bash +upsmon -V; grep SHUTDOWNEXIT /etc/nut/upsmon.conf; systemctl show nut-monitor -p Before | tr ' ' '\n' | grep pve-guests; systemctl is-active nut-monitor +``` + +**You should see** a NUT version of 2.8.1 or later (`SHUTDOWNEXIT` is not +understood before that), the `SHUTDOWNEXIT no` line, `pve-guests.service`, +and `active`. 3.5's `upsc` must still read `OL`. Step 5 is the proof that +this works: 5.3's two intervals show whether `morpheus` waited. + +On `morpheus`, raise `HOSTSYNC` in 1.4's upsmon box from 120 to **600**, +which is the 9½-minute bound rounded up, then save. 1.6's readback should now +show `HOSTSYNC 600`. If `Saruman` logs out sooner, `morpheus` does not +wait the full 600. + +### 3.8 Raise the card's threshold to cover the bound now + +**Do not leave this for §6.5.** Step 0 left 8 minutes on the card, and with +3.6 and 3.7 the worst case to `morpheus` down is about **10½ minutes**: 9½ +to `Saruman` down, plus `FINALDELAY 30`, plus the firewall's own halt. A +real cut between now and 6.5 would run the pack out partway through. + +Set the low battery duration, the field from 0.2, to the bound plus half +again: **16 minutes**. Confirm with 0.3, which should read +`0:0:16:00.00`, and 1.7, where `battery.runtime.low` should read `960`. +§6.5 refines this after the measurement, and never sets it below this +rule. **`phoenix` is a guest here, and it is where a Claude Code session runs.** Anything working from `phoenix` goes away when step 5 halts `Saruman`, and @@ -646,14 +779,136 @@ interface that day. --- +## Step 4b — Subscribe `trinity` + +Added 2026-11-08 by +[ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md), +after the rack move put `trinity` on the PDU. **Not built yet.** Do it +before step 5, so step 5 proves three secondaries and not two. + +`trinity` and the listener are both on Winterfell, so nothing here crosses a +segment. The rules keep the rest of Winterfell — `prometheus`, `oracle` and +the card itself — off the listener. + +### 4b.1 Add the rules first + +**Firewall → Rules → WINTERFELL**, twice, at `https://10.0.99.1` from a +Hicks workstation. + +| Setting | Pass rule | Block rule | +| --- | --- | --- | +| Action | Pass | Block | +| Protocol | TCP | TCP | +| Source | `10.0.99.40` | Winterfell net | +| Destination | `10.0.99.1` | `10.0.99.1` | +| Port | `3493` | `3493` | +| Description | `Allow NUT from trinity` | `Block NUT to pfSense` | + +Drag them so the pass is directly above the block, and both are up with +*Block HTTPS to pfSense*, above Winterfell's egress rule. Then **Apply +Changes**. + +The rules go in before the listener. In the other order, the listener would +be reachable from all of Winterfell until the rules were in. + +### 4b.2 Add the listener + +**Services → UPS → Display Advanced**. This is the only place the third +line is added. 1.4 keeps two lines so that a fresh build never opens the +listener before 4b.1. Add it to the upsd.conf box, so it reads exactly: + +```text +LISTEN 10.0.30.1 3493 +LISTEN 10.0.40.1 3493 +LISTEN 10.0.99.1 3493 +``` + +**Save.** The save also regenerates the package's own monitor password, +which is harmless. Then re-run the readback from 1.6, which leaves out the +`MONITOR` line on purpose. **`sockstat` should now show four bindings.** + +### 4b.3 Verify the rules from the firewall, and the block from `prometheus` + +On `prometheus`: + +```bash +ssh admin@10.0.99.1 'pfctl -sr | grep -E "igc0\.99" | grep -nE "port = nut|Block HTTPS to pfSense"' +``` + +**You should see** the `pass` on `port = nut` before the `block` on +`port = nut`, near the HTTPS block. Then, still on `prometheus`, which is +on Winterfell and is not `trinity`: + +```bash +timeout 5 bash -c '@10.0.99.40`, with `sudo`: + +```bash +sudo apt install nut-client +``` + +Replace `/etc/nut/nut.conf` with the one line `MODE=netclient`, as in 3.2. +Then replace `/etc/nut/upsmon.conf` with 3.3's block. The only change is the +address, which here is `morpheus`'s Winterfell address: + +```text +MONITOR mjolnir@10.0.99.1 1 secondary +MINSUPPLIES 1 +SHUTDOWNCMD "/sbin/shutdown -h +0" +POWERDOWNFLAG /etc/killpower +POLLFREQ 5 +POLLFREQALERT 5 +DEADTIME 15 +``` + +It is the same `` as the other two (ADR-0107 says why). Type the +password into the editor. Do not echo it from a shell. + +```bash +sudo chown root:nut /etc/nut/upsmon.conf && sudo chmod 640 /etc/nut/upsmon.conf && sudo systemctl enable --now nut-monitor +``` + +`shutdown -h` stops Docker, and Docker stops every container in parallel +within its stop timeout, so `trinity` halts in seconds, not minutes. + +### 4b.5 Verify + +On `trinity`: + +```bash +upsc mjolnir@10.0.99.1 ups.status +``` + +**You should see `OL`.** Then, from `prometheus`: + +```bash +ssh admin@10.0.99.1 'upsc -c mjolnir@localhost' +``` + +**It should list `10.0.99.40`** beside `10.0.40.30`, `10.0.30.20` and `::1`. + +The *If you see* lines under 3.5 apply here too, with 4b.1 in place of 2.1 +and `LISTEN 10.0.99.1` in place of `LISTEN 10.0.30.1`. + +--- + ## Step 5 — Prove the shutdown order This halts everything. Your window must be open and you must be able to power -the three hosts back on by hand. +the four hosts back on by hand. ### 5.0 Silence the estate for the window -Step 5 halts four hosts on purpose. Left alone, `InstanceDown` pages `urgent` +Step 5 halts `Saruman` and its guests, `smaug`, `trinity` and `morpheus` on +purpose. Left alone, `InstanceDown` pages `urgent` for each of them, the DNS and gateway rules cascade behind it once `morpheus` goes, and the estate spends its notification budget telling you about something you are doing deliberately. @@ -664,7 +919,7 @@ END=$(date -u -d '+2 hours' +%Y-%m-%dT%H:%M:%SZ) ``` ```bash -curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"category","value":"power","isRegex":false,"isEqual":false},{"name":"alertname","value":"Watchdog","isRegex":false,"isEqual":false}],"startsAt":"%s","endsAt":"%s","createdBy":"#574 prove the UPS shutdown sequence","comment":"#574 Planned halt of Saruman, its guests, smaug and morpheus for steps 5 and 6. Everything except category=power is suppressed, so UpsOnBattery still pages in 6.2, and Watchdog is excluded so the heartbeat keeps arriving. Delete at 6.6, not on expiry."}' "$START" "$END")" | python3 -m json.tool +curl -sS -X POST http://localhost:9093/api/v2/silences -H 'Content-Type: application/json' --data "$(printf '{"matchers":[{"name":"category","value":"power","isRegex":false,"isEqual":false},{"name":"alertname","value":"Watchdog","isRegex":false,"isEqual":false}],"startsAt":"%s","endsAt":"%s","createdBy":"#574 prove the UPS shutdown sequence","comment":"#574 Planned halt of Saruman, its guests, smaug, trinity and morpheus for steps 5 and 6. Everything except category=power is suppressed, so UpsOnBattery still pages in 6.2, and Watchdog is excluded so the heartbeat keeps arriving. Delete at 6.6, not on expiry."}' "$START" "$END")" | python3 -m json.tool ``` **Record the UUID it returns.** 6.6 deletes it. @@ -705,13 +960,14 @@ takes about three minutes. 5.3 says what the alerts are still good for. Both loops print UTC to the second and both use **addresses, never names** — DNS goes away with `morpheus`. -On `prometheus`, which can reach `smaug` and `morpheus`: +On `prometheus`, which can reach `smaug`, `trinity` and `morpheus`: ```bash while :; do curl -s -m 2 -o /dev/null http://10.0.40.30:9100/metrics 2>/dev/null && s=up || s='---' + nc -z -w2 10.0.99.40 443 >/dev/null 2>&1 && t=up || t='---' nc -z -w2 10.0.99.1 22 >/dev/null 2>&1 && m=up || m='---' - printf '%s smaug=%s morpheus=%s\n' "$(date -u +%H:%M:%S)" "$s" "$m" + printf '%s smaug=%s trinity=%s morpheus=%s\n' "$(date -u +%H:%M:%S)" "$s" "$t" "$m" sleep 2 done | tee ~/ups-fsd-timing.log ``` @@ -720,6 +976,13 @@ done | tee ~/ups-fsd-timing.log accepts the connection and then never answers when the NAS has a disk fault — a port check would read healthy while the host was not. +`trinity` is checked on Caddy's 443, the one port it publishes to +Winterfell. It is on the same segment as `prometheus`, so its column keeps +working after `morpheus` goes. **It times Docker stopping, not the host +halting**: 443 closes when the containers stop, a few seconds before the +kernel does. After 5.4, read the halt itself from `trinity`'s previous boot +with `journalctl -b -1 -n 30 --no-pager`. + On the Mac on Hicks, the one you used for step 3, because **nothing on VLAN 99 can reach `Saruman`**: the single pass between those segments runs the other way. It has to be Hicks and not a Mac on VLAN 30 for the reason step 3 @@ -780,16 +1043,20 @@ and stays there is when that host stopped answering. **The order you should see:** -1. `saruman=---` and `smaug=---`, within a few seconds of each other. Both are - secondaries and both start halting on the same flag. +1. `trinity=---` first, within seconds, then `saruman=---` and `smaug=---`. + All three are secondaries and start halting on the same flag. `trinity` + only has containers to stop. `Saruman` has its guests, which took 2 min + 10 s on 2026-10-06, so expect it last of the three by a margin. 2. `morpheus=---` last, once every secondary has disconnected or `HOSTSYNC` - has elapsed, plus `FINALDELAY`. + has elapsed, plus `FINALDELAY`. Because of 3.7, `Saruman` disconnects + when its guests are down, so `morpheus=---` should come after + `saruman=---`, not a fixed delay after 5.2. **Write down two intervals**, both measured from the time you noted in 5.2: -- **to `Saruman` down** — the number `HOSTSYNC` in 1.4 has to exceed. +- **to `Saruman` down** — this has to stay under `HOSTSYNC` (600 after 3.7). - **to `morpheus` down** — the end-to-end time. §6.5 sets the card's - low-battery threshold from it, and step 7 corrects `HOSTSYNC` to it. + low-battery threshold from it. One trap in the `prometheus` log: once `morpheus` is down, that host cannot reach VLAN 40 at all. A `smaug=---` appearing in the same second as @@ -805,7 +1072,8 @@ time the sequence, which is the whole reason the loops exist: | `InstanceDown` | `smaug`, then `morpheus`, `shiva`, `neo` | about 5 minutes after each stops answering | `for: 5m` on a bare `up == 0`, so it covers the `snmp` job too. `morpheus` arrives here as `10.0.99.1` rather than by name, and `shiva` and `neo` follow it because their scrapes route through the firewall — `neo` is still powered and still switching, only unreachable from the monitoring host | | `SnmpTargetUnreachable` | the same three | about 10 minutes after | `for: 10m`, so it trails `InstanceDown` on the same targets rather than arriving first. There is no `SnmpTargetDown` | | `RemoteWriteJobStale` | `Saruman` | about 20 minutes after | Five minutes of lookback plus `for: 15m`. `InstanceDown` cannot see an agent that pushes, so `Saruman` never appears under it | -| `HypervisorGuestStopped` | `alexander`, `phoenix` | not during this test | `for: 1h`, and `homelab_guest_running` stops arriving the moment `Saruman` halts, so it never matures | +| `RemoteWriteJobStale` | `trinity` | about 20 minutes after | `trinity` pushes through its Alloy agent like `Saruman`, so it is seen the same way, and its blackbox probes fail as the containers stop | +| `HypervisorGuestStopped` | every guest on `Saruman` | not during this test | `for: 1h`, and `homelab_guest_running` stops arriving the moment `Saruman` halts, so it never matures | | `UpsOnBattery` | — | must not fire at all | Nothing in step 5 writes to the card | **The UPS itself stays on.** `UpsOnBattery` firing here would mean something @@ -813,11 +1081,16 @@ took real power away, and that is not this test. ### 5.4 Power everything back on, in this order -1. `morpheus`, from the KVM in U23. -2. `Saruman`, from the iLO at `10.0.30.10`. -3. `smaug`, from the button on the front. +1. `morpheus`, from the KVM in U23, port 8. +2. `trinity`, from its power button. Watch KVM port 7: if the TPM does not + unseal, the LUKS passphrase prompt is there. It comes before `Saruman` + because AdGuard on it is `morpheus`'s only forwarder (ADR-0055). +3. `Saruman`, from the iLO at `10.0.30.10`. +4. `smaug`, from the button on the front. -`Saruman`'s guests come back by themselves if *Start at boot* is set. +This is the order of +[`move-into-the-32u-rack.md`](move-into-the-32u-rack.md#7-power-up-in-this-order) +§7. `Saruman`'s guests come back by themselves if *Start at boot* is set. ### 5.5 Check nothing leaked @@ -874,10 +1147,24 @@ Worked example: 100 → 64 in 18 minutes gives 18 × 100 / 36 = **50 minutes**. You now know two things you did not in step 0: how long the whole shutdown takes (5.3) and what a minute of pack is worth (6.4). -Set the card's low battery duration, the same field as 0.2, to **the step 5.3 -end-to-end time plus half again**, rounded up to the next whole minute. - -Worked example: if 5.3 measured 3 minutes 10 seconds, set 5 minutes. +Set the card's low battery duration, the same field as 0.2, to **the larger +of the step 5.3 end-to-end time and the worst-case bound from 3.8, plus half +again**, rounded up to the next whole minute. +[ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) +says why the bound counts. One good run is the typical case, and a real cut +can hang a guest in every stage. + +Worked example: 5.3 measures 3 minutes 10 seconds to `morpheus` down, and +3.8's bound is 10½ minutes. The larger is 10½, plus half again is 15¾, so +the duration stays at the **16 minutes** 3.8 set. It only moves if step 5 +measured longer than the bound. In that case the bound is wrong: find out +why before you raise the duration. + +At the 23 % load read on 2026-10-09, the card claimed about 44 minutes of +runtime, so 16 minutes still lets the estate ride out the first half hour +of a cut before anything halts. If 6.4 measured much less than the card +claims, the duration still comes from this rule. What changes is how long a +cut can last before the halt begins, so write that down in step 7. Confirm with the command from 0.3. @@ -912,7 +1199,10 @@ this work exists to catch. | [`hardware.md`](../hardware.md) Accessories, UPS entry | The low battery duration you set in 6.5, and that 0.1 read 8 minutes before any change here (factory value not observed). Also the card's second SNMPv3 access-control entry, `10.0.99.1` read-only, from 1.0 | | [`security.md`](../security.md) mains-loss row | It no longer says nothing shuts down on the signal | | [`fit-the-ups-battery.md`](fit-the-ups-battery.md) §5 | Whether the card's runtime series tracked your measurement | -| §1.4 of this file | `HOSTSYNC`, corrected down to the 5.3 measurement plus margin | +| §1.4 and §3.7 of this file | `HOSTSYNC` stays at 600 as long as 3.6's bound is 9½ minutes. Lower it only if the bound itself falls. A measured run shorter than the bound is not a reason to lower it | +| [`hardware.md`](../hardware.md) outlet-group paragraph | The three groups' turn-off, turn-on and load-shedding settings read under *Before you start* | +| [`network.md`](../network.md) Winterfell notes, and ADR-0013's accounting | The `trinity` → `10.0.99.1:3493` pass/block pair from 4b.1 | +| [`architecture.md`](../architecture.md) `trinity` row | It is a NUT secondary of `morpheus` | Then close [#574](https://github.com/Gerrrt/HomeLab/issues/574). @@ -927,15 +1217,32 @@ address (check 1.6) or the pass rule is below the block or the catch-all **`upsc` says `Access denied`.** The credential in 1.4 and the one in 3.3 or 4.2 do not match. Not a firewall problem. -**`morpheus` halted before `Saruman` finished.** `HOSTSYNC` is shorter than -the guests take. The two logs from 5.1 are how you know: `morpheus=---` -appears before `saruman=---`. Raise it in 1.4 and re-run step 5. +**`morpheus` halted before `Saruman` finished.** The two logs from 5.1 are +how you know: `morpheus=---` appears before `saruman=---`. Raising +`HOSTSYNC` alone will not fix this. `Saruman`'s monitor logged out before its +guests were down, so check 3.7 first: + +- the `SHUTDOWNEXIT no` line, and that `upsmon -V` is 2.8.1 or later; +- that the drop-in is loaded: `systemctl show nut-monitor -p Before` + lists `pve-guests.service`; +- on `Saruman`, `journalctl -b -1 -u nut-monitor -u pve-guests` should show + `nut-monitor` stopping after `pve-guests`. + +If all three hold and `morpheus` still went first, the wait reached +`HOSTSYNC`. The guests took longer than 3.6's bound, so find the stage that +overran before raising anything. Raise 3.8's duration to match any increase +in `HOSTSYNC`, then re-run step 5. **`smaug` did not halt.** Look at its system log under *System → Advanced → System Log*. A slave that never saw the shutdown flag was never connected; the block counter on CasaBonita from 2.5 says whether the firewall refused it. +**`trinity=---` never appeared, or `trinity` came back with no journal of a +halt.** Run `journalctl -u nut-monitor -b -1` on `trinity`. A secondary that +never logged in never saw the flag. Check 4b.5's `upsc -c` listing, and that +4b.1's pass is above its block. + **`UpsOnBattery` did not fire within a minute in 6.2.** The scrape is every 60 seconds and the rule waits 30, so two minutes is the outside limit. Past that, the card is not reporting the transfer — plug back in and stop. @@ -1022,6 +1329,16 @@ long cord, and since the move into the 32U rack it is on that PDU directly irreplaceable data on spinning disks that was going to stop uncleanly every time the pack ran out. +### Why `trinity` is a subscriber, and `gandalf` is not + +Both came onto the PDU in the move into the 32U rack, on 2026-11-07. +`trinity` holds the household's own data, every service of it on a database +or a disk that a hard stop can leave half-written, so it halts with the +first group +([ADR-0107](../adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md)). +`gandalf` is the range: disposable by decision, off between sessions, and +the only copy of nothing. + ### What this does not prove Whether each host powers itself back on when mains returns is a BIOS setting diff --git a/docs/security.md b/docs/security.md index 7cb5f143..1573933f 100644 --- a/docs/security.md +++ b/docs/security.md @@ -26,7 +26,7 @@ What this network is actually built to survive: | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | | Losing visibility of a failure | 179 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | -| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, and since the move into the 32U rack on 2026-11-06 also `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | +| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, and since the move into the 32U rack on 2026-11-06 also `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) adds `trinity` to the first group and leaves `gandalf` and the console to the pack, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | What it explicitly does **not** defend against: a determined attacker with diff --git a/tofu/guests.tf b/tofu/guests.tf index dbee574d..4bbc18b9 100644 --- a/tofu/guests.tf +++ b/tofu/guests.tf @@ -195,7 +195,7 @@ module "guest" { output "startup_orders" { description = "What root on Saruman sets after an apply, one `qm set` per guest that boots with the host." value = { - for name, g in local.guests : name => "qm set ${g.vm_id} --startup order=${g.startup},up=120" + for name, g in local.guests : name => "qm set ${g.vm_id} --startup order=${g.startup},up=120,down=90" if g.startup != null } }