Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ is a very different thing from an overlooked one. Full detail in
| Alertmanager webhook URL and the MokerLink SNMP community, in a local Claude Code session transcript | Found 2026-08-20 by a value-level sweep of the host. Redacted in place; mode 600, never committed or synced. The webhook topic was rotated — on the public ntfy instance the topic name *is* the credential, there is nothing to revoke — and delivery re-verified end to end. The switch community deliberately was not: rotating it means the `neo` residual above all over again. **That reasoning is spent, and this value now retires with the hardware.** The overwrite it was afraid of was done on 2026-09-12 and persisted, and the MokerLink is being replaced by a switch commissioned with SNMP v2c off entirely — so what retires this community is not a fresh community but the absence of one. The thing that must not be carried across is any value the MokerLink held, passphrases included. **Retired 2026-11-05**, when the CRS326 replaced it with no v2c community at all and v3 passphrases generated fresh ([#444](https://github.com/Gerrrt/HomeLab/issues/444)); the switch section above has the proof. |
| Pre-purge objects still served by GitHub after the history rewrite | The 2026-08-19 rewrite (`021d2b6`) removed both secrets above from every *reachable* commit, but GitHub still serves the orphaned objects by SHA. Verified 2026-08-26: `647d90a`, `21afcad`, `efb2632` and `ee3d443` all still resolve through the API, and the tree at `21afcad` still lists `certificates/Gandalf.Gondor.Lab/ca-key.pem` and `cert-key.pem`. Garbage collection requested from GitHub Support on 2026-08-26 — **pending**; this is the [purge runbook](docs/runbooks/purge-git-history.md)'s *Afterwards* step, and it is the last one outstanding. The repository has no forks and a network count of 0, so nothing else is perpetuating them. Both credentials were rotated *before* the rewrite, so this changes nothing about their status: the old keys and the old community remain superseded and must still be treated as public. Re-check with `gh api repos/Gerrrt/HomeLab/commits/647d90a --jq .sha` — a `404` means GitHub has collected them. |
| Alertmanager published on `0.0.0.0`, letting anyone who could reach it silence an alert | Fixed 2026-08-30 — 9093 now binds to `127.0.0.1` ([#70](https://github.com/Gerrrt/HomeLab/issues/70), [ADR-0012](docs/adr/0012-publish-only-ports-with-an-off-host-consumer.md)). This was the sharpest of the three because a silence switches off monitoring and the record of it lives in the system being switched off. Nothing off-host ever used the port: silences are reached through Grafana, which proxies Alertmanager over the compose network behind a login, so closing it cost no capability. |
| Prometheus and Loki published on `0.0.0.0` with no authentication | Fixed — [#182](https://github.com/Gerrrt/HomeLab/issues/182), [ADR-0067](docs/adr/0067-authenticate-the-ingest-ports-with-a-token-per-client.md). Until then anything that could route to `10.0.99.20:9090` or `:3100` could read every metric and log line, inject metrics through the remote-write receiver, and delete log ranges through Loki's delete API, with firewall default-deny as the whole control. Prometheus and Loki now publish on `127.0.0.1` only. The host's address on both ports is an ingest proxy that serves a push to one token per agent (`oracle`, `trinity`, `Saruman`), a query to one reader token (Glance, Home Assistant, the deploy script's arrival check), and the admin and delete APIs to nobody. `IngestAuthNotEnforced` pages if the published address ever answers a token-less request with anything but the proxy's 401. The tokens no longer cross the wire in cleartext: both ports serve TLS under the estate CA, and every client verifies it ([#764](https://github.com/Gerrrt/HomeLab/issues/764), [ADR-0086](docs/adr/0086-serve-the-ingest-ports-over-tls-under-the-estate-ca.md)). **One thing remains, deliberately:** loopback is unauthenticated, because a local user on the monitoring host already holds the SOPS key. The firewall still narrows who can try: Winterfell (99) itself and `Saruman`'s one pass from `10.0.30.110`. |
| Prometheus and Loki published on `0.0.0.0` with no authentication | Fixed — [#182](https://github.com/Gerrrt/HomeLab/issues/182), [ADR-0067](docs/adr/0067-authenticate-the-ingest-ports-with-a-token-per-client.md). Until then anything that could route to `10.0.99.20:9090` or `:3100` could read every metric and log line, inject metrics through the remote-write receiver, and delete log ranges through Loki's delete API, with firewall default-deny as the whole control. Prometheus and Loki now publish on `127.0.0.1` only. The host's address on both ports is an ingest proxy that serves a push to one token per agent (`oracle`, `trinity`, `Saruman`), a query to one reader token (Glance, Home Assistant, the deploy script's arrival check), and the admin and delete APIs to nobody. `IngestAuthNotEnforced` pages if the published address ever answers a token-less request with anything but the proxy's 401. The tokens no longer cross the wire in cleartext: both ports serve TLS under the estate CA, and every client verifies it ([#764](https://github.com/Gerrrt/HomeLab/issues/764), [ADR-0086](docs/adr/0086-serve-the-ingest-ports-over-tls-under-the-estate-ca.md)). **One thing remains, deliberately:** loopback is unauthenticated, because a local user on the monitoring host already holds the SOPS key. The firewall still narrows who can try: Winterfell (99) itself and `Saruman`'s one pass from `10.0.30.20`. |
| The lab's Prometheus and Loki published to VLAN 30 with no authentication | **Open until the rollout lands.** The fix is authored ([#834](https://github.com/Gerrrt/HomeLab/issues/834)), and none of the four hosts converges, so it is live only once `stacks/lab/README.md`'s ordered rollout has run and its three `curl`s return 401. `alexander` published both to the segment that exists to hold attackers (#436, 2026-09). Anything on it could `POST /-/quit`, forge series and delete log ranges in the lab's evidence store. Once rolled out, they are on `127.0.0.1`, behind `stacks/lab/Caddyfile`: one token each for `odin`, `phoenix` and `fenrir`, and a reader token, all the lab's own and never the estate's. **Two residuals.** The tokens cross VLAN 30 as plain HTTP, where ARP spoofing is expected (#764's question). And nothing pages if the proxy is bypassed, because the lab has no Alertmanager (ADR-0020, [#858](https://github.com/Gerrrt/HomeLab/issues/858)). |
| The monitoring host's disk and swap are unencrypted | **Accepted residual, not a fix in progress** — see below. |
| Loki's log volume has no size ceiling | **Accepted residual, not a fix in progress** — see below. Prometheus has a byte ceiling since [#184](https://github.com/Gerrrt/HomeLab/issues/184); Loki has no size-based retention to set, only time. `HostDiskWillFillIn24h` is the control. |
Expand Down
7 changes: 7 additions & 0 deletions docs/adr/0007-defensive-estate-and-offensive-range.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,13 @@
> filesystems are crosses, read by the hypervisor through the guest agent and
> sent over `Saruman`'s existing pass. No path is opened, and nothing else a
> guest produces crosses. The text here is left as written, per ADR-0001.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,13 @@
> ImaginationLAN of the shape ADR-0053 gives `golem`
> ([ADR-0072](0072-put-the-iso-store-on-smaug-over-nfs-to-saruman-alone.md)).
> Same kind of pass, host- and port-scoped, and the table is not edited.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,13 @@
> ([#451](https://github.com/Gerrrt/HomeLab/issues/451)). The collector still
> runs from the range when there is one. The text here is left as written, per
> ADR-0001.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

**Status:** Accepted · 2026-09

> [!NOTE]
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

[ADR-0007](0007-defensive-estate-and-offensive-range.md) gave `Saruman` the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@
> `qm guest exec`. This closes the half the first consequence below leaves
> open, for those services only. Everything else in "What services it runs,
> and their health" stays in the lab.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@
> converted. The argument here is unchanged: the mirror is on `Saruman`'s own
> bridge, so what removes Zeek's view is moving the domain guests off it, not
> a cluster.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ power source [#413](https://github.com/Gerrrt/HomeLab/issues/413) owed
> the intent of reusing it, and it is now true on the card as well as here.
> A user of NUT's own remains the way to separate them. The decision is
> unchanged.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,13 @@
> [ADR-0069](0069-vendor-the-ja4-scripts-into-the-sensor-stack-rather-than-build-an-image.md)
> decides. [`stacks/sensor/README.md`](../../stacks/sensor/README.md) has the
> query that reads them. The text below is left as written, per ADR-0001.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@ on ImaginationLAN rather than Hicks or Winterfell; decides
> §2b). `phoenix` holds read-only `PVEAuditor` on the store, so it cannot
> write to it either. What this costs is up to a day between a replacement
> and the page. The text below is left as written, per ADR-0001.
>
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

**Status:** Accepted · 2026-10

> [!NOTE]
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

`HypervisorGuestStopped` (`stacks/observability/prometheus/rules/host.rules.yaml`)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

**Status:** Accepted · 2026-10

> [!NOTE]
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

Four management consoles serve self-signed certificates that nothing watched
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@
[ADR-0014](0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)
and [ADR-0017](0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)

> [!NOTE]
> **`Saruman`'s address.** `Saruman` moved from `10.0.30.110` to
> **`10.0.30.20`** on 2026-11-07, as
> [`build-the-playground.md`](../runbooks/build-the-playground.md) §3 has it
> ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). Each rule, export,
> alias and command below that names `.110` now names `.20`, and the decision
> is unchanged. The text below is left as written, per ADR-0001.

## Context

[ADR-0014](0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,7 +185,7 @@ hole from the monitoring VLAN into the monitored one.
| Host | VLAN | Stack | Contents |
| --- | --- | --- | --- |
| `prometheus` (10.0.99.20) | 🔴 99 | [`stacks/observability`](../stacks/observability) | Prometheus, Alertmanager, Loki, Grafana, snmp-exporter, blackbox-exporter, speedtest-tracker, docker-socket-proxy, Alloy |
| `Saruman` (10.0.30.110) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
| `Saruman` (10.0.30.20) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, hosting twenty-one guests: `alexander`, built 2026-09-05 ([#262](https://github.com/Gerrrt/HomeLab/issues/262)); `phoenix`, built 2026-09-20 ([#436](https://github.com/Gerrrt/HomeLab/issues/436)); `odin`, built 2026-09-27 for the security tooling (ADR-0030); `fenrir`, built 2026-09-30 as the Zeek sensor ([#437](https://github.com/Gerrrt/HomeLab/issues/437)); `golem`, built 2026-10-03 as the backup server ([#485](https://github.com/Gerrrt/HomeLab/issues/485)); `eden`, built 2026-10-07 as the BloodHound guest ([#451](https://github.com/Gerrrt/HomeLab/issues/451)), off between sessions; `garuda`, built 2026-10-10 as the analyst workstation ([#921](https://github.com/Gerrrt/HomeLab/issues/921)); and ADR-0029's six-machine domain, built by hand 2026-09-24 to 2026-09-25 ([#414](https://github.com/Gerrrt/HomeLab/issues/414)), of which `carbuncle` and `siren` run per session; and all eight of [ADR-0090](adr/0090-test-the-dotfiles-os-layers-on-on-demand-saruman-guests.md)'s on-demand dotfiles test VMs, `dot-debian`, `dot-fedora` and `dot-windows`, built 2026-10-08, and `dot-opensuse`, `dot-arch`, `dot-alpine`, `dot-gentoo` and `dot-nixos`, built 2026-10-09 ([#920](https://github.com/Gerrrt/HomeLab/issues/920)). Alloy agent (native package). It runs no compose stack by decision, not by omission: Docker would rewrite the iptables its own firewall relies on ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md)), which is why the agent here is the native package and why `stacks/lab` runs in a guest |
| `gandalf` (10.0.30.30) | 🟢 30 | *(none — and none intended)* | Proxmox VE 9, the range host, installed 2026-11-07 ([#421](https://github.com/Gerrrt/HomeLab/issues/421)). No guests yet. Guest disks go on the ZFS pool `range`, targets go on the portless `vmbr1`, and it is off between sessions ([ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md), [ADR-0106](adr/0106-name-the-range-host-gandalf-and-leave-ifrit-to-its-disabled-amt.md)) |
| `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy and its Docker socket proxy, and Caddy as the ingest proxy in front of the first two ([#834](https://github.com/Gerrrt/HomeLab/issues/834)) — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) |
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
Expand Down
Loading
Loading