Skip to content

Phase 4: observability and docs match production - #424

Merged
espadonne merged 11 commits into
trunkfrom
ops/phase4-observability
Sep 2, 2026
Merged

espadonne merged 11 commits into
trunkfrom
ops/phase4-observability

Conversation

@espadonne

Copy link
Copy Markdown
Contributor

Availability campaign Phase 4 (docs/internal/retro/2026-09-02-availability-sitrep.md). Docs described a WireGuard mesh and a monitoring droplet that never existed; production is one box pushing metrics to Grafana Cloud with no alerting.

  • bc62c4b5 docs: single-box reference deployment (topology + memory budget) in deploy.md / architecture.md; real Alloy → Grafana Cloud pipeline in observability.md / runbooks/observability.md; no Alertmanager / no log shipping / no worker metrics stated plainly in alerts.md and incidents.md; multi-host design kept as a marked aspirational section. Adds a memory-pressure runbook section (sar -r, systemctl show -p MemoryCurrent, /metrics gauges).
  • c9b992b5 deploy/monitoring/README.md: nothing there is deployed, which alerts are therefore inert, why some could not fire even with a Prometheus, and what adopting them would take. Nothing deleted.
  • 02df85e4 backup scripts emit timestamped start/end/exit-status to the cron-redirected stream and write a success-only heartbeat; set -euo pipefail preserved (failed rclone exits non-zero, previous heartbeat untouched).
  • ef456602 shithub_backup_last_success_seconds{job} reads those heartbeats; BackupOverdue named shithubd_backup_last_success_seconds, which never existed.
  • 1e204081 scripts/lint-shell.sh (bash -n sweep) + scripts/test-backup-scripts.sh in make ci and CI.
  • 3e9149ec postgresql.conf.j2 sized for a shared 4 GB box: shared_buffers=256MB, work_mem=4MB, effective_cache_size=1GB, maintenance_work_mem=64MB, max_connections=60, pg_stat_statements preloaded with track=top.
  • 95080d3f db.md: template has never been applied; live-vs-template table; safe-apply procedure (restart, 05:15–06:00 window, never a blind make deploy).
  • c37a7a13 drift script tracks /etc/postgresql/16/main/postgresql.conf and reports it plus the web.env Stripe hand-edit as KNOWN drift.
  • d040bd11 scripts/lint-docs-topology.sh fails CI on WireGuard / wg0 / 10.50.0. in docs/internal outside a marked aspirational block (docs/internal/retro/ excluded).
  • 2c22f929 sitrep Phase 4 ticked; operator-only items collected into an "Operator to-do" section with exact commands.

Tests

  • make ci green locally (golangci-lint 2.13, gofumpt, markdown/policy/secret/spdx/unused/migration/systemd lints, new shell + topology lints, go test ./..., build)
  • scripts/test-backup-scripts.sh — 17 assertions over both cron scripts with stubbed rclone/pg_dump: success logs + heartbeat, rclone failure exits non-zero with no heartbeat and no refresh of a stale one, rclone chatter stays out of the cron log
  • internal/infra/metrics/backupobserver_test.go — gauge value, absent series for missing/garbage/zero heartbeat, missing dir does not break a scrape
  • scripts/lint-docs-topology.sh — verified it fails on an injected unmarked mention and on an unclosed block
  • scripts/lint-shell.sh — 35 scripts parse
  • CI integration suite (Postgres service) — runs on this PR; no Postgres locally
  • Nothing applied to production; every box-side step is in the sitrep Operator to-do

Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
@espadonne
espadonne merged commit 20ec4b5 into trunk Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants