74 improve server installer

[Yeetoxic]

This MR implements daemon-owned install lifecycle tracking and warmup for Issue #74. The daemon now owns install phases, warmup, post-warmup cleanup, reinstall, and install lifecycle logs. Protocube proxies the lifecycle endpoint and install log requests so clients do not call daemon directly. Paper and Spigot definitions no longer embed first-boot warmup logic in the install script; they install/configure files and delegate first boot to daemon warmup.

Known limitation: local WSL + Docker Desktop cannot fully validate runtime Paper boot because daemon-created overlay mounts do not propagate to sibling containers. Install and warmup lifecycle validation passes; full runtime boot should be verified on native Linux Docker Engine.

[Yeetoxic]

Issue #74 Merge Notes: Daemon-Owned Install/Warmup Lifecycle

Summary

This change moves install lifecycle visibility and first-boot warmup ownership into the daemon, with Protocube acting as the client-facing proxy. The daemon now tracks install phases, exposes install metadata and lifecycle logs, runs warmup and post-warmup containers after install, and supports reinstall through the same lifecycle path.

What Changed

• Added daemon install lifecycle phases:
  • idle
  • installing
  • warming
  • post_warmup
  • install_failed
  • warmup_failed
  • post_warmup_failed
  • ready
• Added daemon install state tracking with container name/id, Docker status, exit code, timestamps, failure reason, and recent lifecycle log content.
• Added GET /api/servers/:server/install on daemon and Protocube.
• Added install lifecycle log support:
  • /api/servers/:server/install?size=N returns content.
  • /api/servers/:server/logs?type=install&size=N returns install/warmup logs.
  • size and lines are both accepted, capped at 100.
• Added Protocube forwarding for install info and install-log query parameters.
• Added daemon-owned warmup support using the normal server image, environment, base folder mount, startup matcher, timeout, and retry policy.
• Added post-warmup script support.
• Added POST /api/servers/:server/reinstall through Protocube and daemon.
• Updated Paper and Spigot software definitions so their install scripts install/build/configure files only; daemon now handles warmup.
• Fixed Paper empty BUILD_NUMBER handling so an empty build value falls back to the latest Paper build instead of producing an invalid PaperMC download URL.
• Updated local compose files to build local images and support the local daemon test environment.
• Persisted daemon lifecycle logs under the host-mounted daemon log directory so install logs survive daemon rebuilds.

How It Works

Install begins in daemon Server.install(). The daemon creates an installer container, records phase/status/container metadata, writes installer output to a stable log file, and reports install success/failure back to Protocube.

After install succeeds, RunWarmup() checks the software warmup settings. If warmup is enabled, the daemon creates a warmup container using the same server image/env/base folder and waits for the configured online signal. Timeout or process failure marks warmup_failed unless the policy allows continuing. When configured, a post-warmup container runs the cleanup script and records post_warmup status.

Clients should call Protocube, not the daemon directly:

GET  /api/servers/:id/install
GET  /api/servers/:id/logs?type=install
POST /api/servers/:id/reinstall

Direct daemon API calls require the daemon's internal node token, not an admin Protocube API key.

Compliance Checklist

• Install lifecycle visibility: complete.
• Required phases: complete.
• Daemon GET /api/servers/:server/install: complete.
• Protocube proxy for install info: complete.
• Docker status/exit/timestamps/failure reason/container metadata: complete.
• Recent install/warmup logs exposed: complete.
• Warmup schema fields: complete.
• Defaults:
  • warmup-timeout: 300 seconds.
  • warmup-retries: 0.
  • post-warmup-timeout: 120 seconds.
  • warmup_failure_policy: fail.
• Daemon-owned warmup with normal server image/env/mounts/invocation/online signal: complete.
• Warmup retries: complete for warmup_failure_policy: retry.
• continue policy: complete.
• Post-warmup script: complete.
• Paper/Spigot cleanup: complete.
• Old software YAML without warmup fields remains valid: complete.
• skip_scripts behavior preserved: complete.
• Reinstall endpoint: included as related issue scope and complete.
• SLS updater idea from comments: intentionally out of scope.

Manual Validation

Local manual validation was run through Protocube on http://localhost:5620.

Run these commands from the repository root.

Useful commands:

export SLS_KEY='...'
export SERVER_ID='...'

Rebuild:

docker compose -f protocube/docker-compose.yml up -d --build
docker compose -f daemon/docker-compose.yml up -d --build

Trigger reinstall:

curl -i -X POST \
  -H "Authorization: Bearer $SLS_KEY" \
  "http://localhost:5620/api/servers/$SERVER_ID/reinstall"

Poll lifecycle:

watch -n 2 "curl -s -H \"Authorization: Bearer $SLS_KEY\" \
  'http://localhost:5620/api/servers/$SERVER_ID/install?size=100' | jq '{phase,status,exit_code,started_at,finished_at,lines:(.content // [] | length)}'"

Expected final state:

{
  "phase": "ready",
  "status": "exited",
  "exit_code": 0,
  "lines": 1
}

Check install logs through install endpoint:

curl -s \
  -H "Authorization: Bearer $SLS_KEY" \
  "http://localhost:5620/api/servers/$SERVER_ID/install?size=100" | jq -r '.content[]?'

Check install logs through logs endpoint:

curl -s \
  -H "Authorization: Bearer $SLS_KEY" \
  "http://localhost:5620/api/servers/$SERVER_ID/logs?type=install&size=100" | jq -r '.data[]'

Check runtime logs separately:

curl -s \
  -H "Authorization: Bearer $SLS_KEY" \
  "http://localhost:5620/api/servers/$SERVER_ID/logs?size=100" | jq -r '.data[]'

Validate persisted lifecycle logs:

sudo find /var/log/sls/daemon/install -maxdepth 1 -type f -name "$SERVER_ID*" -printf '%p %s bytes\n'

Expected files:

/var/log/sls/daemon/install/<SERVER_ID>.log
/var/log/sls/daemon/install/<SERVER_ID>-warmup.log
/var/log/sls/daemon/install/<SERVER_ID>-post-warmup.log

Automated Test Evidence

Focused tests for changed daemon packages:

docker run --rm -v "$PWD:/src" -w /src/daemon golang:1.25.8 go test ./server ./api/router

Result: pass.

Focused tests for changed Protocube packages:

docker run --rm -v "$PWD:/src" -w /src/protocube golang:1.25.8 go test ./client ./server ./api/router

Result: pass.

Broader Protocube suite:

docker run --rm -v "$PWD:/src" -w /src/protocube golang:1.25.8 go test ./...

Result: pass.

Broader daemon suite:

docker run --rm -v "$PWD:/src" -w /src/daemon golang:1.25.8 go test ./...

Result: existing unrelated failure in daemon/server/filesystem/archive_test.go; it panics with archive not supported at this time. Other daemon packages completed successfully before that package failure. This failure is not introduced by the install lifecycle work.

Known Local Environment Caveat

Full Minecraft runtime boot is blocked in the current WSL + Docker Desktop environment by overlay mount propagation.

Observed behavior:

• Daemon install/warmup reaches ready.
• server.jar exists in the installed base folder.
• Runtime server container may still log:

Error: Unable to access jarfile server.jar

Root cause from local testing:

• Overlay mounts created inside the daemon-style container were visible inside that container.
• The same overlay mount was not visible to sibling Docker containers launched by Docker Desktop.
• Even with rshared, Docker Desktop did not propagate the daemon-created overlay mount to the sibling server container.

Review implication:

• Issue Improve server installer #74 lifecycle and warmup behavior validates locally.
• Full runtime Paper boot should be validated on native Linux Docker Engine, or a separate dev fallback should be added for Docker Desktop if that environment must support full runtime smoke tests.