From e30e675d13bc17e52e18e45ee2def06b1b5237ea Mon Sep 17 00:00:00 2001 From: Erik Thorelli Date: Thu, 15 Oct 2026 16:06:36 -0700 Subject: [PATCH 1/3] feat(booth): keep the booth game running with a watchdog, Mac kiosk setup and a staff runbook [Agent: Claude Code | Model: Opus 5.5 | Thinking: High | Acting on behalf of: Erik | Skill: none] Booth mode (#43) handles idle players, but nothing restarts the game after a crash or a hang, the Mac still sleeps, updates and shows notifications, and staff have no written procedure. - --heartbeat= (app/booth_heartbeat.gd) rewrites the file with the Unix time every two seconds, also while a game pauses the tree. It writes a temp file and renames it, and warns once on failure. - tools/booth/watchdog.sh (bash 3.2 and Linux) launches the game with --heartbeat, relaunches it when it exits with a doubling backoff that resets after a stable run, and SIGTERMs (then SIGKILLs) a game whose heartbeat goes stale. SIGTERM stops game and watchdog; a stop file stops relaunching for maintenance. - tools/booth/macos/setup.sh prints its plan by default and changes nothing; --apply installs the watchdog as a LaunchAgent (RunAtLoad, KeepAlive), clears the quarantine, and turns off sleep, Power Nap, the screen saver, automatic updates and crash dialogs, with restart after a power cut. --undo reverses it. It lists the settings macOS can't script: automatic login, Do Not Disturb, display, volume. - docs/booth-runbook.md: pack list, setup, mornings, the show, maintenance stops, swapping to the spare, collecting the session logs, troubleshooting and teardown. README and docs/builds.md link it. - Tests: tests/booth_heartbeat.gd, plus unit tests that run the real watchdog against a stand-in game (exit, hang, healthy, stop file, arguments, usage) and setup.sh's plan and undo (never --apply on a Mac). CI shellchecks both scripts. --- .github/workflows/ci.yml | 4 + README.md | 2 + docs/booth-runbook.md | 97 ++++++++ docs/builds.md | 1 + hoppys-adventure-arcade/app/app_router.gd | 18 ++ .../app/booth_heartbeat.gd | 53 ++++ .../app/booth_heartbeat.gd.uid | 1 + .../tests/booth_heartbeat.gd | 79 ++++++ .../tests/booth_heartbeat.gd.uid | 1 + tools/booth/macos/setup.sh | 226 ++++++++++++++++++ tools/booth/test_macos_setup.py | 123 ++++++++++ tools/booth/test_watchdog.py | 169 +++++++++++++ tools/booth/watchdog.sh | 206 ++++++++++++++++ 13 files changed, 980 insertions(+) create mode 100644 docs/booth-runbook.md create mode 100644 hoppys-adventure-arcade/app/booth_heartbeat.gd create mode 100644 hoppys-adventure-arcade/app/booth_heartbeat.gd.uid create mode 100644 hoppys-adventure-arcade/tests/booth_heartbeat.gd create mode 100644 hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid create mode 100755 tools/booth/macos/setup.sh create mode 100644 tools/booth/test_macos_setup.py create mode 100644 tools/booth/test_watchdog.py create mode 100755 tools/booth/watchdog.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 12357e53..e6a752e3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -106,6 +106,10 @@ jobs: run: python -m unittest discover -s tools/vendor -v - name: Booth tools unit tests run: python -m unittest discover -s tools/booth -v + - name: Shellcheck booth scripts + run: | + command -v shellcheck >/dev/null || sudo apt-get install -y shellcheck + shellcheck tools/booth/watchdog.sh tools/booth/macos/setup.sh gitleaks: name: gitleaks diff --git a/README.md b/README.md index 38c4ba3c..259193da 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,8 @@ HoppysAdventureArcade.exe -- --booth --route=heroes_vs_villains Booth mode runs fullscreen and ignores the menu's **Quit Game**, so staff close the window instead. After 90 seconds without input, any game returns to the hub's attract screen. [Run it at a booth](docs/builds.md#run-it-at-a-booth) lists every route and flag. +For a machine that runs unattended all day, follow the [booth runbook](docs/booth-runbook.md). It sets up a watchdog that restarts the game if it closes or freezes, along with the Mac's kiosk settings, and covers the staff checklist. + ## Docs - Platformer: [Bay Area Blitz](docs/bay-area-blitz.md), [Orbital Ops](docs/orbital-ops.md), [The Enchanted Repo](docs/enchanted-repo.md) diff --git a/docs/booth-runbook.md b/docs/booth-runbook.md new file mode 100644 index 00000000..94ae4790 --- /dev/null +++ b/docs/booth-runbook.md @@ -0,0 +1,97 @@ +# Booth runbook + +This runbook is for the staff who run Hoppy’s Adventure Arcade unattended at a booth, such as booth 528 at re:Invent. It covers a Mac running the game, with a second Mac set up the same way as a spare. The watchdog also runs on Linux; Windows isn't covered. + +## Pack list + +- The booth Mac and a spare Mac, each set up and tested with the steps below. +- A USB stick with the macOS build zip from [Releases or Actions](builds.md#get-a-build) and a copy of this repository, which has the booth tools in `tools/booth/`. +- The controllers plus one spare, with their cables or wireless dongles, the display cable and a power strip. +- A keyboard and mouse for staff. Keep them out of players' reach. + +## Set up a machine (once, about 20 minutes) + +1. Sign in to the account the booth will use. +2. Unzip the build and move **Hoppy’s Adventure Arcade.app** to **Applications**. +3. In Terminal, from the copy of this repository, see what the setup will change: + ```sh + tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" --route heroes_vs_villains + ``` + Nothing changes yet. To apply it, run the same command with `--apply` and enter the administrator password when asked. The game starts. +4. Do the manual steps the script lists at the end: automatic login, Do Not Disturb, the display, the volume and the controllers. +5. Restart the Mac. The game should start on its own within a minute. + +The setup does the following: +- It installs the watchdog (`tools/booth/watchdog.sh`) as a login item. The watchdog starts the game in booth mode, relaunches it whenever it closes or crashes, and restarts it when it freezes. +- It removes the download quarantine, so Gatekeeper doesn't block the app. +- It keeps the Mac awake and turns it back on after a power cut. +- It turns off the screen saver, automatic updates and crash dialogs. + +Booth mode itself runs fullscreen, ignores the menu's **Quit Game**, and sends a game nobody has touched for 90 seconds back to the hub's attract screen ([details](builds.md#run-it-at-a-booth)). + +## Every morning + +1. Turn the Mac on. It logs in, and the game appears within a minute. +2. Check that every controller moves Hoppy and that the volume suits the floor. +3. Play one short round. + +## During the show + +| What happens | What you'll see | +|---|---| +| A player walks away mid-game | After 90 seconds the game returns to the hub. | +| The game crashes or closes | The watchdog opens it again within a few seconds. | +| The game freezes | The watchdog restarts it within about a minute. | +| Someone presses Cmd+Q | The game closes, and the watchdog opens it again. | +| The player's controller disconnects | The game pauses and asks for the controller. Plug it back in, or press Start on another controller. | +| You want to clear the Endless Runner scores | **Endless Runner → Leaderboard → Reset scores**. | + +## Stop the game for maintenance + +```sh +touch ~/Library/Application\ Support/HoppyBooth/stop +``` + +Then press Cmd+Q. The game stays closed. To bring it back, restart the Mac or run: + +```sh +launchctl kickstart gui/$(id -u)/ai.coderabbit.arcade.booth +``` + +The watchdog removes the stop file when it starts. + +## Swap to the spare + +1. Move the display cable and the controllers to the spare Mac and turn it on. It starts the game on its own. +2. Copy the session logs off the first Mac when there's time (see below). + +## Collect the session logs + +The game records each session's start and end in `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/booth/`, one file per run of the game. It records the game, the times and the length, and nothing about players. At the end of each day, copy the whole `booth` folder to the USB stick and leave the original in place. To summarize every machine's folder at once: + +```sh +python3 tools/booth/sessions_report.py booth-a/ booth-b/ --csv sessions.csv +``` + +## Troubleshooting + +The watchdog writes launches, exits and restarts to `~/Library/Logs/HoppyBooth/watchdog.log`, and the game's own output to `~/Library/Logs/HoppyBooth/game.log`. + +| Problem | What to do | +|---|---| +| No game after login | Read `watchdog.log`. Run `launchctl print gui/$(id -u)/ai.coderabbit.arcade.booth` to see whether the login item is loaded. If it isn't, run the setup again with `--apply`. | +| macOS says the app can't be opened | Run `xattr -dr com.apple.quarantine "/Applications/Hoppy’s Adventure Arcade.app"`. | +| The game restarts every minute or two | If `watchdog.log` says the game "has not written a heartbeat", the build is older than the watchdog. Install a current build. | +| A controller does nothing | Reconnect it. Check **System Settings → Bluetooth** for wireless pads. | +| Black borders or the wrong size | Check the resolution in **System Settings → Displays**. | +| Notifications appear over the game | Turn on Do Not Disturb in **Control Center → Focus**. | +| Nothing helps | Restart the Mac. If the game still doesn't run, swap to the spare. | + +## After the show + +```sh +tools/booth/macos/setup.sh --undo # see what it will change +tools/booth/macos/setup.sh --undo --apply +``` + +Then turn off automatic login and Do Not Disturb, and copy the session logs. diff --git a/docs/builds.md b/docs/builds.md index 9726d5f8..63e551f8 100644 --- a/docs/builds.md +++ b/docs/builds.md @@ -36,6 +36,7 @@ open -a "Hoppy’s Adventure Arcade.app" --args -- --booth --route=heroes_vs_vil - **Idle reset.** After 90 seconds with no keyboard, mouse, touch or controller input, any game returns to the hub, whose rotating previews act as the attract screen. - **No quitting from the menu.** The menu's **Quit Game** does nothing in booth mode; staff close the window (Cmd+Q, Alt+F4). - **Launch route.** `--route=` opens a route at launch (`heroes_vs_villains`, `code_city_brawl`, `bay_area_blitz`, `orbital_ops`, `enchanted_repo`, `runner`, `platformer`, `beatem_up`, `hub`), with or without `--booth`. An unknown name logs an error and opens the hub. +- **Keep it running.** `tools/booth/watchdog.sh` relaunches the game whenever it exits and restarts it when it hangs. The watchdog passes `--heartbeat=`, which makes the game rewrite that file with the current time every two seconds; a file that stops changing means the game is stuck. On a Mac, `tools/booth/macos/setup.sh` installs the watchdog as a login item and applies the kiosk settings. The [booth runbook](booth-runbook.md) walks staff through setup, the show days and recovery. - **Session log.** Booth mode records each game session's start and end in `user://booth/`, one file per run of the game (`sessions--.jsonl`). Each entry records the game, the local start and end times, the duration, how the session ended (`menu`, `idle` or `quit`) and how many times the player retried. It records nothing about players and never uses the network. A start with no end means the game stopped without quitting: a crash, a power cut or a forced kill. The game prints its file's full path at startup. `--session-log=` writes to another folder and works without `--booth`, for playtests. `user://` is `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/` on macOS, `%APPDATA%\Godot\app_userdata\Hoppy’s Adventure Arcade\` on Windows and `~/.local/share/godot/app_userdata/Hoppy’s Adventure Arcade/` on Linux. To summarize the logs, copy each machine's `booth` folder and run: diff --git a/hoppys-adventure-arcade/app/app_router.gd b/hoppys-adventure-arcade/app/app_router.gd index 85ef6a3d..4de8c207 100644 --- a/hoppys-adventure-arcade/app/app_router.gd +++ b/hoppys-adventure-arcade/app/app_router.gd @@ -1,5 +1,6 @@ extends CanvasLayer const BoothLog := preload("res://app/booth_log.gd") +const BoothHeartbeat := preload("res://app/booth_heartbeat.gd") const ROUTES := { "heroes_vs_villains": "res://scenes/beatem_up/reinvent/heroes_vs_villains.tscn", "code_city_brawl": "res://scenes/beatem_up/code_city_brawl.tscn", @@ -30,12 +31,17 @@ const STICK_DEADZONE := 0.4 ## of them is passed. const SESSION_LOG_PREFIX := "--session-log=" const SESSION_LOG_DEFAULT := "user://booth" +## Watchdog support (app/booth_heartbeat.gd): `--heartbeat=` keeps that file updated with the +## time every two seconds, so tools/booth/watchdog.sh can restart a game that hangs. Off unless +## passed; the watchdog passes it. +const HEARTBEAT_PREFIX := "--heartbeat=" var busy := false var veil: ColorRect var booth := false var idle_limit := IDLE_SECONDS var idle_seconds := 0.0 var session_log: BoothLog +var heartbeat: BoothHeartbeat func _ready() -> void: # The veil tween must keep running while a microgame has the tree paused. process_mode = Node.PROCESS_MODE_ALWAYS @@ -55,6 +61,11 @@ func _ready() -> void: var log_path := session_log_path(args, booth) if log_path != "": start_session_log(log_path) + var beat_path := heartbeat_path(args) + if beat_path != "": + heartbeat = BoothHeartbeat.new(beat_path) + add_child(heartbeat) + print("AppRouter: heartbeat at %s" % ProjectSettings.globalize_path(beat_path)) var start := launch_route(args) if start != "": _launch.call_deferred(start) @@ -125,6 +136,13 @@ func start_session_log(folder: String) -> void: session_log = BoothLog.new(folder) print("AppRouter: session log at %s" % ProjectSettings.globalize_path(session_log.path)) +## The heartbeat file named by `--heartbeat=`, or "" when absent. +static func heartbeat_path(args: PackedStringArray) -> String: + for arg in args: + if arg.begins_with(HEARTBEAT_PREFIX) and arg.length() > HEARTBEAT_PREFIX.length(): + return arg.trim_prefix(HEARTBEAT_PREFIX) + return "" + func _launch(route: String) -> void: await go(route) var scene := get_tree().current_scene diff --git a/hoppys-adventure-arcade/app/booth_heartbeat.gd b/hoppys-adventure-arcade/app/booth_heartbeat.gd new file mode 100644 index 00000000..25cb0026 --- /dev/null +++ b/hoppys-adventure-arcade/app/booth_heartbeat.gd @@ -0,0 +1,53 @@ +extends Node +## Booth watchdog support. While the game's main loop runs, this rewrites a file with the current +## Unix time every INTERVAL seconds. tools/booth/watchdog.sh restarts the game when the file goes +## stale, which catches a hang that leaves the process alive. AppRouter adds it only when the game +## is launched with `--heartbeat=`. +const INTERVAL := 2.0 +var path := "" +var elapsed := 0.0 +## Set after the first failed write, so the warning prints once. Every beat still tries again: a +## heartbeat that stays stale makes the watchdog restart the game. +var warned := false + +func _init(file_path: String) -> void: + path = file_path + name = "BoothHeartbeat" + # Keep beating while a game has the tree paused: a paused game is not a hung one. + process_mode = Node.PROCESS_MODE_ALWAYS + +func _ready() -> void: + var directory := path.get_base_dir() + if directory != "" and not DirAccess.dir_exists_absolute(directory): + DirAccess.make_dir_recursive_absolute(directory) + beat() + +func _process(delta: float) -> void: + elapsed += delta + if elapsed >= INTERVAL: + elapsed = 0.0 + beat() + +## Write to a temporary file and rename it over the heartbeat, so the watchdog never reads a +## half-written number. +func beat() -> void: + var temporary := path + ".tmp" + var file := FileAccess.open(temporary, FileAccess.WRITE) + if file == null: + _fail(error_string(FileAccess.get_open_error())) + return + var written := file.store_line(str(int(Time.get_unix_time_from_system()))) + file.close() + if not written: + _fail("write failed") + return + var error := DirAccess.rename_absolute(temporary, path) + if error != OK: + _fail(error_string(error)) + +func _fail(reason: String) -> void: + if warned: + return + warned = true + var problem := "Booth heartbeat: cannot write %s (%s)." % [path, reason] + push_warning(problem + " If it stays stale, the watchdog restarts the game.") diff --git a/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid b/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid new file mode 100644 index 00000000..02c9380e --- /dev/null +++ b/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid @@ -0,0 +1 @@ +uid://dpc1yt8at1sty diff --git a/hoppys-adventure-arcade/tests/booth_heartbeat.gd b/hoppys-adventure-arcade/tests/booth_heartbeat.gd new file mode 100644 index 00000000..a9d374d9 --- /dev/null +++ b/hoppys-adventure-arcade/tests/booth_heartbeat.gd @@ -0,0 +1,79 @@ +extends SceneTree +## Booth heartbeat (app/booth_heartbeat.gd): with --heartbeat= the game keeps that file +## updated with the current Unix time, including while a game has paused the tree; without the flag +## there is no heartbeat; a heartbeat it can't write warns instead of failing. The flag is checked +## by launching the game as a child process, so _ready()'s argument handling is covered. +## Run headless with --script res://tests/booth_heartbeat.gd. +const BoothHeartbeat := preload("res://app/booth_heartbeat.gd") +const BEAT := "user://test_heartbeat/beat" +const CHILD_BEAT := "user://test_heartbeat/child_beat" +var failures := 0 +var router: Node + +func _initialize() -> void: + call_deferred("run") + +func check(ok: bool, message: String) -> void: + print("PASS: " if ok else "FAIL: ", message) + if not ok: + failures += 1 + +func wait(seconds: float) -> void: + await create_timer(seconds, true).timeout + +## Launch the game headless with user args for two simulated seconds and return its exit code. +func launch(user_args: Array[String]) -> int: + var args: Array[String] = ["--headless", "--path", ProjectSettings.globalize_path("res://"), + "--fixed-fps", "60", "--quit-after", "120", "--"] + args.append_array(user_args) + var output: Array = [] + return OS.execute(OS.get_executable_path(), args, output, true) + +func read(path: String) -> String: + return FileAccess.get_file_as_string(path).strip_edges() + +## True when the file holds a Unix time within a minute of now. +func fresh(path: String) -> bool: + var text := read(path) + return text.is_valid_int() and absi(text.to_int() - int(Time.get_unix_time_from_system())) < 60 + +func run() -> void: + router = root.get_node("AppRouter") + await wait(0.3) + check(router.heartbeat == null, "No heartbeat without --heartbeat") + check(router.heartbeat_path(PackedStringArray(["--booth"])) == "", + "--booth alone has no heartbeat") + check(router.heartbeat_path(PackedStringArray(["--heartbeat=user://beat"])) == "user://beat", + "--heartbeat names the file") + check(router.heartbeat_path(PackedStringArray(["--heartbeat="])) == "", + "An empty --heartbeat is ignored") + + check(launch(["--booth", "--heartbeat=" + ProjectSettings.globalize_path(CHILD_BEAT)]) == 0, + "A launch with --heartbeat exits cleanly") + check(fresh(CHILD_BEAT), "--heartbeat makes the game write the current Unix time") + check(not FileAccess.file_exists(CHILD_BEAT + ".tmp"), "The heartbeat leaves no temporary file") + + var beat := BoothHeartbeat.new(BEAT) + root.add_child(beat) + check(fresh(BEAT), "The heartbeat writes as soon as it starts") + var marker := FileAccess.open(BEAT, FileAccess.WRITE) + marker.store_line("0") + marker.close() + paused = true + await wait(beat.INTERVAL + 0.6) + check(fresh(BEAT), "The heartbeat keeps writing while a game has the tree paused") + paused = false + + # The heartbeat file itself where a folder should be: warn, keep running, never an error. + var broken := BoothHeartbeat.new(BEAT + "/nested") + root.add_child(broken) + check(broken.warned, "A heartbeat it can't write warns") + broken.beat() + check(broken.is_inside_tree(), "A failed heartbeat leaves the game running") + + beat.queue_free() + broken.queue_free() + await process_frame + await root.get_node("AudioService").settle() + print("Booth heartbeat failures: ", failures) + quit(1 if failures else 0) diff --git a/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid b/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid new file mode 100644 index 00000000..81b167c6 --- /dev/null +++ b/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid @@ -0,0 +1 @@ +uid://bkg1o2uubr6lh diff --git a/tools/booth/macos/setup.sh b/tools/booth/macos/setup.sh new file mode 100755 index 00000000..f8dff18d --- /dev/null +++ b/tools/booth/macos/setup.sh @@ -0,0 +1,226 @@ +#!/usr/bin/env bash +# Set up a Mac to run Hoppy's Adventure Arcade as an unattended booth kiosk, and put it back after +# the show. Staff steps are in docs/booth-runbook.md. +# +# Usage: +# tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" [--route ] +# Print the plan. Nothing changes. +# tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" [--route ] --apply +# Make the changes. Asks for an administrator password for the system settings. +# tools/booth/macos/setup.sh --undo [--apply] +# Print (or, with --apply, make) the changes that put the Mac back. +# +# --apply does this: +# - copies tools/booth/watchdog.sh to ~/Library/Application Support/HoppyBooth/ +# - installs a LaunchAgent that starts the watchdog, and through it the game in booth mode, at +# login, and restarts the watchdog if it dies; it starts the game right away +# - removes the download quarantine from the app, so Gatekeeper doesn't block it +# - turns off system sleep, display sleep, disk sleep and Power Nap, and restarts after a power cut +# - turns off the screen saver, automatic macOS and App Store updates, and crash report dialogs +# Settings macOS doesn't let a script change are listed at the end (automatic login, notifications, +# display, volume, controllers). + +label="ai.coderabbit.arcade.booth" +support="$HOME/Library/Application Support/HoppyBooth" +logs="$HOME/Library/Logs/HoppyBooth" +agent="$HOME/Library/LaunchAgents/$label.plist" +here="$(cd "$(dirname "$0")" && pwd)" +app="" +route="" +apply=0 +undo=0 + +usage() { + sed -n '2,/^$/p' "$0" | sed -e 's/^# \{0,1\}//' +} + +die() { + echo "setup: $*" >&2 + exit 2 +} + +while [ $# -gt 0 ]; do + case "$1" in + --app | --route) + [ $# -ge 2 ] || die "$1 needs a value" + if [ "$1" = "--app" ]; then app="${2%/}"; else route="$2"; fi + shift 2 + ;; + --apply) apply=1; shift ;; + --undo) undo=1; shift ;; + -h | --help) usage; exit 0 ;; + *) die "unknown option '$1'" ;; + esac +done + +if [ "$apply" -eq 1 ] && [ "$(uname -s)" != "Darwin" ]; then + die "--apply only runs on macOS" +fi + +# Print each command; run it only with --apply. A failed command is reported and the rest go on. +failures=0 +run() { + local line=" \$" arg + for arg in "$@"; do + case "$arg" in + *[[:space:]\'\"\&]*) line="$line \"$arg\"" ;; + *) line="$line $arg" ;; + esac + done + echo "$line" + [ "$apply" -eq 1 ] || return 0 + if ! "$@"; then + echo " ^ failed; the remaining steps continue" >&2 + failures=$((failures + 1)) + fi +} + +step() { + echo + echo "$*" +} + +xml() { + printf '%s' "$1" | sed -e 's/&/\&/g' -e 's//\>/g' +} + +# The app's executable, from its Info.plist. +executable() { + local plist="$1/Contents/Info.plist" name="" + [ -f "$plist" ] || return 0 + if [ -x /usr/libexec/PlistBuddy ]; then + name="$(/usr/libexec/PlistBuddy -c 'Print :CFBundleExecutable' "$plist" 2>/dev/null)" + else + name="$(sed -n '/CFBundleExecutable<\/key>/{n;s/.*\(.*\)<\/string>.*/\1/p;}' "$plist")" + fi + [ -n "$name" ] && printf '%s' "$1/Contents/MacOS/$name" +} + +if [ "$apply" -eq 1 ]; then + echo "Applying. Commands run as they are listed." +else + echo "Plan only: nothing will change. Run again with --apply to make these changes." +fi + +if [ "$undo" -eq 1 ]; then + step "Stop the booth game and remove its login item:" + run launchctl bootout "gui/$(id -u)/$label" + run rm -f "$agent" + step "Restore power, screen saver, update and crash dialog settings:" + run sudo pmset restoredefaults + run defaults -currentHost delete com.apple.screensaver idleTime + for key in AutomaticCheckEnabled AutomaticDownload AutomaticallyInstallMacOSUpdates CriticalUpdateInstall ConfigDataInstall; do + run sudo defaults delete /Library/Preferences/com.apple.SoftwareUpdate "$key" + done + run sudo defaults delete /Library/Preferences/com.apple.commerce AutoUpdate + run defaults delete com.apple.CrashReporter DialogType + step "Left in place: the logs in $logs and the session logs under the game's user data." + echo "Turn automatic login and Do Not Disturb back off by hand if you turned them on." + if [ "$failures" -gt 0 ]; then + echo + echo "$failures step(s) failed (a setting that was never changed fails to delete, which is fine)." + fi + exit 0 +fi + +[ -n "$app" ] || die "--app is required: the path to Hoppy’s Adventure Arcade.app" +binary="$(executable "$app")" +if [ -z "$binary" ]; then + [ "$apply" -eq 0 ] || die "cannot read the app's executable from $app/Contents/Info.plist" + binary="$app/Contents/MacOS/" + echo "(Can't read $app here; the plan shows a placeholder for its executable.)" +fi +[ "$apply" -eq 0 ] || [ -x "$binary" ] || die "$binary is missing or not executable" + +game_args=(-- --booth) +[ -z "$route" ] || game_args+=("--route=$route") +watchdog=("/bin/bash" "$support/watchdog.sh" --log "$logs/watchdog.log" --heartbeat "$support/heartbeat" + --stop-file "$support/stop" -- "$binary" "${game_args[@]}") +plist_arguments="" +for arg in "${watchdog[@]}"; do + plist_arguments="$plist_arguments + $(xml "$arg")" +done +plist=" + + + + Label + $label + ProgramArguments + $plist_arguments + + RunAtLoad + + KeepAlive + + SuccessfulExit + + + LimitLoadToSessionType + Aqua + ProcessType + Interactive + StandardOutPath + $(xml "$logs/game.log") + StandardErrorPath + $(xml "$logs/game.log") + +" + +if [ "$apply" -eq 1 ]; then + sudo -v || die "the system settings need an administrator password" +fi + +step "Install the watchdog:" +run mkdir -p "$support" "$logs" +run cp "$here/../watchdog.sh" "$support/watchdog.sh" +run chmod 755 "$support/watchdog.sh" + +step "Let Gatekeeper open the app (it is ad-hoc signed, not notarized):" +run xattr -dr com.apple.quarantine "$app" + +step "Keep the Mac awake, and start again after a power cut:" +run sudo pmset -a sleep 0 displaysleep 0 disksleep 0 powernap 0 +run sudo pmset -a autorestart 1 + +step "No screen saver, no automatic updates, no crash dialogs over the game:" +run defaults -currentHost write com.apple.screensaver idleTime -int 0 +for key in AutomaticCheckEnabled AutomaticDownload AutomaticallyInstallMacOSUpdates CriticalUpdateInstall ConfigDataInstall; do + run sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate "$key" -bool false +done +run sudo defaults write /Library/Preferences/com.apple.commerce AutoUpdate -bool false +run defaults write com.apple.CrashReporter DialogType -string none + +step "Start the game at login and keep it running ($agent):" +if [ "$apply" -eq 1 ]; then + if ! { mkdir -p "$(dirname "$agent")" && printf '%s\n' "$plist" > "$agent"; }; then + die "cannot write $agent" + fi + echo " wrote $agent" + launchctl bootout "gui/$(id -u)/$label" 2>/dev/null +else + echo " would write:" + printf '%s\n' "$plist" | sed 's/^/ /' +fi +run launchctl bootstrap "gui/$(id -u)" "$agent" + +step "Do these by hand (macOS offers no reliable command for them):" +cat <<'MANUAL' + 1. System Settings > Users & Groups > Automatically log in as: the booth account. + (FileVault must be off for automatic login.) + 2. Control Center > Focus: click Do Not Disturb so it stays on. Then in System Settings > + Notifications, turn off "Allow notifications when mirroring or sharing the display". + 3. System Settings > Displays: the booth resolution, and mirror or extend as the booth needs. + 4. Set the volume for the floor, and pair or plug in the controllers. + 5. Restart the Mac and check that the game starts on its own within a minute. +MANUAL + +if [ "$apply" -eq 1 ]; then + echo + if [ "$failures" -gt 0 ]; then + echo "$failures step(s) failed; see the messages above." + exit 1 + fi + echo "Done. The watchdog's log is $logs/watchdog.log." +fi diff --git a/tools/booth/test_macos_setup.py b/tools/booth/test_macos_setup.py new file mode 100644 index 00000000..4fc47f71 --- /dev/null +++ b/tools/booth/test_macos_setup.py @@ -0,0 +1,123 @@ +"""Tests for macos/setup.sh. Run: python3 -m unittest discover -s tools/booth -v + +Only the plan (the default, which changes nothing) and the argument checks run here; --apply +changes system settings, so these tests never pass it on a Mac. +""" +from __future__ import annotations + +import os +import plistlib +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SETUP = Path(__file__).resolve().parent / "macos" / "setup.sh" +INFO_PLIST = """ + + +\tCFBundleExecutable +\tFake Arcade + + +""" + + +class MacSetupTest(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.dir = Path(folder.name) + self.home = self.dir / "home" + self.home.mkdir() + self.app = self.dir / "Fake & Co’s Arcade.app" + (self.app / "Contents" / "MacOS").mkdir(parents=True) + (self.app / "Contents" / "Info.plist").write_text(INFO_PLIST, encoding="utf-8") + + def setup(self, *args: str) -> subprocess.CompletedProcess[str]: + env = dict(os.environ, HOME=str(self.home)) + return subprocess.run(["bash", str(SETUP), *args], capture_output=True, text=True, env=env, timeout=30) + + def agent(self, output: str) -> dict: + """The LaunchAgent the plan would write.""" + lines = output.splitlines() + start = lines.index(" would write:") + 1 + end = next(i for i in range(start, len(lines)) if lines[i].startswith(" $ launchctl bootstrap")) + return plistlib.loads("\n".join(line[4:] for line in lines[start:end]).encode("utf-8")) + + def test_plan_lists_every_change_and_touches_nothing(self) -> None: + result = self.setup("--app", str(self.app), "--route", "heroes_vs_villains") + self.assertEqual(result.returncode, 0, result.stderr) + out = result.stdout + self.assertTrue(out.startswith("Plan only: nothing will change.")) + for command in ("sudo pmset -a sleep 0 displaysleep 0 disksleep 0 powernap 0", + "sudo pmset -a autorestart 1", + "defaults -currentHost write com.apple.screensaver idleTime -int 0", + "AutomaticallyInstallMacOSUpdates -bool false", + "defaults write com.apple.CrashReporter DialogType -string none", + f'xattr -dr com.apple.quarantine "{self.app}"', + "launchctl bootstrap gui/"): + self.assertIn(command, out) + self.assertIn("Automatically log in as", out) + self.assertEqual(list(self.home.iterdir()), [], "the plan must not write anything") + + def test_agent_runs_the_watchdog_and_the_game_in_booth_mode(self) -> None: + agent = self.agent(self.setup("--app", str(self.app), "--route", "heroes_vs_villains").stdout) + support = f"{self.home}/Library/Application Support/HoppyBooth" + self.assertEqual(agent["Label"], "ai.coderabbit.arcade.booth") + self.assertEqual(agent["ProgramArguments"], [ + "/bin/bash", f"{support}/watchdog.sh", + "--log", f"{self.home}/Library/Logs/HoppyBooth/watchdog.log", + "--heartbeat", f"{support}/heartbeat", + "--stop-file", f"{support}/stop", + "--", f"{self.app}/Contents/MacOS/Fake Arcade", "--", "--booth", "--route=heroes_vs_villains", + ]) + self.assertTrue(agent["RunAtLoad"]) + self.assertEqual(agent["KeepAlive"], {"SuccessfulExit": False}) + self.assertEqual(agent["LimitLoadToSessionType"], "Aqua") + + def test_route_is_optional(self) -> None: + agent = self.agent(self.setup("--app", str(self.app)).stdout) + self.assertEqual(agent["ProgramArguments"][-2:], ["--", "--booth"]) + + def test_unreadable_app_gets_a_placeholder_in_the_plan(self) -> None: + result = self.setup("--app", str(self.dir / "Missing.app")) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("placeholder", result.stdout) + self.assertIn(f"{self.dir}/Missing.app/Contents/MacOS/", self.agent(result.stdout)["ProgramArguments"]) + + def test_undo_plan_removes_the_agent_and_restores_settings(self) -> None: + result = self.setup("--undo") + self.assertEqual(result.returncode, 0, result.stderr) + for command in ("launchctl bootout gui/", "rm -f", "sudo pmset restoredefaults", + "defaults -currentHost delete com.apple.screensaver idleTime", + "defaults delete com.apple.CrashReporter DialogType"): + self.assertIn(command, result.stdout) + self.assertEqual(list(self.home.iterdir()), []) + + def test_the_installed_watchdog_comes_from_this_folder(self) -> None: + out = self.setup("--app", str(self.app)).stdout + line = next(line for line in out.splitlines() if line.startswith(" $ cp ")) + source = Path(line.split()[2]) + self.assertTrue(source.is_file(), source) + self.assertEqual(source.resolve(), (SETUP.parent.parent / "watchdog.sh").resolve()) + + def test_rejects_bad_usage(self) -> None: + for args, message in ((["--route", "hub"], "--app is required"), + (["--bogus"], "unknown option"), + (["--app"], "needs a value")): + result = self.setup(*args) + self.assertEqual(result.returncode, 2, args) + self.assertIn(message, result.stderr, args) + + @unittest.skipIf(sys.platform == "darwin", "--apply would change this Mac's settings") + def test_apply_refuses_off_macos(self) -> None: + result = self.setup("--app", str(self.app), "--apply") + self.assertEqual(result.returncode, 2) + self.assertIn("only runs on macOS", result.stderr) + self.assertEqual(list(self.home.iterdir()), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/booth/test_watchdog.py b/tools/booth/test_watchdog.py new file mode 100644 index 00000000..a9aa08b5 --- /dev/null +++ b/tools/booth/test_watchdog.py @@ -0,0 +1,169 @@ +"""Tests for watchdog.sh, run against a stand-in game. Run: python3 -m unittest discover -s tools/booth -v + +Each test starts the real script under bash with short timings and a fake game that exits, hangs +or stays healthy, then checks what the watchdog did. The waits are generous for slow CI runners; +the assertions count launches with lower bounds. +""" +from __future__ import annotations + +import os +import signal +import subprocess +import tempfile +import time +import unittest +from pathlib import Path + +WATCHDOG = Path(__file__).resolve().parent / "watchdog.sh" + +# Records each launch's arguments and pid, then behaves as FAKE_MODE says. +FAKE_GAME = r"""#!/usr/bin/env bash +echo "$*" >> "$FAKE_RUNS" +echo $$ > "$FAKE_PID" +heartbeat="" +for arg in "$@"; do + case "$arg" in --heartbeat=*) heartbeat="${arg#--heartbeat=}" ;; esac +done +case "$FAKE_MODE" in + exit) exit 3 ;; + hang) date +%s > "$heartbeat"; exec sleep 1000 ;; + healthy) while :; do [ -z "$heartbeat" ] || date +%s > "$heartbeat"; sleep 0.2; done ;; +esac +""" + + +class WatchdogTest(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.dir = Path(folder.name) + self.game = self.dir / "fake-game" + self.game.write_text(FAKE_GAME, encoding="utf-8") + self.game.chmod(0o755) + self.runs = self.dir / "runs" + self.pid = self.dir / "pid" + self.log = self.dir / "watchdog.log" + self.heartbeat = self.dir / "beat" / "heartbeat" + self.process: subprocess.Popen[bytes] | None = None + + def tearDown(self) -> None: + if self.process and self.process.poll() is None: + self.process.kill() + self.process.wait() + if self.pid.exists(): + try: + os.kill(int(self.pid.read_text()), signal.SIGKILL) + except (ProcessLookupError, ValueError): + pass + + def start(self, mode: str, *options: str, game_args: tuple[str, ...] = ("--", "--booth")) -> None: + env = dict(os.environ, FAKE_RUNS=str(self.runs), FAKE_PID=str(self.pid), FAKE_MODE=mode) + command = ["bash", str(WATCHDOG), "--log", str(self.log), "--heartbeat", str(self.heartbeat), + "--poll", "0.2", *options, "--", str(self.game), *game_args] + # The watchdog writes its events to --log; the fake game's own output is discarded. + self.process = subprocess.Popen(command, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + def stop(self) -> int: + assert self.process is not None + self.process.send_signal(signal.SIGTERM) + return self.process.wait(timeout=20) + + def launches(self) -> list[str]: + return self.runs.read_text(encoding="utf-8").splitlines() if self.runs.exists() else [] + + def events(self) -> str: + return self.log.read_text(encoding="utf-8") if self.log.exists() else "" + + def wait_for(self, condition, seconds: float) -> bool: + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + if condition(): + return True + time.sleep(0.1) + return condition() + + def test_relaunches_a_game_that_exits(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1", "--max-backoff", "1") + self.assertTrue(self.wait_for(lambda: len(self.launches()) >= 3, 15), self.events()) + self.assertEqual(self.stop(), 0) + self.assertIn("the game exited after", self.events()) + self.assertIn("(status 3)", self.events()) + self.assertIn("relaunching in 1 s", self.events()) + + def test_waits_longer_after_each_quick_exit(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1", "--max-backoff", "2") + self.assertTrue(self.wait_for(lambda: "relaunching in 2 s" in self.events(), 15), self.events()) + self.assertTrue(self.wait_for(lambda: self.events().count("relaunching in 2 s") >= 2, 15), self.events()) + self.stop() + self.assertNotIn("relaunching in 4 s", self.events(), "the wait is capped at --max-backoff") + + def test_restarts_a_hung_game(self) -> None: + self.start("hang", "--hang", "1", "--grace", "0", "--backoff", "1", "--term-wait", "2") + self.assertTrue(self.wait_for(lambda: len(self.launches()) >= 2, 20), self.events()) + self.stop() + self.assertIn("has not written a heartbeat for", self.events()) + self.assertIn("(signal 15)", self.events()) + + def test_leaves_a_healthy_game_alone_and_stops_it_on_sigterm(self) -> None: + self.start("healthy", "--hang", "2", "--grace", "0") + time.sleep(5) + self.assertEqual(len(self.launches()), 1, self.events()) + game = int(self.pid.read_text()) + self.assertEqual(self.stop(), 0) + self.assertIn("stopping the game", self.events()) + with self.assertRaises(ProcessLookupError): + os.kill(game, 0) + + def test_gives_the_game_its_heartbeat_after_the_separator(self) -> None: + self.start("exit", "--backoff", "1") + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], f"-- --booth --heartbeat={self.heartbeat}") + + def test_adds_a_separator_when_the_game_has_none(self) -> None: + self.start("exit", "--backoff", "1", game_args=()) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], f"-- --heartbeat={self.heartbeat}") + + def test_hang_zero_passes_no_heartbeat(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1") + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], "-- --booth") + + def test_stop_file_ends_the_watch_when_the_game_exits(self) -> None: + stop_file = self.dir / "stop" + stop_file.write_text("left from yesterday", encoding="utf-8") + self.start("healthy", "--hang", "0", "--stop-file", str(stop_file)) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.assertIn("removed the stop file left from before", self.events()) + stop_file.touch() + os.kill(int(self.pid.read_text()), signal.SIGTERM) # staff close the game + assert self.process is not None + self.assertEqual(self.process.wait(timeout=10), 0) + self.assertIn("not relaunching", self.events()) + self.assertEqual(len(self.launches()), 1) + + def test_rejects_bad_usage(self) -> None: + for args, message in ( + ([], "no game command"), + (["--bogus", "--", "true"], "unknown option"), + (["--hang", "soon", "--", "true"], "whole number"), + (["--backoff", "0", "--", "true"], "at least 1"), + (["--poll", "0", "--", "true"], "more than 0"), + (["--log"], "needs a value"), + ): + result = subprocess.run(["bash", str(WATCHDOG), *args], capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 2, args) + self.assertIn(message, result.stderr, args) + + def test_help_prints_the_usage(self) -> None: + result = subprocess.run(["bash", str(WATCHDOG), "--help"], capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 0) + self.assertIn("Usage:", result.stdout) + self.assertIn("--stop-file", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/booth/watchdog.sh b/tools/booth/watchdog.sh new file mode 100755 index 00000000..8c971d3d --- /dev/null +++ b/tools/booth/watchdog.sh @@ -0,0 +1,206 @@ +#!/usr/bin/env bash +# Keep the booth game running unattended. The watchdog launches the game, relaunches it whenever it +# exits (a crash, a forced quit, Cmd+Q), and kills and relaunches it when it hangs. The game +# rewrites a heartbeat file every two seconds (app/booth_heartbeat.gd); when the file stops +# changing, the main loop is stuck. Runs on macOS's bash 3.2 and on Linux. +# +# Usage: +# tools/booth/watchdog.sh [options] -- [game arguments] +# +# Example (macOS; tools/booth/macos/setup.sh installs this as a login item): +# tools/booth/watchdog.sh --log ~/booth-watchdog.log -- \ +# "/Applications/Hoppy’s Adventure Arcade.app/Contents/MacOS/" -- --booth +# +# Options: +# --heartbeat heartbeat file for the game (default: $TMPDIR/hoppy-booth-heartbeat) +# --hang restart the game when its heartbeat is older than this; 0 turns hang +# checks off, for builds without --heartbeat (default: 45) +# --grace how long a fresh launch has to write its first heartbeat (default: 60) +# --stop-file if this file exists when the game exits, stop instead of relaunching +# (exit 0); a stop file left from before is removed at startup +# --log append the watchdog's events here instead of standard output +# --backoff first wait before a relaunch; doubles after each short run (default: 2) +# --max-backoff longest wait before a relaunch (default: 30) +# --stable a run at least this long resets the wait to --backoff (default: 60) +# --term-wait how long the game gets to exit after SIGTERM before SIGKILL (default: 10) +# --poll how often to check the game; may be fractional (default: 1) +# +# The game gets --heartbeat= after its other user arguments (a "--" is added if missing). +# SIGTERM, SIGINT or SIGHUP stops the game, then the watchdog, with exit status 0. + +heartbeat="${TMPDIR:-/tmp}" +heartbeat="${heartbeat%/}/hoppy-booth-heartbeat" +hang=45 +grace=60 +stop_file="" +log_file="" +backoff=2 +max_backoff=30 +stable=60 +term_wait=10 +poll=1 + +usage() { + sed -n '2,/^$/p' "$0" | sed -e 's/^# \{0,1\}//' +} + +die() { + echo "watchdog: $*" >&2 + exit 2 +} + +whole_number() { + case "$2" in + '' | *[!0-9]*) die "$1 needs a whole number of seconds, not '$2'" ;; + esac +} + +while [ $# -gt 0 ]; do + case "$1" in + --heartbeat | --hang | --grace | --stop-file | --log | --backoff | --max-backoff | --stable | --term-wait | --poll) + [ $# -ge 2 ] || die "$1 needs a value" + case "$1" in + --heartbeat) heartbeat="$2" ;; + --hang) whole_number "$1" "$2"; hang="$2" ;; + --grace) whole_number "$1" "$2"; grace="$2" ;; + --stop-file) stop_file="$2" ;; + --log) log_file="$2" ;; + --backoff) + whole_number "$1" "$2" + [ "$2" -ge 1 ] || die "--backoff must be at least 1 second" + backoff="$2" + ;; + --max-backoff) whole_number "$1" "$2"; max_backoff="$2" ;; + --stable) whole_number "$1" "$2"; stable="$2" ;; + --term-wait) whole_number "$1" "$2"; term_wait="$2" ;; + --poll) + case "$2" in + '' | *[!0-9.]* | *.*.* | .) die "--poll needs a number of seconds, not '$2'" ;; + esac + awk -v p="$2" 'BEGIN { exit !(p > 0) }' || die "--poll must be more than 0 seconds" + poll="$2" + ;; + esac + shift 2 + ;; + -h | --help) usage; exit 0 ;; + --) shift; break ;; + *) die "unknown option '$1' (the game command goes after --)" ;; + esac +done +[ $# -gt 0 ] || die "no game command; pass it after --" +[ "$max_backoff" -ge "$backoff" ] || max_backoff="$backoff" + +game=("$@") +if [ "$hang" -gt 0 ]; then + separator=0 + for arg in "$@"; do + [ "$arg" = "--" ] && separator=1 + done + [ "$separator" -eq 1 ] || game+=("--") + game+=("--heartbeat=$heartbeat") +fi + +say() { + local line + line="$(date '+%Y-%m-%dT%H:%M:%S') watchdog: $*" + if [ -n "$log_file" ]; then + echo "$line" >> "$log_file" + else + echo "$line" + fi +} + +child="" + +# SIGTERM, then SIGKILL if the game is still there after --term-wait seconds. The caller reaps it. +terminate() { + local waited=0 + kill -TERM "$1" 2>/dev/null + while kill -0 "$1" 2>/dev/null && [ "$waited" -lt "$term_wait" ]; do + sleep 1 + waited=$((waited + 1)) + done + if kill -0 "$1" 2>/dev/null; then + say "the game (pid $1) ignored SIGTERM for $term_wait s; sending SIGKILL" + kill -KILL "$1" 2>/dev/null + fi +} + +stop() { + trap - TERM INT HUP + if [ -n "$child" ] && kill -0 "$child" 2>/dev/null; then + say "stopping the game (pid $child)" + terminate "$child" + wait "$child" 2>/dev/null + fi + say "stopped" + exit 0 +} +trap stop TERM INT HUP + +# Sleep in one-second steps, so a stop signal is handled within a second. +pause() { + local slept=0 + while [ "$slept" -lt "$1" ]; do + sleep 1 + slept=$((slept + 1)) + done +} + +[ -z "$log_file" ] || mkdir -p "$(dirname "$log_file")" || die "cannot create the folder for $log_file" +if [ "$hang" -gt 0 ]; then + mkdir -p "$(dirname "$heartbeat")" || die "cannot create the folder for $heartbeat" +fi +if [ -n "$stop_file" ] && [ -e "$stop_file" ]; then + rm -f "$stop_file" + say "removed the stop file left from before ($stop_file)" +fi +if [ "$hang" -gt 0 ]; then + say "watching ${game[*]} (restart after $hang s without a heartbeat)" +else + say "watching ${game[*]} (hang checks off)" +fi + +wait_before_relaunch="$backoff" +while :; do + rm -f "$heartbeat" + "${game[@]}" & + child=$! + launched=$(date +%s) + say "launched the game (pid $child)" + while kill -0 "$child" 2>/dev/null; do + sleep "$poll" + [ "$hang" -gt 0 ] || continue + now=$(date +%s) + [ $((now - launched)) -ge "$grace" ] || continue + beat=$(cat "$heartbeat" 2>/dev/null) + case "$beat" in + '' | *[!0-9]*) last="$launched" ;; + *) last="$beat" ;; + esac + if [ $((now - last)) -gt "$hang" ]; then + say "the game has not written a heartbeat for $((now - last)) s; restarting it" + terminate "$child" + break + fi + done + wait "$child" 2>/dev/null + status=$? + child="" + ran=$(($(date +%s) - launched)) + if [ "$status" -gt 128 ]; then + say "the game exited after $ran s (signal $((status - 128)))" + else + say "the game exited after $ran s (status $status)" + fi + if [ -n "$stop_file" ] && [ -e "$stop_file" ]; then + say "stop file $stop_file is present; not relaunching" + exit 0 + fi + [ "$ran" -lt "$stable" ] || wait_before_relaunch="$backoff" + say "relaunching in $wait_before_relaunch s" + pause "$wait_before_relaunch" + wait_before_relaunch=$((wait_before_relaunch * 2)) + [ "$wait_before_relaunch" -le "$max_backoff" ] || wait_before_relaunch="$max_backoff" +done From 6321db65f6d92939101d909c44c0dc6eb6e98e70 Mon Sep 17 00:00:00 2001 From: Erik Thorelli Date: Thu, 15 Oct 2026 17:25:54 -0700 Subject: [PATCH 2/3] fix(booth): harden the watchdog and make the Mac undo exact [Agent: Claude Code | Model: Opus 5.5 | Thinking: High | Acting on behalf of: Erik | Skill: none] Addresses CodeRabbit review 5541466824 on #76. watchdog.sh: - The signal handler only sets a flag, checked right after the launch and in every wait, so a signal between `&` and `child=$!` still stops the game. - Without --heartbeat, the heartbeat lives in a new mktemp -d folder (mode 700) removed at exit: nobody can plant a link in it, and two watchdogs never read each other's file. - A game command that sets --heartbeat= is refused; numbers are read as base 10 ("08"); a stop file that can't be removed at startup stops the watchdog instead of being logged as removed. setup.sh: - --apply records each preference and power setting it changes, once, in settings-before.tsv. --undo restores exactly those values (absent keys deleted, others rewritten with their type), instead of pmset restoredefaults and blanket deletes. Without the record it changes no setting, lists what to restore by hand and exits 1; any failed restore exits 1 and keeps the record. - A relative --app becomes absolute, since launchd doesn't run from this folder. Extra settings share one table for apply, record, undo. The game removes a leftover temporary heartbeat before writing, so a planted link is replaced rather than written through. Tests: --apply and --undo --apply run against logging stand-ins for sudo, pmset, defaults, launchctl, xattr and uname (Linux only, so a missed stand-in can never touch a Mac); the private heartbeat folder; leading zeros; a refused --heartbeat; atomic fake heartbeats; the stop file's removal; shlex parsing; a relative --app. The heartbeat suite uses its own emptied folder, bounds the child launch with a wall-clock deadline, and checks a planted link is left untouched. --- docs/booth-runbook.md | 2 +- .../app/booth_heartbeat.gd | 7 +- .../tests/booth_heartbeat.gd | 52 ++++-- tools/booth/macos/setup.sh | 148 ++++++++++++++++-- tools/booth/test_macos_setup.py | 131 +++++++++++++++- tools/booth/test_watchdog.py | 44 +++++- tools/booth/watchdog.sh | 61 ++++++-- 7 files changed, 384 insertions(+), 61 deletions(-) diff --git a/docs/booth-runbook.md b/docs/booth-runbook.md index 94ae4790..4f405890 100644 --- a/docs/booth-runbook.md +++ b/docs/booth-runbook.md @@ -94,4 +94,4 @@ tools/booth/macos/setup.sh --undo # see what it will change tools/booth/macos/setup.sh --undo --apply ``` -Then turn off automatic login and Do Not Disturb, and copy the session logs. +`--undo` puts back the settings the first setup recorded on that Mac. Then turn off automatic login and Do Not Disturb, and copy the session logs. diff --git a/hoppys-adventure-arcade/app/booth_heartbeat.gd b/hoppys-adventure-arcade/app/booth_heartbeat.gd index 25cb0026..9fa6d89e 100644 --- a/hoppys-adventure-arcade/app/booth_heartbeat.gd +++ b/hoppys-adventure-arcade/app/booth_heartbeat.gd @@ -29,9 +29,14 @@ func _process(delta: float) -> void: beat() ## Write to a temporary file and rename it over the heartbeat, so the watchdog never reads a -## half-written number. +## half-written number. A leftover temporary file is removed first, so a link planted in its place +## is replaced rather than written through. The watchdog keeps the heartbeat in a private folder. func beat() -> void: var temporary := path + ".tmp" + var folder := DirAccess.open(path.get_base_dir()) + var leftover := temporary.get_file() + if folder != null and (folder.file_exists(leftover) or folder.is_link(leftover)): + folder.remove(leftover) var file := FileAccess.open(temporary, FileAccess.WRITE) if file == null: _fail(error_string(FileAccess.get_open_error())) diff --git a/hoppys-adventure-arcade/tests/booth_heartbeat.gd b/hoppys-adventure-arcade/tests/booth_heartbeat.gd index a9d374d9..688c5e04 100644 --- a/hoppys-adventure-arcade/tests/booth_heartbeat.gd +++ b/hoppys-adventure-arcade/tests/booth_heartbeat.gd @@ -1,12 +1,16 @@ extends SceneTree ## Booth heartbeat (app/booth_heartbeat.gd): with --heartbeat= the game keeps that file ## updated with the current Unix time, including while a game has paused the tree; without the flag -## there is no heartbeat; a heartbeat it can't write warns instead of failing. The flag is checked -## by launching the game as a child process, so _ready()'s argument handling is covered. -## Run headless with --script res://tests/booth_heartbeat.gd. +## there is no heartbeat; a link planted where the temporary file goes is replaced, not written +## through; a heartbeat it can't write warns instead of failing. The flag is checked by launching +## the game as a child process, so _ready()'s argument handling is covered. Every run starts from an +## empty folder of its own. Run headless with --script res://tests/booth_heartbeat.gd. const BoothHeartbeat := preload("res://app/booth_heartbeat.gd") -const BEAT := "user://test_heartbeat/beat" -const CHILD_BEAT := "user://test_heartbeat/child_beat" +const FOLDER := "user://test_heartbeat" +const BEAT := FOLDER + "/beat" +const CHILD_BEAT := FOLDER + "/child_beat" +## Wall-clock limit for the child launch, for runs outside tools/tests/run_tests.py's timeout. +const CHILD_SECONDS := 60.0 var failures := 0 var router: Node @@ -21,13 +25,20 @@ func check(ok: bool, message: String) -> void: func wait(seconds: float) -> void: await create_timer(seconds, true).timeout -## Launch the game headless with user args for two simulated seconds and return its exit code. +## Launch the game headless with user args for two simulated seconds and return its exit code, +## or -1 if it outlives CHILD_SECONDS and had to be killed. func launch(user_args: Array[String]) -> int: var args: Array[String] = ["--headless", "--path", ProjectSettings.globalize_path("res://"), "--fixed-fps", "60", "--quit-after", "120", "--"] args.append_array(user_args) - var output: Array = [] - return OS.execute(OS.get_executable_path(), args, output, true) + var pid := OS.create_process(OS.get_executable_path(), args) + var deadline := Time.get_ticks_msec() + int(CHILD_SECONDS * 1000.0) + while OS.is_process_running(pid): + if Time.get_ticks_msec() > deadline: + OS.kill(pid) + return -1 + await wait(0.1) + return OS.get_process_exit_code(pid) func read(path: String) -> String: return FileAccess.get_file_as_string(path).strip_edges() @@ -37,8 +48,19 @@ func fresh(path: String) -> bool: var text := read(path) return text.is_valid_int() and absi(text.to_int() - int(Time.get_unix_time_from_system())) < 60 +func clean() -> void: + var dir := DirAccess.open(FOLDER) + if dir == null: + return + dir.include_hidden = true + for file_name in dir.get_files(): + dir.remove(file_name) + DirAccess.remove_absolute(FOLDER) + func run() -> void: router = root.get_node("AppRouter") + clean() + DirAccess.make_dir_recursive_absolute(FOLDER) await wait(0.3) check(router.heartbeat == null, "No heartbeat without --heartbeat") check(router.heartbeat_path(PackedStringArray(["--booth"])) == "", @@ -48,14 +70,23 @@ func run() -> void: check(router.heartbeat_path(PackedStringArray(["--heartbeat="])) == "", "An empty --heartbeat is ignored") - check(launch(["--booth", "--heartbeat=" + ProjectSettings.globalize_path(CHILD_BEAT)]) == 0, - "A launch with --heartbeat exits cleanly") + var code := await launch(["--booth", "--heartbeat=" + ProjectSettings.globalize_path(CHILD_BEAT)]) + check(code == 0, "A launch with --heartbeat exits cleanly (exit %d)" % code) check(fresh(CHILD_BEAT), "--heartbeat makes the game write the current Unix time") check(not FileAccess.file_exists(CHILD_BEAT + ".tmp"), "The heartbeat leaves no temporary file") + # A link where the temporary file goes: the heartbeat replaces it instead of writing through it. + var decoy := FOLDER + "/decoy" + var keep := FileAccess.open(decoy, FileAccess.WRITE) + keep.store_line("keep me") + keep.close() + var linked := DirAccess.open(FOLDER).create_link(ProjectSettings.globalize_path(decoy), + ProjectSettings.globalize_path(BEAT + ".tmp")) == OK var beat := BoothHeartbeat.new(BEAT) root.add_child(beat) check(fresh(BEAT), "The heartbeat writes as soon as it starts") + if linked: + check(read(decoy) == "keep me", "A link planted at the temporary path is not written through") var marker := FileAccess.open(BEAT, FileAccess.WRITE) marker.store_line("0") marker.close() @@ -74,6 +105,7 @@ func run() -> void: beat.queue_free() broken.queue_free() await process_frame + clean() await root.get_node("AudioService").settle() print("Booth heartbeat failures: ", failures) quit(1 if failures else 0) diff --git a/tools/booth/macos/setup.sh b/tools/booth/macos/setup.sh index f8dff18d..aaa75c4b 100755 --- a/tools/booth/macos/setup.sh +++ b/tools/booth/macos/setup.sh @@ -8,10 +8,13 @@ # tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" [--route ] --apply # Make the changes. Asks for an administrator password for the system settings. # tools/booth/macos/setup.sh --undo [--apply] -# Print (or, with --apply, make) the changes that put the Mac back. +# Print (or, with --apply, make) the changes that put the Mac back: the settings recorded +# before setup are restored exactly. Without that record it changes no system setting and +# lists what to restore by hand. # # --apply does this: -# - copies tools/booth/watchdog.sh to ~/Library/Application Support/HoppyBooth/ +# - copies tools/booth/watchdog.sh to ~/Library/Application Support/HoppyBooth/, and records the +# settings below as they were, for --undo (only the first time) # - installs a LaunchAgent that starts the watchdog, and through it the game in booth mode, at # login, and restarts the watchdog if it dies; it starts the game right away # - removes the download quarantine from the app, so Gatekeeper doesn't block it @@ -24,6 +27,21 @@ label="ai.coderabbit.arcade.booth" support="$HOME/Library/Application Support/HoppyBooth" logs="$HOME/Library/Logs/HoppyBooth" agent="$HOME/Library/LaunchAgents/$label.plist" +snapshot="$support/settings-before.tsv" +# The preferences the setup changes: scope, domain, key, type, booth value. "host" is the +# -currentHost domain; "system" is machine-wide and needs sudo to write. +SETTINGS=( + "host com.apple.screensaver idleTime int 0" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticCheckEnabled bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticallyInstallMacOSUpdates bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate CriticalUpdateInstall bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate ConfigDataInstall bool false" + "system /Library/Preferences/com.apple.commerce AutoUpdate bool false" + "user com.apple.CrashReporter DialogType string none" +) +# The power settings the setup changes. +PMSET_KEYS="sleep displaysleep disksleep powernap autorestart" here="$(cd "$(dirname "$0")" && pwd)" app="" route="" @@ -56,6 +74,11 @@ done if [ "$apply" -eq 1 ] && [ "$(uname -s)" != "Darwin" ]; then die "--apply only runs on macOS" fi +# launchd doesn't run from this folder, so the agent needs an absolute path to the app. +case "$app" in + "" | /*) ;; + *) app="$PWD/$app" ;; +esac # Print each command; run it only with --apply. A failed command is reported and the rest go on. failures=0 @@ -84,6 +107,52 @@ xml() { printf '%s' "$1" | sed -e 's/&/\&/g' -e 's//\>/g' } +# The `defaults` command line that reads (r) or writes (w) a setting in this scope. +reader() { + if [ "$1" = host ]; then r=(defaults -currentHost); else r=(defaults); fi +} +writer() { + case "$1" in + host) w=(defaults -currentHost) ;; + system) w=(sudo defaults) ;; + *) w=(defaults) ;; + esac +} + +# Write each setting as it is now to the record, once: re-running the setup keeps the first one. +record_settings() { + local entry scope domain key type value read_type + if [ -e "$snapshot" ]; then + echo " kept the record from the first setup" + return 0 + fi + { + for entry in "${SETTINGS[@]}"; do + read -r scope domain key type value <<<"$entry" + reader "$scope" + if value=$("${r[@]}" read "$domain" "$key" 2>/dev/null); then + read_type=$("${r[@]}" read-type "$domain" "$key" 2>/dev/null) + case "$read_type" in + *boolean) type=bool ;; + *integer) type=int ;; + *float) type=float ;; + *string) type=string ;; + *) type=unsupported ;; + esac + else + type=absent + value="" + fi + printf 'defaults\t%s\t%s\t%s\t%s\t%s\n' "$scope" "$domain" "$key" "$type" "$value" + done + for key in $PMSET_KEYS; do + value=$(pmset -g | awk -v k="$key" '$1 == k { print $2; exit }') + [ -z "$value" ] || printf 'pmset\t%s\t%s\n' "$key" "$value" + done + } > "$snapshot" || die "cannot write the settings record $snapshot" + echo " recorded in $snapshot" +} + # The app's executable, from its Info.plist. executable() { local plist="$1/Contents/Info.plist" name="" @@ -103,22 +172,61 @@ else fi if [ "$undo" -eq 1 ]; then + if [ "$apply" -eq 1 ] && [ -f "$snapshot" ]; then + sudo -v || die "the system settings need an administrator password" + fi step "Stop the booth game and remove its login item:" run launchctl bootout "gui/$(id -u)/$label" run rm -f "$agent" - step "Restore power, screen saver, update and crash dialog settings:" - run sudo pmset restoredefaults - run defaults -currentHost delete com.apple.screensaver idleTime - for key in AutomaticCheckEnabled AutomaticDownload AutomaticallyInstallMacOSUpdates CriticalUpdateInstall ConfigDataInstall; do - run sudo defaults delete /Library/Preferences/com.apple.SoftwareUpdate "$key" - done - run sudo defaults delete /Library/Preferences/com.apple.commerce AutoUpdate - run defaults delete com.apple.CrashReporter DialogType + unrecorded=0 + if [ -f "$snapshot" ]; then + step "Put back the settings recorded before setup ($snapshot):" + while IFS=$'\t' read -r kind a b c d e <&3; do + case "$kind" in + defaults) + writer "$a" + reader "$a" + case "$d" in + absent) + # Only delete what setup added; a key already gone needs nothing. + if [ "$apply" -eq 0 ] || "${r[@]}" read "$b" "$c" >/dev/null 2>&1; then + run "${w[@]}" delete "$b" "$c" + fi + ;; + bool) + case "$e" in 1 | [Tt]rue | YES | [Yy]es) e=true ;; *) e=false ;; esac + run "${w[@]}" write "$b" "$c" -bool "$e" + ;; + int | float | string) run "${w[@]}" write "$b" "$c" "-$d" "$e" ;; + *) + echo " restore $b $c by hand: it held a value this script can't rewrite" + unrecorded=1 + ;; + esac + ;; + pmset) run sudo pmset -a "$a" "$b" ;; + esac + done 3< "$snapshot" + [ "$failures" -gt 0 ] || run rm -f "$snapshot" + else + unrecorded=1 + step "No record of this Mac's settings before setup ($snapshot), so none are changed." + cat <<'BY_HAND' + Restore them by hand if the setup changed them: + - System Settings > Lock Screen: the display and sleep times you want. + - System Settings > Energy (or Battery): Power Nap and starting up after a power failure. + - System Settings > Screen Saver: the screen saver and when it starts. + - System Settings > General > Software Update > Automatic Updates: switch each one back on. + - In Terminal, to get crash dialogs back: defaults delete com.apple.CrashReporter DialogType +BY_HAND + fi step "Left in place: the logs in $logs and the session logs under the game's user data." echo "Turn automatic login and Do Not Disturb back off by hand if you turned them on." - if [ "$failures" -gt 0 ]; then + if [ "$apply" -eq 1 ] && { [ "$failures" -gt 0 ] || [ "$unrecorded" -eq 1 ]; }; then echo - echo "$failures step(s) failed (a setting that was never changed fails to delete, which is fine)." + [ "$failures" -eq 0 ] || echo "$failures step(s) failed; see the messages above." + [ "$unrecorded" -eq 0 ] || echo "Some settings need restoring by hand; see above." + exit 1 fi exit 0 fi @@ -177,6 +285,13 @@ run mkdir -p "$support" "$logs" run cp "$here/../watchdog.sh" "$support/watchdog.sh" run chmod 755 "$support/watchdog.sh" +step "Record the current settings, so --undo can put them back ($snapshot):" +if [ "$apply" -eq 1 ]; then + record_settings +else + echo " would record the preferences and power settings changed below, unless a record exists" +fi + step "Let Gatekeeper open the app (it is ad-hoc signed, not notarized):" run xattr -dr com.apple.quarantine "$app" @@ -185,12 +300,11 @@ run sudo pmset -a sleep 0 displaysleep 0 disksleep 0 powernap 0 run sudo pmset -a autorestart 1 step "No screen saver, no automatic updates, no crash dialogs over the game:" -run defaults -currentHost write com.apple.screensaver idleTime -int 0 -for key in AutomaticCheckEnabled AutomaticDownload AutomaticallyInstallMacOSUpdates CriticalUpdateInstall ConfigDataInstall; do - run sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate "$key" -bool false +for entry in "${SETTINGS[@]}"; do + read -r scope domain key type value <<<"$entry" + writer "$scope" + run "${w[@]}" write "$domain" "$key" "-$type" "$value" done -run sudo defaults write /Library/Preferences/com.apple.commerce AutoUpdate -bool false -run defaults write com.apple.CrashReporter DialogType -string none step "Start the game at login and keep it running ($agent):" if [ "$apply" -eq 1 ]; then diff --git a/tools/booth/test_macos_setup.py b/tools/booth/test_macos_setup.py index 4fc47f71..080d3dbf 100644 --- a/tools/booth/test_macos_setup.py +++ b/tools/booth/test_macos_setup.py @@ -1,12 +1,16 @@ """Tests for macos/setup.sh. Run: python3 -m unittest discover -s tools/booth -v -Only the plan (the default, which changes nothing) and the argument checks run here; --apply -changes system settings, so these tests never pass it on a Mac. +The plan (the default, which changes nothing) and the argument checks run everywhere. The --apply +tests replace every command that changes settings (sudo, pmset, defaults, launchctl, xattr) and +uname with logging stand-ins, and still run only off macOS, so a missed stand-in can never change +a real Mac. """ from __future__ import annotations import os import plistlib +import shlex +import shutil import subprocess import sys import tempfile @@ -24,6 +28,27 @@ """ +# Stand-ins: each logs its arguments. `defaults` keeps values as files under $STUB_STORE and fails +# writes to a key named in $STUB_FAIL; `pmset -g` reports fixed power settings. +STUBS = { + "uname": 'echo Darwin', + "sudo": '[ "$1" = -v ] && exit 0; exec "$@"', + "launchctl": 'echo "launchctl $*" >> "$STUB_LOG"', + "xattr": 'echo "xattr $*" >> "$STUB_LOG"', + "pmset": """echo "pmset $*" >> "$STUB_LOG" +if [ "$1" = -g ]; then printf ' sleep 10\n displaysleep 5\n autorestart 0\n'; fi""", + "defaults": """echo "defaults $*" >> "$STUB_LOG" +host=""; [ "$1" = -currentHost ] && { host="host-"; shift; } +verb="$1"; key_file="$STUB_STORE/$host$(printf '%s' "$2" | tr '/' '_')-$3" +case "$verb" in + read) [ -f "$key_file" ] && cut -d' ' -f2- "$key_file" ;; + read-type) [ -f "$key_file" ] && echo "Type is $(cut -d' ' -f1 "$key_file")" ;; + write) [ "$3" = "${STUB_FAIL:-}" ] && exit 1; echo "$4 $5" > "$key_file" ;; + delete) [ -f "$key_file" ] && rm "$key_file" ;; +esac""", +} + + class MacSetupTest(unittest.TestCase): def setUp(self) -> None: folder = tempfile.TemporaryDirectory() @@ -87,19 +112,40 @@ def test_unreadable_app_gets_a_placeholder_in_the_plan(self) -> None: self.assertIn("placeholder", result.stdout) self.assertIn(f"{self.dir}/Missing.app/Contents/MacOS/", self.agent(result.stdout)["ProgramArguments"]) - def test_undo_plan_removes_the_agent_and_restores_settings(self) -> None: + def test_undo_without_a_record_changes_no_setting(self) -> None: result = self.setup("--undo") self.assertEqual(result.returncode, 0, result.stderr) - for command in ("launchctl bootout gui/", "rm -f", "sudo pmset restoredefaults", - "defaults -currentHost delete com.apple.screensaver idleTime", - "defaults delete com.apple.CrashReporter DialogType"): - self.assertIn(command, result.stdout) + self.assertIn("launchctl bootout gui/", result.stdout) + self.assertIn("No record of this Mac's settings before setup", result.stdout) + self.assertNotIn("pmset", result.stdout.split("No record")[0]) + self.assertNotIn("restoredefaults", result.stdout) self.assertEqual(list(self.home.iterdir()), []) + def test_undo_plan_restores_the_recorded_settings(self) -> None: + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + record.parent.mkdir(parents=True) + record.write_text("defaults\thost\tcom.apple.screensaver\tidleTime\tint\t300\n" + "defaults\tsystem\t/Library/Preferences/com.apple.SoftwareUpdate\tAutomaticDownload\tbool\t1\n" + "defaults\tuser\tcom.apple.CrashReporter\tDialogType\tabsent\t\n" + "pmset\tsleep\t10\n", encoding="utf-8") + out = self.setup("--undo").stdout + for command in ("defaults -currentHost write com.apple.screensaver idleTime -int 300", + "sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload -bool true", + "defaults delete com.apple.CrashReporter DialogType", "sudo pmset -a sleep 10"): + self.assertIn(command, out) + self.assertTrue(record.exists(), "the plan must leave the record alone") + + def test_a_relative_app_becomes_an_absolute_path_for_launchd(self) -> None: + env = dict(os.environ, HOME=str(self.home)) + result = subprocess.run(["bash", str(SETUP), "--app", self.app.name], capture_output=True, text=True, + env=env, cwd=self.dir, timeout=30) + executable = self.agent(result.stdout)["ProgramArguments"][-3] + self.assertEqual(Path(executable), self.app.resolve() / "Contents" / "MacOS" / "Fake Arcade") + def test_the_installed_watchdog_comes_from_this_folder(self) -> None: out = self.setup("--app", str(self.app)).stdout line = next(line for line in out.splitlines() if line.startswith(" $ cp ")) - source = Path(line.split()[2]) + source = Path(shlex.split(line)[2]) self.assertTrue(source.is_file(), source) self.assertEqual(source.resolve(), (SETUP.parent.parent / "watchdog.sh").resolve()) @@ -111,6 +157,75 @@ def test_rejects_bad_usage(self) -> None: self.assertEqual(result.returncode, 2, args) self.assertIn(message, result.stderr, args) + def stubbed(self, *args: str, fail: str = "") -> tuple[subprocess.CompletedProcess[str], list[str]]: + """Run setup.sh with the stand-ins first on PATH; return the result and the commands they saw.""" + stubs, store, log = self.dir / "stubs", self.dir / "store", self.dir / "stub.log" + if not stubs.exists(): + stubs.mkdir() + store.mkdir() + for name, body in STUBS.items(): + (stubs / name).write_text(f"#!/usr/bin/env bash\n{body}\n", encoding="utf-8") + (stubs / name).chmod(0o755) + (self.app / "Contents" / "MacOS" / "Fake Arcade").write_text("", encoding="utf-8") + (self.app / "Contents" / "MacOS" / "Fake Arcade").chmod(0o755) + path = f"{stubs}{os.pathsep}{os.environ['PATH']}" + for name in STUBS: + self.assertEqual(shutil.which(name, path=path), str(stubs / name), f"{name} must be a stand-in") + log.write_text("", encoding="utf-8") + env = dict(os.environ, HOME=str(self.home), PATH=path, STUB_STORE=str(store), STUB_LOG=str(log), + STUB_FAIL=fail) + result = subprocess.run(["bash", str(SETUP), *args], capture_output=True, text=True, env=env, timeout=60) + return result, log.read_text(encoding="utf-8").splitlines() + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_apply_records_settings_and_undo_puts_them_back(self) -> None: + store = self.dir / "store" + result, _ = self.stubbed("--app", str(self.app), "--apply") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + agent = self.home / "Library" / "LaunchAgents" / "ai.coderabbit.arcade.booth.plist" + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + self.assertTrue(agent.is_file()) + rows = [line.split("\t") for line in record.read_text(encoding="utf-8").splitlines()] + self.assertIn(["defaults", "host", "com.apple.screensaver", "idleTime", "absent", ""], rows) + self.assertIn(["pmset", "sleep", "10"], rows) + self.assertEqual((store / "host-com.apple.screensaver-idleTime").read_text().split(), ["-int", "0"]) + + result, seen = self.stubbed("--undo", "--apply") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("defaults -currentHost delete com.apple.screensaver idleTime", seen) + self.assertIn("pmset -a sleep 10", seen) + self.assertNotIn("pmset restoredefaults", " ".join(seen)) + self.assertFalse(agent.exists()) + self.assertFalse(record.exists(), "a complete undo removes the record") + self.assertEqual(list(store.iterdir()), [], "every preference setup added is gone again") + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_undo_restores_a_value_that_was_there_before(self) -> None: + store = self.dir / "store" + self.stubbed("--undo") # creates the stand-ins and the store + (store / "host-com.apple.screensaver-idleTime").write_text("integer 300\n", encoding="utf-8") + self.stubbed("--app", str(self.app), "--apply") + _, seen = self.stubbed("--undo", "--apply") + self.assertIn("defaults -currentHost write com.apple.screensaver idleTime -int 300", seen) + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_undo_without_a_record_fails_and_changes_nothing(self) -> None: + result, seen = self.stubbed("--undo", "--apply") + self.assertEqual(result.returncode, 1) + self.assertIn("restoring by hand", result.stdout) + self.assertFalse([command for command in seen if command.startswith(("defaults", "pmset"))]) + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_a_failed_restore_fails_the_undo(self) -> None: + self.stubbed("--app", str(self.app), "--apply") + (self.dir / "store" / "host-com.apple.screensaver-idleTime").unlink() + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + record.write_text("defaults\tuser\tcom.apple.CrashReporter\tDialogType\tstring\tbasic\n", encoding="utf-8") + result, _ = self.stubbed("--undo", "--apply", fail="DialogType") + self.assertEqual(result.returncode, 1) + self.assertIn("1 step(s) failed", result.stdout) + self.assertTrue(record.exists(), "a failed undo keeps the record for another try") + @unittest.skipIf(sys.platform == "darwin", "--apply would change this Mac's settings") def test_apply_refuses_off_macos(self) -> None: result = self.setup("--app", str(self.app), "--apply") diff --git a/tools/booth/test_watchdog.py b/tools/booth/test_watchdog.py index a9aa08b5..3ecd086c 100644 --- a/tools/booth/test_watchdog.py +++ b/tools/booth/test_watchdog.py @@ -21,13 +21,16 @@ echo "$*" >> "$FAKE_RUNS" echo $$ > "$FAKE_PID" heartbeat="" -for arg in "$@"; do - case "$arg" in --heartbeat=*) heartbeat="${arg#--heartbeat=}" ;; esac +for arg in "$@"; do # the first non-empty one, as AppRouter.heartbeat_path() reads it + case "$arg" in --heartbeat=?*) [ -n "$heartbeat" ] || heartbeat="${arg#--heartbeat=}" ;; esac done +beat() { # write and rename, as the game does, so the watchdog never reads a half-written number + [ -z "$heartbeat" ] || { date +%s > "$heartbeat.tmp" && mv -f "$heartbeat.tmp" "$heartbeat"; } +} case "$FAKE_MODE" in exit) exit 3 ;; - hang) date +%s > "$heartbeat"; exec sleep 1000 ;; - healthy) while :; do [ -z "$heartbeat" ] || date +%s > "$heartbeat"; sleep 0.2; done ;; + hang) beat; exec sleep 1000 ;; + healthy) while :; do beat; sleep 0.2; done ;; esac """ @@ -56,9 +59,11 @@ def tearDown(self) -> None: except (ProcessLookupError, ValueError): pass - def start(self, mode: str, *options: str, game_args: tuple[str, ...] = ("--", "--booth")) -> None: - env = dict(os.environ, FAKE_RUNS=str(self.runs), FAKE_PID=str(self.pid), FAKE_MODE=mode) - command = ["bash", str(WATCHDOG), "--log", str(self.log), "--heartbeat", str(self.heartbeat), + def start(self, mode: str, *options: str, game_args: tuple[str, ...] = ("--", "--booth"), + heartbeat: bool = True) -> None: + env = dict(os.environ, FAKE_RUNS=str(self.runs), FAKE_PID=str(self.pid), FAKE_MODE=mode, TMPDIR=str(self.dir)) + own = ["--heartbeat", str(self.heartbeat)] if heartbeat else [] + command = ["bash", str(WATCHDOG), "--log", str(self.log), *own, "--poll", "0.2", *options, "--", str(self.game), *game_args] # The watchdog writes its events to --log; the fake game's own output is discarded. self.process = subprocess.Popen(command, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) @@ -138,6 +143,7 @@ def test_stop_file_ends_the_watch_when_the_game_exits(self) -> None: self.start("healthy", "--hang", "0", "--stop-file", str(stop_file)) self.assertTrue(self.wait_for(lambda: self.launches(), 10)) self.assertIn("removed the stop file left from before", self.events()) + self.assertFalse(stop_file.exists()) stop_file.touch() os.kill(int(self.pid.read_text()), signal.SIGTERM) # staff close the game assert self.process is not None @@ -145,6 +151,30 @@ def test_stop_file_ends_the_watch_when_the_game_exits(self) -> None: self.assertIn("not relaunching", self.events()) self.assertEqual(len(self.launches()), 1) + def test_default_heartbeat_is_private_to_each_watchdog(self) -> None: + self.start("healthy", "--hang", "2", "--grace", "0", heartbeat=False) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + path = Path(self.launches()[0].split("--heartbeat=")[1]) + self.assertEqual(path.parent.parent.resolve(), self.dir.resolve()) + self.assertTrue(path.parent.name.startswith("hoppy-booth.")) + self.assertEqual(path.parent.stat().st_mode & 0o777, 0o700) + time.sleep(3) + self.assertEqual(len(self.launches()), 1, "a healthy game on the private heartbeat is left alone") + self.stop() + self.assertFalse(path.parent.exists(), "the private folder goes when the watchdog stops") + + def test_rejects_a_game_command_that_sets_its_own_heartbeat(self) -> None: + result = subprocess.run(["bash", str(WATCHDOG), "--", "game", "--", "--heartbeat=/elsewhere"], + capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 2) + self.assertIn("passes its own --heartbeat", result.stderr) + + def test_reads_numbers_with_leading_zeros_as_decimal(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "01", "--max-backoff", "08") + self.assertTrue(self.wait_for(lambda: "relaunching in 2 s" in self.events(), 15), self.events()) + self.stop() + self.assertNotIn("value too great", self.events()) + def test_rejects_bad_usage(self) -> None: for args, message in ( ([], "no game command"), diff --git a/tools/booth/watchdog.sh b/tools/booth/watchdog.sh index 8c971d3d..55b121a3 100755 --- a/tools/booth/watchdog.sh +++ b/tools/booth/watchdog.sh @@ -12,7 +12,8 @@ # "/Applications/Hoppy’s Adventure Arcade.app/Contents/MacOS/" -- --booth # # Options: -# --heartbeat heartbeat file for the game (default: $TMPDIR/hoppy-booth-heartbeat) +# --heartbeat heartbeat file for the game (default: in a new private folder under +# $TMPDIR, removed when the watchdog stops) # --hang restart the game when its heartbeat is older than this; 0 turns hang # checks off, for builds without --heartbeat (default: 45) # --grace how long a fresh launch has to write its first heartbeat (default: 60) @@ -25,11 +26,12 @@ # --term-wait how long the game gets to exit after SIGTERM before SIGKILL (default: 10) # --poll how often to check the game; may be fractional (default: 1) # -# The game gets --heartbeat= after its other user arguments (a "--" is added if missing). -# SIGTERM, SIGINT or SIGHUP stops the game, then the watchdog, with exit status 0. +# The game gets --heartbeat= after its other user arguments (a "--" is added if missing); the +# game command must not set its own. SIGTERM, SIGINT or SIGHUP stops the game, then the watchdog, +# with exit status 0. -heartbeat="${TMPDIR:-/tmp}" -heartbeat="${heartbeat%/}/hoppy-booth-heartbeat" +heartbeat="" +private_dir="" hang=45 grace=60 stop_file="" @@ -49,10 +51,12 @@ die() { exit 2 } +# Validate a whole number of seconds and print it in base 10 ("08" is 8, not bad octal). whole_number() { case "$2" in '' | *[!0-9]*) die "$1 needs a whole number of seconds, not '$2'" ;; esac + echo $((10#$2)) } while [ $# -gt 0 ]; do @@ -61,18 +65,17 @@ while [ $# -gt 0 ]; do [ $# -ge 2 ] || die "$1 needs a value" case "$1" in --heartbeat) heartbeat="$2" ;; - --hang) whole_number "$1" "$2"; hang="$2" ;; - --grace) whole_number "$1" "$2"; grace="$2" ;; + --hang) hang=$(whole_number "$1" "$2") || exit 2 ;; + --grace) grace=$(whole_number "$1" "$2") || exit 2 ;; --stop-file) stop_file="$2" ;; --log) log_file="$2" ;; --backoff) - whole_number "$1" "$2" - [ "$2" -ge 1 ] || die "--backoff must be at least 1 second" - backoff="$2" + backoff=$(whole_number "$1" "$2") || exit 2 + [ "$backoff" -ge 1 ] || die "--backoff must be at least 1 second" ;; - --max-backoff) whole_number "$1" "$2"; max_backoff="$2" ;; - --stable) whole_number "$1" "$2"; stable="$2" ;; - --term-wait) whole_number "$1" "$2"; term_wait="$2" ;; + --max-backoff) max_backoff=$(whole_number "$1" "$2") || exit 2 ;; + --stable) stable=$(whole_number "$1" "$2") || exit 2 ;; + --term-wait) term_wait=$(whole_number "$1" "$2") || exit 2 ;; --poll) case "$2" in '' | *[!0-9.]* | *.*.* | .) die "--poll needs a number of seconds, not '$2'" ;; @@ -96,7 +99,16 @@ if [ "$hang" -gt 0 ]; then separator=0 for arg in "$@"; do [ "$arg" = "--" ] && separator=1 + case "$arg" in + --heartbeat=?*) die "the game command sets $arg; the watchdog passes its own --heartbeat" ;; + esac done + if [ -z "$heartbeat" ]; then + # A folder only this user can write, new for each watchdog: nobody can plant a symlink in it, + # and two watchdogs never read each other's heartbeat. + private_dir=$(mktemp -d "${TMPDIR:-/tmp}/hoppy-booth.XXXXXX") || die "cannot create a heartbeat folder" + heartbeat="$private_dir/heartbeat" + fi [ "$separator" -eq 1 ] || game+=("--") game+=("--heartbeat=$heartbeat") fi @@ -112,6 +124,7 @@ say() { } child="" +stop_requested=0 # SIGTERM, then SIGKILL if the game is still there after --term-wait seconds. The caller reaps it. terminate() { @@ -128,7 +141,7 @@ terminate() { } stop() { - trap - TERM INT HUP + trap '' TERM INT HUP if [ -n "$child" ] && kill -0 "$child" 2>/dev/null; then say "stopping the game (pid $child)" terminate "$child" @@ -137,7 +150,14 @@ stop() { say "stopped" exit 0 } -trap stop TERM INT HUP + +# The signal handler only records the request; the loop acts on it at these checks. That way a +# signal that lands between starting the game and recording its pid still stops the game. +check_stop() { + [ "$stop_requested" -eq 0 ] || stop +} +trap 'stop_requested=1' TERM INT HUP +trap '[ -z "$private_dir" ] || rm -rf "$private_dir"' EXIT # Sleep in one-second steps, so a stop signal is handled within a second. pause() { @@ -145,6 +165,7 @@ pause() { while [ "$slept" -lt "$1" ]; do sleep 1 slept=$((slept + 1)) + check_stop done } @@ -153,7 +174,8 @@ if [ "$hang" -gt 0 ]; then mkdir -p "$(dirname "$heartbeat")" || die "cannot create the folder for $heartbeat" fi if [ -n "$stop_file" ] && [ -e "$stop_file" ]; then - rm -f "$stop_file" + rm -f "$stop_file" || die "cannot remove the stop file left from before ($stop_file)" + [ ! -e "$stop_file" ] || die "the stop file left from before is still there ($stop_file)" say "removed the stop file left from before ($stop_file)" fi if [ "$hang" -gt 0 ]; then @@ -164,13 +186,16 @@ fi wait_before_relaunch="$backoff" while :; do - rm -f "$heartbeat" + check_stop + [ -z "$heartbeat" ] || rm -f "$heartbeat" "${game[@]}" & child=$! launched=$(date +%s) say "launched the game (pid $child)" + check_stop while kill -0 "$child" 2>/dev/null; do sleep "$poll" + check_stop [ "$hang" -gt 0 ] || continue now=$(date +%s) [ $((now - launched)) -ge "$grace" ] || continue @@ -187,7 +212,9 @@ while :; do done wait "$child" 2>/dev/null status=$? + check_stop child="" +stop_requested=0 ran=$(($(date +%s) - launched)) if [ "$status" -gt 128 ]; then say "the game exited after $ran s (signal $((status - 128)))" From e1de8d5aee5179f314ecf5ce744d19265fb57901 Mon Sep 17 00:00:00 2001 From: Erik Thorelli Date: Thu, 15 Oct 2026 17:36:14 -0700 Subject: [PATCH 3/3] fix(booth): keep a stop request that lands after the last check [Agent: Claude Code | Model: Opus 5.5 | Thinking: High | Acting on behalf of: Erik | Skill: none] Addresses CodeRabbit review 5541562607 on #76: a stray `stop_requested=0` after the post-wait check (a replace that matched two places) could discard a signal and relaunch the game. Only the startup initialization remains. --- tools/booth/watchdog.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/booth/watchdog.sh b/tools/booth/watchdog.sh index 55b121a3..30f82700 100755 --- a/tools/booth/watchdog.sh +++ b/tools/booth/watchdog.sh @@ -214,7 +214,6 @@ while :; do status=$? check_stop child="" -stop_requested=0 ran=$(($(date +%s) - launched)) if [ "$status" -gt 128 ]; then say "the game exited after $ran s (signal $((status - 128)))"