diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 12357e53..e6a752e3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -106,6 +106,10 @@ jobs: run: python -m unittest discover -s tools/vendor -v - name: Booth tools unit tests run: python -m unittest discover -s tools/booth -v + - name: Shellcheck booth scripts + run: | + command -v shellcheck >/dev/null || sudo apt-get install -y shellcheck + shellcheck tools/booth/watchdog.sh tools/booth/macos/setup.sh gitleaks: name: gitleaks diff --git a/README.md b/README.md index 38c4ba3c..259193da 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,8 @@ HoppysAdventureArcade.exe -- --booth --route=heroes_vs_villains Booth mode runs fullscreen and ignores the menu's **Quit Game**, so staff close the window instead. After 90 seconds without input, any game returns to the hub's attract screen. [Run it at a booth](docs/builds.md#run-it-at-a-booth) lists every route and flag. +For a machine that runs unattended all day, follow the [booth runbook](docs/booth-runbook.md). It sets up a watchdog that restarts the game if it closes or freezes, along with the Mac's kiosk settings, and covers the staff checklist. + ## Docs - Platformer: [Bay Area Blitz](docs/bay-area-blitz.md), [Orbital Ops](docs/orbital-ops.md), [The Enchanted Repo](docs/enchanted-repo.md) diff --git a/docs/booth-runbook.md b/docs/booth-runbook.md new file mode 100644 index 00000000..4f405890 --- /dev/null +++ b/docs/booth-runbook.md @@ -0,0 +1,97 @@ +# Booth runbook + +This runbook is for the staff who run Hoppy’s Adventure Arcade unattended at a booth, such as booth 528 at re:Invent. It covers a Mac running the game, with a second Mac set up the same way as a spare. The watchdog also runs on Linux; Windows isn't covered. + +## Pack list + +- The booth Mac and a spare Mac, each set up and tested with the steps below. +- A USB stick with the macOS build zip from [Releases or Actions](builds.md#get-a-build) and a copy of this repository, which has the booth tools in `tools/booth/`. +- The controllers plus one spare, with their cables or wireless dongles, the display cable and a power strip. +- A keyboard and mouse for staff. Keep them out of players' reach. + +## Set up a machine (once, about 20 minutes) + +1. Sign in to the account the booth will use. +2. Unzip the build and move **Hoppy’s Adventure Arcade.app** to **Applications**. +3. In Terminal, from the copy of this repository, see what the setup will change: + ```sh + tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" --route heroes_vs_villains + ``` + Nothing changes yet. To apply it, run the same command with `--apply` and enter the administrator password when asked. The game starts. +4. Do the manual steps the script lists at the end: automatic login, Do Not Disturb, the display, the volume and the controllers. +5. Restart the Mac. The game should start on its own within a minute. + +The setup does the following: +- It installs the watchdog (`tools/booth/watchdog.sh`) as a login item. The watchdog starts the game in booth mode, relaunches it whenever it closes or crashes, and restarts it when it freezes. +- It removes the download quarantine, so Gatekeeper doesn't block the app. +- It keeps the Mac awake and turns it back on after a power cut. +- It turns off the screen saver, automatic updates and crash dialogs. + +Booth mode itself runs fullscreen, ignores the menu's **Quit Game**, and sends a game nobody has touched for 90 seconds back to the hub's attract screen ([details](builds.md#run-it-at-a-booth)). + +## Every morning + +1. Turn the Mac on. It logs in, and the game appears within a minute. +2. Check that every controller moves Hoppy and that the volume suits the floor. +3. Play one short round. + +## During the show + +| What happens | What you'll see | +|---|---| +| A player walks away mid-game | After 90 seconds the game returns to the hub. | +| The game crashes or closes | The watchdog opens it again within a few seconds. | +| The game freezes | The watchdog restarts it within about a minute. | +| Someone presses Cmd+Q | The game closes, and the watchdog opens it again. | +| The player's controller disconnects | The game pauses and asks for the controller. Plug it back in, or press Start on another controller. | +| You want to clear the Endless Runner scores | **Endless Runner → Leaderboard → Reset scores**. | + +## Stop the game for maintenance + +```sh +touch ~/Library/Application\ Support/HoppyBooth/stop +``` + +Then press Cmd+Q. The game stays closed. To bring it back, restart the Mac or run: + +```sh +launchctl kickstart gui/$(id -u)/ai.coderabbit.arcade.booth +``` + +The watchdog removes the stop file when it starts. + +## Swap to the spare + +1. Move the display cable and the controllers to the spare Mac and turn it on. It starts the game on its own. +2. Copy the session logs off the first Mac when there's time (see below). + +## Collect the session logs + +The game records each session's start and end in `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/booth/`, one file per run of the game. It records the game, the times and the length, and nothing about players. At the end of each day, copy the whole `booth` folder to the USB stick and leave the original in place. To summarize every machine's folder at once: + +```sh +python3 tools/booth/sessions_report.py booth-a/ booth-b/ --csv sessions.csv +``` + +## Troubleshooting + +The watchdog writes launches, exits and restarts to `~/Library/Logs/HoppyBooth/watchdog.log`, and the game's own output to `~/Library/Logs/HoppyBooth/game.log`. + +| Problem | What to do | +|---|---| +| No game after login | Read `watchdog.log`. Run `launchctl print gui/$(id -u)/ai.coderabbit.arcade.booth` to see whether the login item is loaded. If it isn't, run the setup again with `--apply`. | +| macOS says the app can't be opened | Run `xattr -dr com.apple.quarantine "/Applications/Hoppy’s Adventure Arcade.app"`. | +| The game restarts every minute or two | If `watchdog.log` says the game "has not written a heartbeat", the build is older than the watchdog. Install a current build. | +| A controller does nothing | Reconnect it. Check **System Settings → Bluetooth** for wireless pads. | +| Black borders or the wrong size | Check the resolution in **System Settings → Displays**. | +| Notifications appear over the game | Turn on Do Not Disturb in **Control Center → Focus**. | +| Nothing helps | Restart the Mac. If the game still doesn't run, swap to the spare. | + +## After the show + +```sh +tools/booth/macos/setup.sh --undo # see what it will change +tools/booth/macos/setup.sh --undo --apply +``` + +`--undo` puts back the settings the first setup recorded on that Mac. Then turn off automatic login and Do Not Disturb, and copy the session logs. diff --git a/docs/builds.md b/docs/builds.md index 9726d5f8..63e551f8 100644 --- a/docs/builds.md +++ b/docs/builds.md @@ -36,6 +36,7 @@ open -a "Hoppy’s Adventure Arcade.app" --args -- --booth --route=heroes_vs_vil - **Idle reset.** After 90 seconds with no keyboard, mouse, touch or controller input, any game returns to the hub, whose rotating previews act as the attract screen. - **No quitting from the menu.** The menu's **Quit Game** does nothing in booth mode; staff close the window (Cmd+Q, Alt+F4). - **Launch route.** `--route=` opens a route at launch (`heroes_vs_villains`, `code_city_brawl`, `bay_area_blitz`, `orbital_ops`, `enchanted_repo`, `runner`, `platformer`, `beatem_up`, `hub`), with or without `--booth`. An unknown name logs an error and opens the hub. +- **Keep it running.** `tools/booth/watchdog.sh` relaunches the game whenever it exits and restarts it when it hangs. The watchdog passes `--heartbeat=`, which makes the game rewrite that file with the current time every two seconds; a file that stops changing means the game is stuck. On a Mac, `tools/booth/macos/setup.sh` installs the watchdog as a login item and applies the kiosk settings. The [booth runbook](booth-runbook.md) walks staff through setup, the show days and recovery. - **Session log.** Booth mode records each game session's start and end in `user://booth/`, one file per run of the game (`sessions--.jsonl`). Each entry records the game, the local start and end times, the duration, how the session ended (`menu`, `idle` or `quit`) and how many times the player retried. It records nothing about players and never uses the network. A start with no end means the game stopped without quitting: a crash, a power cut or a forced kill. The game prints its file's full path at startup. `--session-log=` writes to another folder and works without `--booth`, for playtests. `user://` is `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/` on macOS, `%APPDATA%\Godot\app_userdata\Hoppy’s Adventure Arcade\` on Windows and `~/.local/share/godot/app_userdata/Hoppy’s Adventure Arcade/` on Linux. To summarize the logs, copy each machine's `booth` folder and run: diff --git a/hoppys-adventure-arcade/app/app_router.gd b/hoppys-adventure-arcade/app/app_router.gd index 85ef6a3d..4de8c207 100644 --- a/hoppys-adventure-arcade/app/app_router.gd +++ b/hoppys-adventure-arcade/app/app_router.gd @@ -1,5 +1,6 @@ extends CanvasLayer const BoothLog := preload("res://app/booth_log.gd") +const BoothHeartbeat := preload("res://app/booth_heartbeat.gd") const ROUTES := { "heroes_vs_villains": "res://scenes/beatem_up/reinvent/heroes_vs_villains.tscn", "code_city_brawl": "res://scenes/beatem_up/code_city_brawl.tscn", @@ -30,12 +31,17 @@ const STICK_DEADZONE := 0.4 ## of them is passed. const SESSION_LOG_PREFIX := "--session-log=" const SESSION_LOG_DEFAULT := "user://booth" +## Watchdog support (app/booth_heartbeat.gd): `--heartbeat=` keeps that file updated with the +## time every two seconds, so tools/booth/watchdog.sh can restart a game that hangs. Off unless +## passed; the watchdog passes it. +const HEARTBEAT_PREFIX := "--heartbeat=" var busy := false var veil: ColorRect var booth := false var idle_limit := IDLE_SECONDS var idle_seconds := 0.0 var session_log: BoothLog +var heartbeat: BoothHeartbeat func _ready() -> void: # The veil tween must keep running while a microgame has the tree paused. process_mode = Node.PROCESS_MODE_ALWAYS @@ -55,6 +61,11 @@ func _ready() -> void: var log_path := session_log_path(args, booth) if log_path != "": start_session_log(log_path) + var beat_path := heartbeat_path(args) + if beat_path != "": + heartbeat = BoothHeartbeat.new(beat_path) + add_child(heartbeat) + print("AppRouter: heartbeat at %s" % ProjectSettings.globalize_path(beat_path)) var start := launch_route(args) if start != "": _launch.call_deferred(start) @@ -125,6 +136,13 @@ func start_session_log(folder: String) -> void: session_log = BoothLog.new(folder) print("AppRouter: session log at %s" % ProjectSettings.globalize_path(session_log.path)) +## The heartbeat file named by `--heartbeat=`, or "" when absent. +static func heartbeat_path(args: PackedStringArray) -> String: + for arg in args: + if arg.begins_with(HEARTBEAT_PREFIX) and arg.length() > HEARTBEAT_PREFIX.length(): + return arg.trim_prefix(HEARTBEAT_PREFIX) + return "" + func _launch(route: String) -> void: await go(route) var scene := get_tree().current_scene diff --git a/hoppys-adventure-arcade/app/booth_heartbeat.gd b/hoppys-adventure-arcade/app/booth_heartbeat.gd new file mode 100644 index 00000000..9fa6d89e --- /dev/null +++ b/hoppys-adventure-arcade/app/booth_heartbeat.gd @@ -0,0 +1,58 @@ +extends Node +## Booth watchdog support. While the game's main loop runs, this rewrites a file with the current +## Unix time every INTERVAL seconds. tools/booth/watchdog.sh restarts the game when the file goes +## stale, which catches a hang that leaves the process alive. AppRouter adds it only when the game +## is launched with `--heartbeat=`. +const INTERVAL := 2.0 +var path := "" +var elapsed := 0.0 +## Set after the first failed write, so the warning prints once. Every beat still tries again: a +## heartbeat that stays stale makes the watchdog restart the game. +var warned := false + +func _init(file_path: String) -> void: + path = file_path + name = "BoothHeartbeat" + # Keep beating while a game has the tree paused: a paused game is not a hung one. + process_mode = Node.PROCESS_MODE_ALWAYS + +func _ready() -> void: + var directory := path.get_base_dir() + if directory != "" and not DirAccess.dir_exists_absolute(directory): + DirAccess.make_dir_recursive_absolute(directory) + beat() + +func _process(delta: float) -> void: + elapsed += delta + if elapsed >= INTERVAL: + elapsed = 0.0 + beat() + +## Write to a temporary file and rename it over the heartbeat, so the watchdog never reads a +## half-written number. A leftover temporary file is removed first, so a link planted in its place +## is replaced rather than written through. The watchdog keeps the heartbeat in a private folder. +func beat() -> void: + var temporary := path + ".tmp" + var folder := DirAccess.open(path.get_base_dir()) + var leftover := temporary.get_file() + if folder != null and (folder.file_exists(leftover) or folder.is_link(leftover)): + folder.remove(leftover) + var file := FileAccess.open(temporary, FileAccess.WRITE) + if file == null: + _fail(error_string(FileAccess.get_open_error())) + return + var written := file.store_line(str(int(Time.get_unix_time_from_system()))) + file.close() + if not written: + _fail("write failed") + return + var error := DirAccess.rename_absolute(temporary, path) + if error != OK: + _fail(error_string(error)) + +func _fail(reason: String) -> void: + if warned: + return + warned = true + var problem := "Booth heartbeat: cannot write %s (%s)." % [path, reason] + push_warning(problem + " If it stays stale, the watchdog restarts the game.") diff --git a/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid b/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid new file mode 100644 index 00000000..02c9380e --- /dev/null +++ b/hoppys-adventure-arcade/app/booth_heartbeat.gd.uid @@ -0,0 +1 @@ +uid://dpc1yt8at1sty diff --git a/hoppys-adventure-arcade/tests/booth_heartbeat.gd b/hoppys-adventure-arcade/tests/booth_heartbeat.gd new file mode 100644 index 00000000..688c5e04 --- /dev/null +++ b/hoppys-adventure-arcade/tests/booth_heartbeat.gd @@ -0,0 +1,111 @@ +extends SceneTree +## Booth heartbeat (app/booth_heartbeat.gd): with --heartbeat= the game keeps that file +## updated with the current Unix time, including while a game has paused the tree; without the flag +## there is no heartbeat; a link planted where the temporary file goes is replaced, not written +## through; a heartbeat it can't write warns instead of failing. The flag is checked by launching +## the game as a child process, so _ready()'s argument handling is covered. Every run starts from an +## empty folder of its own. Run headless with --script res://tests/booth_heartbeat.gd. +const BoothHeartbeat := preload("res://app/booth_heartbeat.gd") +const FOLDER := "user://test_heartbeat" +const BEAT := FOLDER + "/beat" +const CHILD_BEAT := FOLDER + "/child_beat" +## Wall-clock limit for the child launch, for runs outside tools/tests/run_tests.py's timeout. +const CHILD_SECONDS := 60.0 +var failures := 0 +var router: Node + +func _initialize() -> void: + call_deferred("run") + +func check(ok: bool, message: String) -> void: + print("PASS: " if ok else "FAIL: ", message) + if not ok: + failures += 1 + +func wait(seconds: float) -> void: + await create_timer(seconds, true).timeout + +## Launch the game headless with user args for two simulated seconds and return its exit code, +## or -1 if it outlives CHILD_SECONDS and had to be killed. +func launch(user_args: Array[String]) -> int: + var args: Array[String] = ["--headless", "--path", ProjectSettings.globalize_path("res://"), + "--fixed-fps", "60", "--quit-after", "120", "--"] + args.append_array(user_args) + var pid := OS.create_process(OS.get_executable_path(), args) + var deadline := Time.get_ticks_msec() + int(CHILD_SECONDS * 1000.0) + while OS.is_process_running(pid): + if Time.get_ticks_msec() > deadline: + OS.kill(pid) + return -1 + await wait(0.1) + return OS.get_process_exit_code(pid) + +func read(path: String) -> String: + return FileAccess.get_file_as_string(path).strip_edges() + +## True when the file holds a Unix time within a minute of now. +func fresh(path: String) -> bool: + var text := read(path) + return text.is_valid_int() and absi(text.to_int() - int(Time.get_unix_time_from_system())) < 60 + +func clean() -> void: + var dir := DirAccess.open(FOLDER) + if dir == null: + return + dir.include_hidden = true + for file_name in dir.get_files(): + dir.remove(file_name) + DirAccess.remove_absolute(FOLDER) + +func run() -> void: + router = root.get_node("AppRouter") + clean() + DirAccess.make_dir_recursive_absolute(FOLDER) + await wait(0.3) + check(router.heartbeat == null, "No heartbeat without --heartbeat") + check(router.heartbeat_path(PackedStringArray(["--booth"])) == "", + "--booth alone has no heartbeat") + check(router.heartbeat_path(PackedStringArray(["--heartbeat=user://beat"])) == "user://beat", + "--heartbeat names the file") + check(router.heartbeat_path(PackedStringArray(["--heartbeat="])) == "", + "An empty --heartbeat is ignored") + + var code := await launch(["--booth", "--heartbeat=" + ProjectSettings.globalize_path(CHILD_BEAT)]) + check(code == 0, "A launch with --heartbeat exits cleanly (exit %d)" % code) + check(fresh(CHILD_BEAT), "--heartbeat makes the game write the current Unix time") + check(not FileAccess.file_exists(CHILD_BEAT + ".tmp"), "The heartbeat leaves no temporary file") + + # A link where the temporary file goes: the heartbeat replaces it instead of writing through it. + var decoy := FOLDER + "/decoy" + var keep := FileAccess.open(decoy, FileAccess.WRITE) + keep.store_line("keep me") + keep.close() + var linked := DirAccess.open(FOLDER).create_link(ProjectSettings.globalize_path(decoy), + ProjectSettings.globalize_path(BEAT + ".tmp")) == OK + var beat := BoothHeartbeat.new(BEAT) + root.add_child(beat) + check(fresh(BEAT), "The heartbeat writes as soon as it starts") + if linked: + check(read(decoy) == "keep me", "A link planted at the temporary path is not written through") + var marker := FileAccess.open(BEAT, FileAccess.WRITE) + marker.store_line("0") + marker.close() + paused = true + await wait(beat.INTERVAL + 0.6) + check(fresh(BEAT), "The heartbeat keeps writing while a game has the tree paused") + paused = false + + # The heartbeat file itself where a folder should be: warn, keep running, never an error. + var broken := BoothHeartbeat.new(BEAT + "/nested") + root.add_child(broken) + check(broken.warned, "A heartbeat it can't write warns") + broken.beat() + check(broken.is_inside_tree(), "A failed heartbeat leaves the game running") + + beat.queue_free() + broken.queue_free() + await process_frame + clean() + await root.get_node("AudioService").settle() + print("Booth heartbeat failures: ", failures) + quit(1 if failures else 0) diff --git a/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid b/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid new file mode 100644 index 00000000..81b167c6 --- /dev/null +++ b/hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid @@ -0,0 +1 @@ +uid://bkg1o2uubr6lh diff --git a/tools/booth/macos/setup.sh b/tools/booth/macos/setup.sh new file mode 100755 index 00000000..aaa75c4b --- /dev/null +++ b/tools/booth/macos/setup.sh @@ -0,0 +1,340 @@ +#!/usr/bin/env bash +# Set up a Mac to run Hoppy's Adventure Arcade as an unattended booth kiosk, and put it back after +# the show. Staff steps are in docs/booth-runbook.md. +# +# Usage: +# tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" [--route ] +# Print the plan. Nothing changes. +# tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" [--route ] --apply +# Make the changes. Asks for an administrator password for the system settings. +# tools/booth/macos/setup.sh --undo [--apply] +# Print (or, with --apply, make) the changes that put the Mac back: the settings recorded +# before setup are restored exactly. Without that record it changes no system setting and +# lists what to restore by hand. +# +# --apply does this: +# - copies tools/booth/watchdog.sh to ~/Library/Application Support/HoppyBooth/, and records the +# settings below as they were, for --undo (only the first time) +# - installs a LaunchAgent that starts the watchdog, and through it the game in booth mode, at +# login, and restarts the watchdog if it dies; it starts the game right away +# - removes the download quarantine from the app, so Gatekeeper doesn't block it +# - turns off system sleep, display sleep, disk sleep and Power Nap, and restarts after a power cut +# - turns off the screen saver, automatic macOS and App Store updates, and crash report dialogs +# Settings macOS doesn't let a script change are listed at the end (automatic login, notifications, +# display, volume, controllers). + +label="ai.coderabbit.arcade.booth" +support="$HOME/Library/Application Support/HoppyBooth" +logs="$HOME/Library/Logs/HoppyBooth" +agent="$HOME/Library/LaunchAgents/$label.plist" +snapshot="$support/settings-before.tsv" +# The preferences the setup changes: scope, domain, key, type, booth value. "host" is the +# -currentHost domain; "system" is machine-wide and needs sudo to write. +SETTINGS=( + "host com.apple.screensaver idleTime int 0" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticCheckEnabled bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate AutomaticallyInstallMacOSUpdates bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate CriticalUpdateInstall bool false" + "system /Library/Preferences/com.apple.SoftwareUpdate ConfigDataInstall bool false" + "system /Library/Preferences/com.apple.commerce AutoUpdate bool false" + "user com.apple.CrashReporter DialogType string none" +) +# The power settings the setup changes. +PMSET_KEYS="sleep displaysleep disksleep powernap autorestart" +here="$(cd "$(dirname "$0")" && pwd)" +app="" +route="" +apply=0 +undo=0 + +usage() { + sed -n '2,/^$/p' "$0" | sed -e 's/^# \{0,1\}//' +} + +die() { + echo "setup: $*" >&2 + exit 2 +} + +while [ $# -gt 0 ]; do + case "$1" in + --app | --route) + [ $# -ge 2 ] || die "$1 needs a value" + if [ "$1" = "--app" ]; then app="${2%/}"; else route="$2"; fi + shift 2 + ;; + --apply) apply=1; shift ;; + --undo) undo=1; shift ;; + -h | --help) usage; exit 0 ;; + *) die "unknown option '$1'" ;; + esac +done + +if [ "$apply" -eq 1 ] && [ "$(uname -s)" != "Darwin" ]; then + die "--apply only runs on macOS" +fi +# launchd doesn't run from this folder, so the agent needs an absolute path to the app. +case "$app" in + "" | /*) ;; + *) app="$PWD/$app" ;; +esac + +# Print each command; run it only with --apply. A failed command is reported and the rest go on. +failures=0 +run() { + local line=" \$" arg + for arg in "$@"; do + case "$arg" in + *[[:space:]\'\"\&]*) line="$line \"$arg\"" ;; + *) line="$line $arg" ;; + esac + done + echo "$line" + [ "$apply" -eq 1 ] || return 0 + if ! "$@"; then + echo " ^ failed; the remaining steps continue" >&2 + failures=$((failures + 1)) + fi +} + +step() { + echo + echo "$*" +} + +xml() { + printf '%s' "$1" | sed -e 's/&/\&/g' -e 's//\>/g' +} + +# The `defaults` command line that reads (r) or writes (w) a setting in this scope. +reader() { + if [ "$1" = host ]; then r=(defaults -currentHost); else r=(defaults); fi +} +writer() { + case "$1" in + host) w=(defaults -currentHost) ;; + system) w=(sudo defaults) ;; + *) w=(defaults) ;; + esac +} + +# Write each setting as it is now to the record, once: re-running the setup keeps the first one. +record_settings() { + local entry scope domain key type value read_type + if [ -e "$snapshot" ]; then + echo " kept the record from the first setup" + return 0 + fi + { + for entry in "${SETTINGS[@]}"; do + read -r scope domain key type value <<<"$entry" + reader "$scope" + if value=$("${r[@]}" read "$domain" "$key" 2>/dev/null); then + read_type=$("${r[@]}" read-type "$domain" "$key" 2>/dev/null) + case "$read_type" in + *boolean) type=bool ;; + *integer) type=int ;; + *float) type=float ;; + *string) type=string ;; + *) type=unsupported ;; + esac + else + type=absent + value="" + fi + printf 'defaults\t%s\t%s\t%s\t%s\t%s\n' "$scope" "$domain" "$key" "$type" "$value" + done + for key in $PMSET_KEYS; do + value=$(pmset -g | awk -v k="$key" '$1 == k { print $2; exit }') + [ -z "$value" ] || printf 'pmset\t%s\t%s\n' "$key" "$value" + done + } > "$snapshot" || die "cannot write the settings record $snapshot" + echo " recorded in $snapshot" +} + +# The app's executable, from its Info.plist. +executable() { + local plist="$1/Contents/Info.plist" name="" + [ -f "$plist" ] || return 0 + if [ -x /usr/libexec/PlistBuddy ]; then + name="$(/usr/libexec/PlistBuddy -c 'Print :CFBundleExecutable' "$plist" 2>/dev/null)" + else + name="$(sed -n '/CFBundleExecutable<\/key>/{n;s/.*\(.*\)<\/string>.*/\1/p;}' "$plist")" + fi + [ -n "$name" ] && printf '%s' "$1/Contents/MacOS/$name" +} + +if [ "$apply" -eq 1 ]; then + echo "Applying. Commands run as they are listed." +else + echo "Plan only: nothing will change. Run again with --apply to make these changes." +fi + +if [ "$undo" -eq 1 ]; then + if [ "$apply" -eq 1 ] && [ -f "$snapshot" ]; then + sudo -v || die "the system settings need an administrator password" + fi + step "Stop the booth game and remove its login item:" + run launchctl bootout "gui/$(id -u)/$label" + run rm -f "$agent" + unrecorded=0 + if [ -f "$snapshot" ]; then + step "Put back the settings recorded before setup ($snapshot):" + while IFS=$'\t' read -r kind a b c d e <&3; do + case "$kind" in + defaults) + writer "$a" + reader "$a" + case "$d" in + absent) + # Only delete what setup added; a key already gone needs nothing. + if [ "$apply" -eq 0 ] || "${r[@]}" read "$b" "$c" >/dev/null 2>&1; then + run "${w[@]}" delete "$b" "$c" + fi + ;; + bool) + case "$e" in 1 | [Tt]rue | YES | [Yy]es) e=true ;; *) e=false ;; esac + run "${w[@]}" write "$b" "$c" -bool "$e" + ;; + int | float | string) run "${w[@]}" write "$b" "$c" "-$d" "$e" ;; + *) + echo " restore $b $c by hand: it held a value this script can't rewrite" + unrecorded=1 + ;; + esac + ;; + pmset) run sudo pmset -a "$a" "$b" ;; + esac + done 3< "$snapshot" + [ "$failures" -gt 0 ] || run rm -f "$snapshot" + else + unrecorded=1 + step "No record of this Mac's settings before setup ($snapshot), so none are changed." + cat <<'BY_HAND' + Restore them by hand if the setup changed them: + - System Settings > Lock Screen: the display and sleep times you want. + - System Settings > Energy (or Battery): Power Nap and starting up after a power failure. + - System Settings > Screen Saver: the screen saver and when it starts. + - System Settings > General > Software Update > Automatic Updates: switch each one back on. + - In Terminal, to get crash dialogs back: defaults delete com.apple.CrashReporter DialogType +BY_HAND + fi + step "Left in place: the logs in $logs and the session logs under the game's user data." + echo "Turn automatic login and Do Not Disturb back off by hand if you turned them on." + if [ "$apply" -eq 1 ] && { [ "$failures" -gt 0 ] || [ "$unrecorded" -eq 1 ]; }; then + echo + [ "$failures" -eq 0 ] || echo "$failures step(s) failed; see the messages above." + [ "$unrecorded" -eq 0 ] || echo "Some settings need restoring by hand; see above." + exit 1 + fi + exit 0 +fi + +[ -n "$app" ] || die "--app is required: the path to Hoppy’s Adventure Arcade.app" +binary="$(executable "$app")" +if [ -z "$binary" ]; then + [ "$apply" -eq 0 ] || die "cannot read the app's executable from $app/Contents/Info.plist" + binary="$app/Contents/MacOS/" + echo "(Can't read $app here; the plan shows a placeholder for its executable.)" +fi +[ "$apply" -eq 0 ] || [ -x "$binary" ] || die "$binary is missing or not executable" + +game_args=(-- --booth) +[ -z "$route" ] || game_args+=("--route=$route") +watchdog=("/bin/bash" "$support/watchdog.sh" --log "$logs/watchdog.log" --heartbeat "$support/heartbeat" + --stop-file "$support/stop" -- "$binary" "${game_args[@]}") +plist_arguments="" +for arg in "${watchdog[@]}"; do + plist_arguments="$plist_arguments + $(xml "$arg")" +done +plist=" + + + + Label + $label + ProgramArguments + $plist_arguments + + RunAtLoad + + KeepAlive + + SuccessfulExit + + + LimitLoadToSessionType + Aqua + ProcessType + Interactive + StandardOutPath + $(xml "$logs/game.log") + StandardErrorPath + $(xml "$logs/game.log") + +" + +if [ "$apply" -eq 1 ]; then + sudo -v || die "the system settings need an administrator password" +fi + +step "Install the watchdog:" +run mkdir -p "$support" "$logs" +run cp "$here/../watchdog.sh" "$support/watchdog.sh" +run chmod 755 "$support/watchdog.sh" + +step "Record the current settings, so --undo can put them back ($snapshot):" +if [ "$apply" -eq 1 ]; then + record_settings +else + echo " would record the preferences and power settings changed below, unless a record exists" +fi + +step "Let Gatekeeper open the app (it is ad-hoc signed, not notarized):" +run xattr -dr com.apple.quarantine "$app" + +step "Keep the Mac awake, and start again after a power cut:" +run sudo pmset -a sleep 0 displaysleep 0 disksleep 0 powernap 0 +run sudo pmset -a autorestart 1 + +step "No screen saver, no automatic updates, no crash dialogs over the game:" +for entry in "${SETTINGS[@]}"; do + read -r scope domain key type value <<<"$entry" + writer "$scope" + run "${w[@]}" write "$domain" "$key" "-$type" "$value" +done + +step "Start the game at login and keep it running ($agent):" +if [ "$apply" -eq 1 ]; then + if ! { mkdir -p "$(dirname "$agent")" && printf '%s\n' "$plist" > "$agent"; }; then + die "cannot write $agent" + fi + echo " wrote $agent" + launchctl bootout "gui/$(id -u)/$label" 2>/dev/null +else + echo " would write:" + printf '%s\n' "$plist" | sed 's/^/ /' +fi +run launchctl bootstrap "gui/$(id -u)" "$agent" + +step "Do these by hand (macOS offers no reliable command for them):" +cat <<'MANUAL' + 1. System Settings > Users & Groups > Automatically log in as: the booth account. + (FileVault must be off for automatic login.) + 2. Control Center > Focus: click Do Not Disturb so it stays on. Then in System Settings > + Notifications, turn off "Allow notifications when mirroring or sharing the display". + 3. System Settings > Displays: the booth resolution, and mirror or extend as the booth needs. + 4. Set the volume for the floor, and pair or plug in the controllers. + 5. Restart the Mac and check that the game starts on its own within a minute. +MANUAL + +if [ "$apply" -eq 1 ]; then + echo + if [ "$failures" -gt 0 ]; then + echo "$failures step(s) failed; see the messages above." + exit 1 + fi + echo "Done. The watchdog's log is $logs/watchdog.log." +fi diff --git a/tools/booth/test_macos_setup.py b/tools/booth/test_macos_setup.py new file mode 100644 index 00000000..080d3dbf --- /dev/null +++ b/tools/booth/test_macos_setup.py @@ -0,0 +1,238 @@ +"""Tests for macos/setup.sh. Run: python3 -m unittest discover -s tools/booth -v + +The plan (the default, which changes nothing) and the argument checks run everywhere. The --apply +tests replace every command that changes settings (sudo, pmset, defaults, launchctl, xattr) and +uname with logging stand-ins, and still run only off macOS, so a missed stand-in can never change +a real Mac. +""" +from __future__ import annotations + +import os +import plistlib +import shlex +import shutil +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +SETUP = Path(__file__).resolve().parent / "macos" / "setup.sh" +INFO_PLIST = """ + + +\tCFBundleExecutable +\tFake Arcade + + +""" + + +# Stand-ins: each logs its arguments. `defaults` keeps values as files under $STUB_STORE and fails +# writes to a key named in $STUB_FAIL; `pmset -g` reports fixed power settings. +STUBS = { + "uname": 'echo Darwin', + "sudo": '[ "$1" = -v ] && exit 0; exec "$@"', + "launchctl": 'echo "launchctl $*" >> "$STUB_LOG"', + "xattr": 'echo "xattr $*" >> "$STUB_LOG"', + "pmset": """echo "pmset $*" >> "$STUB_LOG" +if [ "$1" = -g ]; then printf ' sleep 10\n displaysleep 5\n autorestart 0\n'; fi""", + "defaults": """echo "defaults $*" >> "$STUB_LOG" +host=""; [ "$1" = -currentHost ] && { host="host-"; shift; } +verb="$1"; key_file="$STUB_STORE/$host$(printf '%s' "$2" | tr '/' '_')-$3" +case "$verb" in + read) [ -f "$key_file" ] && cut -d' ' -f2- "$key_file" ;; + read-type) [ -f "$key_file" ] && echo "Type is $(cut -d' ' -f1 "$key_file")" ;; + write) [ "$3" = "${STUB_FAIL:-}" ] && exit 1; echo "$4 $5" > "$key_file" ;; + delete) [ -f "$key_file" ] && rm "$key_file" ;; +esac""", +} + + +class MacSetupTest(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.dir = Path(folder.name) + self.home = self.dir / "home" + self.home.mkdir() + self.app = self.dir / "Fake & Co’s Arcade.app" + (self.app / "Contents" / "MacOS").mkdir(parents=True) + (self.app / "Contents" / "Info.plist").write_text(INFO_PLIST, encoding="utf-8") + + def setup(self, *args: str) -> subprocess.CompletedProcess[str]: + env = dict(os.environ, HOME=str(self.home)) + return subprocess.run(["bash", str(SETUP), *args], capture_output=True, text=True, env=env, timeout=30) + + def agent(self, output: str) -> dict: + """The LaunchAgent the plan would write.""" + lines = output.splitlines() + start = lines.index(" would write:") + 1 + end = next(i for i in range(start, len(lines)) if lines[i].startswith(" $ launchctl bootstrap")) + return plistlib.loads("\n".join(line[4:] for line in lines[start:end]).encode("utf-8")) + + def test_plan_lists_every_change_and_touches_nothing(self) -> None: + result = self.setup("--app", str(self.app), "--route", "heroes_vs_villains") + self.assertEqual(result.returncode, 0, result.stderr) + out = result.stdout + self.assertTrue(out.startswith("Plan only: nothing will change.")) + for command in ("sudo pmset -a sleep 0 displaysleep 0 disksleep 0 powernap 0", + "sudo pmset -a autorestart 1", + "defaults -currentHost write com.apple.screensaver idleTime -int 0", + "AutomaticallyInstallMacOSUpdates -bool false", + "defaults write com.apple.CrashReporter DialogType -string none", + f'xattr -dr com.apple.quarantine "{self.app}"', + "launchctl bootstrap gui/"): + self.assertIn(command, out) + self.assertIn("Automatically log in as", out) + self.assertEqual(list(self.home.iterdir()), [], "the plan must not write anything") + + def test_agent_runs_the_watchdog_and_the_game_in_booth_mode(self) -> None: + agent = self.agent(self.setup("--app", str(self.app), "--route", "heroes_vs_villains").stdout) + support = f"{self.home}/Library/Application Support/HoppyBooth" + self.assertEqual(agent["Label"], "ai.coderabbit.arcade.booth") + self.assertEqual(agent["ProgramArguments"], [ + "/bin/bash", f"{support}/watchdog.sh", + "--log", f"{self.home}/Library/Logs/HoppyBooth/watchdog.log", + "--heartbeat", f"{support}/heartbeat", + "--stop-file", f"{support}/stop", + "--", f"{self.app}/Contents/MacOS/Fake Arcade", "--", "--booth", "--route=heroes_vs_villains", + ]) + self.assertTrue(agent["RunAtLoad"]) + self.assertEqual(agent["KeepAlive"], {"SuccessfulExit": False}) + self.assertEqual(agent["LimitLoadToSessionType"], "Aqua") + + def test_route_is_optional(self) -> None: + agent = self.agent(self.setup("--app", str(self.app)).stdout) + self.assertEqual(agent["ProgramArguments"][-2:], ["--", "--booth"]) + + def test_unreadable_app_gets_a_placeholder_in_the_plan(self) -> None: + result = self.setup("--app", str(self.dir / "Missing.app")) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("placeholder", result.stdout) + self.assertIn(f"{self.dir}/Missing.app/Contents/MacOS/", self.agent(result.stdout)["ProgramArguments"]) + + def test_undo_without_a_record_changes_no_setting(self) -> None: + result = self.setup("--undo") + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("launchctl bootout gui/", result.stdout) + self.assertIn("No record of this Mac's settings before setup", result.stdout) + self.assertNotIn("pmset", result.stdout.split("No record")[0]) + self.assertNotIn("restoredefaults", result.stdout) + self.assertEqual(list(self.home.iterdir()), []) + + def test_undo_plan_restores_the_recorded_settings(self) -> None: + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + record.parent.mkdir(parents=True) + record.write_text("defaults\thost\tcom.apple.screensaver\tidleTime\tint\t300\n" + "defaults\tsystem\t/Library/Preferences/com.apple.SoftwareUpdate\tAutomaticDownload\tbool\t1\n" + "defaults\tuser\tcom.apple.CrashReporter\tDialogType\tabsent\t\n" + "pmset\tsleep\t10\n", encoding="utf-8") + out = self.setup("--undo").stdout + for command in ("defaults -currentHost write com.apple.screensaver idleTime -int 300", + "sudo defaults write /Library/Preferences/com.apple.SoftwareUpdate AutomaticDownload -bool true", + "defaults delete com.apple.CrashReporter DialogType", "sudo pmset -a sleep 10"): + self.assertIn(command, out) + self.assertTrue(record.exists(), "the plan must leave the record alone") + + def test_a_relative_app_becomes_an_absolute_path_for_launchd(self) -> None: + env = dict(os.environ, HOME=str(self.home)) + result = subprocess.run(["bash", str(SETUP), "--app", self.app.name], capture_output=True, text=True, + env=env, cwd=self.dir, timeout=30) + executable = self.agent(result.stdout)["ProgramArguments"][-3] + self.assertEqual(Path(executable), self.app.resolve() / "Contents" / "MacOS" / "Fake Arcade") + + def test_the_installed_watchdog_comes_from_this_folder(self) -> None: + out = self.setup("--app", str(self.app)).stdout + line = next(line for line in out.splitlines() if line.startswith(" $ cp ")) + source = Path(shlex.split(line)[2]) + self.assertTrue(source.is_file(), source) + self.assertEqual(source.resolve(), (SETUP.parent.parent / "watchdog.sh").resolve()) + + def test_rejects_bad_usage(self) -> None: + for args, message in ((["--route", "hub"], "--app is required"), + (["--bogus"], "unknown option"), + (["--app"], "needs a value")): + result = self.setup(*args) + self.assertEqual(result.returncode, 2, args) + self.assertIn(message, result.stderr, args) + + def stubbed(self, *args: str, fail: str = "") -> tuple[subprocess.CompletedProcess[str], list[str]]: + """Run setup.sh with the stand-ins first on PATH; return the result and the commands they saw.""" + stubs, store, log = self.dir / "stubs", self.dir / "store", self.dir / "stub.log" + if not stubs.exists(): + stubs.mkdir() + store.mkdir() + for name, body in STUBS.items(): + (stubs / name).write_text(f"#!/usr/bin/env bash\n{body}\n", encoding="utf-8") + (stubs / name).chmod(0o755) + (self.app / "Contents" / "MacOS" / "Fake Arcade").write_text("", encoding="utf-8") + (self.app / "Contents" / "MacOS" / "Fake Arcade").chmod(0o755) + path = f"{stubs}{os.pathsep}{os.environ['PATH']}" + for name in STUBS: + self.assertEqual(shutil.which(name, path=path), str(stubs / name), f"{name} must be a stand-in") + log.write_text("", encoding="utf-8") + env = dict(os.environ, HOME=str(self.home), PATH=path, STUB_STORE=str(store), STUB_LOG=str(log), + STUB_FAIL=fail) + result = subprocess.run(["bash", str(SETUP), *args], capture_output=True, text=True, env=env, timeout=60) + return result, log.read_text(encoding="utf-8").splitlines() + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_apply_records_settings_and_undo_puts_them_back(self) -> None: + store = self.dir / "store" + result, _ = self.stubbed("--app", str(self.app), "--apply") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + agent = self.home / "Library" / "LaunchAgents" / "ai.coderabbit.arcade.booth.plist" + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + self.assertTrue(agent.is_file()) + rows = [line.split("\t") for line in record.read_text(encoding="utf-8").splitlines()] + self.assertIn(["defaults", "host", "com.apple.screensaver", "idleTime", "absent", ""], rows) + self.assertIn(["pmset", "sleep", "10"], rows) + self.assertEqual((store / "host-com.apple.screensaver-idleTime").read_text().split(), ["-int", "0"]) + + result, seen = self.stubbed("--undo", "--apply") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn("defaults -currentHost delete com.apple.screensaver idleTime", seen) + self.assertIn("pmset -a sleep 10", seen) + self.assertNotIn("pmset restoredefaults", " ".join(seen)) + self.assertFalse(agent.exists()) + self.assertFalse(record.exists(), "a complete undo removes the record") + self.assertEqual(list(store.iterdir()), [], "every preference setup added is gone again") + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_undo_restores_a_value_that_was_there_before(self) -> None: + store = self.dir / "store" + self.stubbed("--undo") # creates the stand-ins and the store + (store / "host-com.apple.screensaver-idleTime").write_text("integer 300\n", encoding="utf-8") + self.stubbed("--app", str(self.app), "--apply") + _, seen = self.stubbed("--undo", "--apply") + self.assertIn("defaults -currentHost write com.apple.screensaver idleTime -int 300", seen) + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_undo_without_a_record_fails_and_changes_nothing(self) -> None: + result, seen = self.stubbed("--undo", "--apply") + self.assertEqual(result.returncode, 1) + self.assertIn("restoring by hand", result.stdout) + self.assertFalse([command for command in seen if command.startswith(("defaults", "pmset"))]) + + @unittest.skipIf(sys.platform == "darwin", "stand-ins only: never run --apply on a Mac") + def test_a_failed_restore_fails_the_undo(self) -> None: + self.stubbed("--app", str(self.app), "--apply") + (self.dir / "store" / "host-com.apple.screensaver-idleTime").unlink() + record = self.home / "Library" / "Application Support" / "HoppyBooth" / "settings-before.tsv" + record.write_text("defaults\tuser\tcom.apple.CrashReporter\tDialogType\tstring\tbasic\n", encoding="utf-8") + result, _ = self.stubbed("--undo", "--apply", fail="DialogType") + self.assertEqual(result.returncode, 1) + self.assertIn("1 step(s) failed", result.stdout) + self.assertTrue(record.exists(), "a failed undo keeps the record for another try") + + @unittest.skipIf(sys.platform == "darwin", "--apply would change this Mac's settings") + def test_apply_refuses_off_macos(self) -> None: + result = self.setup("--app", str(self.app), "--apply") + self.assertEqual(result.returncode, 2) + self.assertIn("only runs on macOS", result.stderr) + self.assertEqual(list(self.home.iterdir()), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/booth/test_watchdog.py b/tools/booth/test_watchdog.py new file mode 100644 index 00000000..3ecd086c --- /dev/null +++ b/tools/booth/test_watchdog.py @@ -0,0 +1,199 @@ +"""Tests for watchdog.sh, run against a stand-in game. Run: python3 -m unittest discover -s tools/booth -v + +Each test starts the real script under bash with short timings and a fake game that exits, hangs +or stays healthy, then checks what the watchdog did. The waits are generous for slow CI runners; +the assertions count launches with lower bounds. +""" +from __future__ import annotations + +import os +import signal +import subprocess +import tempfile +import time +import unittest +from pathlib import Path + +WATCHDOG = Path(__file__).resolve().parent / "watchdog.sh" + +# Records each launch's arguments and pid, then behaves as FAKE_MODE says. +FAKE_GAME = r"""#!/usr/bin/env bash +echo "$*" >> "$FAKE_RUNS" +echo $$ > "$FAKE_PID" +heartbeat="" +for arg in "$@"; do # the first non-empty one, as AppRouter.heartbeat_path() reads it + case "$arg" in --heartbeat=?*) [ -n "$heartbeat" ] || heartbeat="${arg#--heartbeat=}" ;; esac +done +beat() { # write and rename, as the game does, so the watchdog never reads a half-written number + [ -z "$heartbeat" ] || { date +%s > "$heartbeat.tmp" && mv -f "$heartbeat.tmp" "$heartbeat"; } +} +case "$FAKE_MODE" in + exit) exit 3 ;; + hang) beat; exec sleep 1000 ;; + healthy) while :; do beat; sleep 0.2; done ;; +esac +""" + + +class WatchdogTest(unittest.TestCase): + def setUp(self) -> None: + folder = tempfile.TemporaryDirectory() + self.addCleanup(folder.cleanup) + self.dir = Path(folder.name) + self.game = self.dir / "fake-game" + self.game.write_text(FAKE_GAME, encoding="utf-8") + self.game.chmod(0o755) + self.runs = self.dir / "runs" + self.pid = self.dir / "pid" + self.log = self.dir / "watchdog.log" + self.heartbeat = self.dir / "beat" / "heartbeat" + self.process: subprocess.Popen[bytes] | None = None + + def tearDown(self) -> None: + if self.process and self.process.poll() is None: + self.process.kill() + self.process.wait() + if self.pid.exists(): + try: + os.kill(int(self.pid.read_text()), signal.SIGKILL) + except (ProcessLookupError, ValueError): + pass + + def start(self, mode: str, *options: str, game_args: tuple[str, ...] = ("--", "--booth"), + heartbeat: bool = True) -> None: + env = dict(os.environ, FAKE_RUNS=str(self.runs), FAKE_PID=str(self.pid), FAKE_MODE=mode, TMPDIR=str(self.dir)) + own = ["--heartbeat", str(self.heartbeat)] if heartbeat else [] + command = ["bash", str(WATCHDOG), "--log", str(self.log), *own, + "--poll", "0.2", *options, "--", str(self.game), *game_args] + # The watchdog writes its events to --log; the fake game's own output is discarded. + self.process = subprocess.Popen(command, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + + def stop(self) -> int: + assert self.process is not None + self.process.send_signal(signal.SIGTERM) + return self.process.wait(timeout=20) + + def launches(self) -> list[str]: + return self.runs.read_text(encoding="utf-8").splitlines() if self.runs.exists() else [] + + def events(self) -> str: + return self.log.read_text(encoding="utf-8") if self.log.exists() else "" + + def wait_for(self, condition, seconds: float) -> bool: + deadline = time.monotonic() + seconds + while time.monotonic() < deadline: + if condition(): + return True + time.sleep(0.1) + return condition() + + def test_relaunches_a_game_that_exits(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1", "--max-backoff", "1") + self.assertTrue(self.wait_for(lambda: len(self.launches()) >= 3, 15), self.events()) + self.assertEqual(self.stop(), 0) + self.assertIn("the game exited after", self.events()) + self.assertIn("(status 3)", self.events()) + self.assertIn("relaunching in 1 s", self.events()) + + def test_waits_longer_after_each_quick_exit(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1", "--max-backoff", "2") + self.assertTrue(self.wait_for(lambda: "relaunching in 2 s" in self.events(), 15), self.events()) + self.assertTrue(self.wait_for(lambda: self.events().count("relaunching in 2 s") >= 2, 15), self.events()) + self.stop() + self.assertNotIn("relaunching in 4 s", self.events(), "the wait is capped at --max-backoff") + + def test_restarts_a_hung_game(self) -> None: + self.start("hang", "--hang", "1", "--grace", "0", "--backoff", "1", "--term-wait", "2") + self.assertTrue(self.wait_for(lambda: len(self.launches()) >= 2, 20), self.events()) + self.stop() + self.assertIn("has not written a heartbeat for", self.events()) + self.assertIn("(signal 15)", self.events()) + + def test_leaves_a_healthy_game_alone_and_stops_it_on_sigterm(self) -> None: + self.start("healthy", "--hang", "2", "--grace", "0") + time.sleep(5) + self.assertEqual(len(self.launches()), 1, self.events()) + game = int(self.pid.read_text()) + self.assertEqual(self.stop(), 0) + self.assertIn("stopping the game", self.events()) + with self.assertRaises(ProcessLookupError): + os.kill(game, 0) + + def test_gives_the_game_its_heartbeat_after_the_separator(self) -> None: + self.start("exit", "--backoff", "1") + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], f"-- --booth --heartbeat={self.heartbeat}") + + def test_adds_a_separator_when_the_game_has_none(self) -> None: + self.start("exit", "--backoff", "1", game_args=()) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], f"-- --heartbeat={self.heartbeat}") + + def test_hang_zero_passes_no_heartbeat(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "1") + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.stop() + self.assertEqual(self.launches()[0], "-- --booth") + + def test_stop_file_ends_the_watch_when_the_game_exits(self) -> None: + stop_file = self.dir / "stop" + stop_file.write_text("left from yesterday", encoding="utf-8") + self.start("healthy", "--hang", "0", "--stop-file", str(stop_file)) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + self.assertIn("removed the stop file left from before", self.events()) + self.assertFalse(stop_file.exists()) + stop_file.touch() + os.kill(int(self.pid.read_text()), signal.SIGTERM) # staff close the game + assert self.process is not None + self.assertEqual(self.process.wait(timeout=10), 0) + self.assertIn("not relaunching", self.events()) + self.assertEqual(len(self.launches()), 1) + + def test_default_heartbeat_is_private_to_each_watchdog(self) -> None: + self.start("healthy", "--hang", "2", "--grace", "0", heartbeat=False) + self.assertTrue(self.wait_for(lambda: self.launches(), 10)) + path = Path(self.launches()[0].split("--heartbeat=")[1]) + self.assertEqual(path.parent.parent.resolve(), self.dir.resolve()) + self.assertTrue(path.parent.name.startswith("hoppy-booth.")) + self.assertEqual(path.parent.stat().st_mode & 0o777, 0o700) + time.sleep(3) + self.assertEqual(len(self.launches()), 1, "a healthy game on the private heartbeat is left alone") + self.stop() + self.assertFalse(path.parent.exists(), "the private folder goes when the watchdog stops") + + def test_rejects_a_game_command_that_sets_its_own_heartbeat(self) -> None: + result = subprocess.run(["bash", str(WATCHDOG), "--", "game", "--", "--heartbeat=/elsewhere"], + capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 2) + self.assertIn("passes its own --heartbeat", result.stderr) + + def test_reads_numbers_with_leading_zeros_as_decimal(self) -> None: + self.start("exit", "--hang", "0", "--backoff", "01", "--max-backoff", "08") + self.assertTrue(self.wait_for(lambda: "relaunching in 2 s" in self.events(), 15), self.events()) + self.stop() + self.assertNotIn("value too great", self.events()) + + def test_rejects_bad_usage(self) -> None: + for args, message in ( + ([], "no game command"), + (["--bogus", "--", "true"], "unknown option"), + (["--hang", "soon", "--", "true"], "whole number"), + (["--backoff", "0", "--", "true"], "at least 1"), + (["--poll", "0", "--", "true"], "more than 0"), + (["--log"], "needs a value"), + ): + result = subprocess.run(["bash", str(WATCHDOG), *args], capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 2, args) + self.assertIn(message, result.stderr, args) + + def test_help_prints_the_usage(self) -> None: + result = subprocess.run(["bash", str(WATCHDOG), "--help"], capture_output=True, text=True, timeout=10) + self.assertEqual(result.returncode, 0) + self.assertIn("Usage:", result.stdout) + self.assertIn("--stop-file", result.stdout) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/booth/watchdog.sh b/tools/booth/watchdog.sh new file mode 100755 index 00000000..30f82700 --- /dev/null +++ b/tools/booth/watchdog.sh @@ -0,0 +1,232 @@ +#!/usr/bin/env bash +# Keep the booth game running unattended. The watchdog launches the game, relaunches it whenever it +# exits (a crash, a forced quit, Cmd+Q), and kills and relaunches it when it hangs. The game +# rewrites a heartbeat file every two seconds (app/booth_heartbeat.gd); when the file stops +# changing, the main loop is stuck. Runs on macOS's bash 3.2 and on Linux. +# +# Usage: +# tools/booth/watchdog.sh [options] -- [game arguments] +# +# Example (macOS; tools/booth/macos/setup.sh installs this as a login item): +# tools/booth/watchdog.sh --log ~/booth-watchdog.log -- \ +# "/Applications/Hoppy’s Adventure Arcade.app/Contents/MacOS/" -- --booth +# +# Options: +# --heartbeat heartbeat file for the game (default: in a new private folder under +# $TMPDIR, removed when the watchdog stops) +# --hang restart the game when its heartbeat is older than this; 0 turns hang +# checks off, for builds without --heartbeat (default: 45) +# --grace how long a fresh launch has to write its first heartbeat (default: 60) +# --stop-file if this file exists when the game exits, stop instead of relaunching +# (exit 0); a stop file left from before is removed at startup +# --log append the watchdog's events here instead of standard output +# --backoff first wait before a relaunch; doubles after each short run (default: 2) +# --max-backoff longest wait before a relaunch (default: 30) +# --stable a run at least this long resets the wait to --backoff (default: 60) +# --term-wait how long the game gets to exit after SIGTERM before SIGKILL (default: 10) +# --poll how often to check the game; may be fractional (default: 1) +# +# The game gets --heartbeat= after its other user arguments (a "--" is added if missing); the +# game command must not set its own. SIGTERM, SIGINT or SIGHUP stops the game, then the watchdog, +# with exit status 0. + +heartbeat="" +private_dir="" +hang=45 +grace=60 +stop_file="" +log_file="" +backoff=2 +max_backoff=30 +stable=60 +term_wait=10 +poll=1 + +usage() { + sed -n '2,/^$/p' "$0" | sed -e 's/^# \{0,1\}//' +} + +die() { + echo "watchdog: $*" >&2 + exit 2 +} + +# Validate a whole number of seconds and print it in base 10 ("08" is 8, not bad octal). +whole_number() { + case "$2" in + '' | *[!0-9]*) die "$1 needs a whole number of seconds, not '$2'" ;; + esac + echo $((10#$2)) +} + +while [ $# -gt 0 ]; do + case "$1" in + --heartbeat | --hang | --grace | --stop-file | --log | --backoff | --max-backoff | --stable | --term-wait | --poll) + [ $# -ge 2 ] || die "$1 needs a value" + case "$1" in + --heartbeat) heartbeat="$2" ;; + --hang) hang=$(whole_number "$1" "$2") || exit 2 ;; + --grace) grace=$(whole_number "$1" "$2") || exit 2 ;; + --stop-file) stop_file="$2" ;; + --log) log_file="$2" ;; + --backoff) + backoff=$(whole_number "$1" "$2") || exit 2 + [ "$backoff" -ge 1 ] || die "--backoff must be at least 1 second" + ;; + --max-backoff) max_backoff=$(whole_number "$1" "$2") || exit 2 ;; + --stable) stable=$(whole_number "$1" "$2") || exit 2 ;; + --term-wait) term_wait=$(whole_number "$1" "$2") || exit 2 ;; + --poll) + case "$2" in + '' | *[!0-9.]* | *.*.* | .) die "--poll needs a number of seconds, not '$2'" ;; + esac + awk -v p="$2" 'BEGIN { exit !(p > 0) }' || die "--poll must be more than 0 seconds" + poll="$2" + ;; + esac + shift 2 + ;; + -h | --help) usage; exit 0 ;; + --) shift; break ;; + *) die "unknown option '$1' (the game command goes after --)" ;; + esac +done +[ $# -gt 0 ] || die "no game command; pass it after --" +[ "$max_backoff" -ge "$backoff" ] || max_backoff="$backoff" + +game=("$@") +if [ "$hang" -gt 0 ]; then + separator=0 + for arg in "$@"; do + [ "$arg" = "--" ] && separator=1 + case "$arg" in + --heartbeat=?*) die "the game command sets $arg; the watchdog passes its own --heartbeat" ;; + esac + done + if [ -z "$heartbeat" ]; then + # A folder only this user can write, new for each watchdog: nobody can plant a symlink in it, + # and two watchdogs never read each other's heartbeat. + private_dir=$(mktemp -d "${TMPDIR:-/tmp}/hoppy-booth.XXXXXX") || die "cannot create a heartbeat folder" + heartbeat="$private_dir/heartbeat" + fi + [ "$separator" -eq 1 ] || game+=("--") + game+=("--heartbeat=$heartbeat") +fi + +say() { + local line + line="$(date '+%Y-%m-%dT%H:%M:%S') watchdog: $*" + if [ -n "$log_file" ]; then + echo "$line" >> "$log_file" + else + echo "$line" + fi +} + +child="" +stop_requested=0 + +# SIGTERM, then SIGKILL if the game is still there after --term-wait seconds. The caller reaps it. +terminate() { + local waited=0 + kill -TERM "$1" 2>/dev/null + while kill -0 "$1" 2>/dev/null && [ "$waited" -lt "$term_wait" ]; do + sleep 1 + waited=$((waited + 1)) + done + if kill -0 "$1" 2>/dev/null; then + say "the game (pid $1) ignored SIGTERM for $term_wait s; sending SIGKILL" + kill -KILL "$1" 2>/dev/null + fi +} + +stop() { + trap '' TERM INT HUP + if [ -n "$child" ] && kill -0 "$child" 2>/dev/null; then + say "stopping the game (pid $child)" + terminate "$child" + wait "$child" 2>/dev/null + fi + say "stopped" + exit 0 +} + +# The signal handler only records the request; the loop acts on it at these checks. That way a +# signal that lands between starting the game and recording its pid still stops the game. +check_stop() { + [ "$stop_requested" -eq 0 ] || stop +} +trap 'stop_requested=1' TERM INT HUP +trap '[ -z "$private_dir" ] || rm -rf "$private_dir"' EXIT + +# Sleep in one-second steps, so a stop signal is handled within a second. +pause() { + local slept=0 + while [ "$slept" -lt "$1" ]; do + sleep 1 + slept=$((slept + 1)) + check_stop + done +} + +[ -z "$log_file" ] || mkdir -p "$(dirname "$log_file")" || die "cannot create the folder for $log_file" +if [ "$hang" -gt 0 ]; then + mkdir -p "$(dirname "$heartbeat")" || die "cannot create the folder for $heartbeat" +fi +if [ -n "$stop_file" ] && [ -e "$stop_file" ]; then + rm -f "$stop_file" || die "cannot remove the stop file left from before ($stop_file)" + [ ! -e "$stop_file" ] || die "the stop file left from before is still there ($stop_file)" + say "removed the stop file left from before ($stop_file)" +fi +if [ "$hang" -gt 0 ]; then + say "watching ${game[*]} (restart after $hang s without a heartbeat)" +else + say "watching ${game[*]} (hang checks off)" +fi + +wait_before_relaunch="$backoff" +while :; do + check_stop + [ -z "$heartbeat" ] || rm -f "$heartbeat" + "${game[@]}" & + child=$! + launched=$(date +%s) + say "launched the game (pid $child)" + check_stop + while kill -0 "$child" 2>/dev/null; do + sleep "$poll" + check_stop + [ "$hang" -gt 0 ] || continue + now=$(date +%s) + [ $((now - launched)) -ge "$grace" ] || continue + beat=$(cat "$heartbeat" 2>/dev/null) + case "$beat" in + '' | *[!0-9]*) last="$launched" ;; + *) last="$beat" ;; + esac + if [ $((now - last)) -gt "$hang" ]; then + say "the game has not written a heartbeat for $((now - last)) s; restarting it" + terminate "$child" + break + fi + done + wait "$child" 2>/dev/null + status=$? + check_stop + child="" + ran=$(($(date +%s) - launched)) + if [ "$status" -gt 128 ]; then + say "the game exited after $ran s (signal $((status - 128)))" + else + say "the game exited after $ran s (status $status)" + fi + if [ -n "$stop_file" ] && [ -e "$stop_file" ]; then + say "stop file $stop_file is present; not relaunching" + exit 0 + fi + [ "$ran" -lt "$stable" ] || wait_before_relaunch="$backoff" + say "relaunching in $wait_before_relaunch s" + pause "$wait_before_relaunch" + wait_before_relaunch=$((wait_before_relaunch * 2)) + [ "$wait_before_relaunch" -le "$max_backoff" ] || wait_before_relaunch="$max_backoff" +done