feat(booth): keep the booth game running with a watchdog, Mac kiosk setup and a staff runbook

.github/workflows/ci.yml

@@ -106,6 +106,10 @@ jobs:
         run: python -m unittest discover -s tools/vendor -v
       - name: Booth tools unit tests
         run: python -m unittest discover -s tools/booth -v
+      - name: Shellcheck booth scripts
+        run: |
+          command -v shellcheck >/dev/null || sudo apt-get install -y shellcheck
+          shellcheck tools/booth/watchdog.sh tools/booth/macos/setup.sh
 
   gitleaks:
     name: gitleaks

README.md

@@ -42,6 +42,8 @@ HoppysAdventureArcade.exe -- --booth --route=heroes_vs_villains
 
 Booth mode runs fullscreen and ignores the menu's **Quit Game**, so staff close the window instead. After 90 seconds without input, any game returns to the hub's attract screen. [Run it at a booth](docs/builds.md#run-it-at-a-booth) lists every route and flag.
 
+For a machine that runs unattended all day, follow the [booth runbook](docs/booth-runbook.md). It sets up a watchdog that restarts the game if it closes or freezes, along with the Mac's kiosk settings, and covers the staff checklist.
+
 ## Docs
 
 - Platformer: [Bay Area Blitz](docs/bay-area-blitz.md), [Orbital Ops](docs/orbital-ops.md), [The Enchanted Repo](docs/enchanted-repo.md)

docs/booth-runbook.md

@@ -0,0 +1,97 @@
+# Booth runbook
+
+This runbook is for the staff who run Hoppy’s Adventure Arcade unattended at a booth, such as booth 528 at re:Invent. It covers a Mac running the game, with a second Mac set up the same way as a spare. The watchdog also runs on Linux; Windows isn't covered.
+
+## Pack list
+
+- The booth Mac and a spare Mac, each set up and tested with the steps below.
+- A USB stick with the macOS build zip from [Releases or Actions](builds.md#get-a-build) and a copy of this repository, which has the booth tools in `tools/booth/`.
+- The controllers plus one spare, with their cables or wireless dongles, the display cable and a power strip.
+- A keyboard and mouse for staff. Keep them out of players' reach.
+
+## Set up a machine (once, about 20 minutes)
+
+1. Sign in to the account the booth will use.
+2. Unzip the build and move **Hoppy’s Adventure Arcade.app** to **Applications**.
+3. In Terminal, from the copy of this repository, see what the setup will change:
+   ```sh
+   tools/booth/macos/setup.sh --app "/Applications/Hoppy’s Adventure Arcade.app" --route heroes_vs_villains
+   ```
+   Nothing changes yet. To apply it, run the same command with `--apply` and enter the administrator password when asked. The game starts.
+4. Do the manual steps the script lists at the end: automatic login, Do Not Disturb, the display, the volume and the controllers.
+5. Restart the Mac. The game should start on its own within a minute.
+
+The setup does the following:
+- It installs the watchdog (`tools/booth/watchdog.sh`) as a login item. The watchdog starts the game in booth mode, relaunches it whenever it closes or crashes, and restarts it when it freezes.
+- It removes the download quarantine, so Gatekeeper doesn't block the app.
+- It keeps the Mac awake and turns it back on after a power cut.
+- It turns off the screen saver, automatic updates and crash dialogs.
+
+Booth mode itself runs fullscreen, ignores the menu's **Quit Game**, and sends a game nobody has touched for 90 seconds back to the hub's attract screen ([details](builds.md#run-it-at-a-booth)).
+
+## Every morning
+
+1. Turn the Mac on. It logs in, and the game appears within a minute.
+2. Check that every controller moves Hoppy and that the volume suits the floor.
+3. Play one short round.
+
+## During the show
+
+| What happens | What you'll see |
+|---|---|
+| A player walks away mid-game | After 90 seconds the game returns to the hub. |
+| The game crashes or closes | The watchdog opens it again within a few seconds. |
+| The game freezes | The watchdog restarts it within about a minute. |
+| Someone presses Cmd+Q | The game closes, and the watchdog opens it again. |
+| The player's controller disconnects | The game pauses and asks for the controller. Plug it back in, or press Start on another controller. |
+| You want to clear the Endless Runner scores | **Endless Runner → Leaderboard → Reset scores**. |
+
+## Stop the game for maintenance
+
+```sh
+touch ~/Library/Application\ Support/HoppyBooth/stop
+```
+
+Then press Cmd+Q. The game stays closed. To bring it back, restart the Mac or run:
+
+```sh
+launchctl kickstart gui/$(id -u)/ai.coderabbit.arcade.booth
+```
+
+The watchdog removes the stop file when it starts.
+
+## Swap to the spare
+
+1. Move the display cable and the controllers to the spare Mac and turn it on. It starts the game on its own.
+2. Copy the session logs off the first Mac when there's time (see below).
+
+## Collect the session logs
+
+The game records each session's start and end in `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/booth/`, one file per run of the game. It records the game, the times and the length, and nothing about players. At the end of each day, copy the whole `booth` folder to the USB stick and leave the original in place. To summarize every machine's folder at once:
+
+```sh
+python3 tools/booth/sessions_report.py booth-a/ booth-b/ --csv sessions.csv
+```
+
+## Troubleshooting
+
+The watchdog writes launches, exits and restarts to `~/Library/Logs/HoppyBooth/watchdog.log`, and the game's own output to `~/Library/Logs/HoppyBooth/game.log`.
+
+| Problem | What to do |
+|---|---|
+| No game after login | Read `watchdog.log`. Run `launchctl print gui/$(id -u)/ai.coderabbit.arcade.booth` to see whether the login item is loaded. If it isn't, run the setup again with `--apply`. |
+| macOS says the app can't be opened | Run `xattr -dr com.apple.quarantine "/Applications/Hoppy’s Adventure Arcade.app"`. |
+| The game restarts every minute or two | If `watchdog.log` says the game "has not written a heartbeat", the build is older than the watchdog. Install a current build. |
+| A controller does nothing | Reconnect it. Check **System Settings → Bluetooth** for wireless pads. |
+| Black borders or the wrong size | Check the resolution in **System Settings → Displays**. |
+| Notifications appear over the game | Turn on Do Not Disturb in **Control Center → Focus**. |
+| Nothing helps | Restart the Mac. If the game still doesn't run, swap to the spare. |
+
+## After the show
+
+```sh
+tools/booth/macos/setup.sh --undo           # see what it will change
+tools/booth/macos/setup.sh --undo --apply
+```
+
+`--undo` puts back the settings the first setup recorded on that Mac. Then turn off automatic login and Do Not Disturb, and copy the session logs.

docs/builds.md

@@ -36,6 +36,7 @@ open -a "Hoppy’s Adventure Arcade.app" --args -- --booth --route=heroes_vs_vil
 - **Idle reset.** After 90 seconds with no keyboard, mouse, touch or controller input, any game returns to the hub, whose rotating previews act as the attract screen.
 - **No quitting from the menu.** The menu's **Quit Game** does nothing in booth mode; staff close the window (Cmd+Q, Alt+F4).
 - **Launch route.** `--route=<name>` opens a route at launch (`heroes_vs_villains`, `code_city_brawl`, `bay_area_blitz`, `orbital_ops`, `enchanted_repo`, `runner`, `platformer`, `beatem_up`, `hub`), with or without `--booth`. An unknown name logs an error and opens the hub.
+- **Keep it running.** `tools/booth/watchdog.sh` relaunches the game whenever it exits and restarts it when it hangs. The watchdog passes `--heartbeat=<path>`, which makes the game rewrite that file with the current time every two seconds; a file that stops changing means the game is stuck. On a Mac, `tools/booth/macos/setup.sh` installs the watchdog as a login item and applies the kiosk settings. The [booth runbook](booth-runbook.md) walks staff through setup, the show days and recovery.
 - **Session log.** Booth mode records each game session's start and end in `user://booth/`, one file per run of the game (`sessions-<date and time>-<process id>.jsonl`). Each entry records the game, the local start and end times, the duration, how the session ended (`menu`, `idle` or `quit`) and how many times the player retried. It records nothing about players and never uses the network. A start with no end means the game stopped without quitting: a crash, a power cut or a forced kill. The game prints its file's full path at startup. `--session-log=<folder>` writes to another folder and works without `--booth`, for playtests.
 
 `user://` is `~/Library/Application Support/Godot/app_userdata/Hoppy’s Adventure Arcade/` on macOS, `%APPDATA%\Godot\app_userdata\Hoppy’s Adventure Arcade\` on Windows and `~/.local/share/godot/app_userdata/Hoppy’s Adventure Arcade/` on Linux. To summarize the logs, copy each machine's `booth` folder and run:

hoppys-adventure-arcade/app/app_router.gd

@@ -1,5 +1,6 @@
 extends CanvasLayer
 const BoothLog := preload("res://app/booth_log.gd")
+const BoothHeartbeat := preload("res://app/booth_heartbeat.gd")
 const ROUTES := {
 	"heroes_vs_villains": "res://scenes/beatem_up/reinvent/heroes_vs_villains.tscn",
 	"code_city_brawl": "res://scenes/beatem_up/code_city_brawl.tscn",
@@ -30,12 +31,17 @@ const STICK_DEADZONE := 0.4
 ## of them is passed.
 const SESSION_LOG_PREFIX := "--session-log="
 const SESSION_LOG_DEFAULT := "user://booth"
+## Watchdog support (app/booth_heartbeat.gd): `--heartbeat=<path>` keeps that file updated with the
+## time every two seconds, so tools/booth/watchdog.sh can restart a game that hangs. Off unless
+## passed; the watchdog passes it.
+const HEARTBEAT_PREFIX := "--heartbeat="
 var busy := false
 var veil: ColorRect
 var booth := false
 var idle_limit := IDLE_SECONDS
 var idle_seconds := 0.0
 var session_log: BoothLog
+var heartbeat: BoothHeartbeat
 func _ready() -> void:
 	# The veil tween must keep running while a microgame has the tree paused.
 	process_mode = Node.PROCESS_MODE_ALWAYS
@@ -55,6 +61,11 @@ func _ready() -> void:
 	var log_path := session_log_path(args, booth)
 	if log_path != "":
 		start_session_log(log_path)
+	var beat_path := heartbeat_path(args)
+	if beat_path != "":
+		heartbeat = BoothHeartbeat.new(beat_path)
+		add_child(heartbeat)
+		print("AppRouter: heartbeat at %s" % ProjectSettings.globalize_path(beat_path))
 	var start := launch_route(args)
 	if start != "":
 		_launch.call_deferred(start)
@@ -125,6 +136,13 @@ func start_session_log(folder: String) -> void:
 	session_log = BoothLog.new(folder)
 	print("AppRouter: session log at %s" % ProjectSettings.globalize_path(session_log.path))
 
+## The heartbeat file named by `--heartbeat=<path>`, or "" when absent.
+static func heartbeat_path(args: PackedStringArray) -> String:
+	for arg in args:
+		if arg.begins_with(HEARTBEAT_PREFIX) and arg.length() > HEARTBEAT_PREFIX.length():
+			return arg.trim_prefix(HEARTBEAT_PREFIX)
+	return ""
+
 func _launch(route: String) -> void:
 	await go(route)
 	var scene := get_tree().current_scene

hoppys-adventure-arcade/app/booth_heartbeat.gd

@@ -0,0 +1,58 @@
+extends Node
+## Booth watchdog support. While the game's main loop runs, this rewrites a file with the current
+## Unix time every INTERVAL seconds. tools/booth/watchdog.sh restarts the game when the file goes
+## stale, which catches a hang that leaves the process alive. AppRouter adds it only when the game
+## is launched with `--heartbeat=<path>`.
+const INTERVAL := 2.0
+var path := ""
+var elapsed := 0.0
+## Set after the first failed write, so the warning prints once. Every beat still tries again: a
+## heartbeat that stays stale makes the watchdog restart the game.
+var warned := false
+
+func _init(file_path: String) -> void:
+	path = file_path
+	name = "BoothHeartbeat"
+	# Keep beating while a game has the tree paused: a paused game is not a hung one.
+	process_mode = Node.PROCESS_MODE_ALWAYS
+
+func _ready() -> void:
+	var directory := path.get_base_dir()
+	if directory != "" and not DirAccess.dir_exists_absolute(directory):
+		DirAccess.make_dir_recursive_absolute(directory)
+	beat()
+
+func _process(delta: float) -> void:
+	elapsed += delta
+	if elapsed >= INTERVAL:
+		elapsed = 0.0
+		beat()
+
+## Write to a temporary file and rename it over the heartbeat, so the watchdog never reads a
+## half-written number. A leftover temporary file is removed first, so a link planted in its place
+## is replaced rather than written through. The watchdog keeps the heartbeat in a private folder.
+func beat() -> void:
+	var temporary := path + ".tmp"
+	var folder := DirAccess.open(path.get_base_dir())
+	var leftover := temporary.get_file()
+	if folder != null and (folder.file_exists(leftover) or folder.is_link(leftover)):
+		folder.remove(leftover)
+	var file := FileAccess.open(temporary, FileAccess.WRITE)
+	if file == null:
+		_fail(error_string(FileAccess.get_open_error()))
+		return
+	var written := file.store_line(str(int(Time.get_unix_time_from_system())))
+	file.close()
+	if not written:
+		_fail("write failed")
+		return
+	var error := DirAccess.rename_absolute(temporary, path)
+	if error != OK:
+		_fail(error_string(error))
+
+func _fail(reason: String) -> void:
+	if warned:
+		return
+	warned = true
+	var problem := "Booth heartbeat: cannot write %s (%s)." % [path, reason]
+	push_warning(problem + " If it stays stale, the watchdog restarts the game.")

hoppys-adventure-arcade/app/booth_heartbeat.gd.uid

@@ -0,0 +1 @@
+uid://dpc1yt8at1sty

hoppys-adventure-arcade/tests/booth_heartbeat.gd

@@ -0,0 +1,111 @@
+extends SceneTree
+## Booth heartbeat (app/booth_heartbeat.gd): with --heartbeat=<path> the game keeps that file
+## updated with the current Unix time, including while a game has paused the tree; without the flag
+## there is no heartbeat; a link planted where the temporary file goes is replaced, not written
+## through; a heartbeat it can't write warns instead of failing. The flag is checked by launching
+## the game as a child process, so _ready()'s argument handling is covered. Every run starts from an
+## empty folder of its own. Run headless with --script res://tests/booth_heartbeat.gd.
+const BoothHeartbeat := preload("res://app/booth_heartbeat.gd")
+const FOLDER := "user://test_heartbeat"
+const BEAT := FOLDER + "/beat"
+const CHILD_BEAT := FOLDER + "/child_beat"
+## Wall-clock limit for the child launch, for runs outside tools/tests/run_tests.py's timeout.
+const CHILD_SECONDS := 60.0
+var failures := 0
+var router: Node
+
+func _initialize() -> void:
+	call_deferred("run")
+
+func check(ok: bool, message: String) -> void:
+	print("PASS: " if ok else "FAIL: ", message)
+	if not ok:
+		failures += 1
+
+func wait(seconds: float) -> void:
+	await create_timer(seconds, true).timeout
+
+## Launch the game headless with user args for two simulated seconds and return its exit code,
+## or -1 if it outlives CHILD_SECONDS and had to be killed.
+func launch(user_args: Array[String]) -> int:
+	var args: Array[String] = ["--headless", "--path", ProjectSettings.globalize_path("res://"),
+		"--fixed-fps", "60", "--quit-after", "120", "--"]
+	args.append_array(user_args)
+	var pid := OS.create_process(OS.get_executable_path(), args)
+	var deadline := Time.get_ticks_msec() + int(CHILD_SECONDS * 1000.0)
+	while OS.is_process_running(pid):
+		if Time.get_ticks_msec() > deadline:
+			OS.kill(pid)
+			return -1
+		await wait(0.1)
+	return OS.get_process_exit_code(pid)
+
+func read(path: String) -> String:
+	return FileAccess.get_file_as_string(path).strip_edges()
+
+## True when the file holds a Unix time within a minute of now.
+func fresh(path: String) -> bool:
+	var text := read(path)
+	return text.is_valid_int() and absi(text.to_int() - int(Time.get_unix_time_from_system())) < 60
+
+func clean() -> void:
+	var dir := DirAccess.open(FOLDER)
+	if dir == null:
+		return
+	dir.include_hidden = true
+	for file_name in dir.get_files():
+		dir.remove(file_name)
+	DirAccess.remove_absolute(FOLDER)
+
+func run() -> void:
+	router = root.get_node("AppRouter")
+	clean()
+	DirAccess.make_dir_recursive_absolute(FOLDER)
+	await wait(0.3)
+	check(router.heartbeat == null, "No heartbeat without --heartbeat")
+	check(router.heartbeat_path(PackedStringArray(["--booth"])) == "",
+		"--booth alone has no heartbeat")
+	check(router.heartbeat_path(PackedStringArray(["--heartbeat=user://beat"])) == "user://beat",
+		"--heartbeat names the file")
+	check(router.heartbeat_path(PackedStringArray(["--heartbeat="])) == "",
+		"An empty --heartbeat is ignored")
+
+	var code := await launch(["--booth", "--heartbeat=" + ProjectSettings.globalize_path(CHILD_BEAT)])
+	check(code == 0, "A launch with --heartbeat exits cleanly (exit %d)" % code)
+	check(fresh(CHILD_BEAT), "--heartbeat makes the game write the current Unix time")
+	check(not FileAccess.file_exists(CHILD_BEAT + ".tmp"), "The heartbeat leaves no temporary file")
+
+	# A link where the temporary file goes: the heartbeat replaces it instead of writing through it.
+	var decoy := FOLDER + "/decoy"
+	var keep := FileAccess.open(decoy, FileAccess.WRITE)
+	keep.store_line("keep me")
+	keep.close()
+	var linked := DirAccess.open(FOLDER).create_link(ProjectSettings.globalize_path(decoy),
+		ProjectSettings.globalize_path(BEAT + ".tmp")) == OK
+	var beat := BoothHeartbeat.new(BEAT)
+	root.add_child(beat)
+	check(fresh(BEAT), "The heartbeat writes as soon as it starts")
+	if linked:
+		check(read(decoy) == "keep me", "A link planted at the temporary path is not written through")
+	var marker := FileAccess.open(BEAT, FileAccess.WRITE)
+	marker.store_line("0")
+	marker.close()
+	paused = true
+	await wait(beat.INTERVAL + 0.6)
+	check(fresh(BEAT), "The heartbeat keeps writing while a game has the tree paused")
+	paused = false
+
+	# The heartbeat file itself where a folder should be: warn, keep running, never an error.
+	var broken := BoothHeartbeat.new(BEAT + "/nested")
+	root.add_child(broken)
+	check(broken.warned, "A heartbeat it can't write warns")
+	broken.beat()
+	check(broken.is_inside_tree(), "A failed heartbeat leaves the game running")
+
+	beat.queue_free()
+	broken.queue_free()
+	await process_frame
+	clean()
+	await root.get_node("AudioService").settle()
+	print("Booth heartbeat failures: ", failures)
+	quit(1 if failures else 0)

hoppys-adventure-arcade/tests/booth_heartbeat.gd.uid

@@ -0,0 +1 @@
+uid://bkg1o2uubr6lh