Sound: audio.play in @threert/gameplay, played by the engine on every platform

docs/RUNTIME.md

@@ -387,6 +387,7 @@ Messages a game posts:
 | `net.close` | `id`, `code?`, `reason?` | closes it (1000 unless `code` is 3000–4999); answers `net.closed` |
 | `net.fetch` | `id`, `url`, `method?`, `headers?`, `body?` | one HTTP request (`GET` unless `method`); answers `net.fetched` |
 | `hud` | `hud` | draws the HUD it describes ("HUD", below). `hud.set` posts it; a game does not post it by hand |
+| `audio.*` | below | plays sound ("Sound", below), with the engine's `audio` module. `audio.play` and the rest post them |
 
 Messages an app sends:
 
@@ -418,11 +419,11 @@ a binary WebSocket frame is dropped. Which addresses a game may reach is the app
 
 The apps in this repository:
 
-| app | input | `net.*` | `launch` | HUD |
-| --- | --- | --- | --- | --- |
-| `threert-render play` (`view.rs`) | keys by physical position, mouse buttons, wheel; the pointer captured for `"look"` (a click; Esc lets go, Esc again quits) | WebSockets and HTTP on a thread each (`src/net.rs`: tungstenite and ureq on rustls; text frames of at most 1 MiB, bodies of at most 4 MiB, 16 sockets, 32 requests at once), any address | `--launch '{"room":"K7Q4MZ"}'` | not drawn; `--hud-log` prints each description |
-| `threert-render live` (`live/session.rs`) | whatever the driver sends | none: the driver is the app | `--launch` | on stdout with every other post |
-| the three.js view (`tools/three/view.ts`, `host.ts`, `hud.ts`) | keyboard, mouse, the Pointer Lock API | `WebSocket`, `fetch` (the browser's rules: same-origin policy, CORS) | the page's query (`?room=K7Q4MZ&name=Ada`) | DOM over the canvas |
+| app | input | `net.*` | `launch` | HUD | sound |
+| --- | --- | --- | --- | --- | --- |
+| `threert-render play` (`view.rs`) | keys by physical position, mouse buttons, wheel; the pointer captured for `"look"` (a click; Esc lets go, Esc again quits) | WebSockets and HTTP on a thread each (`src/net.rs`: tungstenite and ureq on rustls; text frames of at most 1 MiB, bodies of at most 4 MiB, 16 sockets, 32 requests at once), any address | `--launch '{"room":"K7Q4MZ"}'` | not drawn; `--hud-log` prints each description | played (`audio` feature) |
+| `threert-render live` (`live/session.rs`) | whatever the driver sends | none: the driver is the app | `--launch` | on stdout with every other post | on stdout: the driver's |
+| the three.js view (`tools/three/view.ts`, `host.ts`, `hud.ts`) | keyboard, mouse, the Pointer Lock API | `WebSocket`, `fetch` (the browser's rules: same-origin policy, CORS) | the page's query (`?room=K7Q4MZ&name=Ada`) | DOM over the canvas | Web Audio by the same rules (`audio.ts`) |
 
 - `live` speaks the channel as JSON lines, so a game's use of it is testable without a window: on stdin
   `{"cmd":"host","messages":[{"t":"net.opened","id":"room"}, ...]}`, on stdout one line `{"host":[...]}` with what
@@ -553,6 +554,58 @@ hud.flash("damage", 1.2);        // the player was hit from 1.2 rad clockwise of
 - **Text is text.** An app shows a game's strings as they are, never as markup.
 - A scene built anew in the same runtime posts its first description whatever the last one was.
 
+### Sound
+
+Unlike the HUD, sound is not left to each app: a shot is to sound the same in ThreeRT for macOS, on a Steam Deck and
+on an Android handheld. The game names a file and a place (`packages/gameplay/src/audio.ts`); the engine's library
+decodes, places and mixes it (`renderer/src/audio`, the `audio` feature, on by default and in the clients), and an
+app only hands it the messages and the camera. The device is the one part that differs a platform.
+
+```ts
+import { audio } from "@threert/gameplay";
+
+setup() { audio.preload(["assets/sounds/shot.ogg"]); ... }       // decoded ahead, off the frame
+frame(f) {
+  if (fired) audio.play("assets/sounds/shot.ogg", { at: muzzle, gain: 0.9, pitch: 0.97 + 0.06 * rng(), falloff: 8 });
+  audio.play("assets/sounds/tick.wav");                            // no `at`: in the player's head
+  audio.play("assets/sounds/fire.ogg", { at: brazier, loop: true, key: "brazier" });   // then audio.move / audio.stop
+}
+```
+
+- **Messages** (script → app; nothing comes back):
+
+  | `t` | fields | |
+  | --- | --- | --- |
+  | `audio.play` | `sound`, `at?` `[x, y, z]`, `gain?` (0..4, 1), `pitch?` (0.25..4, 1), `loop?`, `key?`, `falloff?` (m, 2) | starts a sound |
+  | `audio.stop` | `key?`, `fade?` (s, 0.05) | fades out the sounds of `key`, or all of them |
+  | `audio.move` | `key`, `at` | moves the sounds of `key` |
+  | `audio.preload` | `sounds` | decodes these files on a thread of their own |
+- **Files.** A sound is a file of the game, named as the scene names its files (`assets/sounds/shot.ogg`, relative to
+  the package; nothing outside it): WAV (PCM or float), Ogg Vorbis, MP3, FLAC, at any rate, one or two channels, at
+  most ten minutes. A package carries everything under `assets/`. Decoded once (by Symphonia: the same samples on
+  every platform), kept until the game closes; a file that is missing or unreadable is said once in the log and then
+  plays nothing.
+- **The listener** is the camera the frame was drawn from: its position, forward and right. A game sends nothing for
+  it.
+- **Placing** (`renderer/src/audio/mix.rs`): a sound at `at` keeps its level out to `falloff` metres, then falls as
+  falloff / distance (−6 dB a doubling) and fades out between 100 and 120 m. It is panned by its direction from the
+  listener's forward, with equal power and never more than 80 % to one side, and comes back towards the middle within
+  `falloff`. Far and behind it is duller: a one-pole low pass from 18 kHz, divided by 1 + distance / 40 m, halved
+  behind. Without `at` a sound is in the middle at its own gain.
+- **Limits:** 32 plays a frame (more are dropped), 48 voices at once (the quietest one-shot goes), 8 of one file (its
+  oldest goes). Levels ramp over each block; a stop fades. The app's volume, then a soft limiter above −2 dB.
+- **Presentation only.** Play sounds from the frame script and from host listeners, never from a handler: handlers
+  are replayed and verified, and a sound is not state. A node with no app (a test, a render node) drops the posts.
+- **Apps:** each message the game posts goes to `audio::Audio::post` (it answers false for what is not `audio.*`), and
+  once a frame `Audio::listener(camera)`. The device opens on the first sound and closes with the game (`close`) or
+  while the app is in the background (`set_paused`; loops go on where they were, one-shots in between are dropped);
+  `set_volume` is the app's own control. Devices (`renderer/src/audio/out.rs`): Core Audio and WASAPI by cpal, ALSA's
+  `default` on Linux and AAudio on Android, both loaded at run time, so a Linux or Android build links no sound
+  library and is silent, not broken, where there is none. `threert-render play` plays a game's sound the same way.
+  The three.js view plays it on Web Audio by the same numbers (`tools/three/audio.ts`, held to mix.rs by
+  `tools/three/test/audio.test.ts`; its low pass is a biquad, the one difference), resumed on the page's first key or
+  click as browsers require. The engine's library in a browser (wasm32) has no `audio` module.
+
 ## The runtime's shape: Rust owns the world, the script engine is a guest
 
 Decided by the user on 2026-11-10, with the measurements below.

packages/gameplay/README.md

@@ -62,6 +62,7 @@ export default defineGame({
 - **Deterministic and replayable.** Events are logged with what they read and wrote; `replayLog` and `verifyLog` rebuild a world from its log, and seeded randomness (`seededRandom`) gives the same result on every machine.
 - **World handlers** (`on`) reach the engine's own world: physics, a ray cast into the scene, objects added and removed as an event arrives.
 - **Input, a HUD and the app.** A game with `pointer: "look"` reads the keys and the mouse every frame (`f.input.move`, `f.input.look`, `f.input.pressed("Space")`); `hud.set(...)` describes a crosshair, figures, bars and a scoreboard for the app to draw; `host.post` / `host.on` reach what the app owns (a WebSocket, an HTTP request, how the game was opened).
+- **Sound.** `audio.play("assets/sounds/shot.ogg", { at: muzzle, pitch: 1.04 })` plays a file of the game at a place in the world (or, without `at`, in the player's head); the engine decodes, places and mixes it the same way on every platform, with the camera as the ears. `audio.preload`, `audio.stop(key)` and `audio.move(key, at)` for the rest.
 
 How scene code runs in the engine is in [RUNTIME.md](https://github.com/meshy-dev/ThreeRT/blob/main/docs/RUNTIME.md).
 

packages/gameplay/src/audio.ts

@@ -0,0 +1,103 @@
+// Sound: what the game plays, from its package's files, at a place in the world or in the player's head. The engine
+// decodes, places and mixes it (renderer/src/audio): the same code in every app that runs the game, so a shot sounds
+// the same in ThreeRT for macOS, on a Steam Deck and on an Android handheld. `audio.play(...)` posts
+// `{ t: "audio.play", ... }` on the host channel (host.ts); docs/RUNTIME.md "Sound" is the contract.
+//
+//   setup() {
+//     audio.preload(["assets/sounds/shot.ogg", "assets/sounds/step-1.ogg"]);   // decoded ahead, off the frame
+//     ...
+//   },
+//   frame(f) {
+//     if (fired) audio.play("assets/sounds/shot.ogg", { at: muzzle, pitch: 0.97 + 0.06 * rng() });
+//     if (f.input.pressed("Escape")) audio.play("assets/sounds/ui-tick.ogg");            // no `at`: in the head
+//     audio.play("assets/sounds/fire.ogg", { at: brazier, loop: true, key: "brazier" }); // once; then
+//     audio.move("brazier", brazier); audio.stop("brazier", 0.5);
+//   }
+//
+// - A sound is a file of the game (WAV, Ogg Vorbis, MP3 or FLAC), named as the scene names its files
+//   ("assets/sounds/shot.ogg"); a package carries everything under assets/. It is decoded once and kept while the game
+//   is open: when first played, or ahead after `preload` (do that in setup for what plays in the first seconds).
+// - The ears are the camera the frame is drawn from: nothing to send. `at` is in metres, world space; the sound keeps
+//   its level out to `falloff` metres (2 by default) and falls as falloff / distance beyond, panned by its direction.
+// - `gain` (0 to 4, 1 by default) and `pitch` (0.25 to 4: a playback rate, 1.1 is a little higher and shorter) vary a
+//   sound that repeats: a step, a shot.
+// - Sound is presentation: play it from the frame script and from host listeners, never from a handler (handlers are
+//   replayed and verified; a sound is not state). An app that plays no sound ignores the messages, and the game goes
+//   on the same.
+// - At most 32 plays a frame, 48 voices at once, 8 of one sound (the oldest goes); the rest is dropped.
+
+import { host } from "./host.ts";
+
+/** A point: `[x, y, z]` or anything with x, y, z (a Vector3). */
+export type AudioPoint = readonly [number, number, number] | { x: number; y: number; z: number };
+
+export interface PlayOptions {
+  /** Where in the world, metres. Without it the sound is in the player's head (the middle, its own level). */
+  at?: AudioPoint;
+  /** 0 to 4; 1 by default. */
+  gain?: number;
+  /** Playback rate, 0.25 to 4; 1 by default. */
+  pitch?: number;
+  /** Plays until stopped (`key` names it for `stop` and `move`). */
+  loop?: boolean;
+  /** The name `stop` and `move` reach this sound by (several sounds may share one). */
+  key?: string;
+  /** Metres out to which the sound keeps its level (2 by default): larger for a loud one, a gunshot, an engine. */
+  falloff?: number;
+}
+
+/** The messages, as posted. */
+export type AudioPost =
+  | { t: "audio.play"; sound: string; at?: [number, number, number]; gain?: number; pitch?: number; loop?: boolean; key?: string; falloff?: number }
+  | { t: "audio.stop"; key?: string; fade?: number }
+  | { t: "audio.move"; key: string; at: [number, number, number] }
+  | { t: "audio.preload"; sounds: string[] };
+
+export interface Audio {
+  /** Plays `sound` (a file of the game). */
+  play(sound: string, options?: PlayOptions): void;
+  /** Fades out the sounds of `key` (every sound without one) over `fade` seconds (0.05 by default). */
+  stop(key?: string, fade?: number): void;
+  /** Moves the sounds of `key` (a loop that follows something). */
+  move(key: string, at: AudioPoint): void;
+  /** Decodes these files ahead, off the frame. */
+  preload(sounds: readonly string[]): void;
+}
+
+const round = (v: number) => Math.round(v * 1000) / 1000;
+
+function point(p: AudioPoint): [number, number, number] {
+  const [x, y, z] = Array.isArray(p) ? (p as readonly number[]) : [(p as { x: number }).x, (p as { y: number }).y, (p as { z: number }).z];
+  if (![x, y, z].every((v) => typeof v === "number" && Number.isFinite(v))) throw new Error(`audio: not a point: ${JSON.stringify(p)}`);
+  return [round(x!), round(y!), round(z!)];
+}
+
+function name(sound: unknown): string {
+  if (typeof sound !== "string" || sound === "") throw new Error(`audio: a sound is a file's path, not ${JSON.stringify(sound)}`);
+  return sound;
+}
+
+export const audio: Audio = {
+  play(sound, o = {}) {
+    const m: Extract<AudioPost, { t: "audio.play" }> = { t: "audio.play", sound: name(sound) };
+    if (o.at !== undefined) m.at = point(o.at);
+    if (o.gain !== undefined && o.gain !== 1) m.gain = round(o.gain);
+    if (o.pitch !== undefined && o.pitch !== 1) m.pitch = round(o.pitch);
+    if (o.loop) m.loop = true;
+    if (o.key !== undefined) m.key = String(o.key);
+    if (o.falloff !== undefined) m.falloff = round(o.falloff);
+    host.post(m);
+  },
+  stop(key, fade) {
+    const m: Extract<AudioPost, { t: "audio.stop" }> = { t: "audio.stop" };
+    if (key !== undefined) m.key = String(key);
+    if (fade !== undefined) m.fade = round(fade);
+    host.post(m);
+  },
+  move(key, at) {
+    host.post({ t: "audio.move", key: String(key), at: point(at) });
+  },
+  preload(sounds) {
+    host.post({ t: "audio.preload", sounds: sounds.map(name) });
+  },
+};

packages/gameplay/src/host.ts

@@ -21,8 +21,8 @@
 // - The same object is `globalThis.__threert_host`, for a library that does not import this package.
 //
 // The message families an app may implement are in docs/RUNTIME.md ("The host channel"): `net.*` (sockets and HTTP
-// the app owns), `hud` (hud.ts), `launch` (how the game was opened), and `input` (input.ts), which the node host
-// takes itself.
+// the app owns), `hud` (hud.ts), `audio.*` (audio.ts: played by the engine's mixer), `launch` (how the game was
+// opened), and `input` (input.ts), which the node host takes itself.
 
 import type { WorldContext } from "./world.ts";
 

packages/gameplay/src/index.ts

@@ -8,6 +8,8 @@ export { host } from "./host.ts";
 export type { Host, HostEvents, HostListener, HostMessage, LaunchParams, NetPost } from "./host.ts";
 export { InputState } from "./input.ts";
 export type { Input, InputMessage } from "./input.ts";
+export { audio } from "./audio.ts";
+export type { Audio, AudioPoint, AudioPost, PlayOptions } from "./audio.ts";
 export { hud } from "./hud.ts";
 export type { Hud, HudAnchor, HudBar, HudCrosshair, HudFlash, HudItem, HudLine, HudList, HudMessage, HudState, HudTable, HudText, HudTone } from "./hud.ts";
 export { runWorldHandler } from "./world.ts";

packages/gameplay/test/audio.test.ts

@@ -0,0 +1,44 @@
+// Sound (src/audio.ts): what the game asks for reaches the app's queue as the messages docs/RUNTIME.md "Sound"
+// describes, points as arrays, defaults left out, and a sound that is not a path refused.
+import { test } from "node:test";
+import assert from "node:assert/strict";
+import { Vector3 } from "@threert/scene";
+import { audio } from "../src/index.ts";
+
+function app(): { posts: { t: string; [k: string]: unknown }[]; close(): void } {
+  const posts: { t: string }[] = [];
+  const g = globalThis as { __host_post?: (text: string) => unknown };
+  g.__host_post = (text) => void posts.push(JSON.parse(text));
+  return { posts, close: () => void delete g.__host_post };
+}
+
+test("play, stop, move and preload post their messages", () => {
+  const a = app();
+  try {
+    audio.preload(["assets/sounds/shot.ogg"]);
+    audio.play("assets/sounds/shot.ogg", { at: new Vector3(1, 2.00049, -3), gain: 0.8, pitch: 1, falloff: 6 });
+    audio.play("assets/sounds/tick.wav");
+    audio.play("assets/sounds/fire.ogg", { at: [0, 1, 0], loop: true, key: "fire" });
+    audio.move("fire", { x: 1, y: 1, z: 0 });
+    audio.stop("fire", 0.5);
+    audio.stop();
+    assert.deepEqual(a.posts, [
+      { t: "audio.preload", sounds: ["assets/sounds/shot.ogg"] },
+      { t: "audio.play", sound: "assets/sounds/shot.ogg", at: [1, 2, -3], gain: 0.8, falloff: 6 },
+      { t: "audio.play", sound: "assets/sounds/tick.wav" },
+      { t: "audio.play", sound: "assets/sounds/fire.ogg", at: [0, 1, 0], loop: true, key: "fire" },
+      { t: "audio.move", key: "fire", at: [1, 1, 0] },
+      { t: "audio.stop", key: "fire", fade: 0.5 },
+      { t: "audio.stop" },
+    ]);
+  } finally {
+    a.close();
+  }
+});
+
+test("a sound is a path and a point is three finite numbers", () => {
+  assert.throws(() => audio.play(""), /a sound is a file's path/);
+  assert.throws(() => audio.play("a.ogg", { at: [0, Number.NaN, 0] }), /not a point/);
+  // Without an app nothing listens, and nothing throws.
+  audio.play("a.ogg");
+});