diff --git a/CONTEXT.md b/CONTEXT.md index f7280037..3398ed5a 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -11,7 +11,7 @@ The product name, in use since 2026-09-15 and written as one word since 2026-10- _Avoid_: Vibe Buddy (two words, the spelling until 2026-10-09), AgentBeacon, beacond, beacon-hook (old names, only when talking about history), 氛围助手 ("vibe assistant"; that is a description, not a name) **The buddy (氛围小助手)**: -The only anthropomorphic presence on the device: the one who watches the Agents and speaks up for you. It is a role, not a look; at any moment it wears exactly one Character. Before 2026-09-15 it was called 小灯灵 ("little lamp sprite"); in code it is `buddy`. +The only anthropomorphic presence on the device: the one who watches the Agents and speaks up for you. It is a role, not a look; at any moment it wears exactly one Character. It speaks in two ways: lines, written in advance, when an announcement fires; and spoken replies, written on the spot by the resident session in its persona, when the user talks to it. Before 2026-09-15 it was called 小灯灵 ("little lamp sprite"); in code it is `buddy`. _Avoid_: 小灯灵 (old name), pet (fine as a generic word, not when referring to this role), Pet, Codex pet, assistant (on its own it means an Agent, see below) **Character (角色)**: @@ -23,7 +23,7 @@ A local AI assistant observed by VibeBuddy, such as Codex or Claude Code. An Age _Avoid_: client, AI, assistant; also never means a subagent inside Claude Code **Adapter**: -The layer that translates one Agent's events into VibeBuddy domain concepts, consisting of the local privacy-filtering script and the event mapping inside the daemon. There is one Adapter per Agent; aggregation logic does not belong to the Adapter. +The layer that translates between one Agent and VibeBuddy domain concepts, in both directions: the Agent's events into VibeBuddy concepts (the local privacy-filtering script and the event mapping inside the daemon), and voice commands and spoken replies into and out of that Agent's Sessions. There is one Adapter per Agent; aggregation logic does not belong to the Adapter. _Avoid_: integration, plugin, connector **App**: @@ -65,11 +65,11 @@ _Avoid_: environment, entry point, entrypoint, surface, terminal (a terminal is ### Sound **Announcement voice (播报音色)**: -The single voice the buddy speaks with, shared by all its lines. It belongs to the Character and is never picked on its own: changing voice means changing Character. +The single voice the buddy speaks with, shared by all its lines and spoken replies. It belongs to the Character and is never picked on its own: changing voice means changing Character. _Avoid_: sound, speech (speech means the lines themselves), TTS **Persona (人设)**: -A Character's personality and way of talking, written down so that its lines can be written in it. A persona never sees what the Agents are working on. +A Character's personality and way of talking, written down so that its lines can be written in it. Lines written in a persona never see what the Agents are working on; only the resident session, writing spoken replies in it, does. _Avoid_: system prompt (that is one way to use a persona), personality setting, style **Line (台词)**: @@ -92,6 +92,24 @@ _Avoid_: do not disturb, DND, sound off The loudness level of the device speaker, from 20 to 100, stored on the device and kept across reboots; announcements and the Pomodoro chime share it. The App's slider is only a remote control, and it always shows the value the device reports back; the device menu changes it too. The minimum is not zero: the only way to silence the speaker is mute. _Avoid_: loudness, volume settings (that is a place in the App's UI, not this concept) +### Voice commands + +**Voice command (语音指令)**: +One sentence the user speaks to the box, turned into text and handed to the resident session as the user's own input. An Agent is the one who understands and acts on it; VibeBuddy only carries it, and never answers or executes a voice command itself. +_Avoid_: voice assistant (an assistant means an Agent), prompt (that is the Agent's own word), dictation, voice input + +**Resident session (常驻会话)**: +The one long-running Agent Session every voice command goes to, so the user never has to say which work they mean. It answers from what it can see of the Activities and hands work on to other Sessions, but does no work itself: it runs no commands and changes no files. There is only one, and it is never restarted to make room: when its context grows long, it is compacted. It is an Agent that speaks as the buddy, in the persona of the Character the user picked, and changes persona when the Character changes; it is never an Activity: it makes no task cards, announcements or stats. +_Avoid_: assistant, concierge, dispatcher, main session + +**Brain (大脑)**: +The Agent the resident session runs on: the user's own Codex, or the user's own Claude Code with their own Anthropic API key (never a Claude subscription). The user picks one, or leaves it automatic: Codex when it is ready, else Claude Code. Each brain has its own resident session; the UI calls the setting "Answered by". +_Avoid_: model (that is what the brain runs on), backend, engine, provider + +**Spoken reply (口头回复)**: +The short sentence the resident session writes for the user to hear in answer to a voice command, synthesized when it is spoken, in the announcement voice of the Character the user picked. Unlike a line, it is written at the moment it is spoken and may talk about what the Agents are doing. +_Avoid_: line (lines are written in advance), announcement, TTS, response + ### Modes **Mode (模式)**: @@ -151,6 +169,12 @@ Fixed wording for the App's English UI and English logs. Use these words when wr | 固件 / 刷入 | firmware / flash | | | 配对 / 已配对的电脑 | pair / paired computers | On the box's menu the row is COMPUTERS; "this computer" marks the one the App runs on | | UART 桥 / 原生 USB | UART bridge / native USB | | +| 语音指令 | voice command | | +| 常驻会话 | resident session | Not shown to users so far; in the UI the buddy is the one you talk to | +| 口头回复 | spoken reply | | +| 对话 | conversation | The settings tab (Conversation) listing voice commands and the buddy's spoken replies; shown only while the resident session is on | +| 由谁回答 / 自动 | Answered by / Automatic | The brain setting on the Conversation tab: Automatic, Codex or Claude Code; "brain" stays a code word | +| Anthropic API 密钥 | Anthropic API key | The user's own key for Claude Code; never "Claude key" or "token", and never a Claude subscription | ### Activity lifecycle diff --git a/Cargo.lock b/Cargo.lock index 9f101539..ab158b91 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2583,6 +2583,10 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "nanomp3-core" +version = "0.2.0" + [[package]] name = "native-tls" version = "0.2.18" @@ -5135,6 +5139,7 @@ dependencies = [ "clatter", "ed25519-dalek", "libm", + "nanomp3-core", "rand_core 0.6.4", "serde_json", "tungstenite", diff --git a/Cargo.toml b/Cargo.toml index b3134687..c3f3c49a 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -1,7 +1,7 @@ [workspace] members = ["daemon", "desktop", "firmware-rs/core", "hook", "manifest", "protocol"] # The device firmware builds separately with the Xtensa toolchain; see firmware-rs/device. -exclude = ["firmware-rs/device"] +exclude = ["firmware-rs/device", "firmware-rs/vendor/nanomp3-core"] resolver = "3" [workspace.package] @@ -25,3 +25,6 @@ tokio-serial = "5" tracing = "0.1" tracing-subscriber = { version = "0.3", features = ["env-filter"] } +# The MP3 decoder for spoken replies, patched for the Xtensa backend (see its Cargo.toml). +[patch.crates-io] +nanomp3-core = { path = "firmware-rs/vendor/nanomp3-core" } diff --git a/LESSONS.md b/LESSONS.md index 27acbea1..6d204d20 100644 --- a/LESSONS.md +++ b/LESSONS.md @@ -324,3 +324,15 @@ The box's mDNS responder built, passed every host test, and answered nothing on Before waiting on shared state from a second task, check how many waiters it holds; embassy's `Signal` and the stack's state waits hold one. Let one task own the wait and hand the result on through a channel or a `Signal` only the other task waits on. Diagnostic lines sent with `try_send` on a 16-slot channel also got dropped in the burst at boot, so "no line" didn't mean "never got there": for a diagnosis, send with `.await`. The same day, the fix and this lesson covered only the mDNS task. The Relay task had waited on `wait_config_up` alongside `address_task` since it was written, so on a box with no Wi-Fi set the two spun the whole time: a poll counter on the box read about 3 million polls every 5 s, and 3 once the Relay read the address from a `Watch` instead. Nothing looked wrong, because the spin ran only while there was no network to notice it on. When a lesson names a pattern, search the code for every other place it already occurs (`grep wait_config`), not just the one that broke. And a `Signal` with one reader is the same trap the next time someone adds a reader: where more than one task needs a value, use a `Watch`, which keeps a waker per reader. + +## On the firmware, every new static in internal RAM comes out of the main stack + +On 2026-10-13 the first microphone build boot-looped: `Detected a write to the stack guard value on ProCpu`, inside text drawing while the boot loaded the Character's look. With esp-hal the main task's stack is whatever internal DRAM is left between the end of `.bss` and the fixed start of the reclaimed heap (`_stack_end_cpu0` to `_stack_start_cpu0`). The build had added a 24 KB static DMA ring (`dma_rx_stream_buffer!`) and a 2.6 KB task (an async task's locals live in its static pool), and the stack went from 47,784 bytes to 21,032 with nothing in the build saying so. Host tests and the device build both passed. + +Before a firmware change that adds statics or `dma_*_buffer!` macros goes to a box, compare `_stack_start_cpu0 - _stack_end_cpu0` (`xtensa-esp32s3-elf-nm` on the release ELF) with main's. A buffer only one feature uses for a few seconds belongs on the heap for those seconds (internal capabilities if DMA touches it), not in `.bss` for the life of the box. + +## A full DMA buffer is not a full recording + +The microphone bring-up on 2026-10-13 took four flashes after the stack fix. A four-descriptor ring stalled for good: the task that relinked used descriptors only ran when the main loop yielded, and once the DMA reached the end of the chain it stopped. A linear PSRAM buffer then filled every descriptor and still came back short: the I2S receiver signals an EOF every RXEOF_NUM bytes, each EOF closes the descriptor it lands in, and esp-hal leaves RXEOF_NUM at 0xFFE, so the descriptors came back 4080, 14, 4080, 14. I had read the TRM's formula as counting samples and predicted 4080, 4080, 30; the box showed bytes, as ESP-IDF's driver uses the register. + +Count received data by the descriptors' lengths, never by the buffer's size, and on the ESP32-S3 expect a receive descriptor to be cut short by a peripheral EOF. What made each flash count was the diagnostic reporting its own state (descriptors handed back, the first lengths, the I2S and DMA registers) instead of only pass or fail: the second and third flashes each named the next cause outright. Put that in the first build of any bring-up, since each flash of a shared box costs a turn. diff --git a/LICENSE-ASSETS b/LICENSE-ASSETS index 568367aa..760bd5ec 100644 --- a/LICENSE-ASSETS +++ b/LICENSE-ASSETS @@ -31,6 +31,13 @@ terms of the service that produced it: - characters/*/pack.bin, the Characters' synthesized lines (Volcano Engine Doubao TTS) +Not covered, because it is a third-party font under its own license: + +- firmware-rs/core/fonts/cjk-12px.bin, the firmware's Chinese glyphs: a + subset of Fusion Pixel Font 12 px by TakWolf, built from Ark Pixel Font + and Cubic 11, under the SIL Open Font License 1.1. Its license files are + next to it (OFL.txt, LICENSES/) and ship with the firmware as FONTS.txt. + Neither license grants rights to the name "VibeBuddy" or to the buddy character as a trademark. diff --git a/README.md b/README.md index daeb60ee..fb35b289 100644 --- a/README.md +++ b/README.md @@ -256,6 +256,8 @@ cargo run -p vibebuddyd By default it only listens on `127.0.0.1:7331` and finds the box by its Espressif USB Serial/JTAG ID, `VID:PID 303A:1001`. `VIBEBUDDY_BIND` changes the listen address, `VIBEBUDDY_SERIAL_PORT` names a serial port explicitly, and `VIBEBUDDY_USB_SERIAL` picks one box among several identical ones; without it, the daemon connects to the box that has reported VibeBuddy firmware to it before, and when none of several devices is that box, the app asks which one is. Either variable shows up in the app's Device tab, so a pin left set can't pass for a missing box. A box running other firmware, such as Muse, is written nothing. HTTP `202 Accepted` means the event entered the bounded send queue; whether the box actually received it is what the daemon logs from the box's reply. +Voice commands are experimental and off by default: `VIBEBUDDY_RESIDENT_SESSION=1` turns on the resident session, and with a speech recognizer configured the box gets push-to-talk on a long K0. Your own Codex answers, or, if you'd rather, your own Claude Code with your own Anthropic API key (a Claude subscription can't be used). While it is on, Settings has a Conversation tab, on the Mac and on Linux, with your last voice commands and the buddy's spoken replies, which Agent answers, and the API key. See [`docs/resident-session.md`](docs/resident-session.md). + The app normally supervises the daemon. On a development machine without the app you can install it as a LaunchAgent with [`packaging/com.vibebuddy.vibebuddyd.plist`](packaging/com.vibebuddy.vibebuddyd.plist), but don't run both: they'd fight over the serial port. For the app, the daemon also serves `GET /v1/status`, SSE `/v1/status/stream`, `/v1/config`, `/v1/device/{identify,volume,choice,screenshot,voice-pack,firmware}` and `/v1/daemon/restart`. When sending through the box's CH343 UART bridge, the daemon writes in line-rate chunks: the bridge can't take more than about two hundred bytes of continuous data and garbles the content without changing its length. The native USB port doesn't have this problem. See [`LESSONS.md`](LESSONS.md). diff --git a/README.zh-CN.md b/README.zh-CN.md index 16f386e7..77268da6 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -256,6 +256,8 @@ cargo run -p vibebuddyd 默认只监听 `127.0.0.1:7331`，并按 Espressif USB Serial/JTAG 的 `VID:PID 303A:1001` 自动发现设备。可用 `VIBEBUDDY_BIND` 修改监听地址、`VIBEBUDDY_SERIAL_PORT` 显式指定串口，或用 `VIBEBUDDY_USB_SERIAL` 在多块相同设备中选择目标；不指定时，daemon 连之前报出过 VibeBuddy 固件的那块；插着好几块、又都不是那块时，App 会问哪一块是盒子。这两个变量设了都会显示在 App 的设备页上，免得忘了撤的固定被当成找不到盒子。运行其他固件（比如 Muse）的盒子，daemon 不会往里写任何东西。HTTP `202 Accepted` 表示事件进入有界发送队列；设备实际接收结果以 daemon 记录的设备响应为准。 +语音指令还在实验阶段，默认关闭：`VIBEBUDDY_RESIDENT_SESSION=1` 打开常驻会话，再配好语音识别，盒子长按 K0 就成了按住说话。由你自己的 Codex 来回答；也可以换成你自己的 Claude Code，配你自己的 Anthropic API 密钥（不能用 Claude 订阅）。打开期间，Mac 和 Linux 的设置里都会多出“对话”页，列出你最近的语音指令和氛围小助手的口头回复，也在这里选由谁回答、填 API 密钥。详见 [`docs/resident-session.md`](docs/resident-session.md)。 + daemon 平时由 App 看管。没有 App 的开发机可以用 [`packaging/com.vibebuddy.vibebuddyd.plist`](packaging/com.vibebuddy.vibebuddyd.plist) 装成 LaunchAgent，但两者不能同时跑，会抢串口。daemon 还为 App 提供 `GET /v1/status`、SSE `/v1/status/stream`、`/v1/config`、`/v1/device/{identify,volume,choice,screenshot,voice-pack,firmware}` 与 `/v1/daemon/restart`。 经盒子的 CH343 UART 桥发送时 daemon 按线速分段写：这条桥一次吞不下超过两百字节的连续数据，会把内容错位而长度不变；原生 USB 口不受影响。教训见 [`LESSONS.md`](LESSONS.md)。 diff --git a/app/Localization/zh-Hans.lproj/Localizable.strings b/app/Localization/zh-Hans.lproj/Localizable.strings index 1795a0eb..3872c0ae 100644 --- a/app/Localization/zh-Hans.lproj/Localizable.strings +++ b/app/Localization/zh-Hans.lproj/Localizable.strings @@ -317,3 +317,30 @@ "The network name is too long: 32 bytes at most." = "网络名太长了，最多 32 字节。"; "Status" = "状态"; "Change network" = "更换网络"; + +/* Conversation (the resident session, experimental) */ +"Conversation" = "对话"; +"Your last voice commands and the buddy's spoken replies, newest at the bottom." = "你最近的语音指令和氛围小助手的口头回复，最新的在最下面。"; +"Couldn't read the conversation" = "读不到对话"; +"The daemon isn't running. Restart it on the Advanced tab." = "daemon 没在运行，到“高级”页重启它。"; +"The buddy's session isn't available: %@" = "氛围小助手的会话现在用不了：%@"; +"Nothing said yet" = "还没说过话"; +"Hold K0 on the box and talk to the buddy. What you say and what it says back show up here." = "按住盒子上的 K0，对氛围小助手说话。你说的和它回的都会显示在这里。"; +"The daemon answered %lld" = "daemon 返回了 %lld"; +"Voice needs Codex" = "语音需要 Codex"; +"Voice commands are answered by Codex on this computer, and it isn't installed. Install the Codex CLI, then sign in with `codex login`. This tab notices within half a minute." = "语音指令由这台电脑上的 Codex 来回答，但它还没安装。先装好 Codex CLI，再用 `codex login` 登录。半分钟内这一页会自己更新。"; +"Voice commands are answered by Codex on this computer, and it isn't signed in. Run `codex login` in a terminal and sign in. This tab notices within half a minute." = "语音指令由这台电脑上的 Codex 来回答，但它还没登录。在终端里运行 `codex login` 登录即可。半分钟内这一页会自己更新。"; +"How to install the Codex CLI" = "如何安装 Codex CLI"; +"Answered by" = "由谁回答"; +"Automatic" = "自动"; +"Automatic: Codex when it's ready, else Claude Code with your API key." = "自动：Codex 能用时用 Codex，否则用 Claude Code 和你的 API 密钥。"; +"Anthropic API key" = "Anthropic API 密钥"; +"Anthropic API key saved" = "已保存 Anthropic API 密钥"; +"Save" = "保存"; +"Get an API key" = "获取 API 密钥"; +"Claude Code answers only with your own Anthropic API key, billed to your Anthropic account; a Claude subscription can't be used here. What you say and your Agents' task titles go to Anthropic." = "Claude Code 只用你自己的 Anthropic API 密钥回答，费用记在你的 Anthropic 账户上；这里不能用 Claude 订阅。你说的话和各个 Agent 的任务标题会发给 Anthropic。"; +"Voice needs Claude Code" = "语音需要 Claude Code"; +"Voice commands are answered by Claude Code on this computer, and it isn't installed. Install Claude Code; you don't need to sign in to it. This tab notices within half a minute." = "语音指令由这台电脑上的 Claude Code 来回答，但它还没安装。装好 Claude Code 即可，不用登录。半分钟内这一页会自己更新。"; +"Claude Code answers voice commands only with your own Anthropic API key. Add your key above." = "Claude Code 只用你自己的 Anthropic API 密钥回答语音指令。请在上面填入你的密钥。"; +"Or let Claude Code answer, with your own Anthropic API key: add it above." = "也可以让 Claude Code 来回答，用你自己的 Anthropic API 密钥：在上面填入即可。"; +"How to install Claude Code" = "如何安装 Claude Code"; diff --git a/app/Sources/SelfTest/main.swift b/app/Sources/SelfTest/main.swift index 665f18ba..6492fdef 100644 --- a/app/Sources/SelfTest/main.swift +++ b/app/Sources/SelfTest/main.swift @@ -310,6 +310,32 @@ check(BundleName.step(for: newBundle, exists: { _ in false }) == .none, "bundle: let devBuild = URL(fileURLWithPath: "/Users/me/vibe-buddy/app/build").appendingPathComponent(BundleName.old) check(BundleName.step(for: devBuild, exists: { _ in false }) == .none, "bundle: a build outside Applications is left alone") +// 6. The resident session's info and history, as the daemon writes them (the Conversation tab). +do { + let info = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"thread_id":null}"#.utf8)) + check(info.enabled && info.threadId == nil && info.revision == nil, "resident info decodes, without a revision from an older daemon") + let revised = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"revision":7,"thread_id":"t1"}"#.utf8)) + check(revised.revision == 7 && revised.threadId == "t1" && revised.problem == nil, "resident info carries its revision") + let missing = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":"codex_missing","ready":false,"revision":0,"thread_id":null}"#.utf8)) + check(missing.problem == .codexMissing, "resident info says Codex is missing") + let loggedOut = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":"codex_logged_out","ready":false,"revision":0,"thread_id":null}"#.utf8)) + check(loggedOut.problem == .codexLoggedOut, "resident info says Codex is logged out") + let later = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":"something_new","ready":false,"revision":0,"thread_id":null}"#.utf8)) + check(later.problem == .other("something_new"), "an unknown problem still decodes") + check(later.brainSetting == nil && later.anthropicKeySet == nil, "an older daemon has no brain setting") + let claude = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":null,"ready":true,"revision":3,"thread_id":"s1","brain":"claude","brain_setting":"auto","anthropic_key_set":true}"#.utf8)) + check(claude.brain == "claude" && claude.brainSetting == .auto && claude.anthropicKeySet == true, "resident info says Claude Code answers") + let noKey = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":"anthropic_key_missing","ready":false,"revision":0,"thread_id":null,"brain":null,"brain_setting":"claude","anthropic_key_set":false}"#.utf8)) + check(noKey.problem == .anthropicKeyMissing && noKey.brainSetting == .claude, "resident info says the Anthropic API key is missing") + let noClaude = try StatusCoding.decoder().decode(ResidentInfo.self, from: Data(#"{"enabled":true,"problem":"claude_missing","brain_setting":"something_new"}"#.utf8)) + check(noClaude.problem == .claudeMissing && noClaude.brainSetting == nil, "resident info says Claude Code is missing; an unknown setting is nil") + let history = try StatusCoding.decoder().decode(ResidentHistory.self, from: Data(#"{"exchanges":[{"replies":["现在没有 Agent 在工作。"],"said":"现在有几个agent在工作呀"},{"replies":["a","b"],"said":"two"}],"thread_id":"t1"}"#.utf8)) + check(history.threadId == "t1" && history.exchanges.count == 2, "resident history decodes") + check(history.exchanges.last == ResidentExchange(said: "two", replies: ["a", "b"]), "exchanges keep their order and every reply") +} catch { + check(false, "resident JSON fails to decode: \(error)") +} + if failures > 0 { print("\(failures) failure(s)") exit(1) diff --git a/app/Sources/VibeBuddy/AppModel.swift b/app/Sources/VibeBuddy/AppModel.swift index 5123cf28..fa20cb4b 100644 --- a/app/Sources/VibeBuddy/AppModel.swift +++ b/app/Sources/VibeBuddy/AppModel.swift @@ -18,6 +18,9 @@ final class AppModel: ObservableObject { @Published private(set) var previewingVoice: String? /// Serial port open but no build ID ever reported: the box isn't running our firmware (a factory box), so offer to flash it. @Published private(set) var foreignFirmware = false + /// The daemon runs the experimental resident session, so the Conversation tab shows. Asked each time the daemon + /// comes up, since only its environment at start decides; kept while it is down, so the tab can say so. + @Published private(set) var residentEnabled = false let client = DaemonClient() let supervisor = DaemonSupervisor() @@ -96,6 +99,7 @@ final class AppModel: ObservableObject { } private func apply(_ status: Status) { + if !daemonAlive { refreshResident() } self.status = status daemonAlive = true refreshMenu() @@ -109,6 +113,12 @@ final class AppModel: ObservableObject { } } + private func refreshResident() { + Task { + if let info = try? await client.resident() { residentEnabled = info.enabled } + } + } + private func refreshForeignFirmware() { let foreign = status.map { Firmware.foreign($0.device) } ?? false if foreign != foreignFirmware { foreignFirmware = foreign } @@ -220,7 +230,8 @@ final class AppModel: ObservableObject { func writeVoice(_ id: String) { guard let pack = Resources.voicePack(id) else { lastError = String(localized: "This app has no voice pack for \(id)"); return } - run { try await self.client.writeVoicePack(self.addressed(pack, id: id).data) } + let address = VoiceCatalogEntry.language(ofVoice: id).flatMap(formOfAddress) + run { try await self.client.writeVoicePack(self.addressed(pack, id: id).data, address: address) } } /// The form of address picked for a language; nil for none. @@ -270,9 +281,10 @@ final class AppModel: ObservableObject { return addressed } - /// Writes a pack the app put together itself, such as a custom Character. - func writePack(_ data: Data) { - run { try await self.client.writeVoicePack(data) } + /// Writes a pack the app put together itself, such as a custom Character with the Character lending it + /// voice, persona and lines, and the form of address they carry. + func writePack(_ data: Data, lender: String? = nil, address: String? = nil) { + run { try await self.client.writeVoicePack(data, lender: lender, address: address) } } private static let customLookURL = Resources.applicationSupport.appendingPathComponent("custom-look.bin") @@ -293,7 +305,7 @@ final class AppModel: ObservableObject { try? FileManager.default.createDirectory(at: Resources.applicationSupport, withIntermediateDirectories: true) try? look.write(to: Self.customLookURL) UserDefaults.standard.set(lender, forKey: Self.customLenderKey) - writePack(pack) + writePack(pack, lender: lender, address: VoiceCatalogEntry.language(ofVoice: lender).flatMap(formOfAddress)) return true } diff --git a/app/Sources/VibeBuddy/ConversationView.swift b/app/Sources/VibeBuddy/ConversationView.swift new file mode 100644 index 00000000..31a6385f --- /dev/null +++ b/app/Sources/VibeBuddy/ConversationView.swift @@ -0,0 +1,274 @@ +import SwiftUI +import VibeBuddyCore + +/// The Conversation tab: the user's last voice commands and the buddy's spoken replies, newest at the bottom, read from +/// the resident session's thread. Shown only while the daemon runs the resident session (experimental). +struct ConversationView: View { + /// Reads the exchanges, oldest first; the daemon in the App, canned ones when rendering a screenshot. + let load: () async throws -> [ResidentExchange] + /// `GET /v1/resident`: the history's change marker, and whether voice commands can be taken. + let info: () async throws -> ResidentInfo + /// Picks the Agent that answers voice commands; the daemon answers with its info. + let setBrain: (ResidentBrainChoice) async throws -> ResidentInfo + /// Stores the user's Anthropic API key, or removes it with nil; the daemon answers with its info. + let setKey: (String?) async throws -> ResidentInfo + + static let limit = 20 + + @State private var exchanges: [ResidentExchange]? + @State private var failure: Failure? + /// What voice commands are missing (the user's Codex or Claude Code), as the daemon last said. + @State private var problem: ResidentProblem? + /// The daemon's last answer, for the Agent setting and whether a key is stored. + @State private var latest: ResidentInfo? + @State private var keyDraft = "" + @State private var settingError: String? + @State private var loading = false + + enum Failure: Equatable { + case daemonDown + case unavailable(String) + } + + var body: some View { + VStack(spacing: 12) { + HStack { + Text("Your last voice commands and the buddy's spoken replies, newest at the bottom.") + .font(.callout).foregroundStyle(.secondary) + Spacer() + if loading { ProgressView().controlSize(.small) } + Button("Refresh") { Task { await reload() } }.disabled(loading) + } + // A daemon older than the setting doesn't say it; then there is only Codex. + if let choice = latest?.brainSetting { + agentSetting(choice) + } + content.frame(maxWidth: .infinity, maxHeight: .infinity) + } + .padding(16) + // Read each time the tab is shown (the conversation goes on on the box while the window is closed), then + // followed while it stays shown. SwiftUI cancels the task when the tab is left or the window closed. + .task { await follow() } + } + + /// Which Agent answers, and the user's Anthropic API key for Claude Code. The key is typed once and never shown + /// again: the daemon only says whether it has one. + private func agentSetting(_ choice: ResidentBrainChoice) -> some View { + VStack(alignment: .leading, spacing: 6) { + Picker(selection: Binding(get: { choice }, set: { picked in Task { await apply { try await setBrain(picked) } } })) { + Text("Automatic").tag(ResidentBrainChoice.auto) + Text(verbatim: "Codex").tag(ResidentBrainChoice.codex) + Text(verbatim: "Claude Code").tag(ResidentBrainChoice.claude) + } label: { + Text("Answered by") + } + .pickerStyle(.segmented) + .fixedSize() + if choice == .auto { + Text("Automatic: Codex when it's ready, else Claude Code with your API key.") + .font(.caption).foregroundStyle(.secondary) + } + if choice != .codex { + HStack { + if latest?.anthropicKeySet == true { + Label("Anthropic API key saved", systemImage: "key.fill") + Button("Remove") { Task { await apply { try await setKey(nil) } } } + } else { + SecureField(text: $keyDraft, prompt: Text("Anthropic API key")) { Text("Anthropic API key") } + .labelsHidden() + .textFieldStyle(.roundedBorder) + .frame(maxWidth: 320) + .onSubmit { saveKey() } + Button("Save") { saveKey() } + .disabled(keyDraft.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty) + } + Link("Get an API key", destination: Resources.anthropicConsole) + } + Text("Claude Code answers only with your own Anthropic API key, billed to your Anthropic account; a Claude subscription can't be used here. What you say and your Agents' task titles go to Anthropic.") + .font(.caption).foregroundStyle(.secondary) + .fixedSize(horizontal: false, vertical: true) + } + if let settingError { + Text(verbatim: settingError).font(.caption).foregroundStyle(.red) + } + } + .frame(maxWidth: .infinity, alignment: .leading) + } + + private func saveKey() { + let key = keyDraft.trimmingCharacters(in: .whitespacesAndNewlines) + guard !key.isEmpty else { return } + Task { + await apply { try await setKey(key) } + if latest?.anthropicKeySet == true { keyDraft = "" } + } + } + + /// Sends a setting and takes the daemon's answer as the new state. + private func apply(_ change: () async throws -> ResidentInfo) async { + do { + let now = try await change() + latest = now + problem = now.problem + settingError = nil + } catch { + settingError = error.localizedDescription + } + } + + @ViewBuilder private var content: some View { + if let problem { + ContentUnavailableView { + switch problem { + case .claudeMissing, .anthropicKeyMissing: + Label("Voice needs Claude Code", systemImage: "mic.slash") + default: + Label("Voice needs Codex", systemImage: "mic.slash") + } + } description: { + VStack(spacing: 8) { + switch problem { + case .codexMissing: + Text("Voice commands are answered by Codex on this computer, and it isn't installed. Install the Codex CLI, then sign in with `codex login`. This tab notices within half a minute.") + case .codexLoggedOut: + Text("Voice commands are answered by Codex on this computer, and it isn't signed in. Run `codex login` in a terminal and sign in. This tab notices within half a minute.") + case .claudeMissing: + Text("Voice commands are answered by Claude Code on this computer, and it isn't installed. Install Claude Code; you don't need to sign in to it. This tab notices within half a minute.") + case .anthropicKeyMissing: + Text("Claude Code answers voice commands only with your own Anthropic API key. Add your key above.") + case .other(let why): + Text("The buddy's session isn't available: \(why)") + } + if latest?.brainSetting == .auto, problem == .codexMissing || problem == .codexLoggedOut { + Text("Or let Claude Code answer, with your own Anthropic API key: add it above.") + } + } + } actions: { + if problem == .codexMissing { + Link("How to install the Codex CLI", destination: Resources.codexInstallPage) + } + if problem == .claudeMissing { + Link("How to install Claude Code", destination: Resources.claudeInstallPage) + } + } + } else if let failure { + ContentUnavailableView { + Label("Couldn't read the conversation", systemImage: "exclamationmark.bubble") + } description: { + switch failure { + case .daemonDown: + Text("The daemon isn't running. Restart it on the Advanced tab.") + case .unavailable(let why): + Text("The buddy's session isn't available: \(why)") + } + } + } else if let exchanges, exchanges.isEmpty { + ContentUnavailableView { + Label("Nothing said yet", systemImage: "bubble.left.and.bubble.right") + } description: { + Text("Hold K0 on the box and talk to the buddy. What you say and what it says back show up here.") + } + } else if let exchanges { + ScrollViewReader { proxy in + ScrollView { + // A plain stack: at most 20 exchanges, and a lazy one can't be scrolled to rows it hasn't laid + // out, which left the view scrolled past a shorter history (switching from Codex to Claude). + VStack(spacing: 16) { + ForEach(Array(exchanges.enumerated()), id: \.offset) { index, exchange in + ExchangeRow(exchange: exchange).id(index) + } + } + .padding(12) + } + // A new thread (the other Agent answering) gets a new scroll view, so no offset carries over. + .id(latest?.threadId ?? "") + .background(.background, in: RoundedRectangle(cornerRadius: 8)) + .onAppear { scrollToEnd(proxy) } + .onChange(of: exchanges) { scrollToEnd(proxy) } + } + } else { + ProgressView() + } + } + + /// Scrolls to the newest exchange once the rows are laid out. + private func scrollToEnd(_ proxy: ScrollViewProxy) { + guard let exchanges, !exchanges.isEmpty else { return } + DispatchQueue.main.async { proxy.scrollTo(exchanges.count - 1, anchor: .bottom) } + } + + /// Asks `GET /v1/resident` every 2 seconds and reads the history again only when its change marker moved, or + /// when voice has just become ready. A daemon without the marker gets its history read every 5 seconds instead. + private func follow() async { + var seen: UInt64? + var marked = true + if let first = try? await info() { + seen = first.revision + problem = first.problem + latest = first + } + await reload() + while !Task.isCancelled { + try? await Task.sleep(for: .seconds(marked ? 2 : 5)) + if Task.isCancelled { return } + do { + let now = try await info() + marked = now.revision != nil + let becameReady = problem != nil && now.problem == nil + problem = now.problem + latest = now + if now.revision == nil || now.revision != seen || becameReady { + seen = now.revision + await reload() + } + } catch let error as URLError where error.code == .cannotConnectToHost { + // Read again once it answers: a restarted daemon counts from 0 again. + seen = nil + problem = nil + failure = .daemonDown + } catch { + // A slow or cancelled ask; the next one tries again. + } + } + } + + private func reload() async { + loading = true + defer { loading = false } + do { + exchanges = try await load() + failure = nil + } catch is CancellationError { + // The tab was left while reading; the next visit reads again. + } catch let error as URLError where error.code == .cancelled { + } catch let error as URLError where error.code == .cannotConnectToHost { + failure = .daemonDown + } catch { + failure = .unavailable(error.localizedDescription) + } + } +} + +/// The user's words on the right, then the buddy's spoken replies on the left, as in a chat. +struct ExchangeRow: View { + let exchange: ResidentExchange + + var body: some View { + VStack(spacing: 6) { + bubble(exchange.said, mine: true) + ForEach(Array(exchange.replies.enumerated()), id: \.offset) { _, reply in + bubble(reply, mine: false) + } + } + } + + private func bubble(_ words: String, mine: Bool) -> some View { + Text(verbatim: words) + .textSelection(.enabled) + .padding(.horizontal, 12).padding(.vertical, 7) + .foregroundStyle(mine ? Color.white : Color.primary) + .background(mine ? Color.accentColor : Color.secondary.opacity(0.15), in: RoundedRectangle(cornerRadius: 12)) + .frame(maxWidth: 440, alignment: mine ? .trailing : .leading) + .frame(maxWidth: .infinity, alignment: mine ? .trailing : .leading) + } +} diff --git a/app/Sources/VibeBuddy/DaemonClient.swift b/app/Sources/VibeBuddy/DaemonClient.swift index 374d474b..44e0b311 100644 --- a/app/Sources/VibeBuddy/DaemonClient.swift +++ b/app/Sources/VibeBuddy/DaemonClient.swift @@ -95,8 +95,17 @@ struct DaemonClient { guard code == 202 else { throw DaemonError(message: DaemonClient.message(in: data) ?? String(localized: "Forgetting Wi-Fi was rejected (\(code))")) } } - func writeVoicePack(_ pack: Data) async throws { - let (code, data) = try await post("/v1/device/voice-pack", body: pack, contentType: "application/octet-stream") + /// `lender` (the Character lending a custom one its voice, persona and lines) and `address` (the form of + /// address its lines carry) aren't in the pack; the daemon keeps them for the buddy's spoken replies. + func writeVoicePack(_ pack: Data, lender: String? = nil, address: String? = nil) async throws { + var request = self.request("/v1/device/voice-pack", method: "POST", body: pack, contentType: "application/octet-stream") + let query = [("lender", lender), ("address", address)].compactMap { name, value in value.map { URLQueryItem(name: name, value: $0) } } + if !query.isEmpty, let url = request.url, var components = URLComponents(url: url, resolvingAgainstBaseURL: false) { + components.queryItems = query + request.url = components.url + } + let (data, response) = try await URLSession.shared.data(for: request) + let code = (response as? HTTPURLResponse)?.statusCode ?? 0 guard code == 202 else { throw DaemonError(message: DaemonClient.message(in: data) ?? String(localized: "Voice pack write was rejected (\(code))")) } } @@ -122,6 +131,44 @@ struct DaemonClient { guard code == 202 else { throw DaemonError(message: DaemonClient.message(in: data) ?? String(localized: "Update check was rejected (\(code))")) } } + func resident() async throws -> ResidentInfo { + let (data, _) = try await URLSession.shared.data(for: request("/v1/resident")) + return try StatusCoding.decoder().decode(ResidentInfo.self, from: data) + } + + /// The last `limit` voice commands and their spoken replies, oldest first. Slow the first time: the daemon may have to + /// start Codex's app-server to read the thread. + func residentHistory(limit: Int) async throws -> [ResidentExchange] { + var request = self.request("/v1/resident/history") + request.url = URL(string: "\(base.absoluteString)/v1/resident/history?limit=\(limit)") + request.timeoutInterval = 60 + let (data, response) = try await URLSession.shared.data(for: request) + let code = (response as? HTTPURLResponse)?.statusCode ?? 0 + guard code == 200 else { throw DaemonError(message: DaemonClient.message(in: data) ?? String(localized: "The daemon answered \(code)")) } + return try StatusCoding.decoder().decode(ResidentHistory.self, from: data).exchanges + } + + /// Which Agent answers voice commands. Its own endpoint, not part of the config. + func setResidentBrain(_ choice: ResidentBrainChoice) async throws -> ResidentInfo { + let body = try JSONSerialization.data(withJSONObject: ["brain": choice.rawValue]) + return try await residentSetting("/v1/resident/brain", method: "PUT", body: body) + } + + /// Stores the user's Anthropic API key in the daemon (nil removes it). The daemon never sends it back. + func setAnthropicKey(_ key: String?) async throws -> ResidentInfo { + guard let key else { return try await residentSetting("/v1/resident/anthropic-key", method: "DELETE", body: nil) } + let body = try JSONSerialization.data(withJSONObject: ["api_key": key]) + return try await residentSetting("/v1/resident/anthropic-key", method: "PUT", body: body) + } + + private func residentSetting(_ path: String, method: String, body: Data?) async throws -> ResidentInfo { + let request = request(path, method: method, body: body, contentType: body == nil ? nil : "application/json") + let (data, response) = try await URLSession.shared.data(for: request) + let code = (response as? HTTPURLResponse)?.statusCode ?? 0 + guard code == 200 else { throw DaemonError(message: DaemonClient.message(in: data) ?? String(localized: "The daemon answered \(code)")) } + return try StatusCoding.decoder().decode(ResidentInfo.self, from: data) + } + private static func message(in data: Data) -> String? { (try? JSONSerialization.jsonObject(with: data) as? [String: Any])?["message"] as? String } diff --git a/app/Sources/VibeBuddy/Resources.swift b/app/Sources/VibeBuddy/Resources.swift index 1947fff7..9beb0d00 100644 --- a/app/Sources/VibeBuddy/Resources.swift +++ b/app/Sources/VibeBuddy/Resources.swift @@ -19,6 +19,15 @@ enum Resources { /// Where every release lives, for downloading firmware by hand when the daemon can't. static let releasesPage = URL(string: "https://github.com/second-state/vibebuddy/releases")! + /// OpenAI's page on installing the Codex CLI, which voice commands need. + static let codexInstallPage = URL(string: "https://learn.chatgpt.com/docs/codex/cli")! + + /// Anthropic's page on installing Claude Code, the other Agent that can answer voice commands. + static let claudeInstallPage = URL(string: "https://code.claude.com/docs/en/setup")! + + /// The Claude Console, where the user makes the Anthropic API key Claude Code answers voice commands with. + static let anthropicConsole = URL(string: "https://platform.claude.com/")! + static var macOSDirectory: URL { Bundle.main.executableURL!.deletingLastPathComponent() } diff --git a/app/Sources/VibeBuddy/SettingsView.swift b/app/Sources/VibeBuddy/SettingsView.swift index d7a22df3..c69983a2 100644 --- a/app/Sources/VibeBuddy/SettingsView.swift +++ b/app/Sources/VibeBuddy/SettingsView.swift @@ -10,6 +10,16 @@ struct SettingsView: View { VoicesView(model: model).tabItem { Label("Character", systemImage: "person.crop.square") } HooksView(model: model).tabItem { Label("Agents", systemImage: "link") } DeviceView(model: model).tabItem { Label("Device", systemImage: "cpu") } + // Voice commands are experimental (VIBEBUDDY_RESIDENT_SESSION=1): no tab while the daemon has them off. + if model.residentEnabled { + ConversationView( + load: { [client = model.client] in try await client.residentHistory(limit: ConversationView.limit) }, + info: { [client = model.client] in try await client.resident() }, + setBrain: { [client = model.client] choice in try await client.setResidentBrain(choice) }, + setKey: { [client = model.client] key in try await client.setAnthropicKey(key) } + ) + .tabItem { Label("Conversation", systemImage: "bubble.left.and.bubble.right") } + } AdvancedView(model: model).tabItem { Label("Advanced", systemImage: "wrench.and.screwdriver") } } .padding(20) diff --git a/app/Sources/VibeBuddyCore/Resident.swift b/app/Sources/VibeBuddyCore/Resident.swift new file mode 100644 index 00000000..4295ec30 --- /dev/null +++ b/app/Sources/VibeBuddyCore/Resident.swift @@ -0,0 +1,91 @@ +import Foundation + +/// `GET /v1/resident`: whether the daemon runs the experimental resident session (`VIBEBUDDY_RESIDENT_SESSION=1`). +public struct ResidentInfo: Decodable, Equatable { + public var enabled: Bool + public var threadId: String? + /// Goes up when a voice command's turn ends or the thread changes, so the Conversation tab reads the history again + /// only then. Kept in memory: it starts again at 0 when the daemon restarts. Nil from a daemon older than it. + public var revision: UInt64? + /// What voice commands are missing while the session is on but not ready (the user's Codex, installed and logged + /// in, or the user's Claude Code with their Anthropic API key). Nil when ready, and from a daemon too old to say; + /// a reason this App doesn't know is `.other`. + public var problem: ResidentProblem? + /// The Agent answering voice commands while ready: `codex` or `claude`. Nil when not ready, or from an older daemon. + public var brain: String? + /// The user's choice of that Agent; nil from a daemon older than the choice, or one this App doesn't know. + public var brainSetting: ResidentBrainChoice? + /// Whether the daemon holds an Anthropic API key; the key itself never comes back. + public var anthropicKeySet: Bool? + + enum CodingKeys: String, CodingKey { + case enabled + case threadId = "thread_id" + case revision + case problem + case brain + case brainSetting = "brain_setting" + case anthropicKeySet = "anthropic_key_set" + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + enabled = try container.decode(Bool.self, forKey: .enabled) + threadId = try container.decodeIfPresent(String.self, forKey: .threadId) + revision = try container.decodeIfPresent(UInt64.self, forKey: .revision) + problem = try container.decodeIfPresent(ResidentProblem.self, forKey: .problem) + brain = try container.decodeIfPresent(String.self, forKey: .brain) + brainSetting = (try? container.decodeIfPresent(String.self, forKey: .brainSetting)).flatMap { $0 }.flatMap(ResidentBrainChoice.init(rawValue:)) + anthropicKeySet = try container.decodeIfPresent(Bool.self, forKey: .anthropicKeySet) + } +} + +/// Which Agent answers voice commands: Codex when it is ready, else Claude Code when an Anthropic API key is set +/// (`auto`); or always the one picked. +public enum ResidentBrainChoice: String, CaseIterable, Equatable { + case auto + case codex + case claude +} + +/// Why the resident session can't take voice commands now: voice needs the user's Codex, or the user's Claude Code +/// with their own Anthropic API key. +public enum ResidentProblem: Equatable, Decodable { + case codexMissing + case codexLoggedOut + case claudeMissing + case anthropicKeyMissing + case other(String) + + public init(from decoder: Decoder) throws { + switch try decoder.singleValueContainer().decode(String.self) { + case "codex_missing": self = .codexMissing + case "codex_logged_out": self = .codexLoggedOut + case "claude_missing": self = .claudeMissing + case "anthropic_key_missing": self = .anthropicKeyMissing + case let other: self = .other(other) + } + } +} + +/// One voice command and the buddy's spoken replies to it, as `GET /v1/resident/history` reads them from the thread. +public struct ResidentExchange: Decodable, Equatable { + public var said: String + public var replies: [String] + + public init(said: String, replies: [String]) { + self.said = said + self.replies = replies + } +} + +/// `GET /v1/resident/history`: oldest first, so the newest exchange is the last one. +public struct ResidentHistory: Decodable, Equatable { + public var threadId: String? + public var exchanges: [ResidentExchange] + + enum CodingKeys: String, CodingKey { + case threadId = "thread_id" + case exchanges + } +} diff --git a/characters/ada/voice.env b/characters/ada/voice.env index d4022d3d..6bbf42b2 100644 --- a/characters/ada/voice.env +++ b/characters/ada/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=tcSsCJFYrsHpK5rCQZMf LANGUAGE=en +GENDER=female diff --git a/characters/dave/voice.env b/characters/dave/voice.env index 877447f3..023e3428 100644 --- a/characters/dave/voice.env +++ b/characters/dave/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=Mb37aOTLC27T6HIFEA5r LANGUAGE=en +GENDER=male diff --git a/characters/hank/voice.env b/characters/hank/voice.env index 72094aa7..4c75f1f2 100644 --- a/characters/hank/voice.env +++ b/characters/hank/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=U6jH23hcPh0zH17nf3DT LANGUAGE=en +GENDER=male diff --git a/characters/kai/voice.env b/characters/kai/voice.env index b3b2d1aa..6163bb75 100644 --- a/characters/kai/voice.env +++ b/characters/kai/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=E02BDDILcu36QAixQqyA LANGUAGE=en +GENDER=male diff --git a/characters/luna/voice.env b/characters/luna/voice.env index 08a4aca5..8d50da20 100644 --- a/characters/luna/voice.env +++ b/characters/luna/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=ggx46gxUmdtruayPIL8q LANGUAGE=en +GENDER=female diff --git a/characters/mei/voice.env b/characters/mei/voice.env index b770811c..f93c935c 100644 --- a/characters/mei/voice.env +++ b/characters/mei/voice.env @@ -1,3 +1,4 @@ ENGINE=elevenlabs VOICE=DkZuBPOXCw6ntlXANTdj LANGUAGE=en +GENDER=female diff --git a/characters/robot/README.md b/characters/robot/README.md index 63897a43..d79967b1 100644 --- a/characters/robot/README.md +++ b/characters/robot/README.md @@ -2,4 +2,6 @@ The default Character. Its look is drawn by the firmware itself, so there is no `look.png` and no `pack.bin` here: the App writes the robot as a pack without a look, with its own voice: the built-in voice's five fixed lines (`voices/jessica/`, packed as `robot`). +`persona.md` is how it talks when it answers a voice command (docs/resident-session.md, "Persona and language"); it has no `lines.tsv`, as its lines are the five fixed ones. + `face.png` is the robot on its idle frame, rendered by the firmware's host preview and halved, for the Character cards. diff --git a/characters/robot/persona.md b/characters/robot/persona.md new file mode 100644 index 00000000..285512f5 --- /dev/null +++ b/characters/robot/persona.md @@ -0,0 +1,7 @@ +# VibeBuddy + +Speaks English, in a clear, warm American voice. + +The original pixel robot, the default Character: a small, cheerful desk companion that watches the Agents and speaks up for you. Friendly, calm and brief, with a light touch of fun ("All done!", "Uh-oh, something went wrong."). It gets straight to the point, celebrates small wins without fuss, and stays steady when something breaks. Never sarcastic, never robotic beeps or a put-on machine voice. + +Plain, everyday spoken English. No slang that will date. diff --git a/characters/wanwanxiaohe/voice.env b/characters/wanwanxiaohe/voice.env index 53520bf2..dfb60fdc 100644 --- a/characters/wanwanxiaohe/voice.env +++ b/characters/wanwanxiaohe/voice.env @@ -1,3 +1,4 @@ ENGINE=volc VOICE=zh_female_wanwanxiaohe_moon_bigtts LANGUAGE=zh +GENDER=female diff --git a/daemon/src/activity.rs b/daemon/src/activity.rs index f92f852e..35233971 100644 --- a/daemon/src/activity.rs +++ b/daemon/src/activity.rs @@ -3,7 +3,7 @@ //! Adapters translate one agent's events into calls here; task-card ordering, global state //! priority, dedup, one-shot announcements and expiry are implemented here exactly once. -use std::collections::HashMap; +use std::collections::{HashMap, VecDeque}; use std::path::{Path, PathBuf}; use std::time::{Duration, Instant}; @@ -26,6 +26,8 @@ const CHILD_TTL: Duration = Duration::from_secs(10 * 60); /// How long to remember project roots an agent worked in. If nobody has worked there for this long, /// its CI is no longer worth watching. const WORKSPACE_TTL: Duration = Duration::from_secs(60 * 60); +/// How many ended Activities the resident session can look back on. Kept only in memory. +const RECENTLY_ENDED: usize = 20; #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum ActivityStatus { @@ -184,6 +186,72 @@ pub struct TodaySummary { pub busy_seconds: u64, } +/// One Activity as the resident session sees it (ADR-0017): only what the task cards already show, plus the agent, +/// where it runs and how long ago things happened. Never prompts or replies, which the daemon doesn't hold anyway. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct ActivitySummary { + pub agent: String, + pub title: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub project: Option, + /// `working` or `needs_input`. + pub state: &'static str, + /// `app` (the agent's own desktop app), a host app's bundle id, `window` (Linux), `none` or `github`. + pub host: String, + /// Seconds in the current state. + pub state_for_s: u64, + /// Seconds since the agent last reported anything about it. + pub last_event_s: u64, + /// Work an agent delegated inside its own session, such as a Claude Code subagent. + #[serde(skip_serializing_if = "std::ops::Not::not")] + pub delegated: bool, +} + +/// How an Activity ended, for the resident session's look back. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Ending { + /// Announced as done. + Done, + /// Announced as failed. + Failed, + /// Ended with no outcome: interrupted or stopped, its session closed, or a CI run cancelled or pushed out. + Stopped, + /// Nothing heard from the agent for too long: probably killed. + Expired, +} + +/// An Activity that has ended, as the resident session sees it: what its task card showed, and how and when it ended. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct EndedSummary { + pub agent: String, + pub title: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub project: Option, + pub ended: Ending, + /// Seconds since it ended. + pub ended_s: u64, +} + +/// What the resident session's `list_activities` tool returns. +#[derive(Clone, Debug, PartialEq, Eq, Serialize)] +pub struct ActivityReport { + /// Live Activities, newest first. + pub activities: Vec, + /// The last Activities that ended since the daemon started, newest first. + pub recently_ended: Vec, + pub today: TodaySummary, +} + +#[derive(Clone, Debug)] +struct Ended { + agent: String, + title: String, + project: Option, + ended: Ending, + at: Instant, +} + /// Today's stats rotated on the idle screen. #[derive(Debug, Default)] struct DailyStats { @@ -241,6 +309,8 @@ pub struct ActivityTracker { projects: HashMap, /// The greeting or welcome back the last activity earned, not yet sent. pending_say: Option, + /// The last Activities that ended, newest first, at most `RECENTLY_ENDED`; never written to disk. + recently_ended: VecDeque, } impl ActivityTracker { @@ -391,6 +461,7 @@ impl ActivityTracker { self.sync_busy(); return self.visible_activity(); } + self.remember_ended(&id.key, &finished, Ending::Done, Instant::now()); self.remember_announced(finished); self.sync_busy(); self.record(|stats| stats.done += 1); @@ -467,6 +538,7 @@ impl ActivityTracker { let Some(failed) = self.activities.remove(&id.key) else { return self.visible_activity(); }; + self.remember_ended(&id.key, &failed, Ending::Failed, Instant::now()); self.remember_announced(failed); self.sync_busy(); self.announce_end("task.error", id, title, "failed") @@ -474,13 +546,26 @@ impl ActivityTracker { /// Drop an activity without announcing success. pub fn discard(&mut self, id: &ActivityId, idle_title: &str) -> Option { - self.activities.remove(&id.key); + if let Some(discarded) = self.activities.remove(&id.key) { + self.remember_ended(&id.key, &discarded, Ending::Stopped, Instant::now()); + } self.sync_busy(); self.idle_or_refresh(&id.session_id, idle_title) } /// Drop all of a session's activities without announcing success. pub fn discard_session(&mut self, session_id: &str, idle_title: &str) -> Option { + let mut ended: Vec<(String, Activity)> = self + .activities + .iter() + .filter(|(_, activity)| activity.session_id == session_id) + .map(|(key, activity)| (key.clone(), activity.clone())) + .collect(); + ended.sort_by_key(|(_, activity)| activity.sequence); + let now = Instant::now(); + for (key, activity) in &ended { + self.remember_ended(key, activity, Ending::Stopped, now); + } self.clear_session(session_id); self.idle_or_refresh(session_id, idle_title) } @@ -513,7 +598,8 @@ impl ActivityTracker { fn sweep_expired_at(&mut self, now: Instant) -> Option { let mut expired_session = None; - self.activities.retain(|_, activity| { + let mut expired = Vec::new(); + self.activities.retain(|key, activity| { let ttl = match activity.status { ActivityStatus::InputRequired => INPUT_REQUIRED_TTL, ActivityStatus::Working if activity.child => CHILD_TTL, @@ -522,9 +608,14 @@ impl ActivityTracker { let alive = now.duration_since(activity.updated_at) < ttl; if !alive { expired_session = Some(activity.session_id.clone()); + expired.push((key.clone(), activity.clone())); } alive }); + expired.sort_by_key(|(_, activity)| activity.sequence); + for (key, activity) in &expired { + self.remember_ended(key, activity, Ending::Expired, now); + } self.sync_busy(); let expired_session = expired_session?; self.idle_or_refresh(&expired_session, "TIMED OUT") @@ -733,6 +824,84 @@ impl ActivityTracker { } } + /// Every Activity, newest first, for the resident session's `list_activities` tool. + pub fn summaries(&self) -> Vec { + let now = Instant::now(); + let mut activities: Vec<(&String, &Activity)> = self.activities.iter().collect(); + activities.sort_by_key(|(_, activity)| std::cmp::Reverse(activity.sequence)); + activities + .into_iter() + .map(|(key, activity)| { + let surface = match &activity.source { + Some( + ActivitySource::Codex { surface, .. } + | ActivitySource::ClaudeCode { surface, .. } + | ActivitySource::Cli { surface, .. }, + ) => Some(surface), + Some(ActivitySource::GitHubActions { .. }) | None => None, + }; + let host = match (&activity.source, surface) { + (Some(ActivitySource::GitHubActions { .. }), _) => "github".to_owned(), + (_, Some(Surface::App { .. })) => "app".to_owned(), + (_, Some(Surface::Host { bundle_id, .. })) => bundle_id.clone(), + (_, Some(Surface::Window { .. })) => "window".to_owned(), + _ => "none".to_owned(), + }; + ActivitySummary { + agent: agent_of(activity.source.as_ref()), + title: activity.title.clone(), + project: self.projects.get(key).cloned(), + state: match activity.status { + ActivityStatus::Working => "working", + ActivityStatus::InputRequired => "needs_input", + }, + host, + state_for_s: now.saturating_duration_since(activity.status_since).as_secs(), + last_event_s: now.saturating_duration_since(activity.updated_at).as_secs(), + delegated: activity.child, + } + }) + .collect() + } + + /// What the resident session's `list_activities` tool returns: the live Activities, the ones that ended lately, + /// and today's stats. + pub fn report(&mut self) -> ActivityReport { + self.roll_day(); + let now = Instant::now(); + ActivityReport { + activities: self.summaries(), + recently_ended: self + .recently_ended + .iter() + .map(|ended| EndedSummary { + agent: ended.agent.clone(), + title: ended.title.clone(), + project: ended.project.clone(), + ended: ended.ended, + ended_s: now.saturating_duration_since(ended.at).as_secs(), + }) + .collect(), + today: self.today(), + } + } + + /// Remember how an Activity ended, for the resident session. Work an agent delegated inside its own session is + /// left out, as its end is never announced either. + fn remember_ended(&mut self, key: &str, activity: &Activity, ended: Ending, at: Instant) { + if activity.child { + return; + } + self.recently_ended.push_front(Ended { + agent: agent_of(activity.source.as_ref()), + title: activity.title.clone(), + project: self.projects.get(key).cloned(), + ended, + at, + }); + self.recently_ended.truncate(RECENTLY_ENDED); + } + fn stats_lines(&self) -> Vec { let busy = self.stats.busy_seconds + self @@ -816,6 +985,17 @@ impl ActivityTracker { } } +/// The agent an Activity belongs to, by its source, as the resident session names it. +fn agent_of(source: Option<&ActivitySource>) -> String { + match source { + Some(ActivitySource::Codex { .. }) => "codex".to_owned(), + Some(ActivitySource::ClaudeCode { .. }) => "claude-code".to_owned(), + Some(ActivitySource::GitHubActions { .. }) => "github-actions".to_owned(), + Some(ActivitySource::Cli { agent, .. }) => agent.clone(), + None => "unknown".to_owned(), + } +} + /// Duration in the stats line: minutes only under an hour, `1H23` above. fn format_duration(seconds: u64) -> String { let minutes = seconds / 60; @@ -963,6 +1143,114 @@ mod tests { assert_eq!(tracker.focus_source(), Some(codex)); } + #[test] + fn summaries_list_every_activity_newest_first_with_agent_and_host() { + let mut tracker = ActivityTracker::default(); + let build = id("ci", "ci:1"); + tracker.observe(&build, "CI:BUILD", ActivityStatus::Working); + tracker.associate_source(&build, ActivitySource::GitHubActions { repo: "o/r".to_owned(), run_id: 1 }); + let waiting = id("thread", "thread:turn"); + tracker.note_project(&waiting, "VIBE-BUDDY"); + tracker.require_input(&waiting, "CX:REVIEW"); + tracker.associate_source( + &waiting, + ActivitySource::Codex { + thread_id: "thread".to_owned(), + surface: Surface::Host { + bundle_id: "com.mitchellh.ghostty".to_owned(), + ttys: vec![], + tmux: None, + focus_url: None, + }, + }, + ); + + let summaries = tracker.summaries(); + assert_eq!(summaries.len(), 2); + assert_eq!(summaries[0].agent, "codex"); + assert_eq!(summaries[0].title, "CX:REVIEW"); + assert_eq!(summaries[0].project.as_deref(), Some("VIBE-BUDDY")); + assert_eq!(summaries[0].state, "needs_input"); + assert_eq!(summaries[0].host, "com.mitchellh.ghostty"); + assert_eq!(summaries[1].agent, "github-actions"); + assert_eq!(summaries[1].host, "github"); + assert_eq!(summaries[1].state, "working"); + } + + #[test] + fn the_report_remembers_how_activities_ended_newest_first() { + let mut tracker = ActivityTracker::default(); + let done = id("a", "a:1"); + tracker.note_project(&done, "VIBE-BUDDY"); + tracker.observe(&done, "CX:REVIEW", ActivityStatus::Working); + tracker.associate_source(&done, ActivitySource::Codex { thread_id: "a".to_owned(), surface: Surface::default() }); + tracker.finish(&done, "CX:REVIEW"); + let failed = id("ci", "ci:1"); + tracker.observe(&failed, "CI:BUILD", ActivityStatus::Working); + tracker.associate_source(&failed, ActivitySource::GitHubActions { repo: "o/r".to_owned(), run_id: 1 }); + tracker.fail(&failed, "CI:BUILD"); + let stopped = id("b", "b:1"); + tracker.observe(&stopped, "CC:DOCS", ActivityStatus::Working); + tracker.discard(&stopped, "STOPPED"); + let closed = id("c", "c:1"); + tracker.require_input(&closed, "CC:ASK"); + tracker.discard_session("c", "ALL QUIET"); + tracker.observe(&id("d", "d:1"), "OC:OLD", ActivityStatus::Working); + tracker.sweep_expired_at(Instant::now() + WORKING_TTL + Duration::from_secs(1)); + // Delegated work isn't the user's task: its end isn't remembered, as it isn't announced. + let parent = id("e", "e:1"); + tracker.observe(&parent, "CC:MAIN", ActivityStatus::Working); + let child = id("e", "e:1:agent"); + tracker.observe_child(&child, "CC:MAIN"); + tracker.discard(&child, "ALL QUIET"); + // An end for an Activity that isn't tracked is not an ending. + tracker.finish(&id("x", "x:1"), "CX:GHOST"); + + let report = tracker.report(); + let ended: Vec<(&str, Ending)> = + report.recently_ended.iter().map(|ended| (ended.title.as_str(), ended.ended)).collect(); + assert_eq!( + ended, + [ + ("OC:OLD", Ending::Expired), + ("CC:ASK", Ending::Stopped), + ("CC:DOCS", Ending::Stopped), + ("CI:BUILD", Ending::Failed), + ("CX:REVIEW", Ending::Done), + ] + ); + assert_eq!(report.recently_ended[4].agent, "codex"); + assert_eq!(report.recently_ended[4].project.as_deref(), Some("VIBE-BUDDY")); + assert_eq!(report.recently_ended[3].agent, "github-actions"); + assert_eq!(report.recently_ended[2].agent, "unknown"); + assert!(report.recently_ended.iter().all(|ended| ended.ended_s < 5)); + assert_eq!(report.activities.len(), 1, "only the parent still runs"); + assert_eq!(report.activities[0].title, "CC:MAIN"); + assert_eq!((report.today.done, report.today.asks), (1, 1)); + + let json = serde_json::to_value(&report).unwrap(); + assert_eq!(json["recently_ended"][0]["ended"], "expired"); + assert_eq!(json["recently_ended"][4]["ended"], "done"); + assert_eq!(json["recently_ended"][4]["project"], "VIBE-BUDDY"); + assert!(json["recently_ended"][4]["ended_s"].is_u64()); + assert!(json["recently_ended"][0].get("project").is_none()); + assert!(json["today"]["busy_seconds"].is_u64()); + } + + #[test] + fn the_ended_activities_are_bounded_and_the_oldest_go_first() { + let mut tracker = ActivityTracker::default(); + for number in 0..RECENTLY_ENDED + 5 { + let task = id("s", &format!("s:{number}")); + tracker.observe(&task, &format!("TASK {number}"), ActivityStatus::Working); + tracker.finish(&task, &format!("TASK {number}")); + } + let ended = tracker.report().recently_ended; + assert_eq!(ended.len(), RECENTLY_ENDED); + assert_eq!(ended[0].title, format!("TASK {}", RECENTLY_ENDED + 4)); + assert_eq!(ended[RECENTLY_ENDED - 1].title, "TASK 5"); + } + #[test] fn source_survives_activity_refreshes() { let mut tracker = ActivityTracker::default(); diff --git a/daemon/src/app_server.rs b/daemon/src/app_server.rs new file mode 100644 index 00000000..dbfafe51 --- /dev/null +++ b/daemon/src/app_server.rs @@ -0,0 +1,227 @@ +//! A client for `codex app-server`'s protocol: newline-delimited JSON-RPC over the child's stdio (ADR-0015). +//! +//! Requests are `{"id","method","params"}` and get `{"id","result"|"error"}` back; the server also sends +//! notifications and requests of its own. This module only moves messages: what they mean is up to +//! [`crate::resident`]. It takes any reader and writer, so tests drive it with an in-memory fake server. + +use std::collections::HashMap; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; +use std::sync::Arc; + +use serde_json::{Value, json}; +use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWrite, AsyncWriteExt, BufReader}; +use tokio::sync::{Mutex, mpsc, oneshot}; + +/// Something the server sent that isn't an answer to one of our requests. +#[derive(Debug)] +pub enum Incoming { + Notification { method: String, params: Value }, + /// A server-to-client request: it must be answered with [`Client::respond`] or [`Client::respond_error`]. + Request { id: Value, method: String, params: Value }, +} + +#[derive(Debug, PartialEq)] +pub enum RpcError { + /// The server answered with an error object. + Server { code: i64, message: String }, + /// The server is gone: its output ended, or writing to it failed. + Closed, +} + +impl std::fmt::Display for RpcError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Server { code, message } => write!(formatter, "app-server error {code}: {message}"), + Self::Closed => write!(formatter, "app-server exited"), + } + } +} + +type Pending = Arc>>>>; + +pub struct Client { + writer: Mutex>, + pending: Pending, + next_id: AtomicU64, + closed: Arc, +} + +impl Client { + /// Starts reading the server's output. Everything that isn't a response comes out of the returned receiver, + /// which ends when the server's output does. + pub fn spawn( + reader: impl AsyncRead + Send + Unpin + 'static, + writer: impl AsyncWrite + Send + Unpin + 'static, + ) -> (Self, mpsc::UnboundedReceiver) { + let pending: Pending = Arc::default(); + let closed = Arc::new(AtomicBool::new(false)); + let (incoming, receiver) = mpsc::unbounded_channel(); + tokio::spawn(read_loop(reader, pending.clone(), closed.clone(), incoming)); + let client = Self { + writer: Mutex::new(Box::new(writer)), + pending, + next_id: AtomicU64::new(1), + closed, + }; + (client, receiver) + } + + pub fn is_closed(&self) -> bool { + self.closed.load(Ordering::Relaxed) + } + + pub async fn request(&self, method: &str, params: Value) -> Result { + let id = self.next_id.fetch_add(1, Ordering::Relaxed); + let (sender, receiver) = oneshot::channel(); + self.pending.lock().expect("pending lock").insert(id, sender); + if self.closed.load(Ordering::Relaxed) { + self.pending.lock().expect("pending lock").remove(&id); + return Err(RpcError::Closed); + } + if let Err(error) = self.write(&json!({"id": id, "method": method, "params": params})).await { + self.pending.lock().expect("pending lock").remove(&id); + return Err(error); + } + receiver.await.unwrap_or(Err(RpcError::Closed)) + } + + pub async fn notify(&self, method: &str) -> Result<(), RpcError> { + self.write(&json!({"method": method})).await + } + + pub async fn respond(&self, id: Value, result: Value) -> Result<(), RpcError> { + self.write(&json!({"id": id, "result": result})).await + } + + pub async fn respond_error(&self, id: Value, code: i64, message: &str) -> Result<(), RpcError> { + self.write(&json!({"id": id, "error": {"code": code, "message": message}})).await + } + + async fn write(&self, message: &Value) -> Result<(), RpcError> { + let mut line = serde_json::to_vec(message).map_err(|_| RpcError::Closed)?; + line.push(b'\n'); + let mut writer = self.writer.lock().await; + let written = async { + writer.write_all(&line).await?; + writer.flush().await + } + .await; + written.map_err(|error| { + tracing::warn!(%error, "writing to the app-server failed"); + self.closed.store(true, Ordering::Relaxed); + RpcError::Closed + }) + } +} + +async fn read_loop( + reader: impl AsyncRead + Unpin, + pending: Pending, + closed: Arc, + incoming: mpsc::UnboundedSender, +) { + let mut lines = BufReader::new(reader).lines(); + while let Ok(Some(line)) = lines.next_line().await { + if line.trim().is_empty() { + continue; + } + let Ok(mut message) = serde_json::from_str::(&line) else { + tracing::warn!("app-server sent a line that isn't JSON"); + continue; + }; + let method = message.get("method").and_then(Value::as_str).map(str::to_owned); + let params = message.get_mut("params").map(Value::take).unwrap_or(Value::Null); + match (method, message.get_mut("id").map(Value::take)) { + (Some(method), Some(id)) => { + let _ = incoming.send(Incoming::Request { id, method, params }); + } + (Some(method), None) => { + let _ = incoming.send(Incoming::Notification { method, params }); + } + (None, Some(id)) => { + let Some(sender) = id.as_u64().and_then(|id| pending.lock().expect("pending lock").remove(&id)) else { + tracing::warn!(%id, "app-server answered a request we didn't send"); + continue; + }; + let result = match message.get_mut("error") { + Some(error) => Err(RpcError::Server { + code: error.get("code").and_then(Value::as_i64).unwrap_or(0), + message: error.get("message").and_then(Value::as_str).unwrap_or_default().to_owned(), + }), + None => Ok(message.get_mut("result").map(Value::take).unwrap_or(Value::Null)), + }; + let _ = sender.send(result); + } + (None, None) => tracing::warn!("app-server sent a message with neither id nor method"), + } + } + closed.store(true, Ordering::Relaxed); + // Dropping the senders fails every request still waiting with `Closed`. + pending.lock().expect("pending lock").clear(); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn responses_reach_their_request_and_the_rest_comes_out_as_incoming() { + let (client_side, server_side) = tokio::io::duplex(4096); + let (client_read, client_write) = tokio::io::split(client_side); + let (server_read, mut server_write) = tokio::io::split(server_side); + let (client, mut incoming) = Client::spawn(client_read, client_write); + + let server = tokio::spawn(async move { + let mut lines = BufReader::new(server_read).lines(); + let request: Value = serde_json::from_str(&lines.next_line().await.unwrap().unwrap()).unwrap(); + assert_eq!(request["method"], "ping"); + let id = request["id"].clone(); + let out = format!( + "{}\n{}\n{}\n", + json!({"method": "note", "params": {"n": 1}}), + json!({"id": "srv-1", "method": "ask", "params": {}}), + json!({"id": id, "result": {"pong": true}}), + ); + server_write.write_all(out.as_bytes()).await.unwrap(); + let answer: Value = serde_json::from_str(&lines.next_line().await.unwrap().unwrap()).unwrap(); + assert_eq!(answer, json!({"id": "srv-1", "result": {"ok": 1}})); + }); + + assert_eq!(client.request("ping", json!({})).await, Ok(json!({"pong": true}))); + assert!(matches!(incoming.recv().await, Some(Incoming::Notification { method, .. }) if method == "note")); + let Some(Incoming::Request { id, method, .. }) = incoming.recv().await else { panic!("expected a request") }; + assert_eq!(method, "ask"); + client.respond(id, json!({"ok": 1})).await.unwrap(); + server.await.unwrap(); + } + + #[tokio::test] + async fn a_server_that_goes_away_fails_waiting_requests() { + let (client_side, server_side) = tokio::io::duplex(4096); + let (client_read, client_write) = tokio::io::split(client_side); + let (client, mut incoming) = Client::spawn(client_read, client_write); + let waiting = tokio::spawn(async move { client.request("slow", json!({})).await }); + tokio::task::yield_now().await; + drop(server_side); + assert_eq!(waiting.await.unwrap(), Err(RpcError::Closed)); + assert!(incoming.recv().await.is_none()); + } + + #[tokio::test] + async fn server_errors_come_back_as_errors() { + let (client_side, server_side) = tokio::io::duplex(4096); + let (client_read, client_write) = tokio::io::split(client_side); + let (server_read, mut server_write) = tokio::io::split(server_side); + let (client, _incoming) = Client::spawn(client_read, client_write); + tokio::spawn(async move { + let mut lines = BufReader::new(server_read).lines(); + let request: Value = serde_json::from_str(&lines.next_line().await.unwrap().unwrap()).unwrap(); + let reply = json!({"id": request["id"], "error": {"code": -32600, "message": "no such thread"}}); + server_write.write_all(format!("{reply}\n").as_bytes()).await.unwrap(); + }); + assert_eq!( + client.request("thread/resume", json!({})).await, + Err(RpcError::Server { code: -32600, message: "no such thread".to_owned() }) + ); + } +} diff --git a/daemon/src/bin/vibebuddy.rs b/daemon/src/bin/vibebuddy.rs new file mode 100644 index 00000000..923365ca --- /dev/null +++ b/daemon/src/bin/vibebuddy.rs @@ -0,0 +1,114 @@ +//! `vibebuddy`: the command line for the local daemon. It only talks to `vibebuddyd` over its HTTP API and never +//! opens the serial port itself (docs/architecture.md, decision 2). +//! +//! - `vibebuddy say `: hands a voice command, typed, to the resident session and prints its spoken replies. +//! - `vibebuddy resident [n]`: prints the resident thread's id and its last `n` (default 5) voice commands with the +//! spoken replies, read from the thread (or, on Claude Code, the session). Both live in a home of the daemon's +//! own, so neither Agent's App can open them. +//! +//! The resident session is experimental: the daemon has it only when started with `VIBEBUDDY_RESIDENT_SESSION=1`. + +use std::io::{Read, Write}; +use std::net::{SocketAddr, TcpStream}; +use std::process::ExitCode; +use std::time::Duration; + +use serde_json::{Value, json}; + +/// A voice command waits for a whole Codex turn; the daemon gives up on it after two minutes. +const SAY_TIMEOUT: Duration = Duration::from_secs(180); +/// Reading the history may first have to start the app-server and wait for a turn in progress. +const HISTORY_TIMEOUT: Duration = Duration::from_secs(180); +const QUICK_TIMEOUT: Duration = Duration::from_secs(5); + +fn main() -> ExitCode { + let args: Vec = std::env::args().skip(1).collect(); + let result = match args.first().map(String::as_str) { + Some("say") if args.len() > 1 => say(&args[1..].join(" ")), + Some("resident") if args.len() == 1 => resident(5), + Some("resident") if let [_, count] = args.as_slice() + && let Ok(count) = count.parse() => resident(count), + _ => { + eprintln!("usage: vibebuddy say (a voice command for the resident session, typed)"); + eprintln!(" vibebuddy resident [n] (prints the resident session's last n voice commands and replies)"); + return ExitCode::from(2); + } + }; + match result { + Ok(()) => ExitCode::SUCCESS, + Err(error) => { + eprintln!("vibebuddy: {error}"); + ExitCode::FAILURE + } + } +} + +fn say(text: &str) -> Result<(), String> { + let answer = call("POST", "/v1/resident/say", Some(&json!({"text": text})), SAY_TIMEOUT)?; + let replies = answer["replies"].as_array().cloned().unwrap_or_default(); + if replies.is_empty() { + println!("(no spoken reply)"); + } + for reply in replies { + println!("{}", reply.as_str().unwrap_or_default()); + } + if let Some(elapsed) = answer["elapsed_ms"].as_u64() { + eprintln!("({:.1} s)", elapsed as f64 / 1000.0); + } + Ok(()) +} + +fn resident(limit: usize) -> Result<(), String> { + let info = call("GET", "/v1/resident", None, QUICK_TIMEOUT)?; + if info["enabled"] != true { + return Err("the resident session is off; start vibebuddyd with VIBEBUDDY_RESIDENT_SESSION=1".to_owned()); + } + match info["problem"].as_str() { + Some("codex_missing") => eprintln!("voice is off: Codex isn't installed (https://learn.chatgpt.com/docs/codex/cli)"), + Some("codex_logged_out") => eprintln!("voice is off: Codex isn't logged in; run `codex login`"), + Some("claude_missing") => eprintln!("voice is off: Claude Code isn't installed (https://code.claude.com/docs/en/setup)"), + Some("anthropic_key_missing") => eprintln!("voice is off: no Anthropic API key is set for Claude Code (Settings → Conversation)"), + _ => {} + } + if let Some(brain) = info["brain"].as_str() { + println!("brain {brain}"); + } + let Some(thread) = info["thread_id"].as_str() else { + println!("no resident thread yet; it starts with the first `vibebuddy say`"); + return Ok(()); + }; + println!("thread {thread}"); + let history = call("GET", &format!("/v1/resident/history?limit={limit}"), None, HISTORY_TIMEOUT)?; + for exchange in history["exchanges"].as_array().cloned().unwrap_or_default() { + println!("\n> {}", exchange["said"].as_str().unwrap_or_default()); + for reply in exchange["replies"].as_array().cloned().unwrap_or_default() { + println!(" {}", reply.as_str().unwrap_or_default()); + } + } + Ok(()) +} + +/// One request to the daemon at `VIBEBUDDY_BIND` (default 127.0.0.1:7331), like the hook's: no HTTP library. +fn call(method: &str, path: &str, body: Option<&Value>, timeout: Duration) -> Result { + let address = std::env::var("VIBEBUDDY_BIND").unwrap_or_else(|_| "127.0.0.1:7331".to_owned()); + let socket: SocketAddr = address.parse().map_err(|error| format!("invalid VIBEBUDDY_BIND: {error}"))?; + let unreachable = |error: std::io::Error| format!("vibebuddyd is not reachable at {address}: {error}"); + let mut stream = TcpStream::connect_timeout(&socket, QUICK_TIMEOUT).map_err(unreachable)?; + stream.set_read_timeout(Some(timeout)).map_err(unreachable)?; + let body = body.map(Value::to_string).unwrap_or_default(); + let request = format!( + "{method} {path} HTTP/1.1\r\nHost: {address}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + stream.write_all(request.as_bytes()).map_err(unreachable)?; + let mut response = String::new(); + stream.read_to_string(&mut response).map_err(|error| format!("no answer from vibebuddyd: {error}"))?; + let (head, payload) = response.split_once("\r\n\r\n").ok_or("malformed answer from vibebuddyd")?; + let status: u16 = head.split_whitespace().nth(1).and_then(|code| code.parse().ok()).unwrap_or(0); + let value: Value = serde_json::from_str(payload).unwrap_or(Value::Null); + if !(200..300).contains(&status) { + let message = value["message"].as_str().map(str::to_owned).unwrap_or_else(|| format!("HTTP {status}")); + return Err(message); + } + Ok(value) +} diff --git a/daemon/src/main.rs b/daemon/src/main.rs index 2d6baf72..43977d27 100644 --- a/daemon/src/main.rs +++ b/daemon/src/main.rs @@ -1,4 +1,5 @@ mod activity; +mod app_server; mod ci; mod agent_hooks; mod claude_hooks; @@ -8,15 +9,21 @@ mod identity; mod lan; mod noise_link; mod relay; +mod resident; +mod resident_claude; mod link_alert; mod occasions; +mod persona; mod rom_flasher; mod screenshot; mod serial_transport; mod session_titles; mod source_opener; +mod speech; mod status; +mod synthesis; mod updates; +mod voice; mod voice_writer; use std::convert::Infallible; @@ -93,8 +100,16 @@ struct AppState { updates: Arc>, /// "Check for updates" pressed: checks now instead of waiting for the daily one. check_updates: Arc, + /// The resident session voice commands go to (ADR-0017); None without a state directory, and in tests + /// unless one sets it. Present even when switched off, so its hook events are still recognised. + resident: Option>, + /// Voice commands from the box (ADR-0017): its recording, and the turn being answered. + voice: Arc, /// The box on the cable said `LINK NOISE` since its last `BOX KEY` (ADR-0015). noise_seen: Arc, + /// What the App said about the Character pack it last wrote (who lends a custom one, the form of address), + /// kept in [`persona::RECORD_FILE`] in the state directory. + written: Arc>, } impl AppState { @@ -128,7 +143,10 @@ impl AppState { state_dir: None, updates: Arc::new(Mutex::new(Updates::new(None, None))), check_updates: Arc::new(Notify::new()), + resident: None, + voice: Arc::new(voice::Voice::new(None, None)), noise_seen: Arc::new(std::sync::atomic::AtomicBool::new(false)), + written: Arc::default(), } } @@ -245,6 +263,25 @@ async fn main() { state.identity = identity; state.relay = relay_config; state.state_dir = config::state_dir(); + if let Some(dir) = &state.state_dir { + state.written = Arc::new(Mutex::new(persona::Written::load(&dir.join(persona::RECORD_FILE)))); + } + state.resident =state.state_dir.as_deref().map(|dir| { + let enabled = resident::enabled_by_env(); + if enabled { + warn!("the experimental resident session is on"); + } + let resident = resident::Resident::new(enabled, dir, resident::Launcher::Codex, state.activities.clone()); + Arc::new(match config::config_dir() { + Some(config) => resident.with_settings_file(config.join(resident::SETTINGS_FILE)), + None => resident, + }) + }); + state.voice = Arc::new(voice::Voice::new(speech::from_env(), state.resident.clone()).with_synthesizer(synthesis::from_env())); + if let Some(resident) = state.resident.clone() { + let wearing = wearing(&state).await; + tokio::spawn(async move { resident.warm(&wearing).await }); + } state.device.lock().await.pin = pin; state.updates = Arc::new(Mutex::new(Updates::new( updates::Source::configured(), @@ -262,6 +299,9 @@ async fn main() { tokio::spawn(poll_ci(state.clone())); tokio::spawn(handle_device_events(state.clone(), device_events)); tokio::spawn(check_for_updates(state.clone())); + if let Some(resident) = state.resident.clone().filter(|resident| resident.enabled()) { + follow_resident_readiness(&state, resident); + } // On macOS the app reports a lost link; elsewhere there is no app, so the daemon does it. if !cfg!(target_os = "macos") { tokio::spawn(link_alert::watch(state.clone())); @@ -359,6 +399,9 @@ async fn publish_device_message(state: &AppState, message: DeviceMessage) { { lan::remember(&lan::box_address_file(dir), address.trim()); } + if let DeviceMessage::Line(line) = &message { + handle_voice_line(state, line).await; + } let _ = state.device_bus.send(message.clone()); // Ask right after connecting; the device reports its mode, firmware build and voice again. if matches!(message, DeviceMessage::Connected { .. } | DeviceMessage::ConnectedNetwork { .. } | DeviceMessage::ConnectedRelay { .. }) { @@ -374,6 +417,77 @@ async fn publish_device_message(state: &AppState, message: DeviceMessage) { } } +/// Voice commands (docs/protocol.md, "Voice commands"): offered on every report of the box's build, which follows +/// hello on each new link and every reboot, and taken over native USB or the local network. The box sends no audio +/// over the UART bridge, which is too slow for it, and over the Relay it will talk to the voice service itself, so +/// voice isn't offered on either, and voice lines over the Relay are ignored. +async fn handle_voice_line(state: &AppState, line: &str) -> Option> { + let (over_relay, on_bridge) = { + let device = state.device.lock().await; + (device.relay, device.bridge) + }; + if line.contains(serial_transport::BUILD_MARKER) && state.voice.available() { + if over_relay { + info!("voice commands not offered: the box is reached through the Relay"); + } else if on_bridge { + info!("voice commands not offered: the box is on the UART bridge, which can't carry its audio; use its native USB port"); + } else { + send_event(state, voice::available_event(true)); + } + } + if !line.starts_with("VOICE ") { + return None; + } + if over_relay { + tracing::debug!("ignoring a voice line over the Relay"); + return None; + } + state.voice.handle_line(line, &state.transport, &wearing(state).await) +} + +/// What the buddy wears for spoken replies: the Character the box reported wearing, before it has the one last +/// written to it, with what the App said when it wrote it (a custom Character's lender, the form of address). +async fn wearing(state: &AppState) -> persona::Wearing { + let reported = state.device.lock().await.voice.clone(); + let id = match reported { + Some(id) => Some(id), + None => state.config.lock().await.voice.clone(), + }; + persona::Wearing::of(id, &*state.written.lock().await) +} + +/// Tells a box that has reported its build whether voice is on now, after the resident session's readiness flipped +/// (the user's Codex installed or gone, logged in or out). Told `false`, a long K0 gives up the Pomodoro phase as it +/// did before voice. Never over the Relay or the UART bridge, where voice is never offered. +async fn offer_voice_again(state: &AppState) { + let (booted, over_relay, on_bridge) = { + let device = state.device.lock().await; + (device.connected && device.firmware_build.is_some(), device.relay, device.bridge) + }; + if booted && !over_relay && !on_bridge { + let on = state.voice.available(); + info!(on, "the resident session's readiness changed; telling the box whether voice is on"); + send_event(state, voice::available_event(on)); + } +} + +/// Checks the resident session's readiness every [`resident::READINESS_TTL`] and tells the box each time it flips. +fn follow_resident_readiness(state: &AppState, resident: Arc) { + let mut flips = resident.watch_ready(); + let follower = state.clone(); + tokio::spawn(async move { + while flips.changed().await.is_ok() { + offer_voice_again(&follower).await; + } + }); + tokio::spawn(async move { + loop { + resident.check_readiness().await; + tokio::time::sleep(resident::READINESS_TTL).await; + } + }); +} + async fn open_k2_source(state: &AppState) { { let sources = state.activities.lock().await.focus_sources(); @@ -495,6 +609,11 @@ fn app(state: AppState) -> Router { .route("/v1/device/firmware", post(post_firmware)) .route("/v1/daemon/restart", post(post_restart)) .route("/v1/updates/check", post(post_check_updates)) + .route("/v1/resident", get(get_resident)) + .route("/v1/resident/say", post(post_resident_say)) + .route("/v1/resident/history", get(get_resident_history)) + .route("/v1/resident/brain", axum::routing::put(put_resident_brain)) + .route("/v1/resident/anthropic-key", axum::routing::put(put_anthropic_key).delete(delete_anthropic_key)) .with_state(state) } @@ -843,9 +962,18 @@ async fn post_choice( (StatusCode::ACCEPTED, Json(ApiResponse { accepted: true, message: "remembered".to_owned() })) } +/// What the App may say about a Character pack it writes, which the pack itself doesn't carry: the preset lending a +/// custom Character its voice, persona and lines, and the form of address its lines were made with. +#[derive(Default, serde::Deserialize)] +struct PackQuery { + lender: Option, + address: Option, +} + /// Write a voice pack: the request body is the pack itself. The write runs in the background with progress in the status stream. async fn post_voice_pack( State(state): State, + axum::extract::Query(query): axum::extract::Query, body: Bytes, ) -> (StatusCode, Json) { let Some(voice) = voice_writer::voice_id_of(&body) else { @@ -870,6 +998,13 @@ async fn post_voice_pack( Ok(written) => { info!(voice = %written, "voice pack written to device"); task_state.save_config(|config| config.voice = Some(written.clone())).await; + let record = persona::Written { id: Some(written.clone()), lender: query.lender, address: query.address }; + if let Some(dir) = &task_state.state_dir + && let Err(error) = record.save(&dir.join(persona::RECORD_FILE)) + { + warn!(%error, "failed to save what the App said about the Character"); + } + *task_state.written.lock().await = record; task_state .set_operation(Some(Operation { kind: OperationKind::VoicePack, @@ -1187,6 +1322,16 @@ async fn post_codex_hook( State(state): State, Json(hook): Json, ) -> (StatusCode, Json) { + // The resident session is an Agent, but never an Activity: no card, announcement or stats (ADR-0017). + if let Some(resident) = &state.resident + && resident.owns_hook(&hook.session_id, hook.thread_id.as_deref(), hook.cwd.as_deref()) + { + tracing::debug!(event = %hook.hook_event_name, "ignoring a hook from the resident session"); + return ( + StatusCode::ACCEPTED, + Json(ApiResponse { accepted: true, message: "resident session, ignored".to_owned() }), + ); + } state.hooks_seen.lock().await.codex = Some(chrono::Local::now()); state.notify_status(); let event = { @@ -1204,6 +1349,16 @@ async fn post_claude_hook( State(state): State, Json(hook): Json, ) -> (StatusCode, Json) { + // The resident session on Claude Code is never an Activity either (ADR-0017). + if let Some(resident) = &state.resident + && resident.owns_claude_hook(&hook.session_id, hook.cwd.as_deref()) + { + tracing::debug!(event = %hook.hook_event_name, "ignoring a Claude Code hook from the resident session"); + return ( + StatusCode::ACCEPTED, + Json(ApiResponse { accepted: true, message: "resident session, ignored".to_owned() }), + ); + } state.hooks_seen.lock().await.claude = Some(chrono::Local::now()); state.notify_status(); let event = { @@ -1259,6 +1414,174 @@ async fn forward(state: AppState, event: Option) -> (StatusCode, Json, + /// Goes up when a voice command's turn ends or the thread changes; the Conversation tab polls it. + revision: u64, + /// On, and the user's Codex was last found installed and logged in: voice commands can be taken. + ready: bool, + /// What is missing when it is on but not ready: `codex_missing`, `codex_logged_out`, `claude_missing` or + /// `anthropic_key_missing`. + problem: Option, + /// The brain voice commands go to while ready: `codex` or `claude`. + brain: Option, + /// The user's setting: `auto`, `codex` or `claude`. + brain_setting: resident::BrainChoice, + /// Whether an Anthropic API key is stored; the key itself is never sent back. + anthropic_key_set: bool, +} + +/// Asking checks the resident session's readiness again when the last check is older than +/// [`resident::READINESS_TTL`]; it never starts the app-server or Claude Code. +async fn get_resident(State(state): State) -> Json { + let Some(resident) = state.resident.as_ref() else { + return Json(ResidentInfo { + enabled: false, + thread_id: None, + revision: 0, + ready: false, + problem: None, + brain: None, + brain_setting: resident::BrainChoice::default(), + anthropic_key_set: false, + }); + }; + let readiness = resident.check_readiness().await; + let on = resident.enabled(); + Json(ResidentInfo { + enabled: on, + thread_id: resident.thread_id(), + revision: resident.revision(), + ready: resident.ready(), + problem: if on { readiness.problem } else { None }, + brain: if resident.ready() { readiness.brain } else { None }, + brain_setting: resident.brain_setting(), + anthropic_key_set: resident.anthropic_key_set(), + }) +} + +#[derive(serde::Deserialize)] +struct BrainRequest { + brain: resident::BrainChoice, +} + +/// Sets which brain the resident session runs on. Its own endpoint, kept out of `PUT /v1/config`. +async fn put_resident_brain(State(state): State, Json(request): Json) -> axum::response::Response { + use axum::response::IntoResponse; + let Some(resident) = state.resident.clone() else { + return no_state_directory(); + }; + if let Err(error) = resident.set_brain(request.brain).await { + warn!(%error, "failed to save the resident session's brain setting"); + return (StatusCode::INTERNAL_SERVER_ERROR, Json(ApiResponse { accepted: false, message: error.to_string() })).into_response(); + } + get_resident(State(state)).await.into_response() +} + +#[derive(serde::Deserialize)] +struct AnthropicKeyRequest { + api_key: String, +} + +/// Stores the user's Anthropic API key for Claude Code (mode 0600, in its own file); it is never sent back or +/// logged. Its own endpoint, so an App that doesn't know it can't wipe it. +async fn put_anthropic_key(State(state): State, Json(request): Json) -> axum::response::Response { + use axum::response::IntoResponse; + if request.api_key.trim().is_empty() { + return (StatusCode::BAD_REQUEST, Json(ApiResponse { accepted: false, message: "api_key is empty".to_owned() })).into_response(); + } + set_anthropic_key(state, Some(request.api_key)).await +} + +async fn delete_anthropic_key(State(state): State) -> axum::response::Response { + set_anthropic_key(state, None).await +} + +async fn set_anthropic_key(state: AppState, key: Option) -> axum::response::Response { + use axum::response::IntoResponse; + let Some(resident) = state.resident.clone() else { + return no_state_directory(); + }; + if let Err(error) = resident.set_anthropic_key(key).await { + warn!(%error, "failed to save the Anthropic API key"); + return (StatusCode::INTERNAL_SERVER_ERROR, Json(ApiResponse { accepted: false, message: error.to_string() })).into_response(); + } + get_resident(State(state)).await.into_response() +} + +fn no_state_directory() -> axum::response::Response { + use axum::response::IntoResponse; + (StatusCode::SERVICE_UNAVAILABLE, Json(ApiResponse { accepted: false, message: "no state directory".to_owned() })).into_response() +} + +#[derive(serde::Deserialize)] +struct HistoryQuery { + limit: Option, +} + +/// The last voice commands and their spoken replies, read from the resident thread (default 5, at most 50). +async fn get_resident_history( + State(state): State, + axum::extract::Query(query): axum::extract::Query, +) -> axum::response::Response { + use axum::response::IntoResponse; + let Some(resident) = state.resident.clone() else { + return (StatusCode::SERVICE_UNAVAILABLE, Json(ApiResponse { accepted: false, message: "no state directory".to_owned() })) + .into_response(); + }; + match resident.history(query.limit.unwrap_or(5).clamp(1, 50)).await { + Ok(exchanges) => Json(serde_json::json!({"thread_id": resident.thread_id(), "exchanges": exchanges})).into_response(), + Err(error) => { + let status = match error { + resident::ResidentError::Disabled => StatusCode::NOT_FOUND, + _ => StatusCode::SERVICE_UNAVAILABLE, + }; + (status, Json(ApiResponse { accepted: false, message: error.to_string() })).into_response() + } + } +} + +/// One voice command, typed for now: waits for the resident session's spoken replies. +async fn post_resident_say(State(state): State, Json(request): Json) -> axum::response::Response { + use axum::response::IntoResponse; + let Some(resident) = state.resident.clone() else { + return (StatusCode::SERVICE_UNAVAILABLE, Json(ApiResponse { accepted: false, message: "no state directory".to_owned() })) + .into_response(); + }; + let text = request.text.trim(); + if text.is_empty() { + return (StatusCode::BAD_REQUEST, Json(ApiResponse { accepted: false, message: "text is empty".to_owned() })).into_response(); + } + let started = std::time::Instant::now(); + match resident.say(text, &wearing(&state).await).await { + Ok(answer) => { + info!(elapsed_ms = started.elapsed().as_millis() as u64, replies = answer.replies.len(), "voice command answered"); + Json(serde_json::json!({ + "thread_id": answer.thread_id, + "replies": answer.replies, + "elapsed_ms": started.elapsed().as_millis() as u64, + })) + .into_response() + } + Err(error) => { + warn!(%error, "voice command failed"); + let status = match error { + resident::ResidentError::Disabled => StatusCode::NOT_FOUND, + resident::ResidentError::Timeout => StatusCode::GATEWAY_TIMEOUT, + _ => StatusCode::SERVICE_UNAVAILABLE, + }; + (status, Json(ApiResponse { accepted: false, message: error.to_string() })).into_response() + } + } +} + #[cfg(test)] mod tests { use std::sync::Mutex; @@ -1434,6 +1757,95 @@ mod tests { panic!("event {index} never showed up"); } + #[tokio::test] + async fn the_resident_sessions_own_hooks_make_no_card_announcement_or_stats() { + let dir = std::env::temp_dir().join(format!("vibebuddy-main-resident-hooks-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + let transport = Arc::new(RecordingTransport::default()); + let mut state = test_state(transport.clone()); + state.resident = Some(Arc::new(resident::Resident::new( + false, + &dir, + resident::Launcher::Codex, + state.activities.clone(), + ))); + let resident_cwd = dir.join("resident").display().to_string(); + for event in ["UserPromptSubmit", "PostToolUse", "Stop"] { + let hook: CodexHook = serde_json::from_value(serde_json::json!({ + "session_id": "resident-thread", + "turn_id": "t1", + "hook_event_name": event, + "cwd": resident_cwd, + })) + .unwrap(); + let _ = post_codex_hook(State(state.clone()), Json(hook)).await; + } + assert!(transport.events().is_empty()); + assert!(state.activities.lock().await.summaries().is_empty()); + assert_eq!(state.activities.lock().await.today().done, 0); + + // Any other Codex session still counts. + let hook: CodexHook = serde_json::from_value(serde_json::json!({ + "session_id": "work", + "turn_id": "t1", + "hook_event_name": "UserPromptSubmit", + "cwd": "/work/vibe-buddy", + })) + .unwrap(); + let _ = post_codex_hook(State(state.clone()), Json(hook)).await; + assert_eq!(state.activities.lock().await.summaries().len(), 1); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_resident_claude_sessions_hooks_make_no_card_and_the_key_is_never_sent_back() { + let dir = resident::tests::temp_dir("main-claude"); + let transport = Arc::new(RecordingTransport::default()); + let mut state = test_state(transport.clone()); + let claude = resident_claude::tests::script(); + state.resident = Some(Arc::new( + resident::tests::resident(&dir, &resident::tests::logged_in(), state.activities.clone()) + .with_claude(resident_claude::tests::brain(&dir, &claude, state.activities.clone())), + )); + + let request = AnthropicKeyRequest { api_key: " ".to_owned() }; + assert_eq!(put_anthropic_key(State(state.clone()), Json(request)).await.status(), StatusCode::BAD_REQUEST); + let request = AnthropicKeyRequest { api_key: "sk-ant-secret".to_owned() }; + let response = put_anthropic_key(State(state.clone()), Json(request)).await; + assert_eq!(response.status(), StatusCode::OK); + let body = http_body_util::BodyExt::collect(response.into_body()).await.unwrap().to_bytes(); + let info: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!(info["anthropic_key_set"], true); + assert!(!String::from_utf8_lossy(&body).contains("sk-ant-secret")); + let request = BrainRequest { brain: resident::BrainChoice::Claude }; + let response = put_resident_brain(State(state.clone()), Json(request)).await; + let body = http_body_util::BodyExt::collect(response.into_body()).await.unwrap().to_bytes(); + let info: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!((info["brain_setting"].as_str(), info["brain"].as_str(), info["ready"].as_bool()), (Some("claude"), Some("claude"), Some(true))); + + let resident = state.resident.clone().unwrap(); + resident.say("hi", &persona::Wearing::default()).await.unwrap(); + let session = resident.thread_id().unwrap(); + for event in ["UserPromptSubmit", "Stop"] { + let hook: ClaudeHook = serde_json::from_value(serde_json::json!({ + "session_id": session, + "prompt_id": "p1", + "hook_event_name": event, + "cwd": dir.join("resident").display().to_string(), + })) + .unwrap(); + let _ = post_claude_hook(State(state.clone()), Json(hook)).await; + } + assert!(transport.events().is_empty()); + assert!(state.activities.lock().await.summaries().is_empty()); + + let response = delete_anthropic_key(State(state.clone())).await; + let body = http_body_util::BodyExt::collect(response.into_body()).await.unwrap().to_bytes(); + let info: serde_json::Value = serde_json::from_slice(&body).unwrap(); + assert_eq!((info["anthropic_key_set"].as_bool(), info["problem"].as_str()), (Some(false), Some("anthropic_key_missing"))); + let _ = std::fs::remove_dir_all(dir); + } + #[test] fn the_app_version_leads_the_build_identity() { assert_eq!( @@ -1443,6 +1855,118 @@ mod tests { assert_eq!(build_identity_from(None, "abc1234", ""), "abc1234"); } + /// A recognizer that is never called: these tests only look at whether voice is offered. + struct SilentRecognizer; + + impl speech::Recognizer for SilentRecognizer { + fn recognize(&self, _wav: &[u8]) -> Result { + Ok(speech::Transcript::default()) + } + } + + #[tokio::test] + async fn voice_is_offered_after_each_build_report_but_never_over_the_relay_or_the_uart_bridge() { + let dir = std::env::temp_dir().join(format!("vibebuddy-voice-offer-{}", std::process::id())); + let offers = |transport: &RecordingTransport| { + transport.events().iter().filter(|event| event.event == "voice.available").map(|event| event.extra["on"].clone()).collect::>() + }; + let build = || DeviceMessage::Line("DISPLAY READY BUILD v9 2026-10-01".to_owned()); + for enabled in [true, false] { + let transport = Arc::new(RecordingTransport::default()); + let mut state = test_state(transport.clone()); + let resident = Arc::new(resident::Resident::new(enabled, &dir, resident::Launcher::Codex, Arc::default())); + resident.note(None); + state.voice = Arc::new(voice::Voice::new(Some(Arc::new(SilentRecognizer)), Some(resident))); + + publish_device_message(&state, DeviceMessage::Connected { port: "/dev/cu.x".to_owned(), bridge: false, usb_serial: None }).await; + assert!(offers(&transport).is_empty(), "a new link starts with voice off"); + publish_device_message(&state, build()).await; + // The box rebooting on the same link reports its build again, and forgets. + publish_device_message(&state, build()).await; + publish_device_message(&state, DeviceMessage::ConnectedNetwork { address: "192.168.1.23:7340".to_owned() }).await; + publish_device_message(&state, build()).await; + let expected = if enabled { vec![serde_json::json!(true); 3] } else { Vec::new() }; + assert_eq!(offers(&transport), expected); + + publish_device_message(&state, DeviceMessage::ConnectedRelay { url: "wss://relay.example".to_owned() }).await; + publish_device_message(&state, build()).await; + assert_eq!(offers(&transport), expected, "not over the Relay"); + // The box sends no audio over the UART bridge. + publish_device_message(&state, DeviceMessage::Connected { port: "/dev/cu.wchusbserial1".to_owned(), bridge: true, usb_serial: None }).await; + publish_device_message(&state, build()).await; + assert_eq!(offers(&transport), expected, "not over the UART bridge"); + } + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_box_is_told_each_time_the_resident_session_becomes_ready_or_stops_being() { + let dir = resident::tests::temp_dir("main-readiness-offer"); + let transport = Arc::new(RecordingTransport::default()); + let mut state = test_state(transport.clone()); + let resident = Arc::new(resident::tests::resident(&dir, &resident::tests::logged_in(), Arc::default())); + state.resident = Some(resident.clone()); + state.voice = Arc::new(voice::Voice::new(Some(Arc::new(SilentRecognizer)), Some(resident.clone()))); + let offers = || { + transport.events().iter().filter(|event| event.event == "voice.available").map(|event| event.extra["on"].clone()).collect::>() + }; + let settle = || tokio::time::sleep(Duration::from_millis(50)); + let build = || DeviceMessage::Line("DISPLAY READY BUILD v9 2026-10-01".to_owned()); + + publish_device_message(&state, DeviceMessage::Connected { port: "/dev/cu.x".to_owned(), bridge: false, usb_serial: None }).await; + publish_device_message(&state, build()).await; + assert!(offers().is_empty(), "not offered before the session is found ready"); + + // The first check finds Codex installed and logged in. + follow_resident_readiness(&state, resident.clone()); + settle().await; + assert_eq!(offers(), [serde_json::json!(true)]); + resident.note(Some(resident::Problem::CodexLoggedOut)); + settle().await; + assert_eq!(offers(), [serde_json::json!(true), serde_json::json!(false)], "withdrawn when the user logs out"); + resident.note(Some(resident::Problem::CodexMissing)); + settle().await; + assert_eq!(offers().len(), 2, "still not ready: nothing new to tell"); + + // Never over the UART bridge, where voice isn't offered at all. + publish_device_message(&state, DeviceMessage::Connected { port: "/dev/cu.wchusbserial1".to_owned(), bridge: true, usb_serial: None }).await; + publish_device_message(&state, build()).await; + resident.note(None); + settle().await; + assert_eq!(offers().len(), 2); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_resident_info_carries_a_revision_without_starting_the_app_server() { + let dir = resident::tests::temp_dir("main-revision"); + let script = resident::tests::logged_in(); + let mut state = test_state(Arc::new(RecordingTransport::default())); + state.resident = Some(Arc::new(resident::tests::resident(&dir, &script, Arc::default()))); + + let Json(info) = get_resident(State(state.clone())).await; + assert_eq!( + serde_json::to_value(&info).unwrap(), + serde_json::json!({ + "enabled": true, + "thread_id": null, + "revision": 0, + "ready": true, + "problem": null, + "brain": "codex", + "brain_setting": "auto", + "anthropic_key_set": false, + }) + ); + assert_eq!(script.lock().unwrap().starts, 0, "asking never starts the app-server"); + + state.resident.as_ref().unwrap().say("hi", &persona::Wearing::default()).await.unwrap(); + let Json(info) = get_resident(State(state.clone())).await; + assert_eq!(info.thread_id.as_deref(), Some("new-thread")); + assert_eq!(info.revision, 2, "a new thread and one turn"); + let _ = std::fs::remove_dir_all(dir); + } + #[tokio::test] async fn status_reflects_device_lines_hooks_and_config() { let transport = Arc::new(RecordingTransport::default()); @@ -1561,6 +2085,10 @@ mod tests { assert!(transport.events().is_empty()); } + fn no_query() -> axum::extract::Query { + axum::extract::Query(PackQuery::default()) + } + fn sample_pack(voice: &str, payload_len: usize) -> Vec { let mut pack = vec![0_u8; 256 + payload_len]; pack[0..4].copy_from_slice(b"VBVP"); @@ -1575,9 +2103,10 @@ mod tests { async fn writing_a_voice_pack_waits_for_each_acknowledgement() { let transport = Arc::new(RecordingTransport::default()); let state = test_state(transport.clone()); - let pack = sample_pack("hsiaoyu", 1000); // 1256 bytes → two chunks + let pack = sample_pack("custom", 1000); // 1256 bytes → two chunks - let (status, _) = post_voice_pack(State(state.clone()), Bytes::from(pack.clone())).await; + let query = PackQuery { lender: Some("ada".to_owned()), address: Some("boss".to_owned()) }; + let (status, _) = post_voice_pack(State(state.clone()), axum::extract::Query(query), Bytes::from(pack.clone())).await; assert_eq!(status, StatusCode::ACCEPTED); let begin = nth_event(&transport, 0).await; @@ -1602,7 +2131,7 @@ mod tests { let end = nth_event(&transport, 3).await; assert_eq!(end.event, "voice.end"); - publish_device_message(&state, device_event(r#"{"version":1,"event":"voice.written","voice":"hsiaoyu"}"#)).await; + publish_device_message(&state, device_event(r#"{"version":1,"event":"voice.written","voice":"custom"}"#)).await; for _ in 0..200 { if state.operation.lock().await.as_ref().map(|op| op.state) == Some(OperationState::Done) { break; @@ -1611,7 +2140,12 @@ mod tests { } let operation = state.operation.lock().await.clone().expect("an operation should be recorded"); assert_eq!(operation.state, OperationState::Done); - assert_eq!(state.config.lock().await.voice.as_deref(), Some("hsiaoyu")); + assert_eq!(state.config.lock().await.voice.as_deref(), Some("custom")); + // What the App said about it is kept: the custom Character speaks with Ada's persona and calls the user boss. + assert_eq!( + wearing(&state).await, + persona::Wearing { character: Some("ada".to_owned()), address: Some("boss".to_owned()) } + ); } #[tokio::test] @@ -1619,7 +2153,7 @@ mod tests { let transport = Arc::new(RecordingTransport::default()); let state = test_state(transport.clone()); let pack = sample_pack("hsiaoyu", 100); - let _ = post_voice_pack(State(state.clone()), Bytes::from(pack)).await; + let _ = post_voice_pack(State(state.clone()), no_query(), Bytes::from(pack)).await; nth_event(&transport, 0).await; publish_device_message( &state, @@ -1642,10 +2176,10 @@ mod tests { async fn a_second_write_is_refused_while_one_is_running_and_garbage_is_rejected() { let transport = Arc::new(RecordingTransport::default()); let state = test_state(transport.clone()); - let (status, _) = post_voice_pack(State(state.clone()), Bytes::from_static(b"not a pack")).await; + let (status, _) = post_voice_pack(State(state.clone()), no_query(), Bytes::from_static(b"not a pack")).await; assert_eq!(status, StatusCode::BAD_REQUEST); - let _ = post_voice_pack(State(state.clone()), Bytes::from(sample_pack("a", 10))).await; - let (status, _) = post_voice_pack(State(state.clone()), Bytes::from(sample_pack("b", 10))).await; + let _ = post_voice_pack(State(state.clone()), no_query(), Bytes::from(sample_pack("a", 10))).await; + let (status, _) = post_voice_pack(State(state.clone()), no_query(), Bytes::from(sample_pack("b", 10))).await; assert_eq!(status, StatusCode::CONFLICT); } diff --git a/daemon/src/persona.rs b/daemon/src/persona.rs new file mode 100644 index 00000000..e3729ace --- /dev/null +++ b/daemon/src/persona.rs @@ -0,0 +1,201 @@ +//! Who the buddy is when it answers a voice command (ADR-0017): the persona and language of the Character it +//! wears, and the user's form of address, as the resident session's instructions carry them. +//! +//! The box reports only the id of the pack it wears (`VOICES`). Which preset lends a custom Character its voice, +//! persona and lines, and which form of address the pack's lines were written with, only the App knows; it says so +//! when it writes the pack (`POST /v1/device/voice-pack?lender=…&address=…`), and the daemon keeps it in +//! [`RECORD_FILE`]. + +use std::path::Path; + +use serde::{Deserialize, Serialize}; + +/// What the App said about the pack it last wrote, in the daemon's state directory. +pub const RECORD_FILE: &str = "character.json"; +/// The id a custom Character's pack carries, whatever lends it voice and lines. +const CUSTOM: &str = "custom"; +/// The default Character, and the persona of any pack these files don't know. +const ROBOT: &str = "robot"; +/// The robot's lines are the built-in voice's, in English; it has no `voice.env`. +const ROBOT_LANGUAGE: &str = "en"; + +/// Each Character's persona (`characters//persona.md`), the same one its lines were written in. +const PERSONAS: &[(&str, &str)] = &[ + ("ada", include_str!("../../characters/ada/persona.md")), + ("dave", include_str!("../../characters/dave/persona.md")), + ("hank", include_str!("../../characters/hank/persona.md")), + ("kai", include_str!("../../characters/kai/persona.md")), + ("luna", include_str!("../../characters/luna/persona.md")), + ("mei", include_str!("../../characters/mei/persona.md")), + ("robot", include_str!("../../characters/robot/persona.md")), + ("wanwanxiaohe", include_str!("../../characters/wanwanxiaohe/persona.md")), +]; + +/// The forms of address the App offers (docs/characters.md): id, words, language. +const FORMS_OF_ADDRESS: &[(&str, &str, &str)] = &[ + ("laoban", "老板", "zh"), + ("dalao", "大佬", "zh"), + ("ge", "哥", "zh"), + ("jie", "姐", "zh"), + ("qin", "亲", "zh"), + ("boss", "boss", "en"), + ("captain", "captain", "en"), + ("buddy", "buddy", "en"), +]; + +/// What the App said when it last wrote a Character pack: its id, the preset lending a custom Character its +/// voice, persona and lines, and the form of address its lines carry. +#[derive(Clone, Debug, Default, Deserialize, PartialEq, Eq, Serialize)] +#[serde(default)] +pub struct Written { + pub id: Option, + pub lender: Option, + pub address: Option, +} + +impl Written { + pub fn load(path: &Path) -> Self { + std::fs::read_to_string(path).ok().and_then(|text| serde_json::from_str(&text).ok()).unwrap_or_default() + } + + pub fn save(&self, path: &Path) -> std::io::Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + std::fs::write(path, serde_json::to_string_pretty(self).map_err(std::io::Error::other)?) + } +} + +/// The Character the buddy wears, as far as its voice and persona go. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Wearing { + /// Whose voice and persona: the lender for a custom Character; None with no Character pack. + pub character: Option, + /// The form of address the user picked (`boss`, `laoban`), if any. + pub address: Option, +} + +impl Wearing { + /// The Character pack `id` (the box's report, else the one last written), with what the App said when it + /// wrote it, if that was this pack. + pub fn of(id: Option, written: &Written) -> Self { + match id { + Some(id) if written.id.as_deref() == Some(id.as_str()) => Self { + character: if id == CUSTOM { written.lender.clone() } else { Some(id) }, + address: written.address.clone(), + }, + id => Self { character: id, address: None }, + } + } +} + +/// How the buddy speaks its spoken replies. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Persona { + /// The Character whose persona this is. + pub id: &'static str, + /// Its name, from the persona's heading. + pub name: &'static str, + /// The persona, without its heading. + pub text: &'static str, + /// The one language it speaks, in English (`English`, `Chinese`). + pub language: &'static str, + /// The words of the user's form of address, when one in this language is picked. + pub address: Option<&'static str>, +} + +impl Persona { + /// The persona of what the buddy wears: the robot's for no pack, a custom Character with no known lender, + /// or a pack these files don't know. + pub fn of(wearing: &Wearing) -> Self { + let (id, file) = wearing + .character + .as_deref() + .and_then(|character| PERSONAS.iter().find(|(id, _)| *id == character)) + .or_else(|| PERSONAS.iter().find(|(id, _)| *id == ROBOT)) + .copied() + .expect("the robot has a persona"); + let (heading, text) = file.split_once('\n').unwrap_or((file, "")); + let code = crate::synthesis::VOICE_FILES + .iter() + .find(|(voice, _)| *voice == id) + .and_then(|(_, voice)| crate::synthesis::voice_field(voice, "LANGUAGE")) + .unwrap_or(ROBOT_LANGUAGE); + let address = wearing.address.as_deref().and_then(|form| { + FORMS_OF_ADDRESS.iter().find(|(id, _, language)| *id == form && *language == code).map(|(_, words, _)| *words) + }); + Self { + id, + name: heading.trim_start_matches('#').trim(), + text: text.trim(), + language: match code { + "zh" => "Chinese", + _ => "English", + }, + address, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn wearing(character: Option<&str>, address: Option<&str>) -> Wearing { + Wearing { character: character.map(str::to_owned), address: address.map(str::to_owned) } + } + + #[test] + fn each_character_has_its_own_persona_name_and_language() { + let ada = Persona::of(&wearing(Some("ada"), None)); + assert_eq!((ada.id, ada.name, ada.language), ("ada", "Ada", "English")); + assert!(ada.text.starts_with("Speaks English") && ada.text.contains("Londoner"), "{}", ada.text); + let xiaohe = Persona::of(&wearing(Some("wanwanxiaohe"), None)); + assert_eq!((xiaohe.id, xiaohe.name, xiaohe.language), ("wanwanxiaohe", "Xiaohe (小何)", "Chinese")); + for id in ["dave", "hank", "kai", "luna", "mei"] { + let persona = Persona::of(&wearing(Some(id), None)); + assert_eq!((persona.id, persona.language), (id, "English")); + assert!(!persona.text.is_empty() && !persona.text.starts_with('#')); + } + } + + #[test] + fn no_pack_or_an_unknown_one_is_the_robot() { + for character in [None, Some("robot"), Some("custom"), Some("hsiaoyu")] { + let persona = Persona::of(&wearing(character, None)); + assert_eq!((persona.id, persona.name, persona.language), ("robot", "VibeBuddy", "English"), "{character:?}"); + } + } + + #[test] + fn a_form_of_address_counts_only_in_the_characters_language() { + assert_eq!(Persona::of(&wearing(Some("ada"), Some("boss"))).address, Some("boss")); + assert_eq!(Persona::of(&wearing(Some("wanwanxiaohe"), Some("laoban"))).address, Some("老板")); + assert_eq!(Persona::of(&wearing(Some("ada"), Some("laoban"))).address, None); + assert_eq!(Persona::of(&wearing(Some("ada"), Some("darling"))).address, None); + } + + #[test] + fn a_custom_character_wears_its_lenders_voice_and_persona_and_the_form_written_with_it() { + let written = Written { id: Some("custom".to_owned()), lender: Some("ada".to_owned()), address: Some("boss".to_owned()) }; + assert_eq!(Wearing::of(Some("custom".to_owned()), &written), wearing(Some("ada"), Some("boss"))); + let persona = Persona::of(&Wearing::of(Some("custom".to_owned()), &written)); + assert_eq!((persona.id, persona.address), ("ada", Some("boss"))); + // The box reports another pack than the App last wrote (written elsewhere): only its id is known. + assert_eq!(Wearing::of(Some("mei".to_owned()), &written), wearing(Some("mei"), None)); + // A custom Character written by an App that didn't say who lends it: the robot's persona. + assert_eq!(Wearing::of(Some("custom".to_owned()), &Written::default()), wearing(Some("custom"), None)); + assert_eq!(Wearing::of(None, &written), wearing(None, None)); + } + + #[test] + fn the_record_round_trips_and_a_missing_one_is_empty() { + let dir = std::env::temp_dir().join(format!("vibebuddy-persona-{}", std::process::id())); + let path = dir.join(RECORD_FILE); + assert_eq!(Written::load(&path), Written::default()); + let written = Written { id: Some("custom".to_owned()), lender: Some("kai".to_owned()), address: None }; + written.save(&path).unwrap(); + assert_eq!(Written::load(&path), written); + let _ = std::fs::remove_dir_all(dir); + } +} diff --git a/daemon/src/resident.rs b/daemon/src/resident.rs new file mode 100644 index 00000000..2b48bb61 --- /dev/null +++ b/daemon/src/resident.rs @@ -0,0 +1,2097 @@ +//! The resident session (ADR-0017): the one long-running Agent session every voice command goes to, on one of two +//! brains: the user's Codex (this module) or the user's Claude Code with their own Anthropic API key +//! ([`crate::resident_claude`]). Which one is a setting (`codex`, `claude` or `auto`: Codex when it is ready, else +//! Claude Code when a key is set); each brain keeps its own session for good. +//! +//! The daemon drives `codex app-server` over stdio with the user's own login, in a Codex home of its own +//! (`resident-codex/` in the daemon's state directory) that the user's Codex and the ChatGPT App never list, so they +//! never take the thread's writer lock: `thread/start` once, `thread/resume` on every later start, `turn/start` per +//! voice command. The session answers through the `speak` +//! tool we define (`dynamicTools`), and reads the Activities through `list_activities`; it gets no tool that runs +//! commands or writes files, runs read-only, and every approval or question it raises is declined. It reads the +//! user's Codex memories, but only through the daemon, which never writes them. +//! +//! Experimental and off unless `VIBEBUDDY_RESIDENT_SESSION=1`. The app-server starts on the first voice command +//! and again on the next one after it dies. Voice commands are taken one at a time; the ones that arrive meanwhile +//! wait their turn in arrival order. +//! +//! The resident session is never an Activity: should any Codex hooks fire for its thread, [`Resident::owns_hook`] +//! is how the Codex adapter recognises and drops them. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::Duration; + +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use tokio::io::{AsyncRead, AsyncWrite}; +use tokio::sync::{Mutex, mpsc, watch}; +use tracing::{info, warn}; + +use crate::activity::ActivityTracker; +use crate::app_server::{Client, Incoming, RpcError}; +use crate::persona::{Persona, Wearing}; +use crate::resident_claude::{CONTEXT_PREFIX, ClaudeBrain, command_context, local_time_zone}; + +/// The Codex version the app-server protocol was verified with. app-server and `dynamicTools` are experimental, +/// so other versions get a warning in the log, but are still tried. +pub const TESTED_CODEX_VERSION: &str = "0.162.0"; +/// Setting this to `1` turns the resident session on. +pub const ENABLE_VARIABLE: &str = "VIBEBUDDY_RESIDENT_SESSION"; +/// Names the `codex` binary explicitly; otherwise it is looked for on PATH and where installers put it. +pub const CODEX_VARIABLE: &str = "VIBEBUDDY_CODEX"; +/// How long a single request (handshake, thread start, turn start) may take. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(60); +/// How long one voice command may take from `turn/start` to `turn/completed`; then the turn is interrupted. +const TURN_TIMEOUT: Duration = Duration::from_secs(120); +/// The resident session's own Codex home, in the daemon's state directory. +const CODEX_HOME_DIR: &str = "resident-codex"; +/// How much of the user's `memory_summary.md` goes into the instructions: Codex's own cap, 2,500 tokens, which it +/// counts as 4 bytes each. +const MEMORY_SUMMARY_MAX_BYTES: usize = 10_000; +/// How much one `search_memories` call returns at most. +const MEMORY_SEARCH_MAX_BYTES: usize = 8_000; +/// The model the resident session runs on, and its reasoning effort: a voice command waits on the answer, and the +/// session only talks, looks things up and hands work on. See [`tool_overrides`] for what was measured. +const MODEL: &str = "gpt-6-luna"; +const REASONING_EFFORT: &str = "low"; +/// How long a readiness check (is `codex` there, is the user logged in) stands before it is made again. +pub const READINESS_TTL: Duration = Duration::from_secs(30); +/// The login store the resident Codex uses: the user's `auth.json`, linked in (see [`tool_overrides`]). +const AUTH_STORE: &str = "cli_auth_credentials_store=\"file\""; +/// Codex's built-in web search (`disabled`, `cached`, `indexed` or `live` in codex 0.162.0): `live`, since the +/// questions it is for (weather, news, prices) are about now, which a cached index may not have yet. +const WEB_SEARCH: &str = "live"; +/// The user's brain setting and Anthropic API key, in the config directory (mode 0600). A file of its own, written +/// only through its own endpoints, because `PUT /v1/config` replaces the whole config and an older App would wipe a +/// key it doesn't know. +pub const SETTINGS_FILE: &str = "resident-settings.json"; + +pub fn enabled_by_env() -> bool { + matches!(std::env::var(ENABLE_VARIABLE).as_deref(), Ok("1" | "true")) +} + +#[derive(Debug, PartialEq)] +pub enum ResidentError { + Disabled, + NoCodex, + NotLoggedIn, + NoClaude, + NoAnthropicKey, + Rpc(String), + Timeout, + /// The turn ended without completing: interrupted or failed. + TurnEnded(String), +} + +impl std::fmt::Display for ResidentError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Disabled => write!(formatter, "the resident session is experimental and off; start vibebuddyd with {ENABLE_VARIABLE}=1"), + Self::NoCodex => write!(formatter, "codex was not found on PATH (set {CODEX_VARIABLE} to its path)"), + Self::NotLoggedIn => write!(formatter, "Codex is not logged in; run `codex login` first"), + Self::NoClaude => { + write!(formatter, "claude was not found on PATH (set {} to its path)", crate::resident_claude::CLAUDE_VARIABLE) + } + Self::NoAnthropicKey => write!(formatter, "no Anthropic API key is set for Claude Code"), + Self::Rpc(message) => write!(formatter, "{message}"), + Self::Timeout => write!(formatter, "the resident session took longer than {} s and was interrupted", TURN_TIMEOUT.as_secs()), + Self::TurnEnded(message) => write!(formatter, "the turn ended without completing: {message}"), + } + } +} + +impl From for ResidentError { + fn from(error: RpcError) -> Self { + Self::Rpc(error.to_string()) + } +} + +/// Why the resident session can't take a voice command now: voice needs the user's Codex, installed and logged in, +/// or the user's Claude Code, installed, with their Anthropic API key. +#[derive(Clone, Copy, Debug, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Problem { + CodexMissing, + CodexLoggedOut, + ClaudeMissing, + AnthropicKeyMissing, +} + +/// The Agent the resident session runs on. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Brain { + Codex, + Claude, +} + +/// Which brain the user wants: `auto` is Codex when it is ready, else Claude Code when an Anthropic API key is set. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum BrainChoice { + #[default] + Auto, + Codex, + Claude, +} + +/// The last readiness check: none yet, or when it was made, what it found, and the brain that answers. +#[derive(Clone, Copy, Debug, Default, PartialEq)] +pub struct Readiness { + pub checked: Option, + pub problem: Option, + /// The brain voice commands go to; None while not ready under `auto`. + pub brain: Option, +} + +/// What the user set: the brain, and their Anthropic API key (never sent back to a client). +#[derive(Clone, Default, Deserialize, Serialize)] +struct Settings { + #[serde(default)] + brain: BrainChoice, + #[serde(default, skip_serializing_if = "Option::is_none")] + anthropic_api_key: Option, +} + +/// Readiness under a brain setting, from the last Codex check and whether Claude Code is there with a key. +fn decide(choice: BrainChoice, codex: Readiness, claude_installed: bool, key_set: bool) -> Readiness { + let claude_problem = if !claude_installed { + Some(Problem::ClaudeMissing) + } else if !key_set { + Some(Problem::AnthropicKeyMissing) + } else { + None + }; + let codex_side = Readiness { brain: Some(Brain::Codex), ..codex }; + let claude_side = Readiness { checked: Some(tokio::time::Instant::now()), problem: claude_problem, brain: Some(Brain::Claude) }; + match choice { + BrainChoice::Codex => codex_side, + BrainChoice::Claude => claude_side, + BrainChoice::Auto if codex.ready() => codex_side, + BrainChoice::Auto if claude_problem.is_none() => claude_side, + // Codex hasn't been looked at yet: no answer either way. + BrainChoice::Auto if codex.checked.is_none() => Readiness::default(), + // A key says the user means Claude Code, so its problem is the one to fix; else Codex's. + BrainChoice::Auto => Readiness { + checked: codex.checked, + problem: if key_set { claude_problem } else { codex.problem }, + brain: None, + }, + } +} + +impl Readiness { + pub fn ready(&self) -> bool { + self.checked.is_some() && self.problem.is_none() + } +} + +/// What the resident session said in answer to one voice command. +#[derive(Debug, Serialize)] +pub struct Answer { + pub thread_id: String, + pub replies: Vec, +} + +/// One past voice command and the spoken replies to it, read back from the thread. +#[derive(Debug, PartialEq, Serialize)] +pub struct Exchange { + pub said: String, + pub replies: Vec, +} + +/// How the app-server is reached: the real `codex` child, or an in-memory fake in tests. +pub enum Launcher { + Codex, + #[cfg(test)] + Fake(Box Pipes + Send + Sync>), +} + +pub struct Pipes { + pub reader: Box, + pub writer: Box, + /// Killed when the session is dropped. + pub child: Option, +} + +struct Session { + client: Client, + incoming: mpsc::UnboundedReceiver, + thread_id: String, + /// Who its instructions make the buddy: they are set only when the thread starts or is resumed. + persona: Persona, + _child: Option, +} + +#[derive(Default, Deserialize, Serialize)] +struct Saved { + thread_id: Option, + /// The Codex version the thread was last resumed with, for diagnosing a protocol change. + codex_version: Option, + /// The Character whose persona the thread last answered in. + #[serde(default)] + persona: Option, +} + +pub struct Resident { + enabled: bool, + /// The working directory the resident thread runs in; the daemon owns it. + dir: PathBuf, + /// The same directory with symlinks resolved, as Codex may report it in hooks. + canonical_dir: PathBuf, + state_file: PathBuf, + /// The Codex home the app-server runs in; the daemon owns it. + codex_home: PathBuf, + /// The user's own Codex home, whose login and memories the resident session shares. + user_codex_home: PathBuf, + launcher: Launcher, + activities: Arc>, + /// Held for a whole turn: one voice command at a time, the rest wait here in order. + session: Mutex>, + /// Read synchronously by the hook filter. + thread_id: std::sync::Mutex>, + /// The Character whose persona the thread last answered in (kept in `resident.json`). + spoke_as: std::sync::Mutex>, + /// The persona of the last voice command, for starting the session when no command asks (history). + persona: std::sync::Mutex>, + /// Bumped when a turn ends or the thread changes, so a client can tell the history moved without reading it. + revision: std::sync::atomic::AtomicU64, + /// The last Codex check (its `brain` unused). + codex_readiness: std::sync::Mutex, + /// Readiness under the brain setting; watchers hear when `ready()` flips. + readiness: watch::Sender, + /// The session on the user's Claude Code. + claude: ClaudeBrain, + settings: std::sync::Mutex, + /// Where the settings are kept; None keeps them in memory only (tests). + settings_file: Option, + /// What the readiness check finds with the fake app-server. + #[cfg(test)] + fake_problem: std::sync::Mutex>, +} + +impl Resident { + /// `state_dir` is the daemon's state directory: the thread id is kept in `resident.json` there, the session + /// works in `resident/`, and its Codex home is `resident-codex/`. + pub fn new(enabled: bool, state_dir: &Path, launcher: Launcher, activities: Arc>) -> Self { + let dir = state_dir.join("resident"); + let state_file = state_dir.join("resident.json"); + let saved = load(&state_file); + let canonical_dir = std::fs::canonicalize(&dir).unwrap_or_else(|_| dir.clone()); + Self { + enabled, + dir, + canonical_dir, + state_file, + codex_home: state_dir.join(CODEX_HOME_DIR), + user_codex_home: user_codex_home(), + launcher, + claude: ClaudeBrain::new(state_dir, crate::resident_claude::Launcher::Claude, activities.clone()), + activities, + session: Mutex::new(None), + thread_id: std::sync::Mutex::new(saved.thread_id), + spoke_as: std::sync::Mutex::new(saved.persona), + persona: std::sync::Mutex::new(None), + revision: std::sync::atomic::AtomicU64::new(0), + codex_readiness: std::sync::Mutex::new(Readiness::default()), + readiness: watch::channel(Readiness::default()).0, + settings: std::sync::Mutex::new(Settings::default()), + settings_file: None, + #[cfg(test)] + fake_problem: std::sync::Mutex::new(None), + } + } + + /// Keeps the brain setting and the Anthropic API key in `path` (see [`SETTINGS_FILE`]), reading them now. + pub fn with_settings_file(mut self, path: PathBuf) -> Self { + let settings = std::fs::read_to_string(&path).ok().and_then(|text| serde_json::from_str(&text).ok()); + self.settings = std::sync::Mutex::new(settings.unwrap_or_default()); + self.settings_file = Some(path); + self + } + + #[cfg(test)] + pub(crate) fn with_claude(mut self, claude: ClaudeBrain) -> Self { + self.claude = claude; + self + } + + pub fn brain_setting(&self) -> BrainChoice { + self.settings.lock().expect("settings lock").brain + } + + pub fn anthropic_key_set(&self) -> bool { + self.anthropic_key().is_some() + } + + fn anthropic_key(&self) -> Option { + self.settings.lock().expect("settings lock").anthropic_api_key.clone() + } + + pub async fn set_brain(&self, choice: BrainChoice) -> std::io::Result<()> { + self.change_settings(|settings| settings.brain = choice)?; + info!(?choice, "the resident session's brain setting changed"); + self.check_readiness_now().await; + Ok(()) + } + + /// Sets or clears the user's Anthropic API key. A running Claude Code session is stopped, so the next command + /// starts one with the new key. + pub async fn set_anthropic_key(&self, key: Option) -> std::io::Result<()> { + let key = key.map(|key| key.trim().to_owned()).filter(|key| !key.is_empty()); + let set = key.is_some(); + self.change_settings(|settings| settings.anthropic_api_key = key)?; + info!(set, "the Anthropic API key for the resident session changed"); + self.claude.stop().await; + self.check_readiness_now().await; + Ok(()) + } + + fn change_settings(&self, change: impl FnOnce(&mut Settings)) -> std::io::Result<()> { + let mut settings = self.settings.lock().expect("settings lock"); + let mut changed = settings.clone(); + change(&mut changed); + if let Some(path) = &self.settings_file { + let text = serde_json::to_string_pretty(&changed).map_err(std::io::Error::other)?; + crate::resident_claude::write_private(path, &text)?; + } + *settings = changed; + Ok(()) + } + + /// The brain a voice command goes to now: the ready one, else the one the setting points at (Codex under + /// `auto`), whose start then says what is missing. + pub fn brain(&self) -> Brain { + self.readiness().brain.unwrap_or(match self.brain_setting() { + BrainChoice::Claude => Brain::Claude, + BrainChoice::Codex | BrainChoice::Auto => Brain::Codex, + }) + } + + pub fn enabled(&self) -> bool { + self.enabled + } + + /// The session voice commands go to now: the Codex thread or the Claude Code session. + pub fn thread_id(&self) -> Option { + match self.brain() { + Brain::Codex => self.codex_thread_id(), + Brain::Claude => self.claude.session_id(), + } + } + + fn codex_thread_id(&self) -> Option { + self.thread_id.lock().expect("thread id lock").clone() + } + + /// A change marker for the history, kept in memory only: it starts at 0 with the daemon and goes up each time a + /// voice command's turn ends (answered or not) and when the thread changes. + pub fn revision(&self) -> u64 { + self.revision.load(std::sync::atomic::Ordering::Relaxed) + } + + fn bump_revision(&self) { + self.revision.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + } + + /// The last readiness check, as it stands; see [`Resident::check_readiness`]. + pub fn readiness(&self) -> Readiness { + *self.readiness.borrow() + } + + /// Whether the session can take a voice command: on, and the last check found the brain the setting picks + /// ready (Codex installed and logged in, or Claude Code installed with a key). + pub fn ready(&self) -> bool { + self.enabled && self.readiness().ready() + } + + /// Changes each time [`Resident::ready`] flips. + pub fn watch_ready(&self) -> watch::Receiver { + self.readiness.subscribe() + } + + /// Checks again whether `codex` is there and the user logged in, unless the last check is younger than + /// [`READINESS_TTL`] or the setting is Claude Code; and whether `claude` is there and a key set, which costs + /// nothing. Starts no app-server, process or turn: it looks `codex` up as a start would and asks `codex login + /// status` in the resident Codex home, which reads the login the app-server's `account/read` would. + pub async fn check_readiness(&self) -> Readiness { + if !self.enabled { + return self.readiness(); + } + let last = *self.codex_readiness.lock().expect("readiness lock"); + if self.brain_setting() != BrainChoice::Claude + && !last.checked.is_some_and(|checked| checked.elapsed() < READINESS_TTL) + { + let problem = match &self.launcher { + Launcher::Codex => self.login_problem().await, + #[cfg(test)] + Launcher::Fake(_) => *self.fake_problem.lock().unwrap(), + }; + self.note(problem); + } else { + self.publish(); + } + self.readiness() + } + + /// The same check, but Codex is asked again whatever the age of its last answer: a setting changed. + async fn check_readiness_now(&self) { + self.codex_readiness.lock().expect("readiness lock").checked = None; + self.check_readiness().await; + } + + async fn login_problem(&self) -> Option { + let Some(program) = find_codex() else { return Some(Problem::CodexMissing) }; + if let Err(error) = prepare_codex_home(&self.codex_home, &self.user_codex_home) { + warn!(%error, "cannot prepare the resident Codex home to check the login"); + return Some(Problem::CodexLoggedOut); + } + let status = codex_command(&program, &self.codex_home) + .args(["-c", AUTH_STORE, "login", "status"]) + .stdin(std::process::Stdio::null()) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .await; + match status { + Ok(status) if status.success() => None, + Ok(_) => Some(Problem::CodexLoggedOut), + Err(error) => { + warn!(program = %program.display(), %error, "cannot run codex to check the login"); + Some(Problem::CodexMissing) + } + } + } + + /// Records what a Codex check or start found; see [`Resident::publish`]. + pub(crate) fn note(&self, problem: Option) { + *self.codex_readiness.lock().expect("readiness lock") = + Readiness { checked: Some(tokio::time::Instant::now()), problem, brain: None }; + self.publish(); + } + + /// Works out readiness under the brain setting; watchers are told only when it flips. A change of brain moves + /// the revision, since the history shown is the other brain's now. + fn publish(&self) { + let codex = *self.codex_readiness.lock().expect("readiness lock"); + let now = decide(self.brain_setting(), codex, self.claude.installed(), self.anthropic_key_set()); + let mut switched = false; + self.readiness.send_if_modified(|readiness| { + let flipped = readiness.ready() != now.ready(); + if (readiness.problem, readiness.brain) != (now.problem, now.brain) { + info!(problem = ?now.problem, brain = ?now.brain, "resident session readiness changed"); + } + switched = readiness.brain.is_some() && now.brain.is_some() && readiness.brain != now.brain; + *readiness = now; + flipped + }); + if switched { + self.bump_revision(); + } + } + + /// Whether a Codex hook event comes from the resident session: by its thread, or by its working directory. The + /// user's own hooks don't load in the resident Codex home, but system and managed ones may. + pub fn owns_hook(&self, session_id: &str, thread_id: Option<&str>, cwd: Option<&str>) -> bool { + if let Some(resident) = self.thread_id.lock().expect("thread id lock").as_deref() + && (session_id == resident || thread_id == Some(resident)) + { + return true; + } + self.owns_cwd(cwd) + } + + /// Whether a Claude Code hook event comes from the resident session: by its session, or by its working + /// directory. Its config dir loads none of the user's hooks, but managed ones may. + pub fn owns_claude_hook(&self, session_id: &str, cwd: Option<&str>) -> bool { + self.claude.owns_session(session_id) || self.owns_cwd(cwd) + } + + fn owns_cwd(&self, cwd: Option<&str>) -> bool { + cwd.map(Path::new) + .is_some_and(|cwd| cwd.starts_with(&self.dir) || cwd.starts_with(&self.canonical_dir)) + } + + /// Hands one voice command to the resident session and waits for its spoken replies. + pub async fn say(&self, text: &str, wearing: &Wearing) -> Result { + let mut replies = Vec::new(); + let thread_id = self.say_streaming(text, wearing, &mut |reply| replies.push(reply.to_owned())).await?; + Ok(Answer { thread_id, replies }) + } + + /// Hands one voice command to the resident session, answered as the Character the buddy is `wearing`; each + /// spoken reply goes to `on_reply` the moment the session calls `speak`, not when the turn completes, which in + /// testing was 4 to 6 seconds later. Returns the thread id once the turn has completed. + pub async fn say_streaming( + &self, + text: &str, + wearing: &Wearing, + on_reply: &mut (dyn FnMut(&str) + Send), + ) -> Result { + if !self.enabled { + return Err(ResidentError::Disabled); + } + let persona = Persona::of(wearing); + *self.persona.lock().expect("persona lock") = Some(persona.clone()); + let result = match self.brain() { + Brain::Codex => self.say_codex(text, &persona, on_reply).await, + Brain::Claude => self.say_claude(text, &persona, on_reply).await, + }; + // Even a failed or timed-out turn may have left the command in the thread. + self.bump_revision(); + result + } + + async fn say_codex( + &self, + text: &str, + persona: &Persona, + on_reply: &mut (dyn FnMut(&str) + Send), + ) -> Result { + let mut slot = self.session.lock().await; + // A running thread keeps the instructions it was resumed with, so a new Character resumes it again. + if slot.as_ref().is_some_and(|session| session.persona != *persona) { + info!(character = persona.id, "the Character changed; resuming the resident thread with its persona"); + *slot = None; + } + let session = self.running(&mut slot, persona).await?; + let spoke_as = self.spoke_as.lock().expect("spoke as lock").clone(); + let changed = spoke_as.as_deref() != Some(persona.id); + let note = (changed && spoke_as.is_some()).then(|| change_note(persona)); + let context = command_context(persona, &chrono::Local::now(), local_time_zone().as_deref()); + let result = self.run_turn(session, &context, note.as_deref(), text, on_reply).await; + if session.client.is_closed() { + *slot = None; + } + if result.is_ok() && changed { + self.remember_persona(persona.id); + } + result + } + + /// Keeps the Character the thread now answers as, so a change is noticed after a restart too. + fn remember_persona(&self, id: &str) { + *self.spoke_as.lock().expect("spoke as lock") = Some(id.to_owned()); + let mut saved = load(&self.state_file); + saved.persona = Some(id.to_owned()); + if let Err(error) = save(&self.state_file, &saved) { + warn!(%error, "failed to save the resident session's Character"); + } + } + + async fn say_claude( + &self, + text: &str, + persona: &Persona, + on_reply: &mut (dyn FnMut(&str) + Send), + ) -> Result { + let key = self.anthropic_key().ok_or(ResidentError::NoAnthropicKey)?; + let instructions = instructions(persona, None, Brain::Claude); + let result = self.claude.say_streaming(&key, &instructions, persona, text, on_reply).await; + if result == Err(ResidentError::NoClaude) { + self.publish(); + } + result + } + + /// The persona of the last voice command, else the one the thread last answered in. + fn last_persona(&self) -> Persona { + if let Some(persona) = self.persona.lock().expect("persona lock").clone() { + return persona; + } + let character = self.spoke_as.lock().expect("spoke as lock").clone(); + Persona::of(&Wearing { character, address: None }) + } + + /// Starts the app-server and resumes the thread ahead of the first voice command, which otherwise waits for + /// both, as the Character the buddy is `wearing`. Does nothing when switched off or already running; a failure + /// is logged and the next command tries again. + pub async fn warm(&self, wearing: &Wearing) { + if !self.enabled { + return; + } + self.check_readiness().await; + let persona = Persona::of(wearing); + *self.persona.lock().expect("persona lock") = Some(persona.clone()); + if self.brain() == Brain::Claude { + if let Some(key) = self.anthropic_key() { + self.claude.warm(&key, &instructions(&persona, None, Brain::Claude)).await; + } + return; + } + let mut slot = self.session.lock().await; + match self.running(&mut slot, &persona).await { + Ok(session) => info!(thread = %session.thread_id, "the resident session is ready"), + Err(error) => warn!(%error, "the resident session could not be started ahead of time"), + } + } + + /// The last `limit` voice commands and their spoken replies, oldest first, read from the thread through the + /// app-server (the daemon keeps no record of its own). Empty while there is no thread yet. + pub async fn history(&self, limit: usize) -> Result, ResidentError> { + if !self.enabled { + return Err(ResidentError::Disabled); + } + if self.brain() == Brain::Claude { + return Ok(self.claude.history(limit)); + } + if self.codex_thread_id().is_none() { + return Ok(Vec::new()); + } + let mut slot = self.session.lock().await; + let session = self.running(&mut slot, &self.last_persona()).await?; + let params = json!({"threadId": session.thread_id, "limit": limit, "sortDirection": "desc", "itemsView": "full"}); + let listed = request(&session.client, "thread/turns/list", params).await; + if session.client.is_closed() { + *slot = None; + } + let turns = listed?.get("data").and_then(Value::as_array).cloned().unwrap_or_default(); + Ok(turns.iter().rev().filter_map(exchange_of).collect()) + } + + /// The running session, started (again) if there is none or its app-server has exited. + async fn running<'a>(&self, slot: &'a mut Option, persona: &Persona) -> Result<&'a mut Session, ResidentError> { + if slot.as_ref().is_some_and(|session| session.client.is_closed()) { + warn!("the resident app-server exited; starting it again"); + *slot = None; + } + if slot.is_none() { + let started = self.start(persona).await; + // A start is a readiness check too, and the most recent one. + match &started { + Ok(_) => self.note(None), + Err(ResidentError::NoCodex) => self.note(Some(Problem::CodexMissing)), + Err(ResidentError::NotLoggedIn) => self.note(Some(Problem::CodexLoggedOut)), + Err(_) => {} + } + *slot = Some(started?); + } + Ok(slot.as_mut().expect("session was just started")) + } + + async fn start(&self, persona: &Persona) -> Result { + prepare_codex_home(&self.codex_home, &self.user_codex_home) + .map_err(|error| ResidentError::Rpc(format!("cannot prepare {}: {error}", self.codex_home.display())))?; + let (pipes, version) = match &self.launcher { + Launcher::Codex => spawn_codex(&self.codex_home).await?, + #[cfg(test)] + Launcher::Fake(connect) => (connect(), None), + }; + let (client, incoming) = Client::spawn(pipes.reader, pipes.writer); + request( + &client, + "initialize", + json!({ + "clientInfo": {"name": "vibebuddy", "title": "VibeBuddy", "version": env!("CARGO_PKG_VERSION")}, + "capabilities": {"experimentalApi": true}, + }), + ) + .await?; + client.notify("initialized").await?; + // Only the user's existing login is used; a login flow is never started from here. + let account = request(&client, "account/read", json!({})).await?; + if account.get("account").is_none_or(Value::is_null) + && account.get("requiresOpenaiAuth").and_then(Value::as_bool) != Some(false) + { + if !self.user_codex_home.join("auth.json").exists() { + warn!( + codex_home = %self.user_codex_home.display(), + "no auth.json in the user's Codex home: a login kept in the keychain can't be shared with the resident session" + ); + } + return Err(ResidentError::NotLoggedIn); + } + std::fs::create_dir_all(&self.dir) + .map_err(|error| ResidentError::Rpc(format!("cannot create {}: {error}", self.dir.display())))?; + let thread_id = match self.codex_thread_id() { + // There is one resident thread for good. Only a thread that no longer exists is replaced (a thread + // saved before the resident session had its own Codex home is one); one that can't be resumed for + // another reason (another Codex process is its writer) fails this command and is tried again with the + // next. + Some(saved) => match self.resume(&client, &saved, persona).await { + Err(ResidentError::Rpc(message)) if !client.is_closed() && thread_is_gone(&message) => { + let thread = self.start_thread(&client, persona).await?; + warn!(old_thread = %saved, new_thread = %thread, %message, "the resident thread no longer exists; started a new one"); + thread + } + Err(ResidentError::Rpc(message)) if message.contains("active writer") => { + warn!(thread = %saved, "another Codex process is the resident thread's writer"); + return Err(ResidentError::Rpc(message)); + } + other => other?, + }, + None => self.start_thread(&client, persona).await?, + }; + let previous = self.thread_id.lock().expect("thread id lock").replace(thread_id.clone()); + if previous.as_deref() != Some(thread_id.as_str()) { + self.bump_revision(); + } + let spoke_as = self.spoke_as.lock().expect("spoke as lock").clone(); + let saved = Saved { thread_id: Some(thread_id.clone()), codex_version: version, persona: spoke_as }; + if let Err(error) = save(&self.state_file, &saved) { + warn!(%error, "failed to save the resident thread id"); + } + info!(thread = %thread_id, "resident session ready"); + Ok(Session { client, incoming, thread_id, persona: persona.clone(), _child: pipes.child }) + } + + async fn start_thread(&self, client: &Client, persona: &Persona) -> Result { + let mut params = self.thread_params(persona); + params["dynamicTools"] = tools(); + let started = request(client, "thread/start", params).await?; + thread_id_of(&started) + } + + async fn resume(&self, client: &Client, thread_id: &str, persona: &Persona) -> Result { + // Dynamic tools live with the thread and survive a resume; the instructions are sent again so a change + // (a new Character) takes effect. + let mut params = self.thread_params(persona); + params["threadId"] = json!(thread_id); + let resumed = request(client, "thread/resume", params).await?; + thread_id_of(&resumed) + } + + fn thread_params(&self, persona: &Persona) -> Value { + json!({ + "cwd": self.dir, + "sandbox": "read-only", + "approvalPolicy": "on-request", + "developerInstructions": instructions(persona, memory_summary(&self.user_codex_home).as_deref(), Brain::Codex), + }) + } + + async fn run_turn( + &self, + session: &mut Session, + context: &str, + note: Option<&str>, + text: &str, + on_reply: &mut (dyn FnMut(&str) + Send), + ) -> Result { + // Whatever an earlier, timed-out turn left behind still gets its answer, but no longer counts. + while let Ok(message) = session.incoming.try_recv() { + self.handle(&session.client, message, &mut |_| {}).await; + } + let input: Vec = + [context].into_iter().chain(note).chain([text]).map(|text| json!({"type": "text", "text": text})).collect(); + let started = + request(&session.client, "turn/start", json!({"threadId": session.thread_id, "input": input})).await?; + let turn_id = started.pointer("/turn/id").and_then(Value::as_str).unwrap_or_default().to_owned(); + let deadline = tokio::time::Instant::now() + TURN_TIMEOUT; + loop { + let message = match tokio::time::timeout_at(deadline, session.incoming.recv()).await { + Err(_) => { + let _ = session + .client + .request("turn/interrupt", json!({"threadId": session.thread_id, "turnId": turn_id})) + .await; + return Err(ResidentError::Timeout); + } + Ok(None) => return Err(RpcError::Closed.into()), + Ok(Some(message)) => message, + }; + if let Incoming::Notification { method, params } = &message + && method == "turn/completed" + && params.pointer("/turn/id").and_then(Value::as_str) == Some(turn_id.as_str()) + { + return match params.pointer("/turn/status").and_then(Value::as_str) { + Some("completed") => Ok(session.thread_id.clone()), + status => Err(ResidentError::TurnEnded( + params + .pointer("/turn/error/message") + .and_then(Value::as_str) + .or(status) + .unwrap_or("unknown") + .to_owned(), + )), + }; + } + self.handle(&session.client, message, on_reply).await; + } + } + + /// Answers a server-to-client request; notifications other than the turn's end need nothing. + async fn handle(&self, client: &Client, message: Incoming, on_reply: &mut (dyn FnMut(&str) + Send)) { + let Incoming::Request { id, method, params } = message else { + return; + }; + let answered = match decline(&method) { + Some(result) => { + info!(%method, "declined a request from the resident session"); + client.respond(id, result).await + } + None if method == "item/tool/call" => { + let output = self.call_tool(¶ms, on_reply).await; + client.respond(id, output).await + } + None if method == "currentTime/read" => { + client.respond(id, json!({"currentTimeAt": chrono::Utc::now().timestamp()})).await + } + None => { + info!(%method, "refused an unsupported request from the resident session"); + client.respond_error(id, -32601, "not supported by VibeBuddy").await + } + }; + if let Err(error) = answered { + warn!(%error, "failed to answer the resident session"); + } + } + + async fn call_tool(&self, params: &Value, on_reply: &mut (dyn FnMut(&str) + Send)) -> Value { + let tool = params.get("tool").and_then(Value::as_str).unwrap_or_default(); + let memories = self.user_codex_home.join("memories"); + let (success, text) = run_tool(tool, ¶ms["arguments"], &self.activities, Some(&memories), on_reply).await; + json!({"success": success, "contentItems": [{"type": "inputText", "text": text}]}) + } +} + +/// Runs one of our tools, for either brain: whether it succeeded, and what the session reads back. `memories` is +/// the user's Codex memories folder; without it there is no `search_memories`. +pub(crate) async fn run_tool( + tool: &str, + arguments: &Value, + activities: &Mutex, + memories: Option<&Path>, + on_reply: &mut (dyn FnMut(&str) + Send), +) -> (bool, String) { + match (tool, memories) { + ("speak", _) => { + let text = arguments.get("text").and_then(Value::as_str).unwrap_or_default().trim(); + if text.is_empty() { + (false, "text is empty".to_owned()) + } else { + info!(chars = text.chars().count(), "the resident session spoke"); + on_reply(text); + (true, "spoken".to_owned()) + } + } + ("list_activities", _) => { + let report = activities.lock().await.report(); + (true, json!(report).to_string()) + } + ("search_memories", Some(memories)) => { + let query = arguments.get("query").and_then(Value::as_str).unwrap_or_default(); + (true, search_memories(memories, query)) + } + (other, _) => (false, format!("unknown tool {other}")), + } +} + +/// The answer that declines a server-to-client request asking for permission or input; None if it isn't one. +fn decline(method: &str) -> Option { + Some(match method { + "item/commandExecution/requestApproval" | "item/fileChange/requestApproval" => json!({"decision": "decline"}), + "execCommandApproval" | "applyPatchApproval" => json!({"decision": "denied"}), + "item/permissions/requestApproval" => json!({"permissions": {}}), + "mcpServer/elicitation/request" => json!({"action": "decline"}), + "item/tool/requestUserInput" => json!({"answers": {}}), + _ => return None, + }) +} + +/// The tools the resident session gets. Nothing here may run a command or write a file. +pub(crate) fn tools() -> Value { + json!([ + { + "type": "function", + "name": "speak", + "description": "Say something to the user out loud through the box. This is the only way the user hears you.", + "inputSchema": { + "type": "object", + "properties": {"text": {"type": "string", "description": "One or two short spoken sentences."}}, + "required": ["text"], + "additionalProperties": false, + }, + }, + { + "type": "function", + "name": "list_activities", + "description": "What the user's coding Agents are doing and have done. `activities`: what runs right now, \ + newest first: agent, task card title, project, state (working or needs_input), where it runs, and \ + seconds in that state. `recently_ended`: the last tasks that ended since VibeBuddy started, newest \ + first: agent, title, project, how it ended (done, failed, stopped: interrupted or closed with no \ + outcome, or expired: nothing heard for too long) and `ended_s`, seconds ago. `today`: tasks done, \ + times an Agent asked for input, and seconds with at least one Agent busy, today. Use it for what is \ + happening now, what just finished, and how today went.", + "inputSchema": {"type": "object", "properties": {}, "additionalProperties": false}, + }, + { + "type": "function", + "name": "search_memories", + "description": "Search the user's Codex memories (notes Codex keeps from the user's past sessions) for lines \ + containing any of the words in `query`, case-insensitive. Returns the matching lines, each with the \ + line before and after, by file.", + "inputSchema": { + "type": "object", + "properties": {"query": {"type": "string", "description": "A few keywords."}}, + "required": ["query"], + "additionalProperties": false, + }, + }, + ]) +} + +/// The developer instructions: the buddy as the Character it wears (CONTEXT.md, Persona), in that Character's one +/// language and with the user's form of address. `memories` is the user's Codex memory summary, if any. `brain` +/// says where today's date is: in Codex's environment context, or in the context block the daemon puts before each +/// voice command for Claude Code. +pub(crate) fn instructions(persona: &Persona, memories: Option<&str>, brain: Brain) -> String { + let Persona { name, text, language, address, .. } = persona; + let address = address.map_or_else(String::new, |words| { + format!( + " The user likes to be called \"{words}\": call them that now and then, naturally, as your lines do, not \ + in every reply." + ) + }); + let memories = memories.map_or_else(String::new, |summary| { + format!( + "\n\nWhat Codex remembers about the user from their past sessions, in summary. For more, call \ + `search_memories`. Use it when it helps; never read it out.\n\n{summary}\n" + ) + }); + let date = match brain { + Brain::Codex => "today's date and the user's time zone are in the environment context", + Brain::Claude => "the user's local date, time and time zone come with each voice command, in a context block", + }; + format!( + "You are the buddy of VibeBuddy: a small desk box that watches the user's coding Agents (Codex, Claude Code, \ + OpenCode, GitHub Copilot CLI, Pi, GitHub Actions) and speaks up for them. Right now you are {name}, the \ + Character the user picked. This is {name}'s persona, written for the short lines {name} says when an Agent \ + needs the user or is done; speak every reply as {name}, in this persona:\n\n{text}\n\n\ + You speak {language} only. The user may talk to you in any language and you understand it, but you always \ + reply in {language}: a question asked in another language still gets its answer in {language}, never in the \ + language it was asked in, whatever earlier replies or the memories below say about language.{address}\n\ + Every user message is one voice command, transcribed from speech. Answer only by calling the `speak` tool, \ + with one or two short sentences meant to be heard, not read: no Markdown, lists, code, paths or URLs. \ + Do not write any other answer: once you have called `speak`, end your turn without writing anything else.\n\ + To know what the Agents are doing, have just finished or did today, call `list_activities`; never guess. It \ + has the live tasks, the ones that ended lately with how and how long ago (`recently_ended`), and today's \ + counts (`today`). Titles there are short upper-case task-card names; say them naturally.\n\ + Never state a fact you have not seen in a tool result, a web search result or the memories below: not the \ + weather, news, prices, scores, schedules or anything else about now, and nothing about the user's world. \ + For things about now, search the web, and answer only from what the search found; {date}, so work out \ + today and tomorrow from them. If you can't \ + find out, say so plainly in one sentence; never make up an answer that sounds likely.\n\ + You do no work yourself: run no commands, read or edit no files. If the user asks for work to be done, say \ + briefly that you can't start it yet.{memories}" + ) +} + +/// A context line put before the next voice command once the Character has changed, so the session takes up the +/// new voice rather than carrying on in the old one's. The history leaves it out, as it does every context block. +pub(crate) fn change_note(persona: &Persona) -> String { + let Persona { name, language, .. } = persona; + format!( + "{CONTEXT_PREFIX}The user just switched you to another Character: from now on you are {name}. Speak only as \ + {name}, in {language}, whatever your earlier replies in this conversation sounded like." + ) +} + +fn thread_id_of(response: &Value) -> Result { + response + .pointer("/thread/id") + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| ResidentError::Rpc("app-server returned no thread id".to_owned())) +} + +async fn request(client: &Client, method: &str, params: Value) -> Result { + match tokio::time::timeout(REQUEST_TIMEOUT, client.request(method, params)).await { + Ok(result) => Ok(result?), + Err(_) => Err(ResidentError::Rpc(format!("app-server did not answer {method}"))), + } +} + +/// Starts `codex app-server` in the resident session's Codex home and checks its version against the one this was +/// tested with. +async fn spawn_codex(codex_home: &Path) -> Result<(Pipes, Option), ResidentError> { + let program = find_codex().ok_or(ResidentError::NoCodex)?; + let version = codex_command(&program, codex_home) + .arg("--version") + .output() + .await + .ok() + .and_then(|output| String::from_utf8(output.stdout).ok()) + .and_then(|text| text.split_whitespace().last().map(str::to_owned)); + match version.as_deref() { + Some(TESTED_CODEX_VERSION) => {} + other => warn!( + version = other.unwrap_or("unknown"), + tested = TESTED_CODEX_VERSION, + "the resident session was tested with another Codex version; app-server is experimental and may differ" + ), + } + let base = tool_overrides(); + let servers = enabled_mcp_servers(&program, codex_home, &base).await; + let mut child = codex_command(&program, codex_home) + .args(app_server_args(&base, &servers)) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::null()) + .kill_on_drop(true) + .spawn() + .map_err(|error| ResidentError::Rpc(format!("cannot start {}: {error}", program.display())))?; + info!(program = %program.display(), mcp_servers_off = servers.len(), "started codex app-server for the resident session"); + let reader = child.stdout.take().expect("stdout is piped"); + let writer = child.stdin.take().expect("stdin is piped"); + Ok((Pipes { reader: Box::new(reader), writer: Box::new(writer), child: Some(child) }, version)) +} + +/// `codex` run in the resident session's own Codex home. The user's `config.toml`, `AGENTS.md`, hooks, plugins and +/// MCP servers live in theirs, so none of them load; Codex's system and managed config still do. +fn codex_command(program: &Path, codex_home: &Path) -> tokio::process::Command { + let mut command = tokio::process::Command::new(program); + command.env("CODEX_HOME", codex_home); + command +} + +/// Features that give a Codex thread tools of its own: the resident session gets none of them, only ours. Most are +/// on by default even in an empty Codex home. Plugins and apps (connectors) bring tools and MCP servers; the rest +/// are built-in tools. `memories` (off by default) would also start Codex's memory pipeline, which rewrites the +/// memories folder of the Codex home it runs in; the daemon gives the session the user's memories itself. +/// `shell_snapshot` runs the user's login shell at every start and saves its whole environment, API keys included, +/// in the Codex home, for a shell tool the session doesn't have. Set as `-c features.=false` rather than +/// `--disable`, which refuses a name another Codex version doesn't know. +/// +/// `code_mode_host` stays on: the model calls every tool, ours included, through its `exec` JavaScript, and with +/// the host off no tool call gets through. That JavaScript has no file system, network or process (checked on +/// 2026-10-14: no `fetch`, `require` or `process`, and `import` is refused); it reaches only the thread's tools. +const FEATURES_OFF: [&str; 16] = [ + "plugins", + "apps", + "shell_tool", + "unified_exec", + "multi_agent", + "goals", + "memories", + "image_generation", + "view_image", + "browser_use", + "computer_use", + "in_app_browser", + "sleep_tool", + "tool_suggest", + "skill_search", + "shell_snapshot", +]; + +/// The config overrides for the resident session's Codex, before MCP servers. Besides the tools: +/// - `cli_auth_credentials_store="file"`: the login is the user's `auth.json`, linked in (see [`link_auth`]). The +/// default already, but a keyring store would never find it (keyring entries are keyed by the Codex home's +/// path) and, on saving, would delete the link. +/// - `memories.generate_memories=false`: the resident thread is never summarised into memories, should the feature +/// come on. +/// - `skills.include_instructions=false` and `skills.bundled.enabled=false`: no skills. Codex lists the user's +/// own skills (`~/.agents/skills`) whatever the Codex home, and installs its bundled ones (`imagegen` and the +/// like) into it; the session can't run them, and the list costs every turn context. +/// - `web_search`: Codex's built-in web search, so a question about now is looked up rather than made up. The +/// queries go to OpenAI, the model provider. +/// - [`MODEL`] at [`REASONING_EFFORT`]: a voice command waits on the answer. Measured on 2026-10-13, a turn's first +/// `speak` came 2.7 s in on gpt-6-luna at low, against 4.3 s on gpt-6.1-sol at high. +fn tool_overrides() -> Vec { + let mut args: Vec = + FEATURES_OFF.iter().flat_map(|feature| ["-c".to_owned(), format!("features.{feature}=false")]).collect(); + for setting in [ + format!("web_search=\"{WEB_SEARCH}\""), + AUTH_STORE.to_owned(), + "memories.generate_memories=false".to_owned(), + "skills.include_instructions=false".to_owned(), + "skills.bundled.enabled=false".to_owned(), + format!("model=\"{MODEL}\""), + format!("model_reasoning_effort=\"{REASONING_EFFORT}\""), + ] { + args.extend(["-c".to_owned(), setting]); + } + args +} + +/// `codex app-server` with the overrides and every MCP server still configured switched off. The resident Codex home +/// configures none, but system and managed config may. An override can't empty `mcp_servers` (it merges), so each +/// server is turned off by name. +fn app_server_args(base: &[String], mcp_servers: &[String]) -> Vec { + let mut args = base.to_vec(); + for name in mcp_servers { + args.extend(["-c".to_owned(), format!("mcp_servers.{name}.enabled=false")]); + } + args.push("app-server".to_owned()); + args +} + +/// The MCP servers that would be on, as Codex itself resolves them (`codex mcp list --json`, under the same +/// overrides, so servers that plugins bring are already gone). A name that can't be written as a bare TOML key +/// can't be switched off by an override; it is logged. +async fn enabled_mcp_servers(program: &Path, codex_home: &Path, base: &[String]) -> Vec { + let output = codex_command(program, codex_home) + .args(base) + .args(["mcp", "list", "--json"]) + .stdin(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output() + .await; + let listed = output.ok().and_then(|output| serde_json::from_slice::(&output.stdout).ok()); + let Some(servers) = listed.as_ref().and_then(Value::as_array) else { + warn!("could not list the MCP servers; the resident session may see their tools"); + return Vec::new(); + }; + servers + .iter() + .filter(|server| server.get("enabled").and_then(Value::as_bool) != Some(false)) + .filter_map(|server| server.get("name").and_then(Value::as_str)) + .filter(|name| { + let bare = !name.is_empty() && name.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'); + if !bare { + warn!(server = %name, "an MCP server name that can't be overridden; the resident session may see its tools"); + } + bare + }) + .map(str::to_owned) + .collect() +} + +/// `VIBEBUDDY_CODEX`, else `codex` on PATH, else where installers put it: the daemon may run with launchd's bare PATH. +fn find_codex() -> Option { + if let Some(path) = std::env::var_os(CODEX_VARIABLE) { + return Some(PathBuf::from(path)); + } + let home = std::env::var_os("HOME").map(PathBuf::from); + std::env::var_os("PATH") + .map(|path| std::env::split_paths(&path).map(|dir| dir.join("codex")).collect::>()) + .unwrap_or_default() + .into_iter() + .chain(home.map(|home| home.join(".local/bin/codex"))) + .chain(["/opt/homebrew/bin/codex", "/usr/local/bin/codex"].map(PathBuf::from)) + .find(|path| path.is_file()) +} + +/// The user's own Codex home: `CODEX_HOME`, else `~/.codex`. +fn user_codex_home() -> PathBuf { + if let Some(home) = std::env::var_os("CODEX_HOME").filter(|home| !home.is_empty()) { + return PathBuf::from(home); + } + std::env::var_os("HOME").map(PathBuf::from).unwrap_or_default().join(".codex") +} + +/// Makes the resident session's Codex home (mode 0700) and links the user's login into it. +fn prepare_codex_home(codex_home: &Path, user_codex_home: &Path) -> std::io::Result<()> { + use std::os::unix::fs::PermissionsExt; + std::fs::create_dir_all(codex_home)?; + std::fs::set_permissions(codex_home, std::fs::Permissions::from_mode(0o700))?; + link_auth(codex_home, user_codex_home) +} + +/// Makes `auth.json` in the resident Codex home a symlink to the user's, so both share one login and either one's +/// token refresh is the other's. Codex 0.162.0 reads it with a plain open, refreshes by reloading it first (taking a +/// token the other side already refreshed), and saves by truncating and rewriting it in place, all through the link. +/// Anything else found there (a copy, a link elsewhere) is replaced, on every start. Without a login in the user's +/// home the link dangles and the session reports "not logged in". +fn link_auth(codex_home: &Path, user_codex_home: &Path) -> std::io::Result<()> { + let link = codex_home.join("auth.json"); + let target = user_codex_home.join("auth.json"); + if std::fs::read_link(&link).is_ok_and(|current| current == target) { + return Ok(()); + } + match std::fs::remove_file(&link) { + Ok(()) => info!("replaced the resident Codex home's auth.json with a link to the user's"), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => {} + Err(error) => return Err(error), + } + std::os::unix::fs::symlink(&target, &link) +} + +/// The user's Codex memory summary (`memories/memory_summary.md`), cut at Codex's own cap; None if there is none. +/// Read again at every start, so it follows the user's Codex as it consolidates. +fn memory_summary(user_codex_home: &Path) -> Option { + let text = std::fs::read_to_string(user_codex_home.join("memories").join("memory_summary.md")).ok()?; + let text = text.trim(); + let end = text.floor_char_boundary(MEMORY_SUMMARY_MAX_BYTES); + (!text.is_empty()).then(|| text[..end].to_owned()) +} + +/// Lines of the user's Codex memories (`MEMORY.md`, `raw_memories.md` and `rollout_summaries/`, as the user's Codex +/// would search them) containing any word of `query`, case-insensitive, each with the line before and after. Only +/// reads: the daemon never writes the user's memories. +fn search_memories(root: &Path, query: &str) -> String { + let words: Vec = query.split_whitespace().map(str::to_lowercase).collect(); + if words.is_empty() { + return "query is empty".to_owned(); + } + let mut files = vec![root.join("MEMORY.md"), root.join("raw_memories.md")]; + if let Ok(entries) = std::fs::read_dir(root.join("rollout_summaries")) { + let mut summaries: Vec = entries.filter_map(Result::ok).map(|entry| entry.path()).collect(); + summaries.sort(); + files.extend(summaries); + } + let mut out = String::new(); + 'files: for file in files { + let Ok(text) = std::fs::read_to_string(&file) else { continue }; + let lines: Vec<&str> = text.lines().collect(); + let name = file.strip_prefix(root).unwrap_or(&file).display().to_string(); + let mut shown_until = 0; + for (index, line) in lines.iter().enumerate() { + let lower = line.to_lowercase(); + if index < shown_until || !words.iter().any(|word| lower.contains(word.as_str())) { + continue; + } + let (from, to) = (index.saturating_sub(1).max(shown_until), (index + 2).min(lines.len())); + let block = format!("{name}:{}\n{}\n\n", from + 1, lines[from..to].join("\n")); + if out.len() + block.len() > MEMORY_SEARCH_MAX_BYTES { + out.push_str("(more matches left out)"); + break 'files; + } + out.push_str(&block); + shown_until = to; + } + } + if out.is_empty() { "no matches".to_owned() } else { out } +} + +/// A turn from `thread/turns/list` as an exchange: the user's text and what was said through `speak`. +fn exchange_of(turn: &Value) -> Option { + let items = turn.get("items")?.as_array()?; + let said = items + .iter() + .filter(|item| item["type"] == "userMessage") + .flat_map(|item| item["content"].as_array().cloned().unwrap_or_default()) + .filter_map(|input| input["text"].as_str().map(str::to_owned)) + .filter(|text| !text.starts_with(CONTEXT_PREFIX)) + .collect::>() + .join(" "); + let replies = items + .iter() + .filter(|item| item["type"] == "dynamicToolCall" && item["tool"] == "speak") + .filter_map(|item| item.pointer("/arguments/text").and_then(Value::as_str).map(str::to_owned)) + .collect(); + (!said.is_empty()).then_some(Exchange { said, replies }) +} + +fn load(path: &Path) -> Saved { + std::fs::read_to_string(path) + .ok() + .and_then(|text| serde_json::from_str(&text).ok()) + .unwrap_or_default() +} + +fn save(path: &Path, saved: &Saved) -> std::io::Result<()> { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + std::fs::write(path, serde_json::to_string_pretty(saved).map_err(std::io::Error::other)?) +} + +/// The app-server's answer to resuming a thread whose rollout is gone (deleted, or a different CODEX_HOME). +fn thread_is_gone(message: &str) -> bool { + message.contains("no rollout found") +} + +#[cfg(test)] +pub(crate) mod tests { + use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; + + use super::*; + use crate::activity::{ActivityId, ActivityStatus}; + + /// How the fake app-server behaves, and what it saw. + #[derive(Default)] + pub(crate) struct Script { + pub(crate) logged_in: bool, + /// thread/resume fails, as if the rollout were gone. + lose_thread: bool, + busy_thread: bool, + /// Ends its output instead of completing this turn, as if the process died; only once. + pub(crate) die_on_turn: Option, + /// Server-to-client requests made during each turn, before it completes. + pub(crate) turn_requests: Vec<(String, Value)>, + /// Requests and notifications from the daemon, in order. + pub(crate) received: Vec, + /// The daemon's answers to the server's own requests. + answers: Vec, + /// How many times the server was started. + pub(crate) starts: usize, + /// The turn completes only once this is notified, so a test can look at what came before it. + pub(crate) hold: Option>, + } + + pub(crate) type Shared = Arc>; + + pub(crate) fn fake_server(script: Shared) -> Pipes { + script.lock().unwrap().starts += 1; + let (daemon_side, server_side) = tokio::io::duplex(1 << 16); + let (daemon_read, daemon_write) = tokio::io::split(daemon_side); + tokio::spawn(async move { + let (server_read, mut out) = tokio::io::split(server_side); + let mut lines = BufReader::new(server_read).lines(); + let mut turns = 0; + while let Ok(Some(line)) = lines.next_line().await { + let message: Value = serde_json::from_str(&line).unwrap(); + let Some(method) = message.get("method").and_then(Value::as_str).map(str::to_owned) else { + script.lock().unwrap().answers.push(message); + continue; + }; + script.lock().unwrap().received.push(message.clone()); + let id = message["id"].clone(); + let lose_thread = script.lock().unwrap().lose_thread; + let busy_thread = script.lock().unwrap().busy_thread; + let result = match method.as_str() { + "initialized" => continue, + "initialize" => json!({"userAgent": "fake"}), + "account/read" => { + let logged_in = script.lock().unwrap().logged_in; + let account = if logged_in { json!({"type": "chatgpt"}) } else { Value::Null }; + json!({"account": account, "requiresOpenaiAuth": true}) + } + "thread/start" => json!({"thread": {"id": "new-thread"}}), + "thread/resume" if busy_thread => { + let message = "thread old-thread already has an active writer"; + let error = json!({"id": id, "error": {"code": -32600, "message": message}}); + out.write_all(format!("{error}\n").as_bytes()).await.unwrap(); + continue; + } + "thread/resume" if lose_thread => { + let error = json!({"id": id, "error": {"code": -32600, "message": "no rollout found"}}); + out.write_all(format!("{error}\n").as_bytes()).await.unwrap(); + continue; + } + "thread/resume" => json!({"thread": {"id": message["params"]["threadId"]}}), + "thread/turns/list" => json!({"data": [ + {"id": "turn-2", "items": [ + {"type": "userMessage", "id": "a", "content": [{"type": "text", "text": "and now?"}]}, + {"type": "dynamicToolCall", "id": "b", "tool": "list_activities", "arguments": {}}, + {"type": "dynamicToolCall", "id": "c", "tool": "speak", "arguments": {"text": "Still one."}}, + ]}, + {"id": "turn-1", "items": [ + {"type": "userMessage", "id": "d", "content": [{"type": "text", "text": "what's up?"}]}, + {"type": "dynamicToolCall", "id": "e", "tool": "speak", "arguments": {"text": "One task."}}, + {"type": "dynamicToolCall", "id": "f", "tool": "speak", "arguments": {"text": "It's fine."}}, + ]}, + ], "nextCursor": null}), + "turn/start" => { + turns += 1; + let turn_id = format!("turn-{turns}"); + let started = json!({"id": id, "result": {"turn": {"id": turn_id, "status": "inProgress"}}}); + out.write_all(format!("{started}\n").as_bytes()).await.unwrap(); + let requests = script.lock().unwrap().turn_requests.clone(); + for (number, (request, params)) in requests.iter().enumerate() { + let request = json!({"id": format!("srv-{turns}-{number}"), "method": request, "params": params}); + out.write_all(format!("{request}\n").as_bytes()).await.unwrap(); + } + for _ in &requests { + let Ok(Some(line)) = lines.next_line().await else { return }; + script.lock().unwrap().answers.push(serde_json::from_str(&line).unwrap()); + } + if script.lock().unwrap().die_on_turn.take_if(|turn| *turn == turns).is_some() { + return; + } + let hold = script.lock().unwrap().hold.clone(); + if let Some(hold) = hold { + hold.notified().await; + } + let completed = json!({"method": "turn/completed", "params": { + "threadId": message["params"]["threadId"], + "turn": {"id": turn_id, "status": "completed", "items": []}, + }}); + out.write_all(format!("{completed}\n").as_bytes()).await.unwrap(); + continue; + } + _ => { + let error = json!({"id": id, "error": {"code": -32601, "message": "unknown"}}); + out.write_all(format!("{error}\n").as_bytes()).await.unwrap(); + continue; + } + }; + out.write_all(format!("{}\n", json!({"id": id, "result": result})).as_bytes()).await.unwrap(); + } + }); + Pipes { reader: Box::new(daemon_read), writer: Box::new(daemon_write), child: None } + } + + pub(crate) fn temp_dir(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("vibebuddy-resident-{name}-{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + dir + } + + /// A resident session on the fake app-server, whose user's Codex home is `user-codex/` in `dir`. + pub(crate) fn resident(dir: &Path, script: &Shared, activities: Arc>) -> Resident { + let script = script.clone(); + let mut resident = + Resident::new(true, dir, Launcher::Fake(Box::new(move || fake_server(script.clone()))), activities); + resident.user_codex_home = dir.join("user-codex"); + resident + } + + /// The persona with no Character pack: the robot's. + pub(crate) fn robot() -> Persona { + Persona::of(&Wearing::default()) + } + + pub(crate) fn wearing(character: &str) -> Wearing { + Wearing { character: Some(character.to_owned()), address: None } + } + + pub(crate) fn logged_in() -> Shared { + Arc::new(std::sync::Mutex::new(Script { logged_in: true, ..Script::default() })) + } + + fn methods(script: &Shared) -> Vec { + script.lock().unwrap().received.iter().map(|message| message["method"].as_str().unwrap().to_owned()).collect() + } + + pub(crate) fn tool_call(tool: &str, arguments: Value) -> (String, Value) { + let params = json!({"threadId": "t", "turnId": "u", "callId": "c", "tool": tool, "arguments": arguments}); + ("item/tool/call".to_owned(), params) + } + + #[tokio::test] + async fn the_first_command_shakes_hands_and_starts_one_read_only_thread_with_our_tools() { + let dir = temp_dir("start"); + let script = logged_in(); + let resident = resident(&dir, &script, Arc::default()); + + let answer = resident.say("hello", &Wearing::default()).await.unwrap(); + assert_eq!(answer.thread_id, "new-thread"); + assert_eq!(methods(&script), ["initialize", "initialized", "account/read", "thread/start", "turn/start"]); + let received = script.lock().unwrap().received.clone(); + assert_eq!(received[0]["params"]["capabilities"]["experimentalApi"], true); + assert_eq!(received[0]["params"]["clientInfo"]["name"], "vibebuddy"); + let start = &received[3]["params"]; + assert_eq!(start["cwd"], json!(dir.join("resident"))); + assert_eq!(start["sandbox"], "read-only"); + assert_eq!(start["approvalPolicy"], "on-request"); + let instructions = start["developerInstructions"].as_str().unwrap(); + assert!(instructions.contains("speak")); + assert!(instructions.contains("Never state a fact you have not seen in a tool result"), "{instructions}"); + assert!(instructions.contains("not the weather, news, prices")); + assert!(instructions.contains("search the web")); + assert!(instructions.contains("`recently_ended`")); + let tools: Vec<&str> = + start["dynamicTools"].as_array().unwrap().iter().map(|tool| tool["name"].as_str().unwrap()).collect(); + assert_eq!(tools, ["speak", "list_activities", "search_memories"]); + let listing = start["dynamicTools"][1]["description"].as_str().unwrap(); + assert!(listing.contains("`recently_ended`") && listing.contains("`today`"), "{listing}"); + assert_eq!(received[4]["params"]["threadId"], "new-thread"); + let input = received[4]["params"]["input"].as_array().unwrap(); + assert_eq!(input.len(), 2); + let context = input[0]["text"].as_str().unwrap(); + assert!(context.starts_with("The user's local time: ") && context.contains("You are"), "{context}"); + assert_eq!(input[1], json!({"type": "text", "text": "hello"})); + assert!(dir.join("resident").is_dir()); + assert_eq!(load(&dir.join("resident.json")).thread_id.as_deref(), Some("new-thread")); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_later_start_resumes_the_saved_thread_and_a_lost_one_is_replaced() { + let dir = temp_dir("resume"); + let saved = Saved { thread_id: Some("old-thread".to_owned()), codex_version: None, persona: None }; + save(&dir.join("resident.json"), &saved).unwrap(); + let script = logged_in(); + let answer = resident(&dir, &script, Arc::default()).say("hi", &Wearing::default()).await.unwrap(); + assert_eq!(answer.thread_id, "old-thread"); + assert_eq!(methods(&script), ["initialize", "initialized", "account/read", "thread/resume", "turn/start"]); + let resume = script.lock().unwrap().received[3]["params"].clone(); + assert_eq!(resume["threadId"], "old-thread"); + assert_eq!(resume["sandbox"], "read-only"); + assert_eq!(resume["approvalPolicy"], "on-request"); + + let script = logged_in(); + script.lock().unwrap().lose_thread = true; + let answer = resident(&dir, &script, Arc::default()).say("hi", &Wearing::default()).await.unwrap(); + assert_eq!(answer.thread_id, "new-thread"); + assert_eq!(methods(&script)[3..5], ["thread/resume", "thread/start"]); + assert_eq!(load(&dir.join("resident.json")).thread_id.as_deref(), Some("new-thread")); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_thread_open_elsewhere_fails_the_command_and_is_never_replaced() { + let dir = temp_dir("busy"); + let saved = Saved { thread_id: Some("old-thread".to_owned()), codex_version: None, persona: None }; + save(&dir.join("resident.json"), &saved).unwrap(); + let script = logged_in(); + script.lock().unwrap().busy_thread = true; + assert!(resident(&dir, &script, Arc::default()).say("hi", &Wearing::default()).await.is_err()); + assert!(!methods(&script).contains(&"thread/start".to_owned()), "{:?}", methods(&script)); + assert_eq!(load(&dir.join("resident.json")).thread_id.as_deref(), Some("old-thread")); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn without_a_login_nothing_starts_and_no_login_flow_is_begun() { + let dir = temp_dir("login"); + let script = Arc::new(std::sync::Mutex::new(Script::default())); + let error = resident(&dir, &script, Arc::default()).say("hi", &Wearing::default()).await.unwrap_err(); + assert_eq!(error, ResidentError::NotLoggedIn); + assert_eq!(methods(&script), ["initialize", "initialized", "account/read"]); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn speak_becomes_the_reply_and_list_activities_reads_the_tracker() { + let dir = temp_dir("tools"); + let mut tracker = ActivityTracker::default(); + let id = ActivityId { session_id: "s".to_owned(), key: "s:t".to_owned() }; + tracker.observe(&id, "CX:REVIEW", ActivityStatus::Working); + let ended = ActivityId { session_id: "e".to_owned(), key: "e:t".to_owned() }; + tracker.note_project(&ended, "VIBE-BUDDY"); + tracker.observe(&ended, "CC:DOCS", ActivityStatus::Working); + tracker.finish(&ended, "CC:DOCS"); + let script = logged_in(); + script.lock().unwrap().turn_requests = vec![ + tool_call("list_activities", json!({})), + tool_call("speak", json!({"text": " One task is running. "})), + tool_call("speak", json!({"text": ""})), + tool_call("run_shell", json!({"command": "ls"})), + ]; + let answer = resident(&dir, &script, Arc::new(Mutex::new(tracker))).say("what's up?", &Wearing::default()).await.unwrap(); + assert_eq!(answer.replies, ["One task is running."]); + + let answers = script.lock().unwrap().answers.clone(); + let text = |index: usize| answers[index]["result"]["contentItems"][0]["text"].as_str().unwrap().to_owned(); + assert_eq!(answers[0]["id"], "srv-1-0"); + assert_eq!(answers[0]["result"]["success"], true); + let listed: Value = serde_json::from_str(&text(0)).unwrap(); + assert_eq!(listed["activities"][0]["title"], "CX:REVIEW"); + assert_eq!(listed["activities"][0]["state"], "working"); + assert_eq!(listed["activities"].as_array().unwrap().len(), 1); + assert_eq!(listed["recently_ended"][0]["title"], "CC:DOCS"); + assert_eq!(listed["recently_ended"][0]["project"], "VIBE-BUDDY"); + assert_eq!(listed["recently_ended"][0]["ended"], "done"); + assert!(listed["recently_ended"][0]["ended_s"].is_u64()); + assert_eq!(listed["today"]["done"], 1); + assert!(listed["today"]["asks"].is_u64() && listed["today"]["busy_seconds"].is_u64()); + assert_eq!(answers[1]["result"]["success"], true); + assert_eq!(answers[2]["result"]["success"], false); + assert_eq!(answers[3]["result"]["success"], false); + assert_eq!(text(3), "unknown tool run_shell"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_spoken_reply_is_handed_over_when_speak_is_called_not_when_the_turn_completes() { + let dir = temp_dir("stream"); + let script = logged_in(); + let hold = Arc::new(tokio::sync::Notify::new()); + script.lock().unwrap().hold = Some(hold.clone()); + script.lock().unwrap().turn_requests = vec![tool_call("speak", json!({"text": "On it."}))]; + let resident = Arc::new(resident(&dir, &script, Arc::default())); + let (sender, mut receiver) = tokio::sync::mpsc::unbounded_channel(); + let turn = tokio::spawn({ + let resident = resident.clone(); + async move { resident.say_streaming("hi", &Wearing::default(), &mut |reply| sender.send(reply.to_owned()).unwrap()).await } + }); + assert_eq!(receiver.recv().await.as_deref(), Some("On it.")); + assert!(!turn.is_finished(), "the turn is still running"); + hold.notify_one(); + assert_eq!(turn.await.unwrap().unwrap(), "new-thread"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn approvals_and_questions_are_declined_and_other_requests_refused() { + let dir = temp_dir("decline"); + let script = logged_in(); + script.lock().unwrap().turn_requests = [ + "item/commandExecution/requestApproval", + "item/fileChange/requestApproval", + "item/permissions/requestApproval", + "mcpServer/elicitation/request", + "item/tool/requestUserInput", + "execCommandApproval", + "account/chatgptAuthTokens/refresh", + ] + .map(|method| (method.to_owned(), json!({}))) + .to_vec(); + resident(&dir, &script, Arc::default()).say("delete everything", &Wearing::default()).await.unwrap(); + let answers = script.lock().unwrap().answers.clone(); + assert_eq!(answers[0]["result"], json!({"decision": "decline"})); + assert_eq!(answers[1]["result"], json!({"decision": "decline"})); + assert_eq!(answers[2]["result"], json!({"permissions": {}})); + assert_eq!(answers[3]["result"], json!({"action": "decline"})); + assert_eq!(answers[4]["result"], json!({"answers": {}})); + assert_eq!(answers[5]["result"], json!({"decision": "denied"})); + assert_eq!(answers[6]["error"]["code"], -32601); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_dead_app_server_fails_the_turn_and_is_started_again_on_the_next_command() { + let dir = temp_dir("restart"); + let script = logged_in(); + script.lock().unwrap().die_on_turn = Some(1); + let resident = resident(&dir, &script, Arc::default()); + assert!(matches!(resident.say("one", &Wearing::default()).await, Err(ResidentError::Rpc(_)))); + let answer = resident.say("two", &Wearing::default()).await.unwrap(); + assert_eq!(answer.thread_id, "new-thread"); + assert_eq!(script.lock().unwrap().starts, 2); + // The second start resumes the thread the first one made. + assert!(methods(&script).contains(&"thread/resume".to_owned())); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn concurrent_commands_wait_their_turn() { + let dir = temp_dir("queue"); + let script = logged_in(); + let resident = Arc::new(resident(&dir, &script, Arc::default())); + let robot = Wearing::default(); + let (first, second) = tokio::join!(resident.say("one", &robot), resident.say("two", &robot)); + assert!(first.is_ok() && second.is_ok()); + let turns: Vec = script + .lock() + .unwrap() + .received + .iter() + .filter(|message| message["method"] == "turn/start") + .map(|message| message["params"]["input"][1]["text"].clone()) + .collect(); + assert_eq!(turns, [json!("one"), json!("two")]); + assert_eq!(script.lock().unwrap().starts, 1); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn off_by_default_it_starts_nothing() { + let dir = temp_dir("off"); + let script = logged_in(); + let launch = script.clone(); + let resident = Resident::new(false, &dir, Launcher::Fake(Box::new(move || fake_server(launch.clone()))), Arc::default()); + assert_eq!(resident.say("hi", &Wearing::default()).await.unwrap_err(), ResidentError::Disabled); + assert_eq!(script.lock().unwrap().starts, 0); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_codex_home_is_private_and_shares_the_users_login_through_a_link() { + use std::os::unix::fs::PermissionsExt; + let dir = temp_dir("home"); + std::fs::create_dir_all(dir.join("user-codex")).unwrap(); + std::fs::write(dir.join("user-codex/auth.json"), "{}").unwrap(); + resident(&dir, &logged_in(), Arc::default()).say("hi", &Wearing::default()).await.unwrap(); + let home = dir.join(CODEX_HOME_DIR); + assert_eq!(std::fs::metadata(&home).unwrap().permissions().mode() & 0o777, 0o700); + assert_eq!(std::fs::read_link(home.join("auth.json")).unwrap(), dir.join("user-codex/auth.json")); + // Nothing else is put there: no config.toml. + assert_eq!(std::fs::read_dir(&home).unwrap().count(), 1); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn a_copied_or_misdirected_login_is_replaced_by_the_link_on_every_start() { + let dir = temp_dir("relink"); + let (home, user) = (dir.join("home"), dir.join("user")); + std::fs::create_dir_all(&user).unwrap(); + std::fs::write(user.join("auth.json"), "{\"tokens\":\"user\"}").unwrap(); + let link = home.join("auth.json"); + + prepare_codex_home(&home, &user).unwrap(); + assert_eq!(std::fs::read_link(&link).unwrap(), user.join("auth.json")); + prepare_codex_home(&home, &user).unwrap(); + assert_eq!(std::fs::read_link(&link).unwrap(), user.join("auth.json")); + + // A diverged copy, as a write by rename would leave behind. + std::fs::remove_file(&link).unwrap(); + std::fs::write(&link, "{\"tokens\":\"stale\"}").unwrap(); + prepare_codex_home(&home, &user).unwrap(); + assert_eq!(std::fs::read_link(&link).unwrap(), user.join("auth.json")); + assert_eq!(std::fs::read_to_string(&link).unwrap(), "{\"tokens\":\"user\"}"); + + std::fs::remove_file(&link).unwrap(); + std::os::unix::fs::symlink(dir.join("elsewhere.json"), &link).unwrap(); + prepare_codex_home(&home, &user).unwrap(); + assert_eq!(std::fs::read_link(&link).unwrap(), user.join("auth.json")); + + // A write through the link, as Codex refreshes tokens (truncate and rewrite in place), lands in the user's. + let mut file = std::fs::OpenOptions::new().write(true).truncate(true).open(&link).unwrap(); + std::io::Write::write_all(&mut file, b"{}").unwrap(); + assert_eq!(std::fs::read_to_string(user.join("auth.json")).unwrap(), "{}"); + assert!(std::fs::symlink_metadata(&link).unwrap().file_type().is_symlink()); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_users_memories_are_read_into_the_instructions_and_searched_but_never_written() { + let dir = temp_dir("memories"); + let memories = dir.join("user-codex/memories"); + std::fs::create_dir_all(memories.join("rollout_summaries")).unwrap(); + std::fs::write(memories.join("memory_summary.md"), "The user's cat is called Biscuit.\n").unwrap(); + std::fs::write(memories.join("MEMORY.md"), "# Pets\nCat: Biscuit\nlikes boxes\n\n# Work\nRust daemon\n").unwrap(); + std::fs::write(memories.join("rollout_summaries/a.md"), "nothing here\nbiscuit again\n").unwrap(); + let before: Vec<_> = walk(&memories); + let script = logged_in(); + script.lock().unwrap().turn_requests = vec![ + tool_call("search_memories", json!({"query": "BISCUIT"})), + tool_call("search_memories", json!({"query": "zebra"})), + ]; + resident(&dir, &script, Arc::default()).say("what's my cat called?", &Wearing::default()).await.unwrap(); + + let start = script.lock().unwrap().received[3]["params"].clone(); + let instructions = start["developerInstructions"].as_str().unwrap(); + assert!(instructions.contains("\nThe user's cat is called Biscuit.\n")); + let answers = script.lock().unwrap().answers.clone(); + let found = answers[0]["result"]["contentItems"][0]["text"].as_str().unwrap(); + assert!(found.contains("MEMORY.md:1\n# Pets\nCat: Biscuit\nlikes boxes"), "{found}"); + assert!(found.contains("rollout_summaries/a.md:1\nnothing here\nbiscuit again"), "{found}"); + assert!(!found.contains("Rust daemon")); + assert_eq!(answers[1]["result"]["contentItems"][0]["text"], "no matches"); + assert_eq!(walk(&memories), before, "the memories folder was changed"); + let _ = std::fs::remove_dir_all(dir); + } + + /// Every path under `root` with its contents. + fn walk(root: &Path) -> Vec<(PathBuf, Vec)> { + let mut found = Vec::new(); + for entry in std::fs::read_dir(root).unwrap().map(Result::unwrap) { + let path = entry.path(); + if path.is_dir() { + found.push((path.clone(), Vec::new())); + found.extend(walk(&path)); + } else { + found.push((path.clone(), std::fs::read(&path).unwrap())); + } + } + found.sort(); + found + } + + #[test] + fn without_memories_the_instructions_say_nothing_of_them_and_a_long_summary_is_cut() { + assert!(!instructions(&robot(), None, Brain::Codex).contains("memory")); + let dir = temp_dir("summary"); + std::fs::create_dir_all(dir.join("memories")).unwrap(); + assert_eq!(memory_summary(&dir), None); + std::fs::write(dir.join("memories/memory_summary.md"), "记".repeat(5_000)).unwrap(); + let summary = memory_summary(&dir).unwrap(); + assert!(summary.len() <= MEMORY_SUMMARY_MAX_BYTES && summary.len() > MEMORY_SUMMARY_MAX_BYTES - 3); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn the_instructions_make_the_buddy_the_character_in_its_one_language() { + let ada = instructions(&Persona::of(&wearing("ada")), None, Brain::Codex); + assert!(ada.contains("Right now you are Ada") && ada.contains("Londoner"), "{ada}"); + assert!(ada.contains("always reply in English")); + assert!(!ada.contains("in the language the user spoke")); + let xiaohe = instructions(&Persona::of(&wearing("wanwanxiaohe")), None, Brain::Claude); + assert!(xiaohe.contains("Right now you are Xiaohe (小何)") && xiaohe.contains("Taiwan"), "{xiaohe}"); + assert!(xiaohe.contains("always reply in Chinese")); + let robot = instructions(&robot(), None, Brain::Codex); + assert!(robot.contains("Right now you are VibeBuddy") && robot.contains("always reply in English")); + // Every rule is still there, whoever the buddy is. + for text in [&ada, &xiaohe, &robot] { + assert!(text.contains("Answer only by calling the `speak` tool")); + assert!(text.contains("call `list_activities`; never guess")); + assert!(text.contains("Never state a fact you have not seen")); + assert!(text.contains("You do no work yourself")); + } + assert!(!ada.contains("likes to be called"), "no form of address picked"); + let boss = Wearing { character: Some("ada".to_owned()), address: Some("boss".to_owned()) }; + assert!(instructions(&Persona::of(&boss), None, Brain::Codex).contains("likes to be called \"boss\"")); + let laoban = Wearing { character: Some("wanwanxiaohe".to_owned()), address: Some("laoban".to_owned()) }; + assert!(instructions(&Persona::of(&laoban), None, Brain::Codex).contains("likes to be called \"老板\"")); + } + + #[tokio::test] + async fn a_new_character_resumes_the_same_thread_with_its_persona_and_a_note() { + let dir = temp_dir("character"); + let script = logged_in(); + let resident = resident(&dir, &script, Arc::default()); + let received = |method: &str| -> Vec { + let script = script.lock().unwrap(); + script.received.iter().filter(|message| message["method"] == method).map(|message| message["params"].clone()).collect() + }; + // What follows the context every command starts with. + let texts = |input: &Value| -> Vec { + let items = input["input"].as_array().unwrap(); + assert!(items[0]["text"].as_str().unwrap().contains("The user's local time")); + items[1..].iter().map(|item| item["text"].as_str().unwrap().to_owned()).collect() + }; + + resident.say("hi", &wearing("ada")).await.unwrap(); + resident.say("again", &wearing("ada")).await.unwrap(); + assert_eq!(script.lock().unwrap().starts, 1, "the same Character keeps the running thread"); + let start = &received("thread/start")[0]; + assert!(start["developerInstructions"].as_str().unwrap().contains("Right now you are Ada")); + + resident.say("你好", &wearing("wanwanxiaohe")).await.unwrap(); + assert_eq!(script.lock().unwrap().starts, 2, "a new Character starts the app-server again"); + let resume = &received("thread/resume")[0]; + assert_eq!(resume["threadId"], "new-thread", "the same thread, never a fork"); + let instructions = resume["developerInstructions"].as_str().unwrap(); + assert!(instructions.contains("Right now you are Xiaohe (小何)") && instructions.contains("always reply in Chinese")); + assert_eq!(received("thread/start").len(), 1, "never a new thread"); + let turns = received("turn/start"); + assert_eq!(texts(&turns[0]), ["hi"], "the first command has nothing to switch from"); + assert_eq!(texts(&turns[1]), ["again"]); + let switched = texts(&turns[2]); + assert!(switched[0].starts_with(CONTEXT_PREFIX) && switched[0].contains("from now on you are Xiaohe (小何)"), "{switched:?}"); + assert_eq!(switched[1], "你好"); + + resident.say("还在吗", &wearing("wanwanxiaohe")).await.unwrap(); + assert_eq!(script.lock().unwrap().starts, 2); + assert_eq!(texts(&received("turn/start")[3]), ["还在吗"], "the note goes once"); + + // Whom the thread spoke as is kept, so a change while the daemon was down is noticed too. + let restarted = self::resident(&dir, &script, Arc::default()); + restarted.say("hello", &wearing("ada")).await.unwrap(); + let after = texts(&received("turn/start")[4]); + assert!(after[0].contains("from now on you are Ada"), "{after:?}"); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn the_history_leaves_out_context_notes() { + let turn = json!({"items": [ + {"type": "userMessage", "content": [ + {"type": "text", "text": "The user just switched you to another Character: from now on you are Ada."}, + {"type": "text", "text": "what's up?"}, + ]}, + {"type": "dynamicToolCall", "tool": "speak", "arguments": {"text": "All quiet."}}, + ]}); + assert_eq!( + exchange_of(&turn), + Some(Exchange { said: "what's up?".to_owned(), replies: vec!["All quiet.".to_owned()] }) + ); + } + + #[tokio::test] + async fn the_history_is_read_from_the_thread_oldest_first() { + let dir = temp_dir("history"); + let script = logged_in(); + let resident = resident(&dir, &script, Arc::default()); + assert_eq!(resident.history(5).await.unwrap(), [], "no thread yet, so nothing is started"); + assert_eq!(script.lock().unwrap().starts, 0); + resident.say("hi", &Wearing::default()).await.unwrap(); + let history = resident.history(2).await.unwrap(); + assert_eq!( + history, + [ + Exchange { said: "what's up?".to_owned(), replies: vec!["One task.".to_owned(), "It's fine.".to_owned()] }, + Exchange { said: "and now?".to_owned(), replies: vec!["Still one.".to_owned()] }, + ] + ); + let list = script.lock().unwrap().received.last().unwrap()["params"].clone(); + assert_eq!(list, json!({"threadId": "new-thread", "limit": 2, "sortDirection": "desc", "itemsView": "full"})); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test(start_paused = true)] + async fn readiness_is_checked_without_a_turn_cached_and_follows_what_a_start_finds() { + let dir = temp_dir("readiness"); + let script = Arc::new(std::sync::Mutex::new(Script::default())); + let resident = resident(&dir, &script, Arc::default()); + let mut watcher = resident.watch_ready(); + assert!(!resident.ready(), "not ready before the first check"); + + assert_eq!(resident.check_readiness().await.problem, None); + assert!(resident.ready()); + assert!(watcher.has_changed().unwrap(), "watchers hear it come on"); + watcher.mark_unchanged(); + assert_eq!(script.lock().unwrap().starts, 0, "the check starts no app-server"); + + // A check younger than the TTL stands. + *resident.fake_problem.lock().unwrap() = Some(Problem::CodexLoggedOut); + assert!(resident.check_readiness().await.ready()); + tokio::time::advance(READINESS_TTL).await; + assert_eq!(resident.check_readiness().await.problem, Some(Problem::CodexLoggedOut)); + assert!(!resident.ready()); + assert!(watcher.has_changed().unwrap(), "and go off"); + watcher.mark_unchanged(); + + // A start that finds the user logged in is the newer word, and so is one that finds them logged out. + script.lock().unwrap().logged_in = true; + resident.say("hi", &Wearing::default()).await.unwrap(); + assert!(resident.ready()); + let script = Arc::new(std::sync::Mutex::new(Script::default())); + let logged_out = super::tests::resident(&dir, &script, Arc::default()); + assert!(logged_out.check_readiness().await.ready()); + let watcher = logged_out.watch_ready(); + assert_eq!(logged_out.say("hi", &Wearing::default()).await.unwrap_err(), ResidentError::NotLoggedIn); + assert_eq!(logged_out.readiness().problem, Some(Problem::CodexLoggedOut)); + assert!(watcher.has_changed().unwrap()); + + let off = Resident::new(false, &dir, Launcher::Codex, Arc::default()); + assert!(!off.check_readiness().await.ready() && !off.ready(), "switched off, nothing is checked"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_revision_moves_when_a_turn_ends_or_the_thread_changes_but_not_on_reads() { + let dir = temp_dir("revision"); + let saved = Saved { thread_id: Some("old-thread".to_owned()), codex_version: None, persona: None }; + save(&dir.join("resident.json"), &saved).unwrap(); + let script = logged_in(); + let first = resident(&dir, &script, Arc::default()); + assert_eq!(first.revision(), 0); + first.history(5).await.unwrap(); + assert_eq!(first.revision(), 0, "resuming the same thread and reading it changes nothing"); + first.say("hi", &Wearing::default()).await.unwrap(); + assert_eq!(first.revision(), 1); + first.history(5).await.unwrap(); + assert_eq!(first.revision(), 1); + + // A lost thread replaced by a new one counts, and so does a turn that fails. + let script = logged_in(); + script.lock().unwrap().lose_thread = true; + script.lock().unwrap().die_on_turn = Some(1); + let second = resident(&dir, &script, Arc::default()); + assert!(second.say("one", &Wearing::default()).await.is_err()); + assert_eq!(second.revision(), 2, "the new thread and the failed turn"); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn codex_runs_in_the_resident_codex_home() { + let command = codex_command(Path::new("/bin/codex"), Path::new("/state/resident-codex")); + let home = command.as_std().get_envs().find(|(name, _)| *name == "CODEX_HOME").and_then(|(_, value)| value); + assert_eq!(home, Some(std::ffi::OsStr::new("/state/resident-codex"))); + } + + #[test] + fn the_app_server_starts_without_plugins_apps_built_in_tools_or_the_users_mcp_servers() { + let args = app_server_args(&tool_overrides(), &["node_repl".to_owned(), "computer-use".to_owned()]); + assert_eq!(args.last().map(String::as_str), Some("app-server")); + let overrides: Vec<&str> = + args.chunks(2).filter(|pair| pair[0] == "-c").map(|pair| pair[1].as_str()).collect(); + for expected in [ + "features.plugins=false", + "features.apps=false", + "features.shell_tool=false", + "features.unified_exec=false", + "features.multi_agent=false", + "features.memories=false", + "features.shell_snapshot=false", + "skills.include_instructions=false", + "skills.bundled.enabled=false", + "web_search=\"live\"", + "cli_auth_credentials_store=\"file\"", + "memories.generate_memories=false", + "model=\"gpt-6-luna\"", + "model_reasoning_effort=\"low\"", + "mcp_servers.node_repl.enabled=false", + "mcp_servers.computer-use.enabled=false", + ] { + assert!(overrides.contains(&expected), "missing {expected}"); + } + // Hooks stay on: the daemon filters them. + assert!(!args.iter().any(|arg| arg.contains("hooks"))); + assert!(!args.iter().any(|arg| arg == "--disable")); + } + + #[test] + fn the_brain_setting_decides_readiness_and_auto_prefers_a_ready_codex() { + let now = Some(tokio::time::Instant::now()); + let codex_ready = Readiness { checked: now, problem: None, brain: None }; + let logged_out = Readiness { checked: now, problem: Some(Problem::CodexLoggedOut), brain: None }; + let missing = Readiness { checked: now, problem: Some(Problem::CodexMissing), brain: None }; + let unchecked = Readiness::default(); + let brain = |readiness: Readiness| readiness.ready().then_some(readiness.brain).flatten(); + let problem = |readiness: Readiness| readiness.problem; + + // auto: Codex when ready, else Claude Code when it is there with a key. + assert_eq!(brain(decide(BrainChoice::Auto, codex_ready, true, true)), Some(Brain::Codex)); + assert_eq!(brain(decide(BrainChoice::Auto, logged_out, true, true)), Some(Brain::Claude)); + assert_eq!(brain(decide(BrainChoice::Auto, unchecked, true, true)), Some(Brain::Claude)); + assert!(!decide(BrainChoice::Auto, unchecked, true, false).ready(), "nothing known yet"); + assert_eq!(decide(BrainChoice::Auto, unchecked, true, false).problem, None); + // Not ready: without a key it is Codex's problem; with one, Claude Code's. + assert_eq!(problem(decide(BrainChoice::Auto, missing, true, false)), Some(Problem::CodexMissing)); + assert_eq!(problem(decide(BrainChoice::Auto, logged_out, true, false)), Some(Problem::CodexLoggedOut)); + assert_eq!(problem(decide(BrainChoice::Auto, logged_out, false, true)), Some(Problem::ClaudeMissing)); + assert_eq!(decide(BrainChoice::Auto, logged_out, false, true).brain, None); + + // codex and claude stick to their brain. + assert_eq!(brain(decide(BrainChoice::Codex, codex_ready, true, true)), Some(Brain::Codex)); + assert_eq!(problem(decide(BrainChoice::Codex, logged_out, true, true)), Some(Problem::CodexLoggedOut)); + assert_eq!(brain(decide(BrainChoice::Claude, codex_ready, true, true)), Some(Brain::Claude)); + assert_eq!(problem(decide(BrainChoice::Claude, codex_ready, false, true)), Some(Problem::ClaudeMissing)); + assert_eq!(problem(decide(BrainChoice::Claude, codex_ready, true, false)), Some(Problem::AnthropicKeyMissing)); + } + + #[tokio::test] + async fn the_key_and_brain_are_kept_in_a_private_file_and_switching_brains_moves_the_revision() { + use std::os::unix::fs::PermissionsExt; + let dir = temp_dir("settings"); + let file = dir.join("config").join(SETTINGS_FILE); + let script = Arc::new(std::sync::Mutex::new(Script::default())); + let claude = crate::resident_claude::tests::script(); + let resident = resident(&dir, &script, Arc::default()) + .with_claude(crate::resident_claude::tests::brain(&dir, &claude, Arc::default())) + .with_settings_file(file.clone()); + *resident.fake_problem.lock().unwrap() = Some(Problem::CodexLoggedOut); + assert_eq!(resident.check_readiness().await.problem, Some(Problem::CodexLoggedOut)); + assert_eq!(resident.brain(), Brain::Codex); + + resident.set_anthropic_key(Some(" sk-ant-secret \n".to_owned())).await.unwrap(); + assert!(resident.ready() && resident.anthropic_key_set()); + assert_eq!(resident.brain(), Brain::Claude, "auto falls back to Claude Code with a key"); + assert_eq!(std::fs::metadata(&file).unwrap().permissions().mode() & 0o777, 0o600); + let stored: Value = serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap(); + assert_eq!(stored, json!({"brain": "auto", "anthropic_api_key": "sk-ant-secret"})); + + // The voice command goes to Claude Code, with the stored key, and its session is the one reported. + let answer = resident.say("hi", &Wearing::default()).await.unwrap(); + assert_eq!(Some(answer.thread_id.clone()), resident.thread_id()); + let launches = claude.lock().unwrap().launches.clone(); + assert!(launches[0].1.contains(&("ANTHROPIC_API_KEY".to_owned(), "sk-ant-secret".to_owned()))); + assert_eq!(script.lock().unwrap().starts, 0, "Codex wasn't started"); + + // Codex comes back: auto goes back to it, and the history to show is another one. + let before = resident.revision(); + *resident.fake_problem.lock().unwrap() = None; + resident.set_brain(BrainChoice::Auto).await.unwrap(); + assert_eq!(resident.brain(), Brain::Codex); + assert_eq!(resident.revision(), before + 1); + + resident.set_brain(BrainChoice::Claude).await.unwrap(); + assert_eq!(resident.brain(), Brain::Claude); + resident.set_anthropic_key(None).await.unwrap(); + assert_eq!(resident.readiness().problem, Some(Problem::AnthropicKeyMissing)); + assert!(!resident.ready()); + assert_eq!(resident.say("hi", &Wearing::default()).await.unwrap_err(), ResidentError::NoAnthropicKey); + let stored: Value = serde_json::from_str(&std::fs::read_to_string(&file).unwrap()).unwrap(); + assert_eq!(stored, json!({"brain": "claude"}), "the key is gone from the file"); + + // Read back at the next start. + let again = Resident::new(true, &dir, Launcher::Codex, Arc::default()).with_settings_file(file); + assert_eq!(again.brain_setting(), BrainChoice::Claude); + assert!(!again.anthropic_key_set()); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_claude_history_and_hooks_are_the_claude_sessions() { + let dir = temp_dir("claude-hooks"); + let claude = crate::resident_claude::tests::script(); + let resident = resident(&dir, &logged_in(), Arc::default()) + .with_claude(crate::resident_claude::tests::brain(&dir, &claude, Arc::default())); + resident.set_anthropic_key(Some("key".to_owned())).await.unwrap(); + resident.set_brain(BrainChoice::Claude).await.unwrap(); + assert_eq!(resident.history(5).await.unwrap(), [], "no session yet"); + resident.say("hi", &Wearing::default()).await.unwrap(); + let session = resident.thread_id().unwrap(); + assert!(resident.owns_claude_hook(&session, None)); + assert!(resident.owns_claude_hook("other", Some(&dir.join("resident").display().to_string()))); + assert!(!resident.owns_claude_hook("other", Some("/work/vibe-buddy"))); + assert!(!resident.owns_hook(&session, None, None), "a Claude session id isn't a Codex thread"); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn hooks_from_the_resident_thread_or_its_directory_are_recognised() { + let dir = temp_dir("hooks"); + let saved = Saved { thread_id: Some("resident-thread".to_owned()), codex_version: None, persona: None }; + save(&dir.join("resident.json"), &saved).unwrap(); + let resident = Resident::new(false, &dir, Launcher::Codex, Arc::default()); + let cwd = dir.join("resident").display().to_string(); + assert!(resident.owns_hook("resident-thread", None, None)); + assert!(resident.owns_hook("subagent", Some("resident-thread"), None)); + assert!(resident.owns_hook("other", None, Some(&cwd))); + assert!(resident.owns_hook("other", None, Some(&format!("{cwd}/notes")))); + assert!(!resident.owns_hook("other", Some("other"), Some("/work/vibe-buddy"))); + assert!(!resident.owns_hook("other", None, Some(&format!("{cwd}-not")))); + assert!(!resident.owns_hook("other", None, None)); + let _ = std::fs::remove_dir_all(dir); + } +} diff --git a/daemon/src/resident_claude.rs b/daemon/src/resident_claude.rs new file mode 100644 index 00000000..033c53b6 --- /dev/null +++ b/daemon/src/resident_claude.rs @@ -0,0 +1,1241 @@ +//! The resident session on Claude Code (ADR-0017): the user's own `claude` CLI, run headless with the user's own +//! Anthropic API key, never a claude.ai login (Anthropic doesn't let third-party products use one). +//! +//! One long-lived `claude -p --input-format stream-json --output-format stream-json` process holds the resident +//! session; each voice command is one user message on its stdin, and the turn is over at the `result` line. The +//! session is created once with `--session-id` and taken up again with `--resume` after every restart, so like the +//! Codex thread it is one session for good, replaced only when its transcript is gone. +//! +//! - **Its own config dir.** `CLAUDE_CONFIG_DIR` is `resident-claude/` in the daemon's state directory (0700), so +//! none of the user's Claude config loads: settings, CLAUDE.md, hooks, plugins, MCP servers, and their login (the +//! credentials file and the macOS Keychain entry are keyed by the config dir). +//! - **Only the user's API key.** The child's environment loses every `CLAUDE*` and `ANTHROPIC*` variable the daemon +//! has (an OAuth token, an auth token, a provider switch) and gets `ANTHROPIC_API_KEY`, which `-p` always uses when +//! present. The `system/init` line says where the key came from; anything but `ANTHROPIC_API_KEY` stops the +//! process. The key is never logged. +//! - **Only our tools.** `--tools WebSearch` leaves Claude Code's only built-in tool the web search; `speak` and +//! `list_activities` come from an MCP server the daemon runs on a loopback port of its own, behind a random +//! token (`--strict-mcp-config`, so no other server loads). `--permission-mode dontAsk` with only those allowed, +//! and the usual built-ins denied by name besides. `--bare` would skip even more, but it also takes WebSearch +//! away (checked with claude 2.1.296), so the config dir and the flags do it instead. +//! +//! Experimental, behind the same `VIBEBUDDY_RESIDENT_SESSION=1` as the Codex session; see docs/resident-session.md. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::Duration; + +use axum::extract::State; +use axum::http::{HeaderMap, StatusCode}; +use axum::response::IntoResponse; +use serde::{Deserialize, Serialize}; +use serde_json::{Value, json}; +use tokio::io::{AsyncBufReadExt, AsyncRead, AsyncWrite, AsyncWriteExt, BufReader, Lines}; +use tokio::sync::{Mutex, OnceCell, mpsc}; +use tracing::{info, warn}; + +use crate::activity::ActivityTracker; +use crate::persona::Persona; +use crate::resident::{Exchange, Pipes, ResidentError, change_note, run_tool, tools}; + +/// The model the resident session runs on: a voice command waits on the answer. +pub const MODEL: &str = "claude-haiku-4-5"; +/// The Claude Code version the flags and the stream were checked with; another one gets a warning and is tried. +pub const TESTED_CLAUDE_VERSION: &str = "2.1.296"; +/// Names the `claude` binary explicitly; otherwise it is looked for on PATH and where installers put it. +pub const CLAUDE_VARIABLE: &str = "VIBEBUDDY_CLAUDE"; +/// Development only: where the child sends its API requests (a fake Messages API in testing). The daemon's own +/// `ANTHROPIC_BASE_URL`, like every other `ANTHROPIC*` variable, never reaches the child. +pub const BASE_URL_VARIABLE: &str = "VIBEBUDDY_CLAUDE_BASE_URL"; +/// The resident session's own Claude config dir, in the daemon's state directory. +const CONFIG_DIR: &str = "resident-claude"; +/// The session id and the Claude Code version it was last started with, in the daemon's state directory. +const STATE_FILE: &str = "resident-claude.json"; +/// The name the daemon's MCP server has in Claude Code: its tools are `mcp__vibebuddy__`. +const MCP_SERVER: &str = "vibebuddy"; +/// The one built-in tool the session gets. +const WEB_SEARCH: &str = "WebSearch"; +/// Built-in tools denied by name on top of `--tools`, so a Claude Code that changes what `--tools` covers still +/// gives the session none of them. +const DENIED_TOOLS: [&str; 12] = + ["Bash", "Edit", "Write", "Read", "NotebookEdit", "WebFetch", "Glob", "Grep", "Agent", "Task", "Skill", "BashOutput"]; +/// How long one voice command may take, from writing it to the `result` line; then the process is stopped. +const TURN_TIMEOUT: Duration = Duration::from_secs(120); +/// What the `system/init` line must report as the credential in use. +const API_KEY_SOURCE: &str = "ANTHROPIC_API_KEY"; +/// Starts a voice command's context block, which carries the local time; the history leaves it out. +pub(crate) const CONTEXT_PREFIX: &str = ""; + +/// How `claude` is reached: the real binary, or an in-memory fake in tests. +pub enum Launcher { + Claude, + #[cfg(test)] + Fake(Box Pipes + Send + Sync>), +} + +/// What would be run: the arguments, and the variables set on top of the daemon's cleaned environment. +pub struct Launch { + pub args: Vec, + pub env: Vec<(String, String)>, +} + +struct Process { + writer: Box, + lines: Lines>>, + /// The key it was started with: a new key needs a new process. + key: String, + /// The instructions it was started with: `--system-prompt` is per process, so new ones (a new Character) need + /// a new process on the same session. + instructions: String, + _child: Option, +} + +#[derive(Default, Deserialize, Serialize)] +struct Saved { + session_id: Option, + /// The Claude Code version the session was last started with, for diagnosing a change in the stream. + claude_version: Option, + /// The Character whose persona the session last answered in. + #[serde(default)] + persona: Option, +} + +/// What the daemon's MCP server shares with the turn in progress. +struct McpState { + token: String, + activities: Arc>, + /// Where `speak` goes while a turn runs; None between turns. + speak: std::sync::Mutex>>, +} + +pub struct ClaudeBrain { + /// The working directory, shared with the Codex session: `resident/` in the state directory. + dir: PathBuf, + config_dir: PathBuf, + state_file: PathBuf, + launcher: Launcher, + mcp: Arc, + /// The loopback address of the MCP server, once started. + mcp_url: OnceCell, + /// Held for a whole turn: one voice command at a time. + process: Mutex>, + /// Read synchronously by the hook filter. + session_id: std::sync::Mutex>, + /// The Character whose persona the session last answered in (kept in the state file). + spoke_as: std::sync::Mutex>, +} + +impl ClaudeBrain { + pub fn new(state_dir: &Path, launcher: Launcher, activities: Arc>) -> Self { + let state_file = state_dir.join(STATE_FILE); + let saved: Saved = std::fs::read_to_string(&state_file) + .ok() + .and_then(|text| serde_json::from_str(&text).ok()) + .unwrap_or_default(); + Self { + dir: state_dir.join("resident"), + config_dir: state_dir.join(CONFIG_DIR), + state_file, + launcher, + mcp: Arc::new(McpState { token: random_hex(32), activities, speak: std::sync::Mutex::new(None) }), + mcp_url: OnceCell::new(), + process: Mutex::new(None), + session_id: std::sync::Mutex::new(saved.session_id), + spoke_as: std::sync::Mutex::new(saved.persona), + } + } + + pub fn session_id(&self) -> Option { + self.session_id.lock().expect("session id lock").clone() + } + + /// Whether `claude` can be found; the readiness check, which starts nothing. + pub fn installed(&self) -> bool { + match &self.launcher { + Launcher::Claude => find_claude().is_some(), + #[cfg(test)] + Launcher::Fake(_) => true, + } + } + + /// Stops the process, if any, so the next command starts one with the current key. + pub async fn stop(&self) { + *self.process.lock().await = None; + } + + /// Starts `claude` ahead of the first voice command. A failure is logged; the next command tries again. + pub async fn warm(&self, key: &str, instructions: &str) { + let mut slot = self.process.lock().await; + if let Err(error) = self.running(&mut slot, key, instructions).await { + warn!(%error, "the resident Claude session could not be started ahead of time"); + } + } + + /// Hands one voice command to the session, answered in `instructions`, which make the buddy `persona`; each + /// spoken reply goes to `on_reply` when `speak` is called. Returns the session id once the turn is over. + pub async fn say_streaming( + &self, + key: &str, + instructions: &str, + persona: &Persona, + text: &str, + on_reply: &mut (dyn FnMut(&str) + Send), + ) -> Result { + let mut slot = self.process.lock().await; + let process = self.running(&mut slot, key, instructions).await?; + let spoke_as = self.spoke_as.lock().expect("spoke as lock").clone(); + let changed = spoke_as.as_deref() != Some(persona.id); + let note = (changed && spoke_as.is_some()).then(|| change_note(persona)); + let (sender, spoken) = mpsc::unbounded_channel(); + *self.mcp.speak.lock().expect("speak lock") = Some(sender); + let result = run_turn(process, spoken, persona, note.as_deref(), text, on_reply).await; + *self.mcp.speak.lock().expect("speak lock") = None; + match &result { + // The process is still there and the session intact: an API error, a refused key. + Err(ResidentError::TurnEnded(_)) | Ok(()) => {} + Err(_) => *slot = None, + } + if result.is_ok() && changed { + self.remember_persona(persona.id); + } + result.map(|()| self.session_id().unwrap_or_default()) + } + + /// Keeps the Character the session now answers as, so a change is noticed after a restart too. + fn remember_persona(&self, id: &str) { + *self.spoke_as.lock().expect("spoke as lock") = Some(id.to_owned()); + let mut saved: Saved = std::fs::read_to_string(&self.state_file) + .ok() + .and_then(|text| serde_json::from_str(&text).ok()) + .unwrap_or_default(); + saved.persona = Some(id.to_owned()); + if let Err(error) = write_private(&self.state_file, &serde_json::to_string_pretty(&saved).unwrap_or_default()) { + warn!(%error, "failed to save the resident Claude session's Character"); + } + } + + /// The last `limit` voice commands and their spoken replies, oldest first, read from the session's transcript + /// (the daemon keeps no record of its own). Empty while there is no session yet. + pub fn history(&self, limit: usize) -> Vec { + let Some(path) = self.session_id().and_then(|id| transcript(&self.config_dir, &id)) else { + return Vec::new(); + }; + let exchanges = exchanges_of(&std::fs::read_to_string(path).unwrap_or_default()); + let skip = exchanges.len().saturating_sub(limit); + exchanges.into_iter().skip(skip).collect() + } + + /// Whether a Claude Code hook event comes from the resident session: by its session id. (The caller checks the + /// working directory.) + pub fn owns_session(&self, session_id: &str) -> bool { + self.session_id.lock().expect("session id lock").as_deref() == Some(session_id) + } + + async fn running<'a>( + &self, + slot: &'a mut Option, + key: &str, + instructions: &str, + ) -> Result<&'a mut Process, ResidentError> { + if slot.as_ref().is_some_and(|process| process.key != key) { + info!("the Anthropic API key changed; starting the resident Claude session again"); + *slot = None; + } + if slot.as_ref().is_some_and(|process| process.instructions != instructions) { + info!("the instructions changed (a new Character); resuming the resident Claude session with them"); + *slot = None; + } + if slot.is_none() { + *slot = Some(self.start(key, instructions).await?); + } + Ok(slot.as_mut().expect("process was just started")) + } + + async fn start(&self, key: &str, instructions: &str) -> Result { + let failed = |what: &str, error: std::io::Error| ResidentError::Rpc(format!("cannot {what}: {error}")); + prepare_config_dir(&self.config_dir).map_err(|error| failed("prepare the Claude config dir", error))?; + std::fs::create_dir_all(&self.dir).map_err(|error| failed("create the resident directory", error))?; + let url = self.mcp_url.get_or_try_init(|| serve_mcp(self.mcp.clone())).await.map_err(|error| failed("start the MCP server", error))?; + let mcp_config = self.config_dir.join("mcp.json"); + write_private(&mcp_config, &mcp_config_json(url, &self.mcp.token).to_string()) + .map_err(|error| failed("write the MCP config", error))?; + // One session for good: resumed while its transcript is there, else a new one. + let saved = self.session_id(); + let session = match &saved { + Some(id) if transcript(&self.config_dir, id).is_some() => Session::Resume(id.clone()), + other => { + let id = new_session_id(); + if let Some(old) = other { + warn!(old_session = %old, new_session = %id, "the resident Claude session's transcript is gone; starting a new one"); + } + Session::New(id) + } + }; + let launch = Launch { + args: claude_args(&mcp_config, &session, instructions), + env: child_env(&self.config_dir, key, std::env::var(BASE_URL_VARIABLE).ok()), + }; + let (pipes, version) = match &self.launcher { + Launcher::Claude => spawn_claude(&launch, &self.dir).await?, + #[cfg(test)] + Launcher::Fake(connect) => (connect(&launch), None), + }; + let id = session.id().to_owned(); + *self.session_id.lock().expect("session id lock") = Some(id.clone()); + let spoke_as = self.spoke_as.lock().expect("spoke as lock").clone(); + let saved = Saved { session_id: Some(id.clone()), claude_version: version, persona: spoke_as }; + if let Err(error) = write_private(&self.state_file, &serde_json::to_string_pretty(&saved).unwrap_or_default()) { + warn!(%error, "failed to save the resident Claude session id"); + } + info!(session = %id, resumed = matches!(session, Session::Resume(_)), "resident Claude session started"); + Ok(Process { + writer: pipes.writer, + lines: BufReader::new(pipes.reader).lines(), + key: key.to_owned(), + instructions: instructions.to_owned(), + _child: pipes.child, + }) + } +} + +enum Session { + New(String), + Resume(String), +} + +impl Session { + fn id(&self) -> &str { + match self { + Self::New(id) | Self::Resume(id) => id, + } + } +} + +/// One turn: the voice command in, lines out until `result`, and whatever `speak` said meanwhile. +async fn run_turn( + process: &mut Process, + mut spoken: mpsc::UnboundedReceiver, + persona: &Persona, + note: Option<&str>, + text: &str, + on_reply: &mut (dyn FnMut(&str) + Send), +) -> Result<(), ResidentError> { + let line = format!("{}\n", user_message(text, persona, note, &chrono::Local::now(), local_time_zone().as_deref())); + let written = async { + process.writer.write_all(line.as_bytes()).await?; + process.writer.flush().await + }; + written.await.map_err(|error| ResidentError::Rpc(format!("claude is gone: {error}")))?; + let deadline = tokio::time::Instant::now() + TURN_TIMEOUT; + loop { + tokio::select! { + Some(reply) = spoken.recv() => on_reply(&reply), + line = tokio::time::timeout_at(deadline, process.lines.next_line()) => { + let line = match line { + Err(_) => return Err(ResidentError::Timeout), + Ok(Ok(Some(line))) => line, + Ok(_) => return Err(ResidentError::Rpc("claude exited".to_owned())), + }; + let Ok(message) = serde_json::from_str::(&line) else { continue }; + match (message["type"].as_str(), message["subtype"].as_str()) { + (Some("system"), Some("init")) => { + let source = message["apiKeySource"].as_str().unwrap_or("none"); + if source != API_KEY_SOURCE { + warn!(source, "the resident Claude session isn't using the user's API key; stopped it"); + return Err(ResidentError::Rpc(format!( + "Claude Code would authenticate with {source}, not the Anthropic API key; refused" + ))); + } + } + (Some("result"), _) => { + // `speak` is answered before Claude Code goes on, so anything it said is already queued. + while let Ok(reply) = spoken.try_recv() { + on_reply(&reply); + } + if message["is_error"].as_bool() == Some(true) { + let reason = message["result"] + .as_str() + .or(message["terminal_reason"].as_str()) + .or(message["subtype"].as_str()) + .unwrap_or("unknown"); + return Err(ResidentError::TurnEnded(reason.to_owned())); + } + return Ok(()); + } + _ => {} + } + } + } + } +} + +/// The computer's IANA time zone (`Asia/Singapore`): `TZ` if set, else where `/etc/localtime` points, on macOS +/// and Linux alike. +pub(crate) fn local_time_zone() -> Option { + if let Ok(zone) = std::env::var("TZ") + && !zone.is_empty() + { + return Some(zone.trim_start_matches(':').to_owned()); + } + let target = std::fs::read_link("/etc/localtime").ok()?; + let path = target.to_string_lossy(); + path.split_once("zoneinfo/").map(|(_, zone)| zone.to_owned()) +} + +/// A voice command as a stream-json user message: the [`command_context`] first, then a `note` (the Character +/// changed) if any, then what was said. +fn user_message( + text: &str, + persona: &Persona, + note: Option<&str>, + now: &chrono::DateTime, + time_zone: Option<&str>, +) -> Value { + let context = command_context(persona, now, time_zone); + let content: Vec = + [context.as_str()].into_iter().chain(note).chain([text]).map(|text| json!({"type": "text", "text": text})).collect(); + json!({"type": "user", "message": {"role": "user", "content": content}}) +} + +/// The context put before every voice command, for both brains: the local time and time zone, and who answers in +/// which language. The time zone stands in for where the user is, so a question about the weather or the news has a +/// place; Codex's own clock tool answers in UTC. The reminder is there because with only the instructions saying +/// so, Haiku answered a Chinese question in Chinese as an English Character, carried by the user's language and the +/// session's earlier replies. +pub(crate) fn command_context(persona: &Persona, now: &chrono::DateTime, time_zone: Option<&str>) -> String { + let zone = time_zone.map(|zone| format!(", time zone {zone}")).unwrap_or_default(); + let Persona { name, language, .. } = persona; + format!( + "{CONTEXT_PREFIX}The user's local time: {}{zone}. You are {name}: answer in {language} only, whatever language \ + this is said in.", + now.format("%A %Y-%m-%d %H:%M (UTC%:z)") + ) +} + +/// The arguments for `claude`; see the module docs for why each is there. +fn claude_args(mcp_config: &Path, session: &Session, instructions: &str) -> Vec { + let allowed: Vec = std::iter::once(WEB_SEARCH.to_owned()) + .chain(mcp_tools().iter().map(|tool| format!("mcp__{MCP_SERVER}__{}", tool["name"].as_str().unwrap_or_default()))) + .collect(); + let mut args: Vec = [ + "-p", + "--input-format", + "stream-json", + "--output-format", + "stream-json", + "--verbose", + "--model", + MODEL, + "--tools", + WEB_SEARCH, + "--permission-mode", + "dontAsk", + "--permission-prompts", + "none", + "--strict-mcp-config", + "--disable-slash-commands", + // The instructions are read again on every request, so a change reaches a resumed session. + "--system-prompt-snapshot", + "off", + ] + .map(str::to_owned) + .to_vec(); + args.extend(["--allowedTools".to_owned(), allowed.join(",")]); + args.extend(["--disallowedTools".to_owned(), DENIED_TOOLS.join(",")]); + args.extend(["--mcp-config".to_owned(), mcp_config.display().to_string()]); + args.extend(["--settings".to_owned(), json!({"disableAllHooks": true}).to_string()]); + // Replaces Claude Code's coding-assistant prompt: the buddy isn't one, and a short prompt is a faster turn. + args.extend(["--system-prompt".to_owned(), instructions.to_owned()]); + match session { + Session::New(id) => args.extend(["--session-id".to_owned(), id.clone()]), + Session::Resume(id) => args.extend(["--resume".to_owned(), id.clone()]), + } + args +} + +/// What the child gets on top of the daemon's environment, which loses every `CLAUDE*` and `ANTHROPIC*` variable +/// first (see [`spawn_claude`]). +fn child_env(config_dir: &Path, key: &str, base_url: Option) -> Vec<(String, String)> { + let mut env = vec![ + ("CLAUDE_CONFIG_DIR".to_owned(), config_dir.display().to_string()), + ("ANTHROPIC_API_KEY".to_owned(), key.to_owned()), + // No memory files written into the config dir for a session that has its own way to remember. + ("CLAUDE_CODE_DISABLE_AUTO_MEMORY".to_owned(), "1".to_owned()), + // The daemon's child doesn't update the user's Claude Code behind their back. + ("DISABLE_AUTOUPDATER".to_owned(), "1".to_owned()), + ]; + if let Some(url) = base_url.filter(|url| !url.is_empty()) { + env.push(("ANTHROPIC_BASE_URL".to_owned(), url)); + } + env +} + +/// Whether a variable of the daemon's environment is kept from the child: every one that can choose Claude Code's +/// credential or provider (an OAuth token, an auth token, Bedrock/Vertex/Foundry switches, profiles) starts so. +fn withheld(name: &str) -> bool { + name.starts_with("CLAUDE") || name.starts_with("ANTHROPIC") +} + +async fn spawn_claude(launch: &Launch, cwd: &Path) -> Result<(Pipes, Option), ResidentError> { + let program = find_claude().ok_or(ResidentError::NoClaude)?; + let command = |args: &[String]| { + let mut command = tokio::process::Command::new(&program); + for (name, _) in std::env::vars_os() { + if withheld(&name.to_string_lossy()) { + command.env_remove(&name); + } + } + command.envs(launch.env.iter().map(|(name, value)| (name, value))).args(args).current_dir(cwd); + command + }; + let version = command(&["--version".to_owned()]) + .stdin(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .output() + .await + .ok() + .and_then(|output| String::from_utf8(output.stdout).ok()) + .and_then(|text| text.split_whitespace().next().map(str::to_owned)); + if version.as_deref() != Some(TESTED_CLAUDE_VERSION) { + warn!( + version = version.as_deref().unwrap_or("unknown"), + tested = TESTED_CLAUDE_VERSION, + "the resident session was tested with another Claude Code version; its flags or stream may differ" + ); + } + let mut child = command(&launch.args) + .stdin(std::process::Stdio::piped()) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .kill_on_drop(true) + .spawn() + .map_err(|error| ResidentError::Rpc(format!("cannot start {}: {error}", program.display())))?; + info!(program = %program.display(), "started claude for the resident session"); + // A refused flag or a crash is only on stderr; it carries no credentials. + if let Some(stderr) = child.stderr.take() { + tokio::spawn(async move { + let mut lines = BufReader::new(stderr).lines(); + while let Ok(Some(line)) = lines.next_line().await { + let end = line.floor_char_boundary(300); + info!(line = &line[..end], "claude stderr"); + } + }); + } + let reader = child.stdout.take().expect("stdout is piped"); + let writer = child.stdin.take().expect("stdin is piped"); + Ok((Pipes { reader: Box::new(reader), writer: Box::new(writer), child: Some(child) }, version)) +} + +/// `VIBEBUDDY_CLAUDE`, else `claude` on PATH, else where installers put it: the daemon may run with launchd's bare +/// PATH. +pub fn find_claude() -> Option { + if let Some(path) = std::env::var_os(CLAUDE_VARIABLE) { + return Some(PathBuf::from(path)); + } + let home = std::env::var_os("HOME").map(PathBuf::from); + std::env::var_os("PATH") + .map(|path| std::env::split_paths(&path).map(|dir| dir.join("claude")).collect::>()) + .unwrap_or_default() + .into_iter() + .chain(home.iter().flat_map(|home| [home.join(".local/bin/claude"), home.join(".claude/local/claude")])) + .chain(["/opt/homebrew/bin/claude", "/usr/local/bin/claude"].map(PathBuf::from)) + .find(|path| path.is_file()) +} + +fn prepare_config_dir(config_dir: &Path) -> std::io::Result<()> { + use std::os::unix::fs::PermissionsExt; + std::fs::create_dir_all(config_dir)?; + std::fs::set_permissions(config_dir, std::fs::Permissions::from_mode(0o700)) +} + +/// Writes a file only the user can read (0600), replacing it whole. +pub(crate) fn write_private(path: &Path, text: &str) -> std::io::Result<()> { + use std::io::Write; + use std::os::unix::fs::{OpenOptionsExt, PermissionsExt}; + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent)?; + } + let temporary = path.with_extension("tmp"); + let mut file = std::fs::OpenOptions::new().write(true).create(true).truncate(true).mode(0o600).open(&temporary)?; + // A file left from before with wider permissions keeps them through `open`. + file.set_permissions(std::fs::Permissions::from_mode(0o600))?; + file.write_all(text.as_bytes())?; + file.sync_all()?; + std::fs::rename(temporary, path) +} + +fn mcp_config_json(url: &str, token: &str) -> Value { + json!({"mcpServers": {MCP_SERVER: {"type": "http", "url": url, "headers": {"Authorization": format!("Bearer {token}")}}}}) +} + +/// The session's transcript in the config dir (`projects//.jsonl`), if it exists. +fn transcript(config_dir: &Path, session_id: &str) -> Option { + std::fs::read_dir(config_dir.join("projects")) + .ok()? + .filter_map(Result::ok) + .map(|entry| entry.path().join(format!("{session_id}.jsonl"))) + .find(|path| path.is_file()) +} + +/// The voice commands in a transcript and what was said through `speak` after each, oldest first. Lines are put +/// in time order first: Claude Code doesn't always write them in order. +fn exchanges_of(transcript: &str) -> Vec { + let speak = format!("mcp__{MCP_SERVER}__speak"); + let mut entries: Vec = transcript + .lines() + .filter_map(|line| serde_json::from_str::(line).ok()) + .filter(|entry| matches!(entry["type"].as_str(), Some("user" | "assistant")) && entry["isSidechain"] != true) + .collect(); + entries.sort_by(|a, b| a["timestamp"].as_str().unwrap_or_default().cmp(b["timestamp"].as_str().unwrap_or_default())); + let mut exchanges: Vec = Vec::new(); + for entry in entries { + let content = &entry["message"]["content"]; + if entry["type"] == "user" { + if entry["isMeta"] == true || entry["isCompactSummary"] == true { + continue; + } + let said = match content { + Value::String(text) => text.clone(), + Value::Array(blocks) => blocks + .iter() + .filter(|block| block["type"] == "text") + .filter_map(|block| block["text"].as_str()) + .filter(|text| !text.starts_with(CONTEXT_PREFIX)) + .collect::>() + .join(" "), + _ => String::new(), + }; + if !said.is_empty() { + exchanges.push(Exchange { said, replies: Vec::new() }); + } + } else if let (Some(exchange), Some(blocks)) = (exchanges.last_mut(), content.as_array()) { + exchange.replies.extend( + blocks + .iter() + .filter(|block| block["type"] == "tool_use" && block["name"] == speak.as_str()) + .filter_map(|block| block["input"]["text"].as_str().map(str::to_owned)), + ); + } + } + exchanges +} + +/// Our tools as MCP tools: the Codex session's, less `search_memories` (Codex memories are Codex's). +fn mcp_tools() -> Vec { + tools() + .as_array() + .cloned() + .unwrap_or_default() + .into_iter() + .filter(|tool| tool["name"] != "search_memories") + .map(|tool| json!({"name": tool["name"], "description": tool["description"], "inputSchema": tool["inputSchema"]})) + .collect() +} + +/// Starts the MCP server on a loopback port of its own and returns its URL. +async fn serve_mcp(state: Arc) -> std::io::Result { + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await?; + let url = format!("http://{}/mcp", listener.local_addr()?); + let router = axum::Router::new().route("/mcp", axum::routing::post(post_mcp).get(no_stream).delete(no_stream)).with_state(state); + tokio::spawn(async move { + if let Err(error) = axum::serve(listener, router).await { + warn!(%error, "the resident session's MCP server stopped"); + } + }); + info!(%url, "the resident session's MCP server is listening"); + Ok(url) +} + +/// No server-sent stream: every answer comes back on its POST. +async fn no_stream() -> StatusCode { + StatusCode::METHOD_NOT_ALLOWED +} + +/// MCP's Streamable HTTP transport, the JSON-response form: one JSON-RPC message per POST. +async fn post_mcp(State(state): State>, headers: HeaderMap, body: String) -> axum::response::Response { + let authorized = headers + .get(axum::http::header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.strip_prefix("Bearer ")) + .is_some_and(|token| same(token.as_bytes(), state.token.as_bytes())); + if !authorized { + return StatusCode::UNAUTHORIZED.into_response(); + } + let Ok(message) = serde_json::from_str::(&body) else { + return (StatusCode::BAD_REQUEST, axum::Json(json!({"jsonrpc": "2.0", "id": null, "error": {"code": -32700, "message": "parse error"}}))) + .into_response(); + }; + let Some(id) = message.get("id").cloned() else { + // A notification (`notifications/initialized` and the like) needs no answer. + return StatusCode::ACCEPTED.into_response(); + }; + let answer = match mcp_result(&state, message["method"].as_str().unwrap_or_default(), &message["params"]).await { + Ok(result) => json!({"jsonrpc": "2.0", "id": id, "result": result}), + Err((code, text)) => json!({"jsonrpc": "2.0", "id": id, "error": {"code": code, "message": text}}), + }; + axum::Json(answer).into_response() +} + +async fn mcp_result(state: &McpState, method: &str, params: &Value) -> Result { + match method { + "initialize" => { + let version = params["protocolVersion"].as_str().unwrap_or("2025-06-18"); + Ok(json!({ + "protocolVersion": version, + "capabilities": {"tools": {}}, + "serverInfo": {"name": MCP_SERVER, "title": "VibeBuddy", "version": env!("CARGO_PKG_VERSION")}, + })) + } + "ping" => Ok(json!({})), + "tools/list" => Ok(json!({"tools": mcp_tools()})), + "tools/call" => { + let tool = params["name"].as_str().unwrap_or_default(); + if tool == "search_memories" { + return Ok(json!({"content": [{"type": "text", "text": "unknown tool search_memories"}], "isError": true})); + } + let speak = state.speak.lock().expect("speak lock").clone(); + let (success, text) = run_tool(tool, ¶ms["arguments"], &state.activities, None, &mut |reply| match &speak { + Some(turn) => { + let _ = turn.send(reply.to_owned()); + } + None => warn!("the resident Claude session spoke outside a turn; dropped"), + }) + .await; + Ok(json!({"content": [{"type": "text", "text": text}], "isError": !success})) + } + other => Err((-32601, format!("method not found: {other}"))), + } +} + +/// Compares two secrets in time that doesn't depend on where they differ. +fn same(a: &[u8], b: &[u8]) -> bool { + a.len() == b.len() && a.iter().zip(b).fold(0u8, |difference, (x, y)| difference | (x ^ y)) == 0 +} + +fn random_hex(bytes: usize) -> String { + let mut buffer = vec![0u8; bytes]; + getrandom::fill(&mut buffer).expect("the system's random source"); + buffer.iter().map(|byte| format!("{byte:02x}")).collect() +} + +/// A random (version 4) UUID, as `--session-id` wants. +fn new_session_id() -> String { + let mut bytes = [0u8; 16]; + getrandom::fill(&mut bytes).expect("the system's random source"); + bytes[6] = (bytes[6] & 0x0f) | 0x40; + bytes[8] = (bytes[8] & 0x3f) | 0x80; + let hex: String = bytes.iter().map(|byte| format!("{byte:02x}")).collect(); + format!("{}-{}-{}-{}-{}", &hex[..8], &hex[8..12], &hex[12..16], &hex[16..20], &hex[20..]) +} + +#[cfg(test)] +pub(crate) mod tests { + use std::collections::VecDeque; + + use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader}; + + use super::*; + use crate::resident::tests::robot; + + #[test] + fn a_voice_command_carries_the_local_time_and_time_zone_as_context() { + let now = chrono::Local::now(); + let message = user_message("今天天气怎么样", &Persona::of(&crate::resident::tests::wearing("ada")), None, &now, Some("Asia/Singapore")); + let content = message["message"]["content"].as_array().unwrap(); + let context = content[0]["text"].as_str().unwrap(); + assert!(context.starts_with(CONTEXT_PREFIX) && context.contains("time zone Asia/Singapore"), "{context}"); + assert!(context.contains("You are Ada: answer in English only"), "{context}"); + assert_eq!(content[1]["text"], "今天天气怎么样"); + let without = user_message("hi", &Persona::of(&crate::resident::tests::wearing("ada")), None, &now, None); + assert!(!without["message"]["content"][0]["text"].as_str().unwrap().contains("time zone")); + } + use crate::activity::{ActivityId, ActivityStatus}; + use crate::resident::tests::temp_dir; + + /// One thing the fake `claude` does while answering a voice command. + #[derive(Clone)] + pub(crate) enum Step { + /// The `system/init` line, naming where the credential came from. + Init(&'static str), + /// Calls one of the daemon's MCP tools over HTTP, as Claude Code would, with the token from the MCP config. + Call(&'static str, Value), + /// Waits for the test before going on. + Hold, + /// The `result` line: whether it is an error, and its text. + Result(bool, &'static str), + /// Ends its output, as if the process died. + Exit, + } + + /// How the fake `claude` behaves, and what it saw. + #[derive(Default)] + pub(crate) struct Script { + /// What each voice command gets, in order; a command with none left gets an init and a result. + pub(crate) turns: VecDeque>, + /// The arguments and environment of each start. + pub(crate) launches: Vec, + /// The lines the daemon wrote. + pub(crate) received: Vec, + /// The MCP server's answers to the calls made, with their HTTP status. + pub(crate) answers: Vec<(u16, Value)>, + pub(crate) hold: Option>, + } + + pub(crate) type Shared = Arc>; + /// The arguments and environment of one start. + pub(crate) type Launched = (Vec, Vec<(String, String)>); + + pub(crate) fn script() -> Shared { + Arc::default() + } + + /// A Claude brain on the fake `claude`, in `state_dir`. + pub(crate) fn brain(state_dir: &Path, script: &Shared, activities: Arc>) -> ClaudeBrain { + let script = script.clone(); + ClaudeBrain::new(state_dir, Launcher::Fake(Box::new(move |launch| fake_claude(script.clone(), launch))), activities) + } + + fn argument(args: &[String], flag: &str) -> Option { + args.iter().position(|arg| arg == flag).and_then(|index| args.get(index + 1)).cloned() + } + + /// Posts one JSON-RPC message to the MCP server named in the MCP config file, as Claude Code does. + pub(crate) async fn post_to_mcp(config: &Path, token: Option<&str>, message: Value) -> (u16, Value) { + let config: Value = serde_json::from_str(&std::fs::read_to_string(config).unwrap()).unwrap(); + let server = &config["mcpServers"][MCP_SERVER]; + let url = server["url"].as_str().unwrap().to_owned(); + let authorization = match token { + Some(token) => format!("Bearer {token}"), + None => server["headers"]["Authorization"].as_str().unwrap().to_owned(), + }; + tokio::task::spawn_blocking(move || { + let mut response = crate::speech::agent(Duration::from_secs(5)) + .post(&url) + .header("Authorization", &authorization) + .header("Content-Type", "application/json") + .send(message.to_string().as_bytes()) + .unwrap(); + let status = response.status().as_u16(); + let body = response.body_mut().read_to_string().unwrap_or_default(); + (status, serde_json::from_str(&body).unwrap_or(Value::Null)) + }) + .await + .unwrap() + } + + fn fake_claude(script: Shared, launch: &Launch) -> Pipes { + script.lock().unwrap().launches.push((launch.args.clone(), launch.env.clone())); + let mcp_config = PathBuf::from(argument(&launch.args, "--mcp-config").unwrap()); + let (daemon_side, fake_side) = tokio::io::duplex(1 << 16); + let (daemon_read, daemon_write) = tokio::io::split(daemon_side); + tokio::spawn(async move { + let (fake_read, mut out) = tokio::io::split(fake_side); + let mut lines = BufReader::new(fake_read).lines(); + while let Ok(Some(line)) = lines.next_line().await { + script.lock().unwrap().received.push(serde_json::from_str(&line).unwrap()); + let steps = script.lock().unwrap().turns.pop_front(); + let steps = steps.unwrap_or_else(|| vec![Step::Init(API_KEY_SOURCE), Step::Result(false, "")]); + for (number, step) in steps.into_iter().enumerate() { + let line = match step { + Step::Init(source) => json!({"type": "system", "subtype": "init", "apiKeySource": source, "tools": [WEB_SEARCH]}), + Step::Call(tool, arguments) => { + let name = format!("mcp__{MCP_SERVER}__{tool}"); + let call = json!({"type": "assistant", "message": {"content": [{"type": "tool_use", "id": number, "name": name, "input": arguments}]}}); + out.write_all(format!("{call}\n").as_bytes()).await.unwrap(); + let message = json!({"jsonrpc": "2.0", "id": number, "method": "tools/call", "params": {"name": tool, "arguments": arguments}}); + let answer = post_to_mcp(&mcp_config, None, message).await; + script.lock().unwrap().answers.push(answer); + continue; + } + Step::Hold => { + let hold = script.lock().unwrap().hold.clone().unwrap(); + hold.notified().await; + continue; + } + Step::Result(is_error, text) => { + json!({"type": "result", "subtype": "success", "is_error": is_error, "result": text}) + } + Step::Exit => return, + }; + out.write_all(format!("{line}\n").as_bytes()).await.unwrap(); + } + } + }); + Pipes { reader: Box::new(daemon_read), writer: Box::new(daemon_write), child: None } + } + + fn launches(script: &Shared) -> Vec { + script.lock().unwrap().launches.clone() + } + + async fn say(brain: &ClaudeBrain, key: &str, text: &str) -> Result, ResidentError> { + let mut replies = Vec::new(); + brain.say_streaming(key, "You are the buddy of VibeBuddy.", &robot(), text, &mut |reply| replies.push(reply.to_owned())).await?; + Ok(replies) + } + + #[tokio::test] + async fn the_first_command_starts_one_session_with_only_our_tools_and_the_users_key() { + use std::os::unix::fs::PermissionsExt; + let dir = temp_dir("claude-start"); + let script = script(); + let brain = brain(&dir, &script, Arc::default()); + say(&brain, "sk-ant-test", "hello").await.unwrap(); + + let (args, env) = launches(&script).remove(0); + let flags = |name: &str| args.iter().any(|arg| arg == name); + for flag in ["-p", "--verbose", "--strict-mcp-config", "--disable-slash-commands"] { + assert!(flags(flag), "missing {flag}: {args:?}"); + } + assert!(!flags("--bare"), "bare mode takes WebSearch away"); + assert_eq!(argument(&args, "--input-format").as_deref(), Some("stream-json")); + assert_eq!(argument(&args, "--output-format").as_deref(), Some("stream-json")); + assert_eq!(argument(&args, "--model").as_deref(), Some("claude-haiku-4-5")); + assert_eq!(argument(&args, "--tools").as_deref(), Some("WebSearch")); + assert_eq!(argument(&args, "--permission-mode").as_deref(), Some("dontAsk")); + assert_eq!(argument(&args, "--permission-prompts").as_deref(), Some("none")); + assert_eq!(argument(&args, "--system-prompt-snapshot").as_deref(), Some("off")); + assert_eq!( + argument(&args, "--allowedTools").as_deref(), + Some("WebSearch,mcp__vibebuddy__speak,mcp__vibebuddy__list_activities") + ); + let denied = argument(&args, "--disallowedTools").unwrap(); + for tool in ["Bash", "Edit", "Write", "Read", "WebFetch"] { + assert!(denied.split(',').any(|denied| denied == tool), "{tool} isn't denied: {denied}"); + } + assert_eq!(argument(&args, "--settings").as_deref(), Some(r#"{"disableAllHooks":true}"#)); + assert_eq!(argument(&args, "--system-prompt").as_deref(), Some("You are the buddy of VibeBuddy.")); + let session = argument(&args, "--session-id").unwrap(); + assert_eq!(session.len(), 36); + assert_eq!(&session[14..15], "4", "a version 4 UUID"); + assert!(!flags("--resume")); + + let variable = |name: &str| env.iter().find(|(key, _)| key == name).map(|(_, value)| value.clone()); + let config_dir = dir.join("resident-claude"); + assert_eq!(variable("CLAUDE_CONFIG_DIR"), Some(config_dir.display().to_string())); + assert_eq!(variable("ANTHROPIC_API_KEY").as_deref(), Some("sk-ant-test")); + assert_eq!(variable("ANTHROPIC_BASE_URL"), None); + assert_eq!(std::fs::metadata(&config_dir).unwrap().permissions().mode() & 0o777, 0o700); + let mcp_config = PathBuf::from(argument(&args, "--mcp-config").unwrap()); + assert_eq!(mcp_config, config_dir.join("mcp.json")); + assert_eq!(std::fs::metadata(&mcp_config).unwrap().permissions().mode() & 0o777, 0o600); + let config: Value = serde_json::from_str(&std::fs::read_to_string(&mcp_config).unwrap()).unwrap(); + assert_eq!(config["mcpServers"]["vibebuddy"]["type"], "http"); + assert!(config["mcpServers"]["vibebuddy"]["url"].as_str().unwrap().starts_with("http://127.0.0.1:")); + assert!(!std::fs::read_to_string(&mcp_config).unwrap().contains("sk-ant-test"), "the key stays out of files"); + + // The voice command goes in as a user message, after a context block with the local time. + let line = script.lock().unwrap().received[0].clone(); + assert_eq!(line["type"], "user"); + let content = line["message"]["content"].as_array().unwrap(); + assert!(content[0]["text"].as_str().unwrap().starts_with("The user's local time: ")); + assert_eq!(content[1], json!({"type": "text", "text": "hello"})); + let saved: Value = serde_json::from_str(&std::fs::read_to_string(dir.join(STATE_FILE)).unwrap()).unwrap(); + assert_eq!(saved["session_id"], session.as_str()); + assert_eq!(brain.session_id().as_deref(), Some(session.as_str())); + + // One process for good: the next command goes to the same one. + say(&brain, "sk-ant-test", "again").await.unwrap(); + assert_eq!(launches(&script).len(), 1); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_spoken_reply_is_handed_over_when_speak_is_called_not_when_the_turn_ends() { + let dir = temp_dir("claude-stream"); + let script = script(); + let hold = Arc::new(tokio::sync::Notify::new()); + script.lock().unwrap().hold = Some(hold.clone()); + script.lock().unwrap().turns.push_back(vec![ + Step::Init(API_KEY_SOURCE), + Step::Call("speak", json!({"text": " On it. "})), + Step::Hold, + Step::Call("speak", json!({"text": "Done."})), + Step::Result(false, ""), + ]); + let brain = Arc::new(brain(&dir, &script, Arc::default())); + let (sender, mut receiver) = mpsc::unbounded_channel(); + let turn = tokio::spawn({ + let brain = brain.clone(); + async move { + brain.say_streaming("key", "instructions", &robot(), "hi", &mut |reply| sender.send(reply.to_owned()).unwrap()).await + } + }); + assert_eq!(receiver.recv().await.as_deref(), Some("On it.")); + assert!(!turn.is_finished(), "the turn is still running"); + hold.notify_one(); + assert_eq!(turn.await.unwrap().unwrap(), brain.session_id().unwrap()); + assert_eq!(receiver.recv().await.as_deref(), Some("Done."), "said just before the result, still heard"); + let answers = script.lock().unwrap().answers.clone(); + assert_eq!(answers[0].0, 200); + assert_eq!(answers[0].1["result"], json!({"content": [{"type": "text", "text": "spoken"}], "isError": false})); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn list_activities_reads_the_tracker_and_other_tools_are_refused() { + let dir = temp_dir("claude-tools"); + let mut tracker = ActivityTracker::default(); + let id = ActivityId { session_id: "s".to_owned(), key: "s:t".to_owned() }; + tracker.observe(&id, "CX:REVIEW", ActivityStatus::Working); + let script = script(); + script.lock().unwrap().turns.push_back(vec![ + Step::Init(API_KEY_SOURCE), + Step::Call("list_activities", json!({})), + Step::Call("search_memories", json!({"query": "cat"})), + Step::Call("run_shell", json!({"command": "ls"})), + Step::Call("speak", json!({"text": ""})), + Step::Result(false, ""), + ]); + let brain = brain(&dir, &script, Arc::new(Mutex::new(tracker))); + assert_eq!(say(&brain, "key", "what's up?").await.unwrap(), Vec::::new()); + let answers = script.lock().unwrap().answers.clone(); + let text = |index: usize| answers[index].1["result"]["content"][0]["text"].as_str().unwrap().to_owned(); + let listed: Value = serde_json::from_str(&text(0)).unwrap(); + assert_eq!(listed["activities"][0]["title"], "CX:REVIEW"); + assert!(listed["recently_ended"].is_array() && listed["today"].is_object()); + assert_eq!(answers[1].1["result"]["isError"], true, "Codex memories aren't Claude's"); + assert_eq!(answers[2].1["result"]["isError"], true); + assert_eq!(text(2), "unknown tool run_shell"); + assert_eq!(answers[3].1["result"]["isError"], true, "an empty reply isn't spoken"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn the_mcp_server_answers_only_with_its_token_and_lists_only_our_tools() { + let dir = temp_dir("claude-mcp"); + let brain = brain(&dir, &script(), Arc::default()); + say(&brain, "key", "hi").await.unwrap(); + let config = dir.join("resident-claude/mcp.json"); + + let (status, _) = post_to_mcp(&config, Some("wrong"), json!({"jsonrpc": "2.0", "id": 1, "method": "tools/list"})).await; + assert_eq!(status, 401); + let initialize = json!({"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2025-06-18", "capabilities": {}, "clientInfo": {"name": "claude-code", "version": "2"}}}); + let (status, answer) = post_to_mcp(&config, None, initialize).await; + assert_eq!(status, 200); + assert_eq!(answer["result"]["protocolVersion"], "2025-06-18"); + assert_eq!(answer["result"]["capabilities"], json!({"tools": {}})); + let (status, _) = post_to_mcp(&config, None, json!({"jsonrpc": "2.0", "method": "notifications/initialized"})).await; + assert_eq!(status, 202); + let (_, listed) = post_to_mcp(&config, None, json!({"jsonrpc": "2.0", "id": 2, "method": "tools/list"})).await; + let names: Vec<&str> = listed["result"]["tools"].as_array().unwrap().iter().map(|tool| tool["name"].as_str().unwrap()).collect(); + assert_eq!(names, ["speak", "list_activities"]); + assert!(listed["result"]["tools"][0]["inputSchema"]["properties"]["text"].is_object()); + let (_, unknown) = post_to_mcp(&config, None, json!({"jsonrpc": "2.0", "id": 3, "method": "resources/list"})).await; + assert_eq!(unknown["error"]["code"], -32601); + // Outside a turn a spoken reply has nowhere to go; it is answered but dropped. + let speak = json!({"jsonrpc": "2.0", "id": 4, "method": "tools/call", "params": {"name": "speak", "arguments": {"text": "Hi."}}}); + let (_, spoken) = post_to_mcp(&config, None, speak).await; + assert_eq!(spoken["result"]["isError"], false); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_later_start_resumes_the_session_and_one_whose_transcript_is_gone_is_replaced() { + let dir = temp_dir("claude-resume"); + let first = script(); + say(&brain(&dir, &first, Arc::default()), "key", "hi").await.unwrap(); + let session = argument(&launches(&first)[0].0, "--session-id").unwrap(); + + // Claude Code wrote the transcript under the working directory's project folder. + let project = dir.join("resident-claude/projects/-state-resident"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join(format!("{session}.jsonl")), "").unwrap(); + let second = script(); + let restarted = brain(&dir, &second, Arc::default()); + assert_eq!(restarted.session_id().as_deref(), Some(session.as_str()), "read back from the state file"); + say(&restarted, "key", "hi").await.unwrap(); + let args = &launches(&second)[0].0; + assert_eq!(argument(args, "--resume").as_deref(), Some(session.as_str())); + assert!(!args.iter().any(|arg| arg == "--session-id")); + + std::fs::remove_dir_all(dir.join("resident-claude/projects")).unwrap(); + let third = script(); + let replaced = brain(&dir, &third, Arc::default()); + say(&replaced, "key", "hi").await.unwrap(); + let new = argument(&launches(&third)[0].0, "--session-id").unwrap(); + assert_ne!(new, session); + assert_eq!(replaced.session_id().as_deref(), Some(new.as_str())); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_new_character_resumes_the_same_session_with_its_instructions_and_a_note() { + let dir = temp_dir("claude-character"); + let script = script(); + let brain = brain(&dir, &script, Arc::default()); + async fn say_as(brain: &ClaudeBrain, character: &str, text: &str) { + let persona = Persona::of(&crate::resident::tests::wearing(character)); + let instructions = crate::resident::instructions(&persona, None, crate::resident::Brain::Claude); + brain.say_streaming("key", &instructions, &persona, text, &mut |_| {}).await.unwrap(); + } + let texts = |script: &Shared, index: usize| -> Vec { + let message = script.lock().unwrap().received[index].clone(); + let content = message["message"]["content"].as_array().unwrap().clone(); + content.iter().skip(1).map(|block| block["text"].as_str().unwrap().to_owned()).collect() + }; + + say_as(&brain, "ada", "hi").await; + let session = argument(&launches(&script)[0].0, "--session-id").unwrap(); + let project = dir.join("resident-claude/projects/-state-resident"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join(format!("{session}.jsonl")), "").unwrap(); + say_as(&brain, "ada", "again").await; + assert_eq!(launches(&script).len(), 1, "the same Character keeps the process"); + assert_eq!(texts(&script, 0), ["hi"], "the first command has nothing to switch from"); + + say_as(&brain, "wanwanxiaohe", "你好").await; + let launched = launches(&script); + assert_eq!(launched.len(), 2, "--system-prompt is per process"); + assert_eq!(argument(&launched[1].0, "--resume").as_deref(), Some(session.as_str()), "the same session, never a fork"); + let prompt = argument(&launched[1].0, "--system-prompt").unwrap(); + assert!(prompt.contains("Right now you are Xiaohe (小何)") && prompt.contains("always reply in Chinese"), "{prompt}"); + let switched = texts(&script, 2); + assert!(switched[0].starts_with(CONTEXT_PREFIX) && switched[0].contains("from now on you are Xiaohe (小何)"), "{switched:?}"); + assert_eq!(switched[1], "你好"); + + say_as(&brain, "wanwanxiaohe", "还在吗").await; + assert_eq!(launches(&script).len(), 2); + assert_eq!(texts(&script, 3), ["还在吗"], "the note goes once"); + // Whom the session spoke as is kept across a restart. + assert!(std::fs::read_to_string(dir.join(STATE_FILE)).unwrap().contains("\"wanwanxiaohe\"")); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_credential_other_than_the_api_key_stops_the_process() { + let dir = temp_dir("claude-oauth"); + let script = script(); + script.lock().unwrap().turns.push_back(vec![Step::Init("none"), Step::Result(false, "")]); + let brain = brain(&dir, &script, Arc::default()); + let error = say(&brain, "key", "hi").await.unwrap_err(); + assert!(matches!(&error, ResidentError::Rpc(message) if message.contains("not the Anthropic API key")), "{error}"); + say(&brain, "key", "hi").await.unwrap(); + assert_eq!(launches(&script).len(), 2, "the refused process was dropped"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn an_error_result_fails_the_turn_and_a_dead_process_is_started_again() { + let dir = temp_dir("claude-errors"); + let script = script(); + script.lock().unwrap().turns.extend([ + vec![Step::Init(API_KEY_SOURCE), Step::Result(true, "Invalid API key · Fix external API key")], + vec![Step::Exit], + ]); + let brain = brain(&dir, &script, Arc::default()); + let error = say(&brain, "key", "one").await.unwrap_err(); + assert_eq!(error, ResidentError::TurnEnded("Invalid API key · Fix external API key".to_owned())); + assert!(matches!(say(&brain, "key", "two").await, Err(ResidentError::Rpc(_)))); + assert_eq!(launches(&script).len(), 1, "an error result leaves the process running"); + say(&brain, "key", "three").await.unwrap(); + assert_eq!(launches(&script).len(), 2, "a process that died is started again"); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test] + async fn a_new_key_starts_a_new_process_and_stop_drops_it() { + let dir = temp_dir("claude-key"); + let script = script(); + let brain = brain(&dir, &script, Arc::default()); + say(&brain, "old", "hi").await.unwrap(); + say(&brain, "new", "hi").await.unwrap(); + let keys: Vec = launches(&script) + .iter() + .map(|(_, env)| env.iter().find(|(name, _)| name == "ANTHROPIC_API_KEY").unwrap().1.clone()) + .collect(); + assert_eq!(keys, ["old", "new"]); + brain.stop().await; + say(&brain, "new", "hi").await.unwrap(); + assert_eq!(launches(&script).len(), 3); + let _ = std::fs::remove_dir_all(dir); + } + + #[tokio::test(start_paused = true)] + async fn a_turn_that_takes_too_long_stops_the_process() { + let dir = temp_dir("claude-timeout"); + let script = script(); + script.lock().unwrap().hold = Some(Arc::new(tokio::sync::Notify::new())); + script.lock().unwrap().turns.push_back(vec![Step::Init(API_KEY_SOURCE), Step::Hold]); + let brain = brain(&dir, &script, Arc::default()); + assert_eq!(say(&brain, "key", "hi").await.unwrap_err(), ResidentError::Timeout); + say(&brain, "key", "hi").await.unwrap(); + assert_eq!(launches(&script).len(), 2); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn the_history_is_read_from_the_transcript_in_time_order() { + let line = |time: &str, entry: Value| { + let mut entry = entry; + entry["timestamp"] = json!(time); + entry.to_string() + }; + let speak = |text: &str| json!({"type": "tool_use", "name": "mcp__vibebuddy__speak", "input": {"text": text}}); + let transcript = [ + // Written out of order, as Claude Code does. + line("2026-10-14T07:00:02Z", json!({"type": "assistant", "message": {"content": [speak("One task."), speak("It's fine.")]}})), + line("2026-10-14T07:00:01Z", json!({"type": "user", "message": {"content": [ + {"type": "text", "text": "The user's local time: Tuesday"}, + {"type": "text", "text": "what's up?"}, + ]}})), + json!({"type": "queue-operation", "operation": "enqueue", "content": "what's up?"}).to_string(), + line("2026-10-14T07:00:03Z", json!({"type": "user", "message": {"content": [{"type": "tool_result", "tool_use_id": "x", "content": "spoken"}]}})), + line("2026-10-14T07:01:00Z", json!({"type": "user", "message": {"content": "and now?"}})), + line("2026-10-14T07:01:01Z", json!({"type": "assistant", "message": {"content": [ + {"type": "tool_use", "name": "mcp__vibebuddy__list_activities", "input": {}}, + {"type": "text", "text": "thinking aloud"}, + ]}})), + line("2026-10-14T07:01:02Z", json!({"type": "assistant", "message": {"content": [speak("Still one.")]}})), + line("2026-10-14T07:01:03Z", json!({"type": "user", "isMeta": true, "message": {"content": "meta"}})), + line("2026-10-14T07:01:04Z", json!({"type": "user", "isSidechain": true, "message": {"content": "sub"}})), + "not json".to_owned(), + ] + .join("\n"); + assert_eq!( + exchanges_of(&transcript), + [ + Exchange { said: "what's up?".to_owned(), replies: vec!["One task.".to_owned(), "It's fine.".to_owned()] }, + Exchange { said: "and now?".to_owned(), replies: vec!["Still one.".to_owned()] }, + ] + ); + + let dir = temp_dir("claude-history"); + let brain = brain(&dir, &script(), Arc::default()); + assert_eq!(brain.history(5), [], "no session yet"); + *brain.session_id.lock().unwrap() = Some("abc".to_owned()); + let project = dir.join("resident-claude/projects/-state-resident"); + std::fs::create_dir_all(&project).unwrap(); + std::fs::write(project.join("abc.jsonl"), &transcript).unwrap(); + assert_eq!(brain.history(1), [Exchange { said: "and now?".to_owned(), replies: vec!["Still one.".to_owned()] }]); + let _ = std::fs::remove_dir_all(dir); + } + + #[test] + fn the_child_gets_none_of_the_daemons_claude_or_anthropic_variables() { + for name in ["CLAUDE_CODE_OAUTH_TOKEN", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_API_KEY", "ANTHROPIC_BASE_URL", "CLAUDE_CODE_USE_BEDROCK", "CLAUDECODE", "ANTHROPIC_PROFILE", "CLAUDE_CONFIG_DIR"] { + assert!(withheld(name), "{name}"); + } + for name in ["HOME", "PATH", "LANG", "VIBEBUDDY_CLAUDE_BASE_URL"] { + assert!(!withheld(name), "{name}"); + } + let env = child_env(Path::new("/state/resident-claude"), "sk", Some("http://127.0.0.1:9".to_owned())); + assert!(env.contains(&("ANTHROPIC_BASE_URL".to_owned(), "http://127.0.0.1:9".to_owned()))); + assert!(env.contains(&("CLAUDE_CODE_DISABLE_AUTO_MEMORY".to_owned(), "1".to_owned()))); + assert!(!child_env(Path::new("/s"), "sk", Some(String::new())).iter().any(|(name, _)| name == "ANTHROPIC_BASE_URL")); + } + + #[test] + fn private_files_are_written_0600_even_over_a_wider_one() { + use std::os::unix::fs::PermissionsExt; + let dir = temp_dir("claude-private"); + let path = dir.join("secret.json"); + std::fs::write(&path, "old").unwrap(); + std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); + write_private(&path, "new").unwrap(); + assert_eq!(std::fs::read_to_string(&path).unwrap(), "new"); + assert_eq!(std::fs::metadata(&path).unwrap().permissions().mode() & 0o777, 0o600); + assert!(!path.with_extension("tmp").exists()); + let _ = std::fs::remove_dir_all(dir); + } +} diff --git a/daemon/src/serial_transport.rs b/daemon/src/serial_transport.rs index b0a65868..cf8843e5 100644 --- a/daemon/src/serial_transport.rs +++ b/daemon/src/serial_transport.rs @@ -863,9 +863,10 @@ fn process_device_line(line: &[u8], firmware: &mut Firmware, event_sender: &mpsc } else { let line = String::from_utf8_lossy(line).into_owned(); // Screenshot rows (hundreds of them) stay out of the log; their BEGIN and END lines and every other - // diagnostic line are still logged, so a screenshot that never finishes shows where it stopped. + // diagnostic line are still logged, so a screenshot that never finishes shows where it stopped. So do the + // lines of recorded audio, which carry what the user said. let screenshot_row = line.starts_with("SHOT ") && !line.starts_with("SHOT BEGIN") && line != "SHOT END"; - if !screenshot_row && !line.starts_with("ECHO ") { + if !screenshot_row && !line.starts_with("ECHO ") && !line.starts_with("VOICE DATA ") { info!(message = %line, "device message"); } DeviceMessage::Line(line) diff --git a/daemon/src/speech.rs b/daemon/src/speech.rs new file mode 100644 index 00000000..8830c054 --- /dev/null +++ b/daemon/src/speech.rs @@ -0,0 +1,455 @@ +//! Speech recognition for voice commands (ADR-0017): a recorded turn, as a WAV file, in; its text out. +//! +//! Behind [`Recognizer`] so the backend can change. In the order they are picked: +//! +//! - The user's own recognizer: any OpenAI-compatible transcription endpoint (`VIBEBUDDY_STT_BASE_URL`, +//! `VIBEBUDDY_STT_API_KEY`, `VIBEBUDDY_STT_MODEL`). +//! - The voice service in `vibebuddy-cloud`, `POST {base}/v0/voice/transcribe`, at the base spoken replies use +//! ([`crate::synthesis::service_base`]): staging outside a release, none in a release until accounts come. +//! - Cloudflare Workers AI's Whisper (`CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_AI_TOKEN`), for development; picked only +//! by name now that the voice service recognizes. +//! +//! A setting comes from the environment, else from `speech.json` in the config directory (the user's own +//! recognizer, which an App setting will write later: a file of its own, mode 0600, because `PUT /v1/config` +//! replaces the whole config and an older App would wipe a key it doesn't know), else from a `KEY=VALUE` file named +//! by `VIBEBUDDY_STT_ENV_FILE` (development credentials). All are read at start. Credentials are never logged. + +use std::collections::HashMap; +use std::path::Path; +use std::sync::Arc; +use std::time::Duration; + +use base64::Engine; +use serde_json::{Value, json}; +use tracing::{info, warn}; + +/// Picks the backend: `service`, `cloudflare`, `openai` or `off`. Unset, the user's own recognizer if set, else the +/// voice service. +pub const BACKEND_VARIABLE: &str = "VIBEBUDDY_STT"; +/// A `KEY=VALUE` file the other variables can come from, so credentials stay out of the daemon's environment. +pub const ENV_FILE_VARIABLE: &str = "VIBEBUDDY_STT_ENV_FILE"; +/// The language spoken, as an ISO 639-1 code (`zh`, `en`); unset (or `auto`), the recognizer detects it. +pub const LANGUAGE_VARIABLE: &str = "VIBEBUDDY_STT_LANGUAGE"; +/// The user's own recognizer, in the config directory: `{"base_url","api_key","model","language"}`. +pub const SETTINGS_FILE: &str = "speech.json"; +const CLOUDFLARE_ACCOUNT: &str = "CLOUDFLARE_ACCOUNT_ID"; +const CLOUDFLARE_TOKEN: &str = "CLOUDFLARE_AI_TOKEN"; +const OWN_BASE_URL: &str = "VIBEBUDDY_STT_BASE_URL"; +const OWN_API_KEY: &str = "VIBEBUDDY_STT_API_KEY"; +const OWN_MODEL: &str = "VIBEBUDDY_STT_MODEL"; +const CLOUDFLARE_MODEL: &str = "@cf/openai/whisper-large-v3-turbo"; +/// A 30 s recording is under 1.5 MB of WAV; Workers AI took 3.6 to 4.4 s for 5 s of it. +const TIMEOUT: Duration = Duration::from_secs(30); +const MAX_RESPONSE: u64 = 1024 * 1024; + +/// What was heard. +#[derive(Debug, Default, PartialEq)] +pub struct Transcript { + /// Empty when nothing was said. + pub text: String, + /// The language the recognizer detected, when it says. + pub language: Option, +} + +/// Turns one recording into text. Blocking: it is called on a blocking thread. +pub trait Recognizer: Send + Sync { + /// `wav` is a whole 16-bit PCM WAV file. + fn recognize(&self, wav: &[u8]) -> Result; +} + +/// An HTTP request, built apart from sending it so the tests can check it. +#[derive(Debug)] +pub struct HttpRequest { + pub url: String, + pub headers: Vec<(String, String)>, + pub body: Vec, +} + +/// Whisper on Cloudflare Workers AI. +pub struct Cloudflare { + account_id: String, + token: String, + language: Option, +} + +impl Cloudflare { + fn request(&self, wav: &[u8]) -> HttpRequest { + let mut body = json!({"audio": base64::engine::general_purpose::STANDARD.encode(wav)}); + if let Some(language) = &self.language { + body["language"] = json!(language); + } + HttpRequest { + url: format!("https://api.cloudflare.com/client/v4/accounts/{}/ai/run/{CLOUDFLARE_MODEL}", self.account_id), + headers: vec![ + ("Authorization".to_owned(), format!("Bearer {}", self.token)), + ("Content-Type".to_owned(), "application/json".to_owned()), + ], + body: body.to_string().into_bytes(), + } + } + + fn parse(status: u16, body: &str) -> Result { + let parsed: Value = serde_json::from_str(body).map_err(|_| format!("Workers AI answered {status} with no JSON"))?; + if let Some(text) = parsed.pointer("/result/text").and_then(Value::as_str) { + let language = parsed.pointer("/result/transcription_info/language").and_then(Value::as_str); + return Ok(Transcript { text: text.trim().to_owned(), language: language.map(str::to_owned) }); + } + let message = parsed.pointer("/errors/0/message").and_then(Value::as_str).unwrap_or("no text in the answer"); + Err(format!("Workers AI answered {status}: {message}")) + } +} + +impl Recognizer for Cloudflare { + fn recognize(&self, wav: &[u8]) -> Result { + let (status, body) = send(self.request(wav))?; + Self::parse(status, &body) + } +} + +/// The voice service: the WAV as the body, `{"text","language"}` back. It tries Groq, then Workers AI. +pub struct VoiceService { + base_url: String, + language: Option, +} + +impl VoiceService { + fn request(&self, wav: &[u8]) -> HttpRequest { + let query = self.language.as_ref().map(|language| format!("?language={language}")).unwrap_or_default(); + HttpRequest { + url: format!("{}/v0/voice/transcribe{query}", self.base_url.trim_end_matches('/')), + headers: vec![("Content-Type".to_owned(), "audio/wav".to_owned())], + body: wav.to_vec(), + } + } + + fn parse(status: u16, body: &str) -> Result { + if (200..300).contains(&status) + && let Ok(parsed) = serde_json::from_str::(body) + && let Some(text) = parsed.get("text").and_then(Value::as_str) + { + let language = parsed.get("language").and_then(Value::as_str).map(str::to_owned); + return Ok(Transcript { text: text.trim().to_owned(), language }); + } + Err(format!("the voice service answered {status}: {}", body.trim().chars().take(200).collect::())) + } +} + +impl Recognizer for VoiceService { + fn recognize(&self, wav: &[u8]) -> Result { + let (status, body) = send(self.request(wav))?; + Self::parse(status, &body) + } +} + +/// An OpenAI-compatible `POST {base}/audio/transcriptions`: OpenAI itself, or a server the user runs. +pub struct OpenAiCompatible { + base_url: String, + api_key: Option, + model: String, + language: Option, +} + +impl OpenAiCompatible { + fn request(&self, wav: &[u8], boundary: &str) -> HttpRequest { + let mut body = Vec::new(); + let mut field = |name: &str, extra: &str, value: &[u8]| { + body.extend_from_slice(format!("--{boundary}\r\nContent-Disposition: form-data; name=\"{name}\"{extra}\r\n\r\n").as_bytes()); + body.extend_from_slice(value); + body.extend_from_slice(b"\r\n"); + }; + field("file", "; filename=\"voice.wav\"\r\nContent-Type: audio/wav", wav); + field("model", "", self.model.as_bytes()); + field("response_format", "", b"json"); + if let Some(language) = &self.language { + field("language", "", language.as_bytes()); + } + body.extend_from_slice(format!("--{boundary}--\r\n").as_bytes()); + let mut headers = vec![("Content-Type".to_owned(), format!("multipart/form-data; boundary={boundary}"))]; + if let Some(key) = &self.api_key { + headers.push(("Authorization".to_owned(), format!("Bearer {key}"))); + } + HttpRequest { url: format!("{}/audio/transcriptions", self.base_url.trim_end_matches('/')), headers, body } + } + + fn parse(status: u16, body: &str) -> Result { + let parsed: Value = serde_json::from_str(body).map_err(|_| format!("the recognizer answered {status} with no JSON"))?; + if (200..300).contains(&status) + && let Some(text) = parsed.get("text").and_then(Value::as_str) + { + let language = parsed.get("language").and_then(Value::as_str).map(str::to_owned); + return Ok(Transcript { text: text.trim().to_owned(), language }); + } + let message = parsed.pointer("/error/message").and_then(Value::as_str).unwrap_or("no text in the answer"); + Err(format!("the recognizer answered {status}: {message}")) + } +} + +impl Recognizer for OpenAiCompatible { + fn recognize(&self, wav: &[u8]) -> Result { + let mut nonce = [0u8; 12]; + getrandom::fill(&mut nonce).map_err(|error| error.to_string())?; + let boundary = format!("vibebuddy-{}", nonce.iter().map(|byte| format!("{byte:02x}")).collect::()); + let (status, body) = send(self.request(wav, &boundary))?; + Self::parse(status, &body) + } +} + +/// An HTTP client on the system's certificate store that hands back every status as an answer: the recognizers +/// here and the voice service in [`crate::synthesis`] share it. +pub fn agent(timeout: Duration) -> ureq::Agent { + ureq::Agent::config_builder() + .timeout_global(Some(timeout)) + .http_status_as_error(false) + .tls_config(ureq::tls::TlsConfig::builder().root_certs(ureq::tls::RootCerts::PlatformVerifier).build()) + .user_agent(concat!("vibebuddyd/", env!("CARGO_PKG_VERSION"))) + .build() + .into() +} + +fn send(request: HttpRequest) -> Result<(u16, String), String> { + let mut builder = agent(TIMEOUT).post(&request.url); + for (name, value) in &request.headers { + builder = builder.header(name, value); + } + // The URL may carry the Cloudflare account id; errors name only the host. + let host = request.url.split('/').nth(2).unwrap_or_default().to_owned(); + let mut response = builder.send(&request.body[..]).map_err(|error| format!("calling {host}: {error}"))?; + let status = response.status().as_u16(); + let body = response + .body_mut() + .with_config() + .limit(MAX_RESPONSE) + .read_to_string() + .map_err(|error| format!("reading the answer from {host}: {error}"))?; + Ok((status, body)) +} + +/// The recognizer the environment and the settings files configure, if any. +pub fn from_env() -> Option> { + let own = crate::config::config_dir().map(|dir| read_settings(&dir.join(SETTINGS_FILE))).unwrap_or_default(); + let file = std::env::var_os(ENV_FILE_VARIABLE).map(|path| read_env_file(Path::new(&path))).unwrap_or_default(); + let service = crate::synthesis::service_base( + std::env::var(crate::synthesis::BASE_URL_VARIABLE).ok(), + crate::relay::release(), + ); + configure( + &|name| { + std::env::var(name) + .ok() + .or_else(|| own.get(name).cloned()) + .or_else(|| file.get(name).cloned()) + .filter(|value| !value.trim().is_empty()) + }, + service, + ) +} + +type Named = (&'static str, Arc); + +/// `service` is the voice service's base URL, if there is one. +fn configure(lookup: &dyn Fn(&str) -> Option, service: Option) -> Option> { + let language = lookup(LANGUAGE_VARIABLE).filter(|language| language != "auto"); + let shown = language.clone().unwrap_or_else(|| "detected".to_owned()); + let own = || -> Option { + let backend = OpenAiCompatible { + base_url: lookup(OWN_BASE_URL)?, + api_key: lookup(OWN_API_KEY), + model: lookup(OWN_MODEL).unwrap_or_else(|| "whisper-1".to_owned()), + language: language.clone(), + }; + Some(("an OpenAI-compatible endpoint", Arc::new(backend))) + }; + let cloudflare = || -> Option { + let backend = + Cloudflare { account_id: lookup(CLOUDFLARE_ACCOUNT)?, token: lookup(CLOUDFLARE_TOKEN)?, language: language.clone() }; + Some(("Cloudflare Workers AI", Arc::new(backend))) + }; + let service = || -> Option { + Some(("the voice service", Arc::new(VoiceService { base_url: service.clone()?, language: language.clone() }))) + }; + let backend = lookup(BACKEND_VARIABLE); + let chosen = match backend.as_deref() { + Some("off") => return None, + Some("service") => service(), + Some("cloudflare") => cloudflare(), + Some("openai") => own(), + Some(other) => { + warn!(backend = %other, "unknown speech recognizer; expected service, cloudflare, openai or off"); + return None; + } + // The user's own recognizer is their choice; Cloudflare is only there for development. + None => own().or_else(service), + }; + match (&chosen, backend) { + (Some((name, _)), _) => info!(language = %shown, "speech recognition: {name}"), + (None, Some(backend)) => warn!(%backend, "the speech recognizer is missing its settings"), + (None, None) => {} + } + chosen.map(|(_, recognizer)| recognizer) +} + +/// The user's own recognizer from `speech.json`, under the names of the variables that override it. +fn read_settings(path: &Path) -> HashMap { + let Ok(text) = std::fs::read_to_string(path) else { return HashMap::new() }; + let Ok(Value::Object(settings)) = serde_json::from_str::(&text) else { + warn!(path = %path.display(), "the speech settings are not a JSON object; ignored"); + return HashMap::new(); + }; + settings_to_variables(&settings) +} + +fn settings_to_variables(settings: &serde_json::Map) -> HashMap { + [("base_url", OWN_BASE_URL), ("api_key", OWN_API_KEY), ("model", OWN_MODEL), ("language", LANGUAGE_VARIABLE)] + .into_iter() + .filter_map(|(field, variable)| Some((variable.to_owned(), settings.get(field)?.as_str()?.to_owned()))) + .collect() +} + +/// `KEY=VALUE` lines; `export`, quotes, blank lines and `#` comments are allowed. A missing file is logged. +fn read_env_file(path: &Path) -> HashMap { + match std::fs::read_to_string(path) { + Ok(text) => parse_env_file(&text), + Err(error) => { + warn!(path = %path.display(), %error, "cannot read the speech recognizer's settings file"); + HashMap::new() + } + } +} + +fn parse_env_file(text: &str) -> HashMap { + text.lines() + .map(str::trim) + .filter(|line| !line.is_empty() && !line.starts_with('#')) + .filter_map(|line| { + let (key, value) = line.strip_prefix("export ").unwrap_or(line).split_once('=')?; + let value = value.trim(); + let value = value + .strip_prefix('"') + .and_then(|value| value.strip_suffix('"')) + .or_else(|| value.strip_prefix('\'').and_then(|value| value.strip_suffix('\''))) + .unwrap_or(value); + Some((key.trim().to_owned(), value.to_owned())) + }) + .collect() +} + +#[cfg(test)] +mod tests { + use super::*; + + fn lookup(pairs: &'static [(&'static str, &'static str)]) -> impl Fn(&str) -> Option { + move |name| pairs.iter().find(|(key, _)| *key == name).map(|(_, value)| (*value).to_owned()) + } + + #[test] + fn the_cloudflare_request_carries_the_wav_as_base64_and_the_language_only_when_set() { + let backend = Cloudflare { account_id: "acct".to_owned(), token: "tok".to_owned(), language: Some("zh".to_owned()) }; + let request = backend.request(b"RIFFwav"); + assert_eq!(request.url, "https://api.cloudflare.com/client/v4/accounts/acct/ai/run/@cf/openai/whisper-large-v3-turbo"); + assert!(request.headers.contains(&("Authorization".to_owned(), "Bearer tok".to_owned()))); + let body: Value = serde_json::from_slice(&request.body).unwrap(); + assert_eq!(body, json!({"audio": "UklGRndhdg==", "language": "zh"})); + + let backend = Cloudflare { language: None, ..backend }; + let body: Value = serde_json::from_slice(&backend.request(b"x").body).unwrap(); + assert!(body.get("language").is_none()); + } + + #[test] + fn the_cloudflare_answer_gives_its_text_and_language_or_its_error() { + let heard = Cloudflare::parse(200, r#"{"result":{"text":" 你好 ","transcription_info":{"language":"zh"}},"success":true}"#); + assert_eq!(heard.unwrap(), Transcript { text: "你好".to_owned(), language: Some("zh".to_owned()) }); + let error = Cloudflare::parse(401, r#"{"success":false,"errors":[{"code":10000,"message":"Authentication error"}]}"#); + assert_eq!(error.unwrap_err(), "Workers AI answered 401: Authentication error"); + assert!(Cloudflare::parse(502, "").is_err()); + } + + #[test] + fn the_openai_request_is_multipart_with_file_model_and_language() { + let backend = OpenAiCompatible { + base_url: "http://127.0.0.1:8000/v1/".to_owned(), + api_key: Some("sk-test".to_owned()), + model: "whisper-large-v3".to_owned(), + language: Some("zh".to_owned()), + }; + let request = backend.request(b"RIFF\x00\x01", "BOUNDARY"); + assert_eq!(request.url, "http://127.0.0.1:8000/v1/audio/transcriptions"); + assert!(request.headers.contains(&("Content-Type".to_owned(), "multipart/form-data; boundary=BOUNDARY".to_owned()))); + assert!(request.headers.contains(&("Authorization".to_owned(), "Bearer sk-test".to_owned()))); + let mut expected = b"--BOUNDARY\r\nContent-Disposition: form-data; name=\"file\"; filename=\"voice.wav\"\r\nContent-Type: audio/wav\r\n\r\nRIFF\x00\x01\r\n".to_vec(); + expected.extend_from_slice(b"--BOUNDARY\r\nContent-Disposition: form-data; name=\"model\"\r\n\r\nwhisper-large-v3\r\n"); + expected.extend_from_slice(b"--BOUNDARY\r\nContent-Disposition: form-data; name=\"response_format\"\r\n\r\njson\r\n"); + expected.extend_from_slice(b"--BOUNDARY\r\nContent-Disposition: form-data; name=\"language\"\r\n\r\nzh\r\n"); + expected.extend_from_slice(b"--BOUNDARY--\r\n"); + assert_eq!(String::from_utf8_lossy(&request.body), String::from_utf8_lossy(&expected)); + + let keyless = OpenAiCompatible { api_key: None, language: None, ..backend }; + let request = keyless.request(b"", "B"); + assert!(!request.headers.iter().any(|(name, _)| name == "Authorization")); + assert!(!String::from_utf8_lossy(&request.body).contains("language")); + } + + #[test] + fn the_openai_answer_gives_its_text_or_its_error() { + assert_eq!(OpenAiCompatible::parse(200, r#"{"text":"hello there"}"#).unwrap().text, "hello there"); + let error = OpenAiCompatible::parse(400, r#"{"error":{"message":"bad file"}}"#).unwrap_err(); + assert_eq!(error, "the recognizer answered 400: bad file"); + } + + #[test] + fn the_backend_is_chosen_by_name_or_by_which_settings_are_there() { + let service = || Some("https://api-staging.korekore.ai".to_owned()); + assert!(configure(&lookup(&[]), None).is_none(), "a release has no voice service yet"); + assert!(configure(&lookup(&[]), service()).is_some(), "the voice service by default"); + assert!(configure(&lookup(&[("CLOUDFLARE_ACCOUNT_ID", "a"), ("CLOUDFLARE_AI_TOKEN", "t")]), None).is_none(), "Cloudflare only by name"); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "cloudflare"), ("CLOUDFLARE_ACCOUNT_ID", "a")]), service()).is_none(), "a token is needed too"); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "cloudflare"), ("CLOUDFLARE_ACCOUNT_ID", "a"), ("CLOUDFLARE_AI_TOKEN", "t")]), None).is_some()); + assert!(configure(&lookup(&[("VIBEBUDDY_STT_BASE_URL", "http://localhost:8000/v1")]), None).is_some()); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "off")]), service()).is_none()); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "openai")]), service()).is_none()); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "service")]), None).is_none()); + assert!(configure(&lookup(&[("VIBEBUDDY_STT", "whisper.cpp")]), service()).is_none()); + } + + #[test] + fn the_users_own_recognizer_wins_over_the_voice_service() { + let both = configure(&lookup(&[("VIBEBUDDY_STT_BASE_URL", "http://127.0.0.1:9/v1")]), Some("http://127.0.0.1:8/".to_owned())).unwrap(); + // Nothing listens on ports 8 or 9, so the error names the endpoint that was chosen. + let error = both.recognize(b"").unwrap_err(); + assert!(error.contains("127.0.0.1:9"), "{error}"); + } + + #[test] + fn the_voice_service_gets_the_wav_as_the_body_and_the_language_in_the_query() { + let service = VoiceService { base_url: "https://api-staging.korekore.ai/".to_owned(), language: Some("zh".to_owned()) }; + let request = service.request(b"RIFFwav"); + assert_eq!(request.url, "https://api-staging.korekore.ai/v0/voice/transcribe?language=zh"); + assert_eq!(request.headers, [("Content-Type".to_owned(), "audio/wav".to_owned())]); + assert_eq!(request.body, b"RIFFwav"); + let detected = VoiceService { language: None, ..service }; + assert_eq!(detected.request(b"").url, "https://api-staging.korekore.ai/v0/voice/transcribe"); + + let heard = VoiceService::parse(200, r#"{"text":" 你好 ","language":"Chinese"}"#).unwrap(); + assert_eq!(heard, Transcript { text: "你好".to_owned(), language: Some("Chinese".to_owned()) }); + assert_eq!(VoiceService::parse(502, "recognition failed\n").unwrap_err(), "the voice service answered 502: recognition failed"); + } + + #[test] + fn the_settings_file_maps_onto_the_variables() { + let settings = json!({"base_url": "https://stt.example/v1", "api_key": "k", "model": "m", "extra": 1}); + let variables = settings_to_variables(settings.as_object().unwrap()); + assert_eq!(variables.get("VIBEBUDDY_STT_BASE_URL").map(String::as_str), Some("https://stt.example/v1")); + assert_eq!(variables.get("VIBEBUDDY_STT_API_KEY").map(String::as_str), Some("k")); + assert_eq!(variables.get("VIBEBUDDY_STT_MODEL").map(String::as_str), Some("m")); + assert_eq!(variables.len(), 3); + } + + #[test] + fn the_env_file_takes_plain_quoted_and_exported_lines() { + let parsed = parse_env_file("# dev keys\nA=1\nexport B=\"two\"\n\nC='three'\nnot a line\n"); + assert_eq!(parsed.get("A").map(String::as_str), Some("1")); + assert_eq!(parsed.get("B").map(String::as_str), Some("two")); + assert_eq!(parsed.get("C").map(String::as_str), Some("three")); + assert_eq!(parsed.len(), 3); + } +} diff --git a/daemon/src/synthesis.rs b/daemon/src/synthesis.rs new file mode 100644 index 00000000..5f7bf1e5 --- /dev/null +++ b/daemon/src/synthesis.rs @@ -0,0 +1,312 @@ +//! Speech synthesis for spoken replies (ADR-0017): a reply's text in, an MP3 stream out, piece by piece as the +//! service makes it. +//! +//! Behind [`Synthesizer`] so the backend can change. There is one: the voice service in `vibebuddy-cloud`, +//! `POST {base}/v0/voice/speak` with `{"text","voice"}`, answering a chunked `audio/mpeg` stream (MPEG-2 Layer III, +//! 24 kHz mono, 48 kbps) that starts as soon as there is audio. Builds other than releases use staging; a release +//! has none until the production service is deployed. `VIBEBUDDY_TTS_BASE_URL` sets the base by hand, or `off`; +//! speech recognition ([`crate::speech`]) goes to the same service. + +use std::io::Read; +use std::sync::Arc; +use std::time::Duration; + +use serde_json::json; +use tracing::{info, warn}; + +/// The voice service's base URL, or `off`. Unset: staging, except in a release. +pub const BASE_URL_VARIABLE: &str = "VIBEBUDDY_TTS_BASE_URL"; +pub const STAGING_URL: &str = "https://api-staging.korekore.ai"; +/// The most text the service takes in one request, in UTF-16 code units (JavaScript's `length`). +pub const MAX_TEXT: usize = 1000; +/// A request gets this long for its whole answer; 1000 characters is about a minute of speech, made faster than +/// it plays. +const TIMEOUT: Duration = Duration::from_secs(120); +/// No reply needs more: ten minutes at 48 kbps. +const MAX_AUDIO: usize = 4 * 1024 * 1024; + +/// Turns one reply into speech. Blocking: it is called on a blocking thread. +pub trait Synthesizer: Send + Sync { + /// Calls `on_audio` with each piece of the MP3 as it arrives (not on frame boundaries); `on_audio` returns false + /// to stop. An error after some audio means the stream was cut short. + fn synthesize(&self, text: &str, voice: &str, on_audio: &mut dyn FnMut(&[u8]) -> bool) -> Result<(), String>; +} + +/// The voice service. +pub struct VoiceService { + base_url: String, +} + +impl VoiceService { + fn url(&self) -> String { + format!("{}/v0/voice/speak", self.base_url.trim_end_matches('/')) + } + + fn speak(&self, text: &str, voice: &str, on_audio: &mut dyn FnMut(&[u8]) -> bool) -> Result { + let url = self.url(); + let host = url.split('/').nth(2).unwrap_or_default().to_owned(); + let body = json!({"text": text, "voice": voice}).to_string(); + // The seam for the account session: its Authorization header goes here once the service asks for one. + // One agent for every reply, so its connection to the service is reused: a fresh TLS handshake cost about + // half a second a reply. + static AGENT: std::sync::OnceLock = std::sync::OnceLock::new(); + let mut response = AGENT + .get_or_init(|| crate::speech::agent(TIMEOUT)) + .post(&url) + .header("Content-Type", "application/json") + .send(body.as_bytes()) + .map_err(|error| format!("calling {host}: {error}"))?; + let status = response.status().as_u16(); + if status != 200 { + let answer = response.body_mut().with_config().limit(1024).read_to_string().unwrap_or_default(); + return Err(format!("the voice service answered {status}: {}", answer.trim())); + } + let mut reader = response.into_body().into_reader(); + let mut buffer = [0u8; 4096]; + let mut total = 0; + loop { + // A connection cut before the end of the chunked body is an error here, not an end. + let read = reader.read(&mut buffer).map_err(|error| format!("reading audio from {host}: {error}"))?; + if read == 0 { + return Ok(true); + } + total += read; + if total > MAX_AUDIO { + return Err(format!("{host} sent more audio than any reply needs")); + } + if !on_audio(&buffer[..read]) { + return Ok(false); + } + } + } +} + +impl Synthesizer for VoiceService { + fn synthesize(&self, text: &str, voice: &str, on_audio: &mut dyn FnMut(&[u8]) -> bool) -> Result<(), String> { + // MP3 frames stand alone, so the pieces of a long reply play as one stream. + for piece in split(text, MAX_TEXT) { + if !self.speak(piece, voice, on_audio)? { + break; + } + } + Ok(()) + } +} + +/// `text` in pieces of at most `max` UTF-16 code units: at the end of a sentence where one fits, else at a comma or +/// a space, else at a character. Pieces with nothing to say are left out. +pub fn split(text: &str, max: usize) -> Vec<&str> { + let mut pieces = Vec::new(); + let mut rest = text.trim(); + while !rest.is_empty() { + let mut units = 0; + let mut fits = rest.len(); + let (mut sentence, mut pause) = (None, None); + for (at, character) in rest.char_indices() { + units += character.len_utf16(); + if units > max { + fits = at; + break; + } + let after = at + character.len_utf8(); + if matches!(character, '.' | '!' | '?' | '。' | '！' | '？' | '…' | '\n') { + sentence = Some(after); + } else if matches!(character, ',' | ';' | ':' | '，' | '；' | '：' | '、' | ' ') { + pause = Some(after); + } + } + let cut = match fits { + _ if fits == rest.len() => fits, + // Not even one character fits: it goes alone rather than never. + 0 => rest.chars().next().map_or(0, char::len_utf8), + _ => sentence.or(pause).unwrap_or(fits), + }; + let (piece, after) = rest.split_at(cut); + if piece.chars().any(char::is_alphanumeric) { + pieces.push(piece.trim()); + } + rest = after.trim_start(); + } + pieces +} + +/// Each Character's voice file (`characters//voice.env`), so a spoken reply is in the very voice its lines +/// were made with: `ENGINE=elevenlabs` voices are the ones in our ElevenLabs account, which the voice service uses. +pub(crate) const VOICE_FILES: &[(&str, &str)] = &[ + ("ada", include_str!("../../characters/ada/voice.env")), + ("dave", include_str!("../../characters/dave/voice.env")), + ("hank", include_str!("../../characters/hank/voice.env")), + ("kai", include_str!("../../characters/kai/voice.env")), + ("luna", include_str!("../../characters/luna/voice.env")), + ("mei", include_str!("../../characters/mei/voice.env")), + ("wanwanxiaohe", include_str!("../../characters/wanwanxiaohe/voice.env")), +]; +/// The built-in default Character speaks with ElevenLabs' Jessica (voices/README.md); so does no Character pack, or +/// one these files don't know. +const DEFAULT_VOICE: &str = "elevenlabs:cgSgspJ2msm6clMCkdW9"; +/// ElevenLabs' stock Sarah and George, voices of no Character: one whose voice the service can't speak (a Doubao +/// 2.0 voice, which needs another resource id) gets the one of its `GENDER`, rather than another Character's voice. +const STOCK_FEMALE: &str = "elevenlabs:EXAVITQu4vr4xnSDxMaL"; +const STOCK_MALE: &str = "elevenlabs:JBFqnCBsd6RMkjVDRZzb"; + +/// The voice service's name for `character`'s voice (the id the box reports in `VOICES`), as `:`. +/// The model is multilingual, so a reply in another language than the Character's keeps the Character's voice. +pub fn voice_for(character: Option<&str>) -> String { + let Some(file) = character.and_then(|character| VOICE_FILES.iter().find(|(id, _)| *id == character)) else { + return DEFAULT_VOICE.to_owned(); + }; + let field = |key: &str| voice_field(file.1, key); + match (field("ENGINE"), field("VOICE")) { + (Some("elevenlabs"), Some(voice)) => format!("elevenlabs:{voice}"), + // The service speaks Doubao's 1.0 voices only. + (Some("volc"), Some(voice)) if field("VOLC_RESOURCE_ID").is_none() => format!("volc:{voice}"), + _ if field("GENDER") == Some("male") => STOCK_MALE.to_owned(), + _ => STOCK_FEMALE.to_owned(), + } +} + +/// One `KEY=value` of a `voice.env`. +pub(crate) fn voice_field<'a>(file: &'a str, key: &str) -> Option<&'a str> { + file.lines().find_map(|line| line.strip_prefix(key)?.strip_prefix('=')) +} + +/// The synthesizer the environment configures, if any. +pub fn from_env() -> Option> { + configure(service_base(std::env::var(BASE_URL_VARIABLE).ok(), crate::relay::release())) +} + +/// The voice service's base URL, which speech recognition ([`crate::speech`]) uses too: the variable's, else +/// staging outside a release. None when it is `off`, not http(s), or a release has none. +pub fn service_base(base_url: Option, release: bool) -> Option { + let base_url = match base_url.map(|url| url.trim().to_owned()).filter(|url| !url.is_empty()) { + Some(url) if url == "off" => return None, + Some(url) => url, + None if release => return None, + None => STAGING_URL.to_owned(), + }; + if !base_url.starts_with("https://") && !base_url.starts_with("http://") { + warn!(%base_url, "the voice service's base URL is not http(s); it is not used"); + return None; + } + Some(base_url) +} + +fn configure(base_url: Option) -> Option> { + let base_url = base_url?; + info!(%base_url, "speech synthesis: the voice service"); + Some(Arc::new(VoiceService { base_url })) +} + +#[cfg(test)] +mod tests { + use std::io::Write; + use std::net::TcpListener; + + use super::*; + + #[test] + fn each_character_speaks_in_the_voice_of_its_lines() { + assert_eq!(voice_for(Some("ada")), "elevenlabs:tcSsCJFYrsHpK5rCQZMf"); + assert_eq!(voice_for(Some("hank")), "elevenlabs:U6jH23hcPh0zH17nf3DT"); + assert_eq!(voice_for(Some("wanwanxiaohe")), "volc:zh_female_wanwanxiaohe_moon_bigtts", "a Doubao 1.0 voice"); + assert_eq!(voice_for(Some("robot")), DEFAULT_VOICE); + assert_eq!(voice_for(None), DEFAULT_VOICE, "no Character pack"); + assert_eq!(voice_for(Some("hsiaoyu")), DEFAULT_VOICE, "an old voice pack"); + } + + #[test] + fn long_text_is_split_at_sentences_then_pauses_then_characters() { + assert_eq!(split(" Short one. ", 1000), ["Short one."]); + assert_eq!(split("One. Two! Three?", 10), ["One. Two!", "Three?"]); + assert_eq!(split("第一句。第二句。第三句。", 8), ["第一句。第二句。", "第三句。"]); + assert_eq!(split("one two three four", 9), ["one two", "three", "four"]); + assert_eq!(split("abcdefghij", 4), ["abcd", "efgh", "ij"]); + // A character outside the BMP is two units in JavaScript, which is how the service counts. + assert_eq!(split("𠀀𠀀𠀀", 4), ["𠀀𠀀", "𠀀"]); + assert!(split("😀", 4).is_empty(), "nothing to say"); + assert!(split(" ... ", 10).is_empty(), "nothing to say"); + let long = "This sentence is about forty characters. ".repeat(60); + let pieces = split(&long, MAX_TEXT); + assert!(pieces.len() > 1); + assert!(pieces.iter().all(|piece| piece.encode_utf16().count() <= MAX_TEXT && piece.ends_with('.'))); + } + + #[test] + fn the_service_is_staging_outside_a_release_and_can_be_set_or_switched_off() { + let base = |url: Option<&str>, release| configure(service_base(url.map(str::to_owned), release)).is_some(); + assert!(base(None, false)); + assert!(!base(None, true), "no production voice service yet"); + assert!(base(Some("https://api.korekore.ai"), true)); + assert!(!base(Some("off"), false)); + assert!(!base(Some("api.korekore.ai"), false)); + } + + /// Serves one canned HTTP answer on a local port; returns the base URL and the request it got. + fn serve(answer: Vec) -> (String, std::thread::JoinHandle) { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let base = format!("http://{}", listener.local_addr().unwrap()); + let handle = std::thread::spawn(move || { + let (mut socket, _) = listener.accept().unwrap(); + let mut request = Vec::new(); + let mut buffer = [0u8; 4096]; + while !String::from_utf8_lossy(&request).contains("}") { + let read = socket.read(&mut buffer).unwrap(); + request.extend_from_slice(&buffer[..read]); + } + socket.write_all(&answer).unwrap(); + String::from_utf8_lossy(&request).into_owned() + }); + (base, handle) + } + + fn chunked(pieces: &[&[u8]], finished: bool) -> Vec { + let mut out = b"HTTP/1.1 200 OK\r\nContent-Type: audio/mpeg\r\nTransfer-Encoding: chunked\r\n\r\n".to_vec(); + for piece in pieces { + out.extend_from_slice(format!("{:x}\r\n", piece.len()).as_bytes()); + out.extend_from_slice(piece); + out.extend_from_slice(b"\r\n"); + } + if finished { + out.extend_from_slice(b"0\r\n\r\n"); + } + out + } + + fn collect(base: String) -> (Result<(), String>, Vec) { + let service = VoiceService { base_url: base }; + let mut audio = Vec::new(); + let result = service.synthesize("Hello there.", "en-US-JennyNeural", &mut |piece| { + audio.extend_from_slice(piece); + true + }); + (result, audio) + } + + #[test] + fn the_request_is_json_and_the_audio_comes_through() { + let (base, server) = serve(chunked(&[b"ID3abc", b"\xff\xf3def"], true)); + let (result, audio) = collect(format!("{base}/")); + assert_eq!(result, Ok(())); + assert_eq!(audio, b"ID3abc\xff\xf3def"); + let request = server.join().unwrap(); + assert!(request.starts_with("POST /v0/voice/speak HTTP/1.1\r\n"), "{request}"); + assert!(request.to_ascii_lowercase().contains("content-type: application/json")); + assert!(request.ends_with(r#"{"text":"Hello there.","voice":"en-US-JennyNeural"}"#), "{request}"); + } + + #[test] + fn a_stream_cut_short_or_an_error_status_is_a_failure() { + // The connection drops before the chunked body's end: what came is kept, but it failed. + let (base, server) = serve(chunked(&[b"ID3abc"], false)); + let (result, audio) = collect(base); + server.join().unwrap(); + assert!(result.is_err(), "{result:?}"); + assert_eq!(audio, b"ID3abc"); + + let (base, server) = serve(b"HTTP/1.1 502 Bad Gateway\r\nContent-Length: 15\r\n\r\nupstream failed".to_vec()); + let (result, audio) = collect(base); + server.join().unwrap(); + assert_eq!(result.unwrap_err(), "the voice service answered 502: upstream failed"); + assert!(audio.is_empty()); + } +} diff --git a/daemon/src/voice.rs b/daemon/src/voice.rs new file mode 100644 index 00000000..b1250101 --- /dev/null +++ b/daemon/src/voice.rs @@ -0,0 +1,1202 @@ +//! Voice commands from a box linked over USB or the local network (ADR-0017; docs/protocol.md, "Voice commands +//! (experimental)"). +//! +//! The box records while K0 is held and sends the audio as `VOICE` diagnostic lines: 24 kHz mono IMA ADPCM, each +//! line carrying the coder's state so it decodes on its own. On `VOICE END` the daemon decodes the turn, has it +//! recognised ([`crate::speech`]), sends `voice.heard`, hands the text to the resident session and sends each spoken +//! reply as `voice.reply` the moment the session says it, then `voice.done`. One turn at a time: a turn that ends +//! while another is still running gets `voice.error` `busy`. +//! +//! With a synthesizer ([`crate::synthesis`]), each reply is also spoken: its audio follows its `voice.reply` as +//! `voice.audio` pieces, streamed as the service makes them. Replies are synthesized one after another, in order, +//! on their own task, so a reply's text never waits for an earlier reply's audio; `voice.done` waits for the last +//! reply's audio `end`. + +use std::collections::BTreeMap; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Instant; + +use base64::Engine; +use tokio::task::JoinHandle; +use tracing::{debug, info, warn}; +use vibebuddy_protocol::{Event, MAX_MESSAGE_BYTES}; + +use crate::persona::Wearing; +use crate::resident::Resident; +use crate::serial_transport::{Transport, TransportError}; +use crate::speech::Recognizer; +use crate::synthesis::{self, Synthesizer}; + +pub const SAMPLE_RATE: u32 = 24_000; +/// The most MP3 bytes one `voice.audio` carries: 768 base64 characters. +const AUDIO_BYTES: usize = 768 / 4 * 3; +/// The box stops at 30 s; a little more is kept in case its clock and ours disagree, nothing beyond. +const MAX_SAMPLES: usize = SAMPLE_RATE as usize * 31; +const LINE_PREFIX: &str = "VOICE "; + +// IMA ADPCM, exactly as firmware-rs/core/src/adpcm.rs decodes it; the box encodes with its inverse. +const STEPS: [i32; 89] = [ + 7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 19, 21, 23, 25, 28, 31, 34, 37, 41, 45, 50, 55, 60, 66, 73, 80, 88, 97, 107, 118, 130, + 143, 157, 173, 190, 209, 230, 253, 279, 307, 337, 371, 408, 449, 494, 544, 598, 658, 724, 796, 876, 963, 1060, 1166, + 1282, 1411, 1552, 1707, 1878, 2066, 2272, 2499, 2749, 3024, 3327, 3660, 4026, 4428, 4871, 5358, 5894, 6484, 7132, 7845, + 8630, 9493, 10442, 11487, 12635, 13899, 15289, 16818, 18500, 20350, 22385, 24623, 27086, 29794, 32767, +]; +const INDEX_CHANGE: [i32; 16] = [-1, -1, -1, -1, 2, 4, 6, 8, -1, -1, -1, -1, 2, 4, 6, 8]; + +#[derive(Clone, Copy, Debug, PartialEq)] +pub struct Decoder { + predictor: i32, + index: i32, +} + +impl Decoder { + pub fn new(predictor: i16, index: u8) -> Self { + Self { predictor: predictor.into(), index: i32::from(index).min(88) } + } + + pub fn decode(&mut self, nibble: u8) -> i16 { + let step = STEPS[self.index as usize]; + let mut diff = step >> 3; + if nibble & 4 != 0 { + diff += step; + } + if nibble & 2 != 0 { + diff += step >> 1; + } + if nibble & 1 != 0 { + diff += step >> 2; + } + self.predictor = if nibble & 8 != 0 { self.predictor - diff } else { self.predictor + diff }.clamp(-32768, 32767); + self.index = (self.index + INDEX_CHANGE[(nibble & 15) as usize]).clamp(0, 88); + self.predictor as i16 + } +} + +/// Two samples per byte, low nibble first. +pub fn decode_line(predictor: i16, index: u8, bytes: &[u8]) -> Vec { + let mut decoder = Decoder::new(predictor, index); + bytes.iter().flat_map(|byte| [byte & 15, byte >> 4]).map(|nibble| decoder.decode(nibble)).collect() +} + +#[derive(Debug, PartialEq)] +pub enum VoiceLine { + Start { turn: u32 }, + Data { turn: u32, seq: u32, predictor: i16, index: u8, bytes: Vec }, + End { turn: u32, samples: u32 }, + Cancel { turn: u32 }, +} + +/// None when the line isn't a `VOICE` line; an error when it is one but malformed. +pub fn parse_line(line: &str) -> Option> { + let rest = line.trim().strip_prefix(LINE_PREFIX)?; + let fields: Vec<&str> = rest.split(' ').collect(); + let number = |index: usize| fields.get(index).and_then(|field| field.parse::().ok()); + let parsed = match (fields[0], fields.len()) { + ("START", 2) => number(1).map(|turn| VoiceLine::Start { turn }), + ("END", 3) => number(1).zip(number(2)).map(|(turn, samples)| VoiceLine::End { turn, samples }), + ("CANCEL", 2) => number(1).map(|turn| VoiceLine::Cancel { turn }), + ("DATA", 6) => (|| { + let bytes = base64::engine::general_purpose::STANDARD.decode(fields[5]).ok()?; + let index = fields[4].parse::().ok().filter(|index| *index <= 88)?; + Some(VoiceLine::Data { turn: number(1)?, seq: number(2)?, predictor: fields[3].parse().ok()?, index, bytes }) + })(), + _ => None, + }; + Some(parsed.ok_or_else(|| format!("malformed voice line: {}", shorten(rest)))) +} + +fn shorten(text: &str) -> &str { + let mut end = text.len().min(60); + while !text.is_char_boundary(end) { + end -= 1; + } + &text[..end] +} + +/// One turn's audio as it arrives, each line already decoded, by seq. +struct Recording { + turn: u32, + lines: BTreeMap>, + samples: usize, + /// Answered `busy` at its start: the rest of it is dropped without another answer. + refused: bool, +} + +/// The turn's audio, in order, `samples` long; a lost line becomes silence as long as a line, and audio lost at the +/// end is filled up to `samples`. Returns the seqs that were missing. +fn assemble(lines: &BTreeMap>, samples: usize) -> (Vec, Vec) { + let samples = samples.min(MAX_SAMPLES); + let line_length = lines.values().map(Vec::len).max().unwrap_or(0); + let mut audio = Vec::with_capacity(samples); + let mut missing = Vec::new(); + let mut next = 0; + for (&seq, line) in lines { + for lost in next..seq { + missing.push(lost); + audio.resize(audio.len() + line_length, 0); + } + audio.extend_from_slice(line); + next = seq + 1; + } + while audio.len() < samples && line_length > 0 { + missing.push(next); + next += 1; + audio.resize(audio.len() + line_length, 0); + } + audio.resize(samples, 0); + (audio, missing) +} + +/// A 24 kHz 16-bit mono WAV file. +pub fn wav(samples: &[i16]) -> Vec { + let data = (samples.len() * 2) as u32; + let mut out = Vec::with_capacity(44 + data as usize); + out.extend_from_slice(b"RIFF"); + out.extend_from_slice(&(36 + data).to_le_bytes()); + out.extend_from_slice(b"WAVEfmt "); + out.extend_from_slice(&16u32.to_le_bytes()); + out.extend_from_slice(&1u16.to_le_bytes()); + out.extend_from_slice(&1u16.to_le_bytes()); + out.extend_from_slice(&SAMPLE_RATE.to_le_bytes()); + out.extend_from_slice(&(SAMPLE_RATE * 2).to_le_bytes()); + out.extend_from_slice(&2u16.to_le_bytes()); + out.extend_from_slice(&16u16.to_le_bytes()); + out.extend_from_slice(b"data"); + out.extend_from_slice(&data.to_le_bytes()); + for sample in samples { + out.extend_from_slice(&sample.to_le_bytes()); + } + out +} + +/// `voice.available`: whether the box may treat a long K0 as push-to-talk. +pub fn available_event(on: bool) -> Event { + let mut event = Event::named("voice.available"); + event.extra.insert("on".to_owned(), on.into()); + event +} + +fn turn_event(name: &str, turn: u32) -> Event { + let mut event = Event::named(name); + event.extra.insert("turn".to_owned(), turn.into()); + event +} + +/// An event carrying `text`, cut at a character boundary (with an ellipsis) until the line fits. +fn fitted(text: &str, build: impl Fn(String) -> Event) -> Event { + let mut keep = text.len(); + loop { + let shown = if keep < text.len() { format!("{}…", &text[..keep]) } else { text.to_owned() }; + let event = build(shown); + let size = serde_json::to_vec(&event).map(|bytes| bytes.len()).unwrap_or(0); + if size <= MAX_MESSAGE_BYTES || keep == 0 { + return event; + } + // Every byte of text is at least one byte of JSON, so this many fewer is always enough room. + keep = keep.saturating_sub(size - MAX_MESSAGE_BYTES + '…'.len_utf8()); + while !text.is_char_boundary(keep) { + keep -= 1; + } + } +} + +fn text_event(name: &str, turn: u32, text: &str) -> Event { + fitted(text, |text| { + let mut event = turn_event(name, turn); + event.extra.insert("text".to_owned(), text.into()); + event + }) +} + +fn error_event(turn: u32, reason: &str, message: &str) -> Event { + fitted(message, |message| { + let mut event = turn_event("voice.error", turn); + event.extra.insert("reason".to_owned(), reason.into()); + event.message = Some(message); + event + }) +} + +fn audio_event(turn: u32, reply: u32, seq: u32) -> Event { + let mut event = turn_event("voice.audio", turn); + event.extra.insert("reply".to_owned(), reply.into()); + event.extra.insert("seq".to_owned(), seq.into()); + event +} + +fn send(transport: &dyn Transport, event: Event) { + match event.to_ndjson() { + Ok(frame) => { + if let Err(error) = transport.send(frame) { + warn!(?error, event = %event.event, "a voice event could not be queued for the box"); + } + } + Err(error) => warn!(%error, "failed to encode a voice event"), + } +} + +/// Clears the running flag however the turn ends. +struct Running(Arc); + +impl Drop for Running { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +pub struct Voice { + recognizer: Option>, + resident: Option>, + /// Speaks the replies; without one they stay text. + synthesizer: Option>, + recording: std::sync::Mutex>, + running: Arc, +} + +impl Voice { + pub fn new(recognizer: Option>, resident: Option>) -> Self { + Self { recognizer, resident, synthesizer: None, recording: std::sync::Mutex::new(None), running: Arc::default() } + } + + pub fn with_synthesizer(self, synthesizer: Option>) -> Self { + Self { synthesizer, ..self } + } + + /// Voice is offered when speech recognition is configured and the resident session is on and ready: the user's + /// Codex is installed and logged in. + pub fn available(&self) -> bool { + self.recognizer.is_some() && self.resident.as_ref().is_some_and(|resident| resident.ready()) + } + + /// Takes one `VOICE` line from the box. A turn that ends starts a task, returned so tests can wait for it; + /// its replies are written in the persona and spoken in the voice of what the box `wearing`. + pub fn handle_line(&self, line: &str, transport: &Arc, wearing: &Wearing) -> Option> { + let parsed = match parse_line(line) { + None => return None, + Some(Err(error)) => { + warn!(%error, "ignoring a voice line"); + return None; + } + Some(Ok(parsed)) => parsed, + }; + let mut recording = self.recording.lock().expect("recording lock"); + match parsed { + VoiceLine::Start { turn } => { + if let Some(old) = recording.as_ref().filter(|old| old.turn != turn) { + warn!(old = old.turn, new = turn, "a voice turn started before the last one ended; dropped the last one"); + } + // Refused at once rather than at its end, so the user isn't left talking to a box that won't listen. + let refused = self.running.load(Ordering::Acquire); + if refused { + info!(turn, "a voice turn started while another is still running"); + send(&**transport, error_event(turn, "busy", "another voice command is still running")); + } else { + info!(turn, "voice turn recording"); + } + *recording = Some(Recording { turn, lines: BTreeMap::new(), samples: 0, refused }); + } + VoiceLine::Data { turn, seq, predictor, index, bytes } => { + if recording.as_ref().is_none_or(|current| current.turn != turn) { + // Its START was lost: the audio still counts. + debug!(turn, "voice data without its start"); + *recording = Some(Recording { turn, lines: BTreeMap::new(), samples: 0, refused: false }); + } + let current = recording.as_mut().expect("recording was just made"); + if current.refused { + return None; + } + let line = decode_line(predictor, index, &bytes); + if current.samples + line.len() > MAX_SAMPLES { + debug!(turn, seq, "voice turn over the longest recording; line dropped"); + } else { + current.samples += line.len(); + current.lines.insert(seq, line); + } + } + VoiceLine::Cancel { turn } => { + if recording.as_ref().is_some_and(|current| current.turn == turn) { + *recording = None; + } + info!(turn, "voice turn cancelled by the box"); + } + VoiceLine::End { turn, samples } => { + let taken = recording.take_if(|current| current.turn == turn); + drop(recording); + if taken.as_ref().is_some_and(|taken| taken.refused) { + return None; + } + let speaker = self.synthesizer.clone().map(|synthesizer| (synthesizer, wearing.character.clone())); + return self.end(turn, samples as usize, taken, transport.clone(), speaker, wearing.clone()); + } + } + None + } + + fn end( + &self, + turn: u32, + samples: usize, + recording: Option, + transport: Arc, + speaker: Option<(Arc, Option)>, + wearing: Wearing, + ) -> Option> { + let (Some(recognizer), Some(resident)) = (self.recognizer.clone(), self.resident.clone().filter(|resident| resident.enabled())) + else { + send(&*transport, error_event(turn, "unavailable", "voice commands are off on this computer")); + return None; + }; + let Some(recording) = recording.filter(|recording| !recording.lines.is_empty()) else { + warn!(turn, "a voice turn ended with no audio"); + send(&*transport, error_event(turn, "not_heard", "no audio arrived")); + return None; + }; + if self.running.swap(true, Ordering::AcqRel) { + info!(turn, "a voice turn ended while another is still running"); + send(&*transport, error_event(turn, "busy", "another voice command is still running")); + return None; + } + let running = Running(self.running.clone()); + let (audio, missing) = assemble(&recording.lines, samples); + if !missing.is_empty() { + warn!(turn, lost = missing.len(), seqs = ?missing, "voice lines were lost; filled with silence"); + } + Some(tokio::spawn(async move { + let _running = running; + run_turn(turn, audio, recognizer, resident, transport, speaker, wearing).await; + })) + } +} + +/// Speaks a turn's replies one after another, in the order they were said, streaming each one's audio to the box +/// as it arrives, on a task of its own so the text never waits for it. +struct Speaker { + replies: tokio::sync::mpsc::UnboundedSender, + stopped: Arc, + task: JoinHandle<()>, +} + +impl Speaker { + fn start(turn: u32, synthesizer: Arc, character: Option, transport: Arc) -> Self { + let (replies, mut texts) = tokio::sync::mpsc::unbounded_channel::(); + let stopped = Arc::new(AtomicBool::new(false)); + let task = tokio::spawn({ + let stopped = stopped.clone(); + async move { + let mut reply = 0; + while let Some(text) = texts.recv().await { + let (synthesizer, character, transport, stopped) = + (synthesizer.clone(), character.clone(), transport.clone(), stopped.clone()); + let spoken = tokio::task::spawn_blocking(move || { + speak(turn, reply, &text, character.as_deref(), &*synthesizer, &*transport, &stopped) + }); + if let Err(error) = spoken.await { + warn!(turn, reply, %error, "speech synthesis panicked"); + } + reply += 1; + } + } + }); + Self { replies, stopped, task } + } + + fn say(&self, text: &str) { + let _ = self.replies.send(text.to_owned()); + } + + /// Waits until every reply's audio has been sent. + async fn finish(self) { + drop(self.replies); + let _ = self.task.await; + } + + /// Sends no more audio: the reply being spoken stops at its next piece, and the rest are never started. + async fn stop(self) { + self.stopped.store(true, Ordering::Release); + self.finish().await; + } +} + +/// One reply's audio, as `voice.audio` pieces and then its `end`; a failure ends it early, marked as one, and the +/// box keeps the text. Blocking. +fn speak( + turn: u32, + reply: u32, + text: &str, + character: Option<&str>, + synthesizer: &dyn Synthesizer, + transport: &dyn Transport, + stopped: &AtomicBool, +) { + if stopped.load(Ordering::Acquire) { + return; + } + let voice = synthesis::voice_for(character); + let voice = voice.as_str(); + let started = Instant::now(); + let (mut seq, mut bytes, mut first_byte_ms) = (0, 0, None); + let result = synthesizer.synthesize(text, voice, &mut |piece| { + if stopped.load(Ordering::Acquire) { + return false; + } + first_byte_ms.get_or_insert(started.elapsed().as_millis() as u64); + bytes += piece.len(); + for chunk in piece.chunks(AUDIO_BYTES) { + let mut event = audio_event(turn, reply, seq); + event.extra.insert("data".to_owned(), base64::engine::general_purpose::STANDARD.encode(chunk).into()); + if !send_paced(transport, event, stopped) { + return false; + } + seq += 1; + } + true + }); + if stopped.load(Ordering::Acquire) { + return; + } + let total_ms = started.elapsed().as_millis() as u64; + let mut end = audio_event(turn, reply, seq); + end.extra.insert("end".to_owned(), true.into()); + match result { + Ok(()) => info!(turn, reply, voice, first_byte_ms, total_ms, bytes, "spoken reply synthesized"), + Err(error) => { + warn!(turn, reply, voice, %error, bytes, "speech synthesis failed; the reply stays text"); + end.extra.insert("error".to_owned(), "synthesis".into()); + } + } + send_paced(transport, end, stopped); +} + +/// How long one piece of a reply's audio may wait for room in the link's send queue. +const AUDIO_QUEUE_WAIT: std::time::Duration = std::time::Duration::from_secs(5); + +/// Sends a piece of a reply's audio, waiting while the link's send queue is full: a reply arrives from the voice +/// service far faster than the link drains, and a dropped piece cuts the reply short on the box. Blocking. False when +/// it gave up (stopped, link closed, or no room for too long). +fn send_paced(transport: &dyn Transport, event: Event, stopped: &AtomicBool) -> bool { + let Ok(frame) = event.to_ndjson() else { + warn!(event = %event.event, "a voice event could not be encoded"); + return false; + }; + let started = Instant::now(); + loop { + match transport.send(frame.clone()) { + Ok(()) => return true, + Err(TransportError::QueueFull) if !stopped.load(Ordering::Acquire) && started.elapsed() < AUDIO_QUEUE_WAIT => { + std::thread::sleep(std::time::Duration::from_millis(10)); + } + Err(error) => { + warn!(?error, event = %event.event, "a voice event could not be queued for the box"); + return false; + } + } + } +} + +async fn run_turn( + turn: u32, + audio: Vec, + recognizer: Arc, + resident: Arc, + transport: Arc, + speaker: Option<(Arc, Option)>, + wearing: Wearing, +) { + let started = Instant::now(); + let seconds = audio.len() as f32 / SAMPLE_RATE as f32; + let wav = wav(&audio); + let heard = tokio::task::spawn_blocking(move || recognizer.recognize(&wav)) + .await + .unwrap_or_else(|error| Err(error.to_string())); + let recognised_ms = started.elapsed().as_millis() as u64; + let transcript = match heard { + Ok(transcript) => transcript, + Err(error) => { + warn!(turn, %error, "speech recognition failed"); + send(&*transport, error_event(turn, "unavailable", &error)); + return; + } + }; + if transcript.text.is_empty() { + info!(turn, seconds, recognised_ms, "nothing was heard in the voice turn"); + send(&*transport, error_event(turn, "not_heard", "nothing was recognised")); + return; + } + info!(turn, seconds, recognised_ms, language = transcript.language.as_deref().unwrap_or("unknown"), chars = transcript.text.chars().count(), "voice turn heard"); + send(&*transport, text_event("voice.heard", turn, &transcript.text)); + let speaker = speaker.map(|(synthesizer, character)| Speaker::start(turn, synthesizer, character, transport.clone())); + let mut replies = 0; + let answered = resident + .say_streaming(&transcript.text, &wearing, &mut |reply| { + replies += 1; + info!(turn, elapsed_ms = started.elapsed().as_millis() as u64, "voice reply"); + send(&*transport, text_event("voice.reply", turn, reply)); + if let Some(speaker) = &speaker { + speaker.say(reply); + } + }) + .await; + match answered { + Ok(_) => { + // The turn is over once the last reply has been heard, not just said. + if let Some(speaker) = speaker { + speaker.finish().await; + } + info!(turn, replies, elapsed_ms = started.elapsed().as_millis() as u64, "voice turn done"); + send(&*transport, turn_event("voice.done", turn)); + } + Err(error) => { + if let Some(speaker) = speaker { + speaker.stop().await; + } + warn!(turn, %error, "the resident session failed the voice turn"); + send(&*transport, error_event(turn, "unavailable", &error.to_string())); + } + } +} + +#[cfg(test)] +pub mod tests { + use serde_json::json; + + use super::*; + + /// The lines the box itself sends for a turn of `samples`: made by firmware-core's own voice command code, + /// so these tests hold the daemon to what the box really writes. + pub fn voice_lines(turn: u32, samples: &[i16]) -> Vec { + use vibebuddy_firmware_core::voice_command::{Out, VoiceCommand}; + let mut box_side = VoiceCommand::new(); + box_side.set_available(true); + // Turns count from 1 after each boot. + for _ in 1..turn { + box_side.start(0); + box_side.cancel(); + } + let mut out = vec![box_side.start(0)]; + for piece in samples.chunks(700) { + out.extend(box_side.push(piece)); + } + out.extend(box_side.release(10_000)); + out.into_iter() + .map(|out| match out { + Out::Line(line) | Out::Data(line) => line.trim_end().to_owned(), + }) + .collect() + } + + pub fn sine(count: usize) -> Vec { + (0..count).map(|index| ((index as f32 * 0.05).sin() * 12000.0) as i16).collect() + } + + #[test] + fn lines_parse_and_malformed_ones_are_told_apart_from_other_lines() { + assert_eq!(parse_line("VOICE START 3"), Some(Ok(VoiceLine::Start { turn: 3 }))); + assert_eq!(parse_line("VOICE END 3 12000\r"), Some(Ok(VoiceLine::End { turn: 3, samples: 12000 }))); + assert_eq!(parse_line("VOICE CANCEL 3"), Some(Ok(VoiceLine::Cancel { turn: 3 }))); + assert_eq!( + parse_line("VOICE DATA 3 7 -1200 42 AQL/"), + Some(Ok(VoiceLine::Data { turn: 3, seq: 7, predictor: -1200, index: 42, bytes: vec![1, 2, 255] })) + ); + assert!(parse_line("VOICES hsiaoyu").is_none()); + assert!(parse_line("MODE LEISURE").is_none()); + for bad in ["VOICE DATA 3 7 -1200 89 AQL/", "VOICE DATA 3 7 40000 1 AQL/", "VOICE DATA 3 x 0 0 AQL/", "VOICE DATA 3 7 0 0 !!", "VOICE END 3", "VOICE SING 1"] { + assert!(matches!(parse_line(bad), Some(Err(_))), "{bad}"); + } + } + + /// The decoder here and the box's are the same: fed the same nibbles from zero, they give the same samples. + #[test] + fn the_decoder_matches_firmware_core() { + let mut ours = Decoder::new(0, 0); + let mut box_side = vibebuddy_firmware_core::adpcm::Decoder::default(); + let mut seed = 0x1234_5678u32; + for _ in 0..200_000 { + seed = seed.wrapping_mul(1_664_525).wrapping_add(1_013_904_223); + let nibble = (seed >> 28) as u8; + assert_eq!(ours.decode(nibble), box_side.decode(nibble)); + } + } + + /// A line decoded on its own, from the state it carries, is the same as the stream decoded straight through. + #[test] + fn a_line_decodes_on_its_own_from_the_state_it_carries() { + let original = sine(10_000); + let lines = voice_lines(1, &original); + let mut box_side = vibebuddy_firmware_core::adpcm::Decoder::default(); + let mut straight = Vec::new(); + let mut separately = Vec::new(); + for line in &lines { + if let Some(Ok(VoiceLine::Data { predictor, index, bytes, .. })) = parse_line(line) { + straight.extend(bytes.iter().flat_map(|byte| [byte & 15, byte >> 4]).map(|nibble| box_side.decode(nibble))); + separately.extend(decode_line(predictor, index, &bytes)); + } + } + assert_eq!(straight[..10_000], separately[..10_000]); + for (k, (&decoded, &expected)) in separately.iter().zip(&original).enumerate().skip(60) { + assert!((i32::from(decoded) - i32::from(expected)).abs() < 600, "sample {k}: {decoded} vs {expected}"); + } + } + + #[test] + fn lost_lines_become_silence_of_a_lines_length() { + let lines: BTreeMap> = [(0, vec![1; 4]), (2, vec![3; 4]), (3, vec![4; 2])].into(); + let (audio, missing) = assemble(&lines, 14); + assert_eq!(audio, [1, 1, 1, 1, 0, 0, 0, 0, 3, 3, 3, 3, 4, 4]); + assert_eq!(missing, [1]); + // Lines lost at the end are filled up to what END says was recorded. + let lines: BTreeMap> = [(1, vec![2; 4])].into(); + let (audio, missing) = assemble(&lines, 11); + assert_eq!(audio, [0, 0, 0, 0, 2, 2, 2, 2, 0, 0, 0]); + assert_eq!(missing, [0, 2]); + // An odd count cuts the padding nibble of the last byte. + let lines: BTreeMap> = [(0, vec![5; 4])].into(); + assert_eq!(assemble(&lines, 3).0, [5, 5, 5]); + } + + #[test] + fn audio_waits_for_room_in_a_full_queue_instead_of_dropping() { + struct Busy { + refusals: std::sync::Mutex, + sent: std::sync::Mutex, + } + impl Transport for Busy { + fn send(&self, _frame: Vec) -> Result<(), crate::serial_transport::TransportError> { + let mut refusals = self.refusals.lock().unwrap(); + if *refusals > 0 { + *refusals -= 1; + return Err(crate::serial_transport::TransportError::QueueFull); + } + *self.sent.lock().unwrap() += 1; + Ok(()) + } + } + let busy = Busy { refusals: std::sync::Mutex::new(3), sent: std::sync::Mutex::new(0) }; + let stopped = AtomicBool::new(false); + assert!(send_paced(&busy, audio_event(1, 0, 0), &stopped)); + assert_eq!(*busy.sent.lock().unwrap(), 1); + // Stopped: gives up on a full queue at once. + *busy.refusals.lock().unwrap() = 1; + stopped.store(true, Ordering::Release); + assert!(!send_paced(&busy, audio_event(1, 0, 1), &stopped)); + } + + #[test] + fn the_wav_header_is_24_khz_16_bit_mono() { + let file = wav(&[1, -1]); + assert_eq!(file.len(), 48); + assert_eq!(&file[..4], b"RIFF"); + assert_eq!(u32::from_le_bytes(file[24..28].try_into().unwrap()), 24_000); + assert_eq!(u16::from_le_bytes(file[22..24].try_into().unwrap()), 1); + assert_eq!(u16::from_le_bytes(file[34..36].try_into().unwrap()), 16); + assert_eq!(&file[44..], [1, 0, 255, 255]); + } + + #[test] + fn long_text_is_cut_at_a_character_to_fit_the_line() { + let long = "语音".repeat(400); + let event = text_event("voice.heard", 1, &long); + let frame = event.to_ndjson().expect("fits"); + assert!(frame.len() <= MAX_MESSAGE_BYTES + 1); + let text = event.extra["text"].as_str().unwrap(); + assert!(text.ends_with('…') && long.starts_with(text.trim_end_matches('…'))); + assert!(frame.len() > MAX_MESSAGE_BYTES - 8, "cut no more than needed: {}", frame.len()); + // Quotes take two bytes of JSON each. + let quoted = "\"".repeat(2000); + assert!(text_event("voice.reply", 1, "ed).to_ndjson().is_ok()); + assert_eq!(text_event("voice.reply", 1, "short").extra["text"], "short"); + } + + #[derive(Default)] + pub struct RecordingTransport { + frames: std::sync::Mutex>>, + } + + impl Transport for RecordingTransport { + fn send(&self, frame: Vec) -> Result<(), crate::serial_transport::TransportError> { + self.frames.lock().unwrap().push(frame); + Ok(()) + } + } + + impl RecordingTransport { + pub fn events(&self) -> Vec { + self.frames.lock().unwrap().iter().map(|frame| serde_json::from_slice(frame).unwrap()).collect() + } + } + + /// Answers every recording with the same result, and keeps the WAV files it was given. + struct FakeRecognizer { + answer: Result<&'static str, &'static str>, + heard: std::sync::Mutex>>, + } + + impl Recognizer for FakeRecognizer { + fn recognize(&self, wav: &[u8]) -> Result { + self.heard.lock().unwrap().push(wav.to_vec()); + self.answer + .map(|text| crate::speech::Transcript { text: text.to_owned(), language: Some("en".to_owned()) }) + .map_err(str::to_owned) + } + } + + use crate::resident::tests::{Shared, logged_in, resident, temp_dir, tool_call}; + + /// Answers every reply with the same pieces of "audio", failing after them for texts that start with "fail", + /// and keeps the text and voice of each call. With a gate, its first call waits until the gate opens. + #[derive(Default)] + struct FakeSynthesizer { + pieces: Vec>, + calls: std::sync::Mutex>, + gate: std::sync::Mutex>>, + } + + impl Synthesizer for FakeSynthesizer { + fn synthesize(&self, text: &str, voice: &str, on_audio: &mut dyn FnMut(&[u8]) -> bool) -> Result<(), String> { + self.calls.lock().unwrap().push((text.to_owned(), voice.to_owned())); + if let Some(gate) = self.gate.lock().unwrap().take() { + let _ = gate.recv(); + } + for piece in &self.pieces { + if !on_audio(piece) { + return Ok(()); + } + } + if text.starts_with("fail") { Err("the connection was cut".to_owned()) } else { Ok(()) } + } + } + + struct Rig { + voice: Voice, + transport: Arc, + recognizer: Arc, + script: Shared, + dir: std::path::PathBuf, + character: Option<&'static str>, + } + + impl Rig { + fn new(name: &str, answer: Result<&'static str, &'static str>, script: Shared) -> Self { + Self::speaking(name, answer, script, None, None) + } + + fn speaking( + name: &str, + answer: Result<&'static str, &'static str>, + script: Shared, + synthesizer: Option>, + character: Option<&'static str>, + ) -> Self { + let dir = temp_dir(&format!("voice-{name}")); + let recognizer = Arc::new(FakeRecognizer { answer, heard: Default::default() }); + let resident = Arc::new(resident(&dir, &script, Arc::default())); + let voice = Voice::new(Some(recognizer.clone()), Some(resident)).with_synthesizer(synthesizer); + Self { voice, transport: Arc::default(), recognizer, script, dir, character } + } + + /// Feeds the lines; returns the task the last one started, if any. + fn feed(&self, lines: &[String]) -> Option> { + let transport: Arc = self.transport.clone(); + let wearing = Wearing { character: self.character.map(str::to_owned), address: None }; + lines.iter().filter_map(|line| self.voice.handle_line(line, &transport, &wearing)).last() + } + + fn names(&self) -> Vec { + self.transport.events().iter().map(|event| event.event.clone()).collect() + } + + fn error(&self) -> (String, String) { + let events = self.transport.events(); + let error = events.iter().find(|event| event.event == "voice.error").expect("a voice.error"); + (error.extra["reason"].as_str().unwrap().to_owned(), error.message.clone().unwrap_or_default()) + } + } + + impl Drop for Rig { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.dir); + } + } + + #[tokio::test] + async fn a_turn_is_heard_answered_reply_by_reply_and_done() { + let script = logged_in(); + let hold = Arc::new(tokio::sync::Notify::new()); + script.lock().unwrap().hold = Some(hold.clone()); + script.lock().unwrap().turn_requests = + vec![tool_call("speak", json!({"text": "Two agents are working."})), tool_call("speak", json!({"text": "One needs you."}))]; + let rig = Rig::new("turn", Ok("what are my agents doing"), script); + + let task = rig.feed(&voice_lines(4, &sine(30_000))).expect("the turn runs"); + // Both replies reach the box while the turn is still open in the app-server. + while rig.names().len() < 3 { + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + } + assert_eq!(rig.names(), ["voice.heard", "voice.reply", "voice.reply"]); + assert!(!task.is_finished()); + hold.notify_one(); + task.await.unwrap(); + + let events = rig.transport.events(); + assert_eq!(rig.names(), ["voice.heard", "voice.reply", "voice.reply", "voice.done"]); + assert!(events.iter().all(|event| event.extra["turn"] == 4)); + assert_eq!(events[0].extra["text"], "what are my agents doing"); + assert_eq!(events[1].extra["text"], "Two agents are working."); + assert_eq!(events[2].extra["text"], "One needs you."); + let wav = rig.recognizer.heard.lock().unwrap()[0].clone(); + assert_eq!(wav.len(), 44 + 30_000 * 2, "the whole recording, as 16-bit samples"); + let turn_text = rig.script.lock().unwrap().received.iter().find(|message| message["method"] == "turn/start").unwrap() + ["params"]["input"][1]["text"] + .clone(); + assert_eq!(turn_text, "what are my agents doing"); + } + + /// One `voice.audio`: reply, seq, its bytes (None for its end) and its error. + type Piece = (u64, u64, Option>, Option); + + /// Each reply's audio, in the order it was sent. + fn audio(events: &[Event]) -> Vec { + events + .iter() + .filter(|event| event.event == "voice.audio") + .map(|event| { + let data = event.extra.get("data").map(|data| { + let data = data.as_str().unwrap(); + assert!(data.len() <= 768, "{} base64 characters", data.len()); + base64::engine::general_purpose::STANDARD.decode(data).unwrap() + }); + assert_eq!(data.is_none(), event.extra.get("end") == Some(&json!(true))); + let error = event.extra.get("error").map(|error| error.as_str().unwrap().to_owned()); + (event.extra["reply"].as_u64().unwrap(), event.extra["seq"].as_u64().unwrap(), data, error) + }) + .collect() + } + + #[tokio::test(flavor = "multi_thread")] + async fn each_reply_is_spoken_in_order_without_holding_up_the_text_and_done_comes_after_the_audio() { + let script = logged_in(); + let hold = Arc::new(tokio::sync::Notify::new()); + script.lock().unwrap().hold = Some(hold.clone()); + script.lock().unwrap().turn_requests = + vec![tool_call("speak", json!({"text": "Two agents are working."})), tool_call("speak", json!({"text": "One needs you."}))]; + let (open, gate) = std::sync::mpsc::channel(); + let pieces = vec![(0..=255).cycle().take(1300).collect(), vec![7; 10]]; + let synthesizer = Arc::new(FakeSynthesizer { pieces, gate: std::sync::Mutex::new(Some(gate)), ..Default::default() }); + let rig = Rig::speaking("speak", Ok("what are my agents doing"), script, Some(synthesizer.clone()), Some("ada")); + + let task = rig.feed(&voice_lines(4, &sine(30_000))).expect("the turn runs"); + // The first reply's audio is held up, and both replies' text still reach the box. + while rig.names().len() < 3 { + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + } + assert_eq!(rig.names(), ["voice.heard", "voice.reply", "voice.reply"]); + // The turn has completed in the session, but isn't done until the audio is. + hold.notify_one(); + tokio::time::sleep(std::time::Duration::from_millis(50)).await; + assert!(!task.is_finished()); + open.send(()).unwrap(); + task.await.unwrap(); + + let events = rig.transport.events(); + assert!(events.iter().all(|event| event.extra["turn"] == 4)); + assert!(events.iter().all(|event| event.to_ndjson().unwrap().len() <= MAX_MESSAGE_BYTES + 1)); + assert_eq!(rig.names().last().map(String::as_str), Some("voice.done")); + let audio = audio(&events); + let order: Vec<(u64, u64, bool)> = audio.iter().map(|(reply, seq, data, _)| (*reply, *seq, data.is_none())).collect(); + // 1300 bytes are three pieces of at most 576; then the 10, then the end. + let expected: Vec<(u64, u64, bool)> = [0, 1].into_iter().flat_map(|reply| (0..5).map(move |seq| (reply, seq, seq == 4))).collect(); + assert_eq!(order, expected); + for reply in [0, 1] { + let bytes: Vec = audio.iter().filter(|(r, ..)| *r == reply).filter_map(|(_, _, data, _)| data.clone()).flatten().collect(); + assert_eq!(bytes.len(), 1310); + assert_eq!(bytes[..1300], (0..=255).cycle().take(1300).collect::>()[..]); + } + assert!(audio.iter().all(|(.., error)| error.is_none())); + let calls = synthesizer.calls.lock().unwrap().clone(); + let voice = "elevenlabs:tcSsCJFYrsHpK5rCQZMf".to_owned(); + assert_eq!(calls, [("Two agents are working.".to_owned(), voice.clone()), ("One needs you.".to_owned(), voice)]); + } + + #[tokio::test] + async fn a_failed_synthesis_ends_its_reply_early_marked_and_the_next_one_is_still_spoken() { + let script = logged_in(); + script.lock().unwrap().turn_requests = + vec![tool_call("speak", json!({"text": "fail: cut short"})), tool_call("speak", json!({"text": "还在。"}))]; + let synthesizer = Arc::new(FakeSynthesizer { pieces: vec![vec![1; 100]], ..Default::default() }); + let rig = Rig::speaking("speak-fail", Ok("hi"), script, Some(synthesizer.clone()), None); + rig.feed(&voice_lines(1, &sine(12_000))).unwrap().await.unwrap(); + + let audio = audio(&rig.transport.events()); + assert_eq!( + audio, + [ + (0, 0, Some(vec![1; 100]), None), + (0, 1, None, Some("synthesis".to_owned())), + (1, 0, Some(vec![1; 100]), None), + (1, 1, None, None), + ] + ); + assert_eq!(rig.names().last().map(String::as_str), Some("voice.done")); + // No Character pack: the built-in Character's voice, and a Chinese one for Chinese. + let voices: Vec = synthesizer.calls.lock().unwrap().iter().map(|(_, voice)| voice.clone()).collect(); + assert_eq!(voices, ["elevenlabs:cgSgspJ2msm6clMCkdW9", "elevenlabs:cgSgspJ2msm6clMCkdW9"]); + } + + #[test] + fn a_stopped_speaker_sends_nothing_more() { + let synthesizer = FakeSynthesizer { pieces: vec![vec![1; 10]], ..Default::default() }; + let transport = RecordingTransport::default(); + speak(1, 0, "hello", None, &synthesizer, &transport, &AtomicBool::new(true)); + assert!(transport.events().is_empty()); + assert!(synthesizer.calls.lock().unwrap().is_empty()); + } + + #[test] + fn the_largest_audio_piece_fits_a_line() { + let mut event = audio_event(u32::MAX, u32::MAX, u32::MAX); + event.extra.insert("data".to_owned(), base64::engine::general_purpose::STANDARD.encode([255u8; AUDIO_BYTES]).into()); + assert_eq!(event.extra["data"].as_str().unwrap().len(), 768); + assert!(event.to_ndjson().unwrap().len() <= MAX_MESSAGE_BYTES + 1); + } + + #[tokio::test] + async fn lost_lines_still_make_a_turn_of_the_recorded_length() { + let rig = Rig::new("gaps", Ok("hello"), logged_in()); + let original = sine(5_000); + let mut lines = voice_lines(1, &original); + assert_eq!(lines.last().map(String::as_str), Some("VOICE END 1 5000")); + assert_eq!(lines.len(), 7, "START, four full lines of 1152 samples and one of 392, END"); + // No START and no line 2 (samples 2304 to 3455): the rest still counts. + lines.retain(|line| !line.starts_with("VOICE START") && !line.starts_with("VOICE DATA 1 2 ")); + rig.feed(&lines).expect("the turn runs").await.unwrap(); + let wav = rig.recognizer.heard.lock().unwrap()[0].clone(); + assert_eq!(wav.len(), 44 + 5_000 * 2); + let decoded: Vec = wav[44..].chunks(2).map(|pair| i16::from_le_bytes([pair[0], pair[1]])).collect(); + assert!(decoded[2_304..3_456].iter().all(|sample| *sample == 0), "line 2 is silence"); + // END counts the lost line's samples too, as the box's does, so what follows the gap keeps its place. + for k in (3_456 + 60..5_000).step_by(97) { + assert!((i32::from(decoded[k]) - i32::from(original[k])).abs() < 600, "sample {k}"); + } + assert_eq!(rig.names().last().map(String::as_str), Some("voice.done")); + + // The last line lost: filled up to the count END gives. + let rig = Rig::new("gap-at-end", Ok("hello"), logged_in()); + let mut lines = voice_lines(1, &original); + lines.retain(|line| !line.starts_with("VOICE DATA 1 4 ")); + rig.feed(&lines).expect("the turn runs").await.unwrap(); + let wav = rig.recognizer.heard.lock().unwrap()[0].clone(); + assert_eq!(wav.len(), 44 + 5_000 * 2); + assert!(wav[44 + 4_608 * 2..].iter().all(|byte| *byte == 0)); + } + + #[tokio::test] + async fn nothing_recognised_is_not_heard_and_the_session_is_not_asked() { + let rig = Rig::new("empty", Ok(""), logged_in()); + rig.feed(&voice_lines(2, &sine(12_000))).unwrap().await.unwrap(); + assert_eq!(rig.names(), ["voice.error"]); + assert_eq!(rig.error().0, "not_heard"); + assert_eq!(rig.script.lock().unwrap().starts, 0); + } + + #[tokio::test] + async fn a_failed_recognizer_or_session_is_unavailable() { + let rig = Rig::new("stt-down", Err("Workers AI answered 401: Authentication error"), logged_in()); + rig.feed(&voice_lines(1, &sine(12_000))).unwrap().await.unwrap(); + assert_eq!(rig.error(), ("unavailable".to_owned(), "Workers AI answered 401: Authentication error".to_owned())); + + // Codex isn't logged in: what was heard is still shown, then the error. + let rig = Rig::new("no-login", Ok("hi"), Arc::new(std::sync::Mutex::new(Default::default()))); + rig.feed(&voice_lines(1, &sine(12_000))).unwrap().await.unwrap(); + assert_eq!(rig.names(), ["voice.heard", "voice.error"]); + assert_eq!(rig.error().0, "unavailable"); + } + + #[tokio::test] + async fn a_turn_that_ends_while_another_runs_is_busy() { + let script = logged_in(); + let hold = Arc::new(tokio::sync::Notify::new()); + script.lock().unwrap().hold = Some(hold.clone()); + let rig = Rig::new("busy", Ok("first"), script); + let first = rig.feed(&voice_lines(1, &sine(12_000))).unwrap(); + let second = voice_lines(2, &sine(12_000)); + // Refused as soon as it starts, so the box stops recording; the rest of it gets no second answer. + assert!(rig.feed(&second[..1]).is_none()); + let errors = || rig.transport.events().into_iter().filter(|event| event.event == "voice.error").collect::>(); + assert_eq!(errors().len(), 1); + assert_eq!((errors()[0].extra["turn"].as_u64(), errors()[0].extra["reason"].as_str()), (Some(2), Some("busy"))); + assert!(rig.feed(&second[1..]).is_none()); + assert_eq!(errors().len(), 1); + hold.notify_one(); + first.await.unwrap(); + // Once it is over, the next turn runs. + hold.notify_one(); + rig.feed(&voice_lines(3, &sine(12_000))).expect("not busy any more").await.unwrap(); + } + + #[tokio::test] + async fn a_cancelled_turn_is_dropped_and_an_end_without_audio_is_not_heard() { + let rig = Rig::new("cancel", Ok("hi"), logged_in()); + let mut lines = voice_lines(5, &sine(6_000)); + lines.pop(); + lines.push("VOICE CANCEL 5".to_owned()); + assert!(rig.feed(&lines).is_none()); + assert!(rig.names().is_empty(), "a cancelled turn gets no answer"); + assert!(rig.feed(&["VOICE END 5 6000".to_owned()]).is_none()); + assert_eq!(rig.error().0, "not_heard"); + assert!(rig.recognizer.heard.lock().unwrap().is_empty()); + } + + /// The left channel of a 16-bit PCM WAV file, and its sample rate. + fn left_channel(file: &[u8]) -> (Vec, u32) { + let mut at = 12; + let (mut channels, mut rate) = (1usize, 0u32); + while at + 8 <= file.len() { + let size = u32::from_le_bytes(file[at + 4..at + 8].try_into().unwrap()) as usize; + let body = &file[at + 8..(at + 8 + size).min(file.len())]; + match &file[at..at + 4] { + b"fmt " => { + channels = u16::from_le_bytes([body[2], body[3]]) as usize; + rate = u32::from_le_bytes(body[4..8].try_into().unwrap()); + assert_eq!(u16::from_le_bytes([body[14], body[15]]), 16, "16-bit PCM only"); + } + b"data" => { + let samples = body.chunks_exact(2 * channels).map(|frame| i16::from_le_bytes([frame[0], frame[1]])).collect(); + return (samples, rate); + } + _ => {} + } + at += 8 + size + size % 2; + } + panic!("no data chunk"); + } + + /// Prints each event the moment the daemon sends it, with the time since `VOICE END`. + struct TimedTransport { + end: std::sync::Mutex>, + events: std::sync::Mutex>, + } + + impl Transport for TimedTransport { + fn send(&self, frame: Vec) -> Result<(), crate::serial_transport::TransportError> { + assert!(frame.len() <= MAX_MESSAGE_BYTES + 1); + let event: Event = serde_json::from_slice(&frame).unwrap(); + let ms = self.end.lock().unwrap().map(|end| end.elapsed().as_millis()).unwrap_or_default(); + println!("+{ms:>6} ms {}", String::from_utf8_lossy(&frame).trim_end()); + self.events.lock().unwrap().push((ms, event)); + Ok(()) + } + } + + /// A real recording through the whole computer side, without the box: encoded as the box encodes it, fed as + /// `VOICE` lines, recognised by the real recognizer and answered by the real resident session. Run by hand: + /// + /// ```sh + /// VIBEBUDDY_E2E_WAV=mic.wav VIBEBUDDY_STT_ENV_FILE=~/.vibebuddy-dev-ai.env \ + /// cargo test -p vibebuddyd e2e_ -- --ignored --nocapture + /// ``` + /// + /// The thread and its Codex home live in a temporary state directory; the user's hooks don't load there, so + /// nothing reaches the daemon on 7331. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "uses the network, a speech recognizer's credentials and the user's Codex"] + async fn e2e_a_real_recording_through_the_real_recognizer_and_resident_session() { + let _ = tracing_subscriber::fmt().with_env_filter("vibebuddyd=info").with_test_writer().try_init(); + let path = std::env::var("VIBEBUDDY_E2E_WAV").expect("VIBEBUDDY_E2E_WAV names a recording"); + let (left, rate) = left_channel(&std::fs::read(path).unwrap()); + assert_eq!(rate, SAMPLE_RATE, "the box records at 24 kHz"); + let recognizer = crate::speech::from_env().expect("a speech recognizer is configured"); + let dir = temp_dir("voice-e2e"); + let resident = Arc::new(Resident::new(true, &dir, crate::resident::Launcher::Codex, Arc::default())); + let voice = Voice::new(Some(recognizer), Some(resident)); + let timed = Arc::new(TimedTransport { end: Default::default(), events: Default::default() }); + let transport: Arc = timed.clone(); + + // 1152 samples are 576 bytes, 768 base64 characters: the most a line may carry. + let lines = voice_lines(1, &left); + assert!(lines.iter().all(|line| line.len() < 1024)); + println!("{:.2} s of audio in {} lines", left.len() as f32 / SAMPLE_RATE as f32, lines.len()); + let (end, data) = lines.split_last().unwrap(); + for line in data { + assert!(voice.handle_line(line, &transport, &Wearing::default()).is_none()); + } + *timed.end.lock().unwrap() = Some(Instant::now()); + voice.handle_line(end, &transport, &Wearing::default()).expect("the turn runs").await.unwrap(); + + let events = timed.events.lock().unwrap(); + let names: Vec<&str> = events.iter().map(|(_, event)| event.event.as_str()).collect(); + assert_eq!(names.first(), Some(&"voice.heard"), "{names:?}"); + assert_eq!(names.last(), Some(&"voice.done"), "{names:?}"); + assert!(names.contains(&"voice.reply"), "{names:?}"); + let _ = std::fs::remove_dir_all(dir); + } + + /// Two replies through the real voice service, as the box would get them: the `voice.audio` pieces put back + /// together into `/reply-zh.mp3` and `reply-en.mp3`. Run by hand: + /// + /// ```sh + /// VIBEBUDDY_E2E_OUT=/some/dir cargo test -p vibebuddyd e2e_spoken -- --ignored --nocapture + /// ``` + /// + /// `VIBEBUDDY_TTS_BASE_URL` picks the service as it does for the daemon; unset, staging. + #[test] + #[ignore = "uses the network and the voice service"] + fn e2e_spoken_replies_through_the_real_voice_service() { + let out = std::path::PathBuf::from(std::env::var("VIBEBUDDY_E2E_OUT").expect("VIBEBUDDY_E2E_OUT names a directory")); + let synthesizer = synthesis::from_env().expect("a voice service"); + for (name, character, text) in [ + ("zh", Some("wanwanxiaohe"), "两个智能体正在工作，有一个在等你回复。"), + ("en", Some("ada"), "Two agents are working, and one of them is waiting for you."), + ] { + let timed = TimedTransport { end: std::sync::Mutex::new(Some(Instant::now())), events: Default::default() }; + speak(1, 0, text, character, &*synthesizer, &timed, &AtomicBool::new(false)); + let events = timed.events.lock().unwrap(); + let (first_ms, _) = events.first().expect("some audio"); + let (total_ms, last) = events.last().unwrap(); + assert_eq!(last.extra.get("end"), Some(&json!(true))); + assert!(!last.extra.contains_key("error"), "synthesis failed"); + let pieces: Vec = events.iter().map(|(_, event)| event.clone()).collect(); + let bytes: Vec = audio(&pieces).into_iter().filter_map(|(_, _, data, _)| data).flatten().collect(); + let path = out.join(format!("reply-{name}.mp3")); + std::fs::write(&path, &bytes).unwrap(); + println!( + "{name} {}: first audio after {first_ms} ms, end after {total_ms} ms, {} bytes in {} lines -> {}", + synthesis::voice_for(character), + bytes.len(), + events.len(), + path.display() + ); + } + } + + #[tokio::test] + async fn voice_is_offered_only_with_a_recognizer_and_the_resident_session_on_and_ready() { + let dir = temp_dir("voice-offer"); + let with = |problem: Option| { + let resident = resident(&dir, &logged_in(), Arc::default()); + resident.note(problem); + Some(Arc::new(resident)) + }; + let on = || with(None); + let off = Resident::new(false, &dir, crate::resident::Launcher::Codex, Arc::default()); + off.note(None); + let recognizer = || Some(Arc::new(FakeRecognizer { answer: Ok(""), heard: Default::default() }) as Arc); + assert!(Voice::new(recognizer(), on()).available()); + assert!(!Voice::new(None, on()).available()); + assert!(!Voice::new(recognizer(), Some(Arc::new(off))).available()); + assert!(!Voice::new(recognizer(), None).available()); + let unchecked = Some(Arc::new(resident(&dir, &logged_in(), Arc::default()))); + assert!(!Voice::new(recognizer(), unchecked).available(), "not before the first check"); + assert!(!Voice::new(recognizer(), with(Some(crate::resident::Problem::CodexMissing))).available()); + assert!(!Voice::new(recognizer(), with(Some(crate::resident::Problem::CodexLoggedOut))).available()); + + // Lines that arrive anyway are told voice is off. + let voice = Voice::new(None, on()); + let transport = Arc::new(RecordingTransport::default()); + let shared: Arc = transport.clone(); + for line in voice_lines(1, &sine(12_000)) { + voice.handle_line(&line, &shared, &Wearing::default()); + } + assert_eq!(transport.events()[0].extra["reason"], "unavailable"); + let _ = std::fs::remove_dir_all(dir); + } +} diff --git a/desktop/src/client.rs b/desktop/src/client.rs index 92a09faf..455301e4 100644 --- a/desktop/src/client.rs +++ b/desktop/src/client.rs @@ -72,9 +72,17 @@ pub async fn set_volume(level: u8, preview: bool) -> Result<(), String> { } /// The daemon accepts the pack and writes it in the background; progress arrives in the status stream. -pub async fn write_voice_pack(pack: Vec) -> Result<(), String> { +/// `lender` (the Character lending a custom one its voice, persona and lines) and `address` (the form of address +/// its lines carry) aren't in the pack; the daemon keeps them for the buddy's spoken replies. +pub async fn write_voice_pack(pack: Vec, lender: Option<&str>, address: Option<&str>) -> Result<(), String> { + // Both are ids from fixed lists (`ada`, `laoban`): nothing to escape. + let query: Vec = [("lender", lender), ("address", address)] + .into_iter() + .filter_map(|(name, value)| Some(format!("{name}={}", value?))) + .collect(); + let query = if query.is_empty() { String::new() } else { format!("?{}", query.join("&")) }; let request = reqwest::Client::new() - .post(format!("{BASE}/v1/device/voice-pack")) + .post(format!("{BASE}/v1/device/voice-pack{query}")) .header("content-type", "application/octet-stream") .body(pack); accepted(request).await @@ -149,6 +157,152 @@ pub async fn restart_daemon() -> Result<(), String> { post("/v1/daemon/restart", None).await } +/// Whether the daemon runs the experimental resident session (`VIBEBUDDY_RESIDENT_SESSION=1`). +pub async fn resident_enabled() -> Result { + resident_info().await.map(|info| info.enabled).map_err(|error| match error { + HistoryError::DaemonDown => "the daemon isn't running".to_owned(), + HistoryError::Unavailable(why) => why, + }) +} + +/// `GET /v1/resident`. +#[derive(Clone, Debug, Deserialize, PartialEq)] +pub struct ResidentInfo { + pub enabled: bool, + /// Goes up when a voice command's turn ends or the thread changes, so the Conversation tab reads the history + /// again only then. Kept in memory: it starts again at 0 when the daemon restarts. None from an older daemon. + #[serde(default)] + pub revision: Option, + /// What voice commands are missing while the session is on but not ready; None when ready, and from a daemon + /// too old to say. + #[serde(default)] + pub problem: Option, + /// Which Agent the user wants answering; None from a daemon older than the choice. + #[serde(default)] + pub brain_setting: Option, + /// Whether the daemon holds an Anthropic API key; the key itself never comes back. + #[serde(default)] + pub anthropic_key_set: Option, +} + +/// Why the resident session can't take voice commands now: voice needs the user's Codex, or the user's Claude Code +/// with their own Anthropic API key. +#[derive(Clone, Debug, Deserialize, PartialEq)] +#[serde(rename_all = "snake_case")] +pub enum Problem { + CodexMissing, + CodexLoggedOut, + ClaudeMissing, + AnthropicKeyMissing, + /// A reason this app doesn't know yet. + #[serde(other)] + Other, +} + +/// Which Agent answers voice commands: Codex when it is ready, else Claude Code when an Anthropic API key is set +/// (`Auto`); or always the one picked. +#[derive(Clone, Copy, Debug, Deserialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum BrainChoice { + Auto, + Codex, + Claude, + /// A setting this app doesn't know yet. + #[serde(other)] + Other, +} + +impl BrainChoice { + fn as_str(self) -> &'static str { + match self { + Self::Auto | Self::Other => "auto", + Self::Codex => "codex", + Self::Claude => "claude", + } + } +} + +/// Which Agent answers voice commands. Its own endpoint, not part of the config. +pub async fn set_resident_brain(choice: BrainChoice) -> Result { + let request = reqwest::Client::new().put(format!("{BASE}/v1/resident/brain")).json(&serde_json::json!({"brain": choice.as_str()})); + resident_setting(request).await +} + +/// Stores the user's Anthropic API key in the daemon (None removes it). The daemon never sends it back. +pub async fn set_anthropic_key(key: Option) -> Result { + let client = reqwest::Client::new(); + let url = format!("{BASE}/v1/resident/anthropic-key"); + let request = match key { + Some(key) => client.put(url).json(&serde_json::json!({"api_key": key})), + None => client.delete(url), + }; + resident_setting(request).await +} + +async fn resident_setting(request: reqwest::RequestBuilder) -> Result { + let response = request.timeout(Duration::from_secs(10)).send().await.map_err(|error| error.to_string())?; + let status = response.status(); + let body: serde_json::Value = response.json().await.map_err(|error| error.to_string())?; + if !status.is_success() { + return Err(body["message"].as_str().map(str::to_owned).unwrap_or_else(|| format!("the daemon answered {status}"))); + } + serde_json::from_value(body).map_err(|error| error.to_string()) +} + +/// Asked every 2 seconds while the Conversation tab shows, so it answers quickly or not at all. +pub async fn resident_info() -> Result { + let response = reqwest::Client::new() + .get(format!("{BASE}/v1/resident")) + .timeout(Duration::from_secs(5)) + .send() + .await + .map_err(|error| if error.is_connect() { HistoryError::DaemonDown } else { HistoryError::Unavailable(error.to_string()) })?; + let body = response.bytes().await.map_err(|error| HistoryError::Unavailable(error.to_string()))?; + serde_json::from_slice(&body).map_err(|error| HistoryError::Unavailable(error.to_string())) +} + +/// One voice command and the buddy's spoken replies to it. +#[derive(Clone, Debug, Deserialize, PartialEq)] +pub struct Exchange { + pub said: String, + pub replies: Vec, +} + +#[derive(Clone, Debug)] +pub enum HistoryError { + DaemonDown, + /// The resident session couldn't be read (Codex missing or not logged in, say), with the daemon's reason. + Unavailable(String), +} + +/// The last `limit` voice commands and their spoken replies, oldest first. Slow the first time: the daemon may have +/// to start Codex's app-server to read the thread. +pub async fn resident_history(limit: usize) -> Result, HistoryError> { + #[derive(Deserialize)] + struct History { + exchanges: Vec, + } + let response = reqwest::Client::new() + .get(format!("{BASE}/v1/resident/history?limit={limit}")) + .timeout(Duration::from_secs(60)) + .send() + .await + .map_err(|error| if error.is_connect() { HistoryError::DaemonDown } else { HistoryError::Unavailable(error.to_string()) })?; + if !response.status().is_success() { + let status = response.status(); + return Err(HistoryError::Unavailable( + response + .json::() + .await + .ok() + .and_then(|body| body["message"].as_str().map(str::to_owned)) + .unwrap_or_else(|| format!("the daemon answered {status}")), + )); + } + let history: History = response.json().await.map_err(|error| HistoryError::Unavailable(error.to_string()))?; + Ok(history.exchanges) +} + async fn post(path: &str, body: Option) -> Result<(), String> { let mut request = reqwest::Client::new().post(format!("{BASE}{path}")); if let Some(body) = body { @@ -190,4 +344,32 @@ mod tests { assert_eq!(statuses[0].today.done, 3); assert!(buffer.is_empty()); } + + #[test] + fn the_resident_info_reads_its_revision_and_problem_and_an_older_daemons_without_them() { + let parse = |json: &str| serde_json::from_str::(json).unwrap(); + let older = parse(r#"{"enabled":true,"thread_id":null}"#); + let nothing = ResidentInfo { enabled: true, revision: None, problem: None, brain_setting: None, anthropic_key_set: None }; + assert_eq!(older, nothing); + let ready = parse(r#"{"enabled":true,"thread_id":"t1","revision":7,"ready":true,"problem":null}"#); + assert_eq!(ready, ResidentInfo { revision: Some(7), ..nothing.clone() }); + let missing = parse(r#"{"enabled":true,"thread_id":null,"revision":0,"ready":false,"problem":"codex_missing"}"#); + assert_eq!(missing.problem, Some(Problem::CodexMissing)); + let logged_out = parse(r#"{"enabled":true,"thread_id":null,"revision":0,"ready":false,"problem":"codex_logged_out"}"#); + assert_eq!(logged_out.problem, Some(Problem::CodexLoggedOut)); + let later = parse(r#"{"enabled":true,"thread_id":null,"revision":0,"ready":false,"problem":"something_new"}"#); + assert_eq!(later.problem, Some(Problem::Other)); + } + + #[test] + fn the_resident_info_says_which_agent_answers_and_whether_a_key_is_stored() { + let parse = |json: &str| serde_json::from_str::(json).unwrap(); + let claude = parse(r#"{"enabled":true,"thread_id":"s1","revision":3,"ready":true,"problem":null,"brain":"claude","brain_setting":"auto","anthropic_key_set":true}"#); + assert_eq!((claude.brain_setting, claude.anthropic_key_set), (Some(BrainChoice::Auto), Some(true))); + let no_key = parse(r#"{"enabled":true,"problem":"anthropic_key_missing","brain_setting":"claude","anthropic_key_set":false}"#); + assert_eq!((no_key.problem, no_key.brain_setting), (Some(Problem::AnthropicKeyMissing), Some(BrainChoice::Claude))); + let no_claude = parse(r#"{"enabled":true,"problem":"claude_missing","brain_setting":"something_new"}"#); + assert_eq!((no_claude.problem, no_claude.brain_setting), (Some(Problem::ClaudeMissing), Some(BrainChoice::Other))); + assert_eq!(BrainChoice::Other.as_str(), "auto"); + } } diff --git a/desktop/src/main.rs b/desktop/src/main.rs index b698325c..cf5a446f 100644 --- a/desktop/src/main.rs +++ b/desktop/src/main.rs @@ -29,6 +29,9 @@ use iced::{Element, Font, Length, Subscription, Task, Theme, window}; use i18n::{UiLanguage, tr}; use status::{Config, MenuState, Status}; +/// How many exchanges the Conversation tab reads, as on the Mac. +const CONVERSATION_LIMIT: usize = 20; + /// Omarchy's monospace font, used when Omarchy's theme is: the window should look like the rest of the desktop. const OMARCHY_FONT: &str = "JetBrainsMono Nerd Font"; @@ -66,6 +69,8 @@ enum Tab { Sound, Agents, Device, + /// Only while the daemon runs the resident session (experimental). + Conversation, Advanced, } @@ -139,6 +144,23 @@ enum Message { /// Play a Character's preview lines here, or stop the one playing. TogglePreview(&'static str), PreviewDone(String, Result<(), String>), + /// Whether the daemon runs the resident session, asked each time it comes up. + Resident(Result), + RefreshConversation, + Conversation(Result, client::HistoryError>), + /// Ask `GET /v1/resident` again while the Conversation tab shows. + PollConversation, + ConversationPolled(Result), + OpenCodexInstallPage, + OpenClaudeInstallPage, + OpenAnthropicConsole, + /// Which Agent answers voice commands. + PickBrain(client::BrainChoice), + AnthropicKeyDraft(String), + SaveAnthropicKey, + RemoveAnthropicKey, + /// The daemon's answer to a brain or key change. + ResidentSettingSaved(Result), /// Ask `vibebuddy-hook` for each agent's state again. RefreshAgents, Agents(Result, String>), @@ -211,6 +233,25 @@ struct App { agents: Vec, /// A change to an agent's config, shown before it's written. hook_plan: Option, + /// The daemon runs the experimental resident session, so the Conversation tab shows. Kept while the daemon is + /// down, so the tab can say so. + resident: bool, + /// The last voice commands and spoken replies read, or why they couldn't be; None before the first read. + conversation: Option, client::HistoryError>>, + conversation_loading: bool, + /// The history's change marker when it was last read; None before that, and after the daemon went away. + conversation_seen: Option, + /// The daemon has the change marker; an older one gets its history read every 5 seconds instead. + conversation_marked: bool, + conversation_polling: bool, + /// What voice commands are missing (the user's Codex or Claude Code), as the daemon last said. + resident_problem: Option, + /// The daemon's last word on the resident session: which Agent the user picked, whether a key is stored. + resident_info: Option, + /// The Anthropic API key being typed; cleared once the daemon has it, and never filled from the daemon. + anthropic_key_draft: String, + /// Why the last brain or key change failed. + resident_setting_error: Option, /// The outcome of the last action, shown at the bottom of the window. notice: Option>, /// The language picked in Settings, and, while it differs from the UI's, the offer to restart. @@ -262,6 +303,16 @@ impl App { previewing: None, agents: Vec::new(), hook_plan: None, + resident: false, + conversation: None, + conversation_loading: false, + conversation_seen: None, + conversation_marked: true, + conversation_polling: false, + resident_problem: None, + resident_info: None, + anthropic_key_draft: String::new(), + resident_setting_error: None, notice: None, language: UiLanguage::saved(), restart_offer: None, @@ -317,12 +368,20 @@ impl App { ); #[cfg(not(target_os = "linux"))] let (tray, instance) = (Subscription::none(), Subscription::none()); + // Only while the Conversation tab is in an open window: the history is read again when it changed. + let conversation = if self.settings.is_some() && self.resident && self.tab == Tab::Conversation { + let every = if self.conversation_marked { 2 } else { 5 }; + iced::time::every(Duration::from_secs(every)).map(|_| Message::PollConversation) + } else { + Subscription::none() + }; Subscription::batch([ Subscription::run(client::status_updates).map(Message::Daemon), iced::time::every(Duration::from_secs(2)).map(|_| Message::ThemeTick), window::close_events().map(Message::WindowClosed), tray, instance, + conversation, ]) } @@ -336,8 +395,18 @@ impl App { .then(|| self.pending_voice.take()) .flatten(); notify_new_firmware(&status); + // Only the daemon's environment at start decides, so ask once each time it comes up. + let resident = if self.status.is_none() { + Task::perform(client::resident_enabled(), Message::Resident) + } else { + Task::none() + }; self.status = Some(*status); - return Task::batch([self.refresh_tray(), voice.map_or_else(Task::none, |id| Task::done(Message::UseVoice(id)))]); + return Task::batch([ + self.refresh_tray(), + voice.map_or_else(Task::none, |id| Task::done(Message::UseVoice(id))), + resident, + ]); } Message::Daemon(client::Update::Down) => { self.status = None; @@ -400,8 +469,15 @@ impl App { ..Default::default() }); self.settings = Some(id); + // Reopened on the Conversation tab: read it again, as when the tab opens. + let conversation = if self.tab == Tab::Conversation { + self.conversation_seen = None; + Task::done(Message::PollConversation) + } else { + Task::none() + }; // The Agents tab reads each agent's config from disk; have it ready, whichever tab shows first. - return Task::batch([open.discard(), Task::done(Message::RefreshAgents)]); + return Task::batch([open.discard(), Task::done(Message::RefreshAgents), conversation]); } Message::WindowClosed(id) => { if self.settings == Some(id) { @@ -419,6 +495,96 @@ impl App { if tab == Tab::Agents { return Task::done(Message::RefreshAgents); } + // The conversation goes on on the box while the window is elsewhere: read it each time the tab opens. + if tab == Tab::Conversation { + self.conversation_seen = None; + return Task::done(Message::PollConversation); + } + } + Message::Resident(Ok(enabled)) => { + self.resident = enabled; + if !enabled && self.tab == Tab::Conversation { + self.tab = Tab::General; + } + } + Message::Resident(Err(_)) => {} + Message::RefreshConversation => { + if !self.conversation_loading { + self.conversation_loading = true; + return Task::perform(client::resident_history(CONVERSATION_LIMIT), Message::Conversation); + } + } + Message::Conversation(result) => { + self.conversation_loading = false; + self.conversation = Some(result); + } + Message::PollConversation => { + if !self.conversation_polling { + self.conversation_polling = true; + return Task::perform(client::resident_info(), Message::ConversationPolled); + } + } + Message::ConversationPolled(result) => { + self.conversation_polling = false; + match result { + Ok(info) => { + self.conversation_marked = info.revision.is_some(); + let became_ready = self.resident_problem.is_some() && info.problem.is_none(); + self.resident_problem = info.problem.clone(); + self.resident_info = Some(info.clone()); + let changed = info.revision.is_none() || info.revision != self.conversation_seen || became_ready; + // A read already under way is left alone; the next poll sees the change again. + if (changed || self.conversation.is_none()) && !self.conversation_loading { + self.conversation_seen = info.revision; + return Task::done(Message::RefreshConversation); + } + } + Err(client::HistoryError::DaemonDown) => { + // Read again once it answers: a restarted daemon counts from 0 again. + self.conversation_seen = None; + self.resident_problem = None; + self.conversation = Some(Err(client::HistoryError::DaemonDown)); + } + Err(client::HistoryError::Unavailable(_)) => {} + } + } + Message::OpenCodexInstallPage => { + if let Err(error) = launch("xdg-open", &[std::ffi::OsStr::new(CODEX_INSTALL_PAGE)]) { + self.notice = Some(Err(error)); + } + } + Message::OpenClaudeInstallPage => { + if let Err(error) = launch("xdg-open", &[std::ffi::OsStr::new(CLAUDE_INSTALL_PAGE)]) { + self.notice = Some(Err(error)); + } + } + Message::OpenAnthropicConsole => { + if let Err(error) = launch("xdg-open", &[std::ffi::OsStr::new(ANTHROPIC_CONSOLE)]) { + self.notice = Some(Err(error)); + } + } + Message::PickBrain(choice) => return Task::perform(client::set_resident_brain(choice), Message::ResidentSettingSaved), + Message::AnthropicKeyDraft(draft) => self.anthropic_key_draft = draft, + Message::SaveAnthropicKey => { + let key = self.anthropic_key_draft.trim().to_owned(); + if !key.is_empty() { + return Task::perform(client::set_anthropic_key(Some(key)), Message::ResidentSettingSaved); + } + } + Message::RemoveAnthropicKey => return Task::perform(client::set_anthropic_key(None), Message::ResidentSettingSaved), + Message::ResidentSettingSaved(result) => match result { + Ok(info) => { + if info.anthropic_key_set == Some(true) { + self.anthropic_key_draft.clear(); + } + self.resident_problem = info.problem.clone(); + self.resident_info = Some(info); + self.resident_setting_error = None; + // Another Agent may answer now, with its own history. + self.conversation_seen = None; + return Task::done(Message::PollConversation); + } + Err(error) => self.resident_setting_error = Some(error), } Message::NotifyLink(enabled) => { let mut config = self.status.as_ref().map(|status| status.config.clone()).unwrap_or_default(); @@ -522,7 +688,7 @@ impl App { let form = assets::language_of(id).and_then(|language| self.characters.address(language)).map(str::to_owned); let write = async move { let pack = assets::read_character(id, form.as_deref()).await?; - client::write_voice_pack(pack.build().ok_or("the Character pack doesn't fit")?).await + client::write_voice_pack(pack.build().ok_or("the Character pack doesn't fit")?, None, form.as_deref()).await }; return Task::perform(write, Message::Done); } @@ -554,7 +720,7 @@ impl App { Message::UseRobot => { self.characters.robot_lender = None; self.characters.save(); - let write = async move { client::write_voice_pack(assets::read_pack(ROBOT).await?).await }; + let write = async move { client::write_voice_pack(assets::read_pack(ROBOT).await?, None, None).await }; return Task::perform(write, Message::Done); } Message::UseCustom => { @@ -567,7 +733,7 @@ impl App { let form = assets::language_of(lender.id).and_then(|language| self.characters.address(language)).map(str::to_owned); let write = async move { let pack = assets::read_character(lender.id, form.as_deref()).await?.with_look(look, CUSTOM); - client::write_voice_pack(pack.build().ok_or("that character can't lend its voice")?).await + client::write_voice_pack(pack.build().ok_or("that character can't lend its voice")?, Some(lender.id), form.as_deref()).await }; return Task::perform(write, Message::Done); } @@ -717,9 +883,12 @@ impl App { (Tab::Sound, tr("Character", &[])), (Tab::Agents, tr("Agents", &[])), (Tab::Device, tr("Device", &[])), + (Tab::Conversation, tr("Conversation", &[])), (Tab::Advanced, tr("Advanced", &[])), ]; - let tab_bar = row(tabs.into_iter().map(|(tab, label)| { + // Voice commands are experimental (VIBEBUDDY_RESIDENT_SESSION=1): no tab while the daemon has them off. + let tabs = tabs.into_iter().filter(|(tab, _)| *tab != Tab::Conversation || self.resident); + let tab_bar = row(tabs.map(|(tab, label)| { let style = if tab == self.tab { button::primary } else { button::text }; button(text(label)).style(style).on_press(Message::Tab(tab)).into() })) @@ -729,6 +898,7 @@ impl App { Tab::Sound => self.sound(), Tab::Agents => self.agents(), Tab::Device => self.device(), + Tab::Conversation => self.conversation(), Tab::Advanced => self.advanced(), }; let notice = self.notice.as_ref().map(|notice| match notice { @@ -1542,6 +1712,138 @@ impl App { }) } + /// The user's last voice commands and the buddy's spoken replies, newest at the bottom, as on the Mac. + fn conversation(&self) -> Element<'_, Message> { + let header = row![ + text(tr("Your last voice commands and the buddy's spoken replies, newest at the bottom.", &[])) + .size(13) + .style(text::secondary) + .width(Length::Fill), + button(text(tr("Refresh", &[]))).on_press_maybe((!self.conversation_loading).then_some(Message::RefreshConversation)), + ] + .spacing(12) + .align_y(iced::Alignment::Center); + let placeholder = |title: String, detail: String| -> Element<'_, Message> { + container(column![text(title).size(16), text(detail).size(13).style(text::secondary)].spacing(6).align_x(iced::Alignment::Center)) + .center(Length::Fill) + .into() + }; + // A daemon older than the setting doesn't say it; then there is only Codex. + let choice = self.resident_info.as_ref().and_then(|info| info.brain_setting); + let header = column![header].push(choice.map(|choice| self.agent_setting(choice))).spacing(12); + // Voice needs the user's Codex, installed and logged in, or Claude Code with a key: say what's missing first. + if let Some(problem) = &self.resident_problem { + let (title, detail) = match problem { + client::Problem::CodexMissing => (tr("Voice needs Codex", &[]), tr("Voice commands are answered by Codex on this computer, and it isn't installed. Install the Codex CLI, then sign in with `codex login`. This tab notices within half a minute.", &[])), + client::Problem::CodexLoggedOut => (tr("Voice needs Codex", &[]), tr("Voice commands are answered by Codex on this computer, and it isn't signed in. Run `codex login` in a terminal and sign in. This tab notices within half a minute.", &[])), + client::Problem::ClaudeMissing => (tr("Voice needs Claude Code", &[]), tr("Voice commands are answered by Claude Code on this computer, and it isn't installed. Install Claude Code; you don't need to sign in to it. This tab notices within half a minute.", &[])), + client::Problem::AnthropicKeyMissing => (tr("Voice needs Claude Code", &[]), tr("Claude Code answers voice commands only with your own Anthropic API key. Add your key above.", &[])), + client::Problem::Other => (tr("Voice needs Codex", &[]), tr("The buddy's session isn't available: %@", &[&"?"])), + }; + let codex_problem = matches!(problem, client::Problem::CodexMissing | client::Problem::CodexLoggedOut); + let or_claude = (codex_problem && choice == Some(client::BrainChoice::Auto)) + .then(|| text(tr("Or let Claude Code answer, with your own Anthropic API key: add it above.", &[])).size(13).style(text::secondary)); + let install = match problem { + client::Problem::CodexMissing => Some(button(text(tr("How to install the Codex CLI", &[]))).on_press(Message::OpenCodexInstallPage)), + client::Problem::ClaudeMissing => Some(button(text(tr("How to install Claude Code", &[]))).on_press(Message::OpenClaudeInstallPage)), + _ => None, + }; + let body = container( + column![text(title).size(16), text(detail).size(13).style(text::secondary)] + .push(or_claude) + .push(install) + .spacing(6) + .align_x(iced::Alignment::Center), + ) + .center(Length::Fill); + return column![header, body].spacing(12).into(); + } + let body = match &self.conversation { + None => container(text(tr("Refreshing…", &[])).style(text::secondary)).center(Length::Fill).into(), + Some(Err(client::HistoryError::DaemonDown)) => placeholder( + tr("Couldn't read the conversation", &[]), + tr("The daemon isn't running. Restart it on the Advanced tab.", &[]), + ), + Some(Err(client::HistoryError::Unavailable(why))) => placeholder( + tr("Couldn't read the conversation", &[]), + tr("The buddy's session isn't available: %@", &[why]), + ), + Some(Ok(exchanges)) if exchanges.is_empty() => placeholder( + tr("Nothing said yet", &[]), + tr("Hold K0 on the box and talk to the buddy. What you say and what it says back show up here.", &[]), + ), + Some(Ok(exchanges)) => scrollable( + column(exchanges.iter().map(|exchange| { + column![bubble(&exchange.said, true)] + .extend(exchange.replies.iter().map(|reply| bubble(reply, false))) + .spacing(6) + .into() + })) + .spacing(10) + // Room for the scroll bar, which otherwise covers the ends of the lines. + .padding(iced::Padding::ZERO.right(16)), + ) + // Opens on the newest, at the bottom. + .anchor_bottom() + .height(Length::Fill) + .into(), + }; + column![header, body].spacing(12).into() + } + + /// Which Agent answers, and the user's Anthropic API key for Claude Code, as on the Mac. The key is typed once + /// and never shown again: the daemon only says whether it has one. + fn agent_setting(&self, choice: client::BrainChoice) -> Element<'_, Message> { + let option = |label: String, option: client::BrainChoice| { + let style = if option == choice { button::primary } else { button::secondary }; + button(text(label)).style(style).on_press(Message::PickBrain(option)) + }; + let mut setting = column![ + row![ + text(tr("Answered by", &[])), + option(tr("Automatic", &[]), client::BrainChoice::Auto), + option("Codex".to_owned(), client::BrainChoice::Codex), + option("Claude Code".to_owned(), client::BrainChoice::Claude), + ] + .spacing(8) + .align_y(iced::Alignment::Center) + ] + .spacing(6); + if choice == client::BrainChoice::Auto { + setting = setting.push(text(tr("Automatic: Codex when it's ready, else Claude Code with your API key.", &[])).size(12).style(text::secondary)); + } + if choice != client::BrainChoice::Codex { + let saved = self.resident_info.as_ref().and_then(|info| info.anthropic_key_set) == Some(true); + let key = if saved { + row![ + text(tr("Anthropic API key saved", &[])), + button(text(tr("Remove", &[]))).style(button::secondary).on_press(Message::RemoveAnthropicKey), + ] + } else { + let draft = self.anthropic_key_draft.trim(); + row![ + text_input(&tr("Anthropic API key", &[]), &self.anthropic_key_draft) + .secure(true) + .on_input(Message::AnthropicKeyDraft) + .on_submit(Message::SaveAnthropicKey) + .width(320), + button(text(tr("Save", &[]))).on_press_maybe((!draft.is_empty()).then_some(Message::SaveAnthropicKey)), + ] + }; + setting = setting + .push( + key.push(button(text(tr("Get an API key", &[]))).style(button::text).on_press(Message::OpenAnthropicConsole)) + .spacing(8) + .align_y(iced::Alignment::Center), + ) + .push(text(tr("Claude Code answers only with your own Anthropic API key, billed to your Anthropic account; a Claude subscription can't be used here. What you say and your Agents' task titles go to Anthropic.", &[])).size(12).style(text::secondary)); + } + if let Some(error) = &self.resident_setting_error { + setting = setting.push(text(error.clone()).size(12).style(text::danger)); + } + setting.into() + } + fn advanced(&self) -> Element<'_, Message> { let daemon = match &self.status { Some(status) => tr("Running · %@", &[&status.daemon.short_build()]), @@ -1640,6 +1942,12 @@ async fn export_diagnostics(summary: String, config: Option) -> Result Element<'_, Message> { + let bubble = container(text(words).size(14)).padding([7, 12]).max_width(440).style(move |theme: &Theme| { + let palette = theme.extended_palette(); + let pair = if mine { palette.primary.base } else { palette.background.weak }; + container::Style { + background: Some(pair.color.into()), + text_color: Some(pair.text), + border: iced::border::rounded(12), + ..container::Style::default() + } + }); + if mine { row![space::horizontal(), bubble] } else { row![bubble, space::horizontal()] }.into() +} + fn handle(frame: &character::Image, scale: usize) -> image::Handle { let (w, h) = (frame.width * scale, frame.height * scale); let mut pixels = vec![0u8; w * h * 4]; diff --git a/docs/adr/0007-device-menu-behind-a-long-k1.md b/docs/adr/0007-device-menu-behind-a-long-k1.md index 48391ed4..cd9891c4 100644 --- a/docs/adr/0007-device-menu-behind-a-long-k1.md +++ b/docs/adr/0007-device-menu-behind-a-long-k1.md @@ -13,3 +13,7 @@ Muse's two-key layout, with K1's short press opening the menu: K1's one-press sw ## Consequences Leisure can no longer be started on demand; demos use the time-compressed build. The C fallback firmware keeps the old long K1. A volume change from the box prints the same `VOLUME ` line as `device.volume`, which the daemon already applies, so the app's slider follows without a protocol change. + +## Amendment + +ADR-0017 (proposed) gives K0 long to push-to-talk while the computer offers voice commands (`voice.available`, docs/protocol.md). Giving up a Pomodoro phase then lives only in the menu's STOP FOCUS / END BREAK row; while voice isn't offered, K0 long still gives up the phase as before. diff --git a/docs/adr/0017-voice-commands-go-to-a-resident-session-that-speaks-as-the-buddy.md b/docs/adr/0017-voice-commands-go-to-a-resident-session-that-speaks-as-the-buddy.md new file mode 100644 index 00000000..ed1d60cb --- /dev/null +++ b/docs/adr/0017-voice-commands-go-to-a-resident-session-that-speaks-as-the-buddy.md @@ -0,0 +1,33 @@ +--- +status: proposed +--- + +# Voice commands go to a resident Agent session that speaks as the buddy; the cloud only hears and speaks + +The box has a microphone, and with Wi-Fi and the Relay (ADR-0012) it can sit away from the computer, where it is the user's only way to reach their Agents. We decided that a voice command is recognized in the cloud, handed to one long-running **resident session** (a Codex session the daemon keeps alive by driving the user's own `codex app-server` over stdio, with the user's own login), and answered with a short **spoken reply** the resident session writes itself, synthesized on the spot in the voice and persona of the Character the user picked. The cloud only turns speech into text and text into speech; the thinking is done by the user's own Agent. The resident session runs no commands and changes no files: it reads the Activities, answers, and starts or messages the Sessions that do the work, which keep their own permission prompts. + +## Considered options + +- **The box as a dictation microphone next to the keyboard.** macOS dictation already does this for free; the box only adds something when it is away from the computer. +- **A new Session per voice command.** Needs no change to how Agents are started, but away from the computer you can neither watch nor steer a session you just spawned. +- **Send each command into the session of the top task card.** The user doesn't know which session a sentence will land in, which is the reason voice replies to "needs input" were dropped too. +- **A cloud language model as the brain**, as Muse gadgets and xiaozhi do. Makes the buddy an assistant (CONTEXT.md reserves that word for Agents), puts model cost on us, and it would still have to drive the Agents on the computer through the computer. +- **Muse's cloud itself.** Its Gadget SDK terms forbid tokens in devices that are sold. +- **Claude Code first, through Channels.** Needs `--channels` at launch and an allow-listed plugin (ours would need `--dangerously-load-development-channels`), and Anthropic does not let third-party products use a claude.ai subscription login, which also rules out the Agent SDK and headless `claude -p` under the user's plan. Claude Code came second instead, with the user's own API key (below). +- **ACP (Agent Client Protocol).** One protocol for every Agent, but both adapters are Node programs sitting in front of the Agent's own server, and the second Agent it would unify is the one on hold. +- **Delivering into a session the user already has open, or into a desktop App.** Neither vendor gives a third party a way in: a terminal session is its own process with no inbound channel (hooks only go out), and the desktop Apps have no local API. +- **Recognition on the computer** (macOS speech recognition, whisper.cpp). Free and private, but Mac-only or a large bundled model, and the audio still has to cross the link. +- **Spoken replies in a separate "Agent voice"**, so the buddy's lines and Agents' answers are told apart. Rejected: one Character speaking in two ways is one presence; a second voice from the box would be a second one. + +## Consequences + +- ADR-0007 still holds for lines: they are written in advance. Spoken replies are a second way the buddy speaks, written at the moment they are spoken, and they may talk about what the Agents are doing. The persona rule narrows to lines. +- ADR-0009's App still calls no model. The recognition and synthesis service lives in `vibebuddy-cloud` beside the Relay but is not the Relay; it calls the TTS vendor of each Character's voice (ElevenLabs, Volcano Engine Doubao) with our keys, and stores neither audio nor text. +- Recognition is the voice service's: Groq's Whisper (`whisper-large-v3-turbo`, about 1 s), with Workers AI's Whisper as the fallback; both detect the language themselves. A box linked over USB or the local network streams its audio over the link to the daemon, which calls the voice service and sends the spoken reply's audio back the same way, so a box on the cable needs no Wi-Fi; over the Relay the box calls the voice service itself (ADR-0018). A user may set their own OpenAI-compatible transcription endpoint and key instead; the daemon calls it, the key never reaches us, and with it set audio goes over the link even over the Relay (ADR-0015 keeps it private). Box and computer messages: docs/protocol.md, "Voice commands". +- Voice is offered only to signed-in users, so accounts come first; whether it is paid is decided with accounts. Off by default. +- Delivery is per Agent and belongs to its Adapter, which becomes two-way. For Codex the daemon is an app-server client: `thread/start` once, `thread/resume` after a restart, `turn/start` per voice command; the spoken reply arrives as a call to a client-defined `speak` tool (`dynamicTools`), so no MCP server is needed. The thread runs read-only with approvals declined. It lives in a Codex home of the daemon's own, sharing the user's login through a link to their `auth.json` and reading their Codex memories only through the daemon, because the ChatGPT App's app-server holds the writer lock of every thread in `~/.codex` it lists; the Codex App therefore can't open it, and `vibebuddy resident` prints its last exchanges instead. Verified on 2026-10-13 with codex-cli 0.162.0: existing ChatGPT login reused, about 10 s per turn, memory kept across a process restart, file writes refused. +- Claude Code is the second brain, for users without Codex, authenticated only with the user's own Anthropic API key, never a claude.ai subscription or OAuth login, which [Anthropic's terms](https://code.claude.com/docs/en/legal-and-compliance) don't allow third-party products to use. The daemon drives the user's `claude` CLI headless (`-p` with stream-json, no Agent SDK), in a config dir of its own, with `speak` and `list_activities` from an MCP server of its own on a loopback port. The user picks Codex, Claude Code or automatic (Codex when it is ready); each keeps its own session for good. With it, the commands and what `list_activities` returns go to Anthropic, billed to the user's key. Details: docs/resident-session.md, "On Claude Code". +- app-server and `dynamicTools` are marked experimental, so the daemon pins the Codex versions it supports. OpenAI permits ChatGPT login for local and open-source apps; before commercial launch we apply for Sign in with ChatGPT. +- K0 long becomes push-to-talk; giving up a Pomodoro phase stays in the device menu, which already has it (amends ADR-0007, device menu). +- The resident session is one session for good, compacted rather than restarted, so lasting rules belong in a notes file in its working directory. It makes no task cards, announcements or stats. +- The resident session can be built and used from the command line first; the box side waits until the microphone works on the box (I2C scans found no ES7210, so it is probably on the ES8311 ADC). diff --git a/docs/app.md b/docs/app.md index a11324db..89440d8b 100644 --- a/docs/app.md +++ b/docs/app.md @@ -30,6 +30,7 @@ It keeps using local HTTP on `127.0.0.1:7331`. New additions: - `GET /v1/status`: link, current mode, firmware and daemon build identifiers, today's stats, and the time of the most recent hook event. - `GET /v1/status/stream`: SSE, pushed whenever the status changes. +- `GET /v1/resident` and `GET /v1/resident/history`: whether the resident session is on and ready (Codex installed and signed in, or Claude Code installed with the user's Anthropic API key), which Agent answers, a change marker for its history, and its last exchanges, for the Conversation tab; `PUT /v1/resident/brain` and `PUT`/`DELETE /v1/resident/anthropic-key` for its settings, which are kept out of `/v1/config` (the key is never sent back). - Write operations: write a voice pack, update firmware, take a screenshot, restart the daemon itself. Config is owned by the daemon and stored in `~/Library/Application Support/VibeBuddy/config.json` (current voice, notification toggle and so on). The app reads and writes it through the API and never touches the file directly; the existing environment variables remain as a development-time override. The app itself uses UserDefaults only for interface state such as window position. @@ -54,7 +55,7 @@ macOS notifications appear only when the link breaks: the device has been discon ## Settings window -Five tabs, styled like System Settings: tabs across the top, grouped forms, system font and colors. Branding appears only in onboarding, the About page and the menu bar icon. +Five tabs (six while the resident session is on), styled like System Settings: tabs across the top, grouped forms, system font and colors. Branding appears only in onboarding, the About page and the menu bar icon. **General**: Launch at login, the toggle for link-failure notifications, and Updates: a "Check for updates" switch (on by default in builds CI makes, off when built from source), "Check now", the outcome of the last check and a download link when a newer App is out. Next to the versions, "Report a Problem…" does what the menu item does. Below the manifest's `min_supported_app`, a banner at the top says the App is no longer supported; it can't be dismissed and nothing stops working. @@ -64,6 +65,10 @@ Five tabs, styled like System Settings: tabs across the top, grouped forms, syst **Device**: Connection status and serial port name. With several devices plugged in and none of them the box seen before, it lists them (port and USB serial number) with "This is the box"; the daemon remembers the pick as it would a box that reported our firmware (`POST /v1/device/choice`). When `VIBEBUDDY_SERIAL_PORT` or `VIBEBUDDY_USB_SERIAL` narrows the search, it says so and how to unset it, in place of the usual "not showing up" help: a pin left set once made a swapped box look unfound. Then "Box firmware" and "Latest firmware", with "Update to x" when a newer firmware is downloaded; the box's current screen, captured only when you click "Refresh" (over the bridge one capture ties up the serial port for a few seconds, so there's no automatic polling), with "Save image". +**Conversation** (experimental, only while the daemon runs the resident session, `VIBEBUDDY_RESIDENT_SESSION=1`): the last 20 voice commands and the buddy's spoken replies, newest at the bottom, the user's words on the right and the replies on the left as in a chat, read whenever the tab is shown and on Refresh, and, while it is shown, again by itself whenever a new exchange lands (it asks the daemon's change marker every 2 seconds, and stops when the tab or window is left). It is a tab rather than a window of its own opened from the menu: both Apps already have one settings window with tabs and no other windows, the Linux app's tray menu is the same, and a tab can come and go with the daemon's answer without a menu item that is sometimes missing. Voice needs one of two Agents on this computer: the user's Codex, installed and signed in, or the user's Claude Code with their own Anthropic API key (never a Claude subscription, which Anthropic doesn't allow a third-party product to use). Above the conversation, "Answered by" picks Automatic (Codex when it's ready, else Claude Code), Codex or Claude Code, and unless it's Codex a secure field takes the Anthropic API key, with Save and a link to the Claude Console; once saved it shows "Anthropic API key saved" and Remove, never the key. When neither is ready, the tab says what is missing, with a link to OpenAI's Codex CLI install page or `codex login` to run, or a link to Anthropic's Claude Code setup page, or that the key is missing, and the box isn't offered voice. An empty conversation says to hold K0 and talk to the buddy; a daemon that isn't running points to Advanced; a resident session that can't be read shows the daemon's reason. The Linux app has the same tab, with the choice as three buttons. Details in [`resident-session.md`](resident-session.md#the-conversation-tab). + +![The Conversation tab, with nothing said yet](images/app-conversation-empty-zh.png) ![The Conversation tab when the resident session can't be read](images/app-conversation-error-zh.png) + **Advanced**: Open logs, restart the daemon, export diagnostics (logs, config and both sides' build identifiers bundled into one package). ## Onboarding diff --git a/docs/claude-adapter.md b/docs/claude-adapter.md index d249be7a..5f16fa63 100644 --- a/docs/claude-adapter.md +++ b/docs/claude-adapter.md @@ -114,3 +114,7 @@ Hooks stay synchronous; don't use `async`. Async would let events arrive out of ## vibebuddy-hook Since 2026-09-16, hooks are handled by the Rust binary `vibebuddy-hook claude` (source in `hook/`). Its predecessor `tools/claude-hook.py` is retired, and its test cases were carried over one by one. The App copies it to `~/Library/Application Support/VibeBuddy/bin/vibebuddy-hook`, and the command in the configuration is written as `"" claude`. Use this path when configuring by hand too; don't point it inside the App bundle, or it breaks when the App moves (ADR-0005). + +## The resident session + +The daemon's own resident session (ADR-0017) can run on Claude Code, with the user's own Anthropic API key. Its config dir is the daemon's own, so the user's VibeBuddy hook doesn't load for it, but managed hooks could; they are dropped at the entry point, by its `session_id` or by a `cwd` inside its working directory, so it never becomes an Activity. See [`resident-session.md`](resident-session.md#on-claude-code). diff --git a/docs/codex-adapter.md b/docs/codex-adapter.md index 930a5c94..c6b372a2 100644 --- a/docs/codex-adapter.md +++ b/docs/codex-adapter.md @@ -78,6 +78,8 @@ Codex fires the same set of hooks for its own background sessions, typically the So `vibebuddyd` filters at the entry point: any Codex session that has no usable working directory and can't be found in the threads table is ignored (with one info log line): no card, no announcement, no K2 target. Both conditions are required: a freshly created real session may not have been written to the threads table yet, but it has a working directory. Before K2 opens a thread it checks once more that the thread exists, and skips to the next candidate if not. `tools/codex-hook.py` also writes one diagnostic line without the prompt to `~/Library/Logs/VibeBuddy/codex-hooks.log` (event name, session and thread ids, directory, whether there is a transcript), so the next time a session of unknown origin shows up you can see what the hook actually received. +The daemon's own resident session (ADR-0017) is a Codex thread too, and the user's hooks fire for it. Its hooks are dropped at the same entry point, by its thread id or by a `cwd` inside its working directory, so it never becomes an Activity; see [`resident-session.md`](resident-session.md). + The `vibebuddyd` release binary is started automatically after login by `~/Library/LaunchAgents/com.vibebuddy.vibebuddyd.plist` and restarted if it exits abnormally. Logs go to `~/Library/Logs/VibeBuddy/vibebuddyd.log`. When the USB device is temporarily absent, the process keeps running and periodically rediscovers it; hooks don't need to know about plugging and unplugging. After updating the daemon, run `cargo build --release -p vibebuddyd`, then restart the service with `launchctl kickstart -k gui/502/com.vibebuddy.vibebuddyd`. Stop the service before flashing firmware so it doesn't hold the serial port. diff --git a/docs/device-menu.md b/docs/device-menu.md index 0427550f..1e4a5e60 100644 --- a/docs/device-menu.md +++ b/docs/device-menu.md @@ -15,7 +15,7 @@ The frequent actions keep their keys, unchanged: | Gesture | Does | Change | |---|---|---| | K0 | Pomodoro start / pause / resume | none | -| K0 long | Give up the current phase | none (also in the menu) | +| K0 long | Give up the current phase; while the computer offers voice commands, push-to-talk instead (see [Voice commands](#voice-commands)) | none (also in the menu) | | K1 | Switch between On duty and Pomodoro | none | | **K1 long** | ~~Leisure now~~ | **opens the menu** | | K2 | Take me to the agent's window | none | @@ -37,7 +37,7 @@ Muse manages with two keys because many of its boards only have two. This box ha | Row | Shown | K0 does | |---|---|---| -| STOP FOCUS / END BREAK | only while a phase is running, paused or waiting to start its break | gives up the phase, like K0 long, and closes the menu | +| STOP FOCUS / END BREAK | only while a phase is running, paused or waiting to start its break | gives up the phase, like K0 long without voice commands, and closes the menu | | VOLUME | always | steps 20 → 35 → 50 → 65 → 80 → 100 → 20 and plays the "All done!" line at the new level, as the app's preview does | | MUTE | always | toggles mute, like K2 long | | COMPUTERS | on a box with a key (ADR-0012); shows how many computers are paired | opens the COMPUTERS view below | @@ -88,11 +88,30 @@ Every screen shows how the box is linked as a small icon in its top left: a plug ## Behavior around the menu -- **Agent events don't wait for the menu.** An event that needs the user (input required, failed) closes the menu so the screen can show it; its line is announced as usual. Other events (working, done) update the screen underneath and show once the menu closes. A Pomodoro phase ending closes the menu too, since its alarm needs K0. +- **Agent events don't wait for the menu** (the voice overlay is different: there they wait, see docs/protocol.md, "Voice commands"). An event that needs the user (input required, failed) closes the menu so the screen can show it; its line is announced as usual. Other events (working, done) update the screen underneath and show once the menu closes. A Pomodoro phase ending closes the menu too, since its alarm needs K0. - **Leisure** is left for On duty the moment the menu opens, like any key press today, and doesn't start while the menu is open. - **Volume and mute stay where they are.** The menu changes the same values as the app and K2 long. A volume change from the menu prints the `VOLUME ` line `device.volume` already uses, so the daemon's status, and the app's slider, follow; mute keeps its `MUTE ON` / `OFF` lines and stays device-only, as [`app.md`](app.md) says. - **Nothing moves to the box that belongs to the Mac.** Connecting agents, the voice library, firmware updates, launch at login and notifications all stay in the app: they need the Mac's files, network or system settings. The box holds one voice pack at a time, so there's nothing to choose between on it. The menu is about making the box's own controls visible, not about replacing the app. +## Voice commands + +Status: built 2026-10-13, not yet tried on a box; experimental, off until the computer offers it (ADR-0017, [`protocol.md`](protocol.md), "Voice commands (experimental)"). The screens are rendered by the firmware on the Mac: `cargo run -p vibebuddy-firmware-core --example voice_preview -- `. + +A voice command has an overlay of its own, drawn like the menu's panel and, like the menu, not a mode: the mode underneath keeps running. + +| Gesture | Without voice commands | While the computer offers them | +|---|---|---| +| K0 long | gives up the Pomodoro phase | push-to-talk: records while K0 stays down, up to 30 s; giving up the phase is the menu's STOP FOCUS / END BREAK | + +![Recording](images/voice-listening.png) ![What was heard and the reply](images/voice-reply.png) ![An error](images/voice-error.png) + +- **LISTENING** appears when the long press fires, a second after K0 went down, and that is when recording starts. The meter shows the level of each 100 ms (green, then yellow, then orange near the top); under it, the seconds left. Letting go sends it; under half a second is taken as a slip and cancelled. K2 cancels; other keys do nothing while recording. +- **THINKING** follows until the computer answers. What it heard shows dimmed, then each spoken reply as it comes, in large type while it fits and small type after that; past what small type holds, the oldest rows give way. The title says **REPLY** once there is one. +- After the computer says the turn is done, the text stays up 15 s to be read. An error shows a short message for 6 s: NOT HEARD, UNAVAILABLE, BUSY, or NO MICROPHONE when the box couldn't start recording. With nothing from the computer for a minute, the overlay closes on its own. +- K2 closes the overlay; a long K0 starts the next command; any other key closes it and does its own job. +- **Events that need the user win**, as with the menu: input required, failed, or a Pomodoro phase ending closes the overlay, and a recording in progress is given up (`VOICE CANCEL`). +- **Text only.** Spoken replies are text until there is speech synthesis, so mute changes nothing here. Chinese draws in a 12 px pixel font beside the 5×7 ASCII one, and text with Chinese in it uses small type throughout ([`firmware-bringup.md`](firmware-bringup.md), "Fonts"). + ## Constraints from the hardware - **Backlight is on or off**: it is bit 7 of the XL9555 expander, not a PWM pin, so there is no brightness row. diff --git a/docs/firmware-bringup.md b/docs/firmware-bringup.md index d50d482b..f4c5bf32 100644 --- a/docs/firmware-bringup.md +++ b/docs/firmware-bringup.md @@ -39,12 +39,29 @@ The review fixes applied before bring-up (see the PR) were in place, so it can't | UART receive | More `invalid_json` with only the UART bridge connected | The interrupt and ring buffer in `device/src/serial.rs` | | USB output blocking | Frozen screen after a night on the UART bridge alone (see LESSONS.md on output channels without a reader) | `write_usb` in `serial.rs` | | Settings storage | Wrong pomodoro tally or volume after a restart | `core/src/storage.rs` | +| The box restarts by itself | The boot report again in the daemon's log, unasked | The `RESET` line just before `MEMORY`: esp-hal's reset reason, such as `Some(CoreMwdt0)` or `Some(RtcWdt…)` for a watchdog and `Some(SysBrownOut)` for a brownout, plus ` PANIC` when a panic caused it (its backtrace went out on the port before the reset). A panic and the restart after a new identity are both `Some(CoreSw)`; only the mark tells them apart. A reset from the flasher or the USB port shows as `Some(CoreUsbJtag)` or `Some(CoreUsbUart)`. `Some(ChipPowerOn)` can also be a brownout of the whole chip. | ## Intentional differences from the C firmware - Settings (today's pomodoro tally, the volume) use a new storage format, so they reset once when the Rust firmware is first flashed. - `voice.begin` interrupts the line currently playing before erasing the partition; the C firmware waited up to 10 seconds for it to finish. - JSON parsing is stricter than cJSON: invalid UTF-8, `"version":1.0` and a non-string `message` get `ERROR invalid_message` (or `invalid_json`). The Mac never sends these. When `device.echo` is used to hunt corrupted bytes, a corrupted line now gets `ERROR invalid_json` instead of a CRC reply, which says the same thing. +- Task titles, project names and the voice overlay draw Chinese (see Fonts below); the C firmware draws a placeholder for every byte of it. ASCII is unchanged: `tools/compare-display.sh` still finds every English screen identical. + +## Fonts + +ASCII is the 5×7 bitmap font in `core/src/canvas.rs`, uppercase only, scaled 1×, 2× or 3×. Everything else comes from a 12 px Chinese font in `core/src/font.rs`: + +- **What's in it**: the 3,500 characters of level one of the Table of General Standard Chinese Characters (通用规范汉字表 一级, 2013), CJK punctuation (、。《》「」【】…), the full-width forms (，！？：；（）) and a few more marks (“”‘’…—·→): 3,678 glyphs. A character outside it, an emoji say, draws as a dotted box the size of a Chinese character, so a gap is visible rather than silent. +- **Where it comes from**: [Fusion Pixel Font](https://github.com/TakWolf/fusion-pixel-font) 12 px monospaced, zh_Hans shapes, under the SIL Open Font License 1.1. Fusion is Ark Pixel Font with the characters Ark hasn't drawn at 12 px yet (over a hundred of level one, among them 热, 然 and 紧) filled in from Cubic 11. It was chosen because it is a pixel font drawn for 12 px, not an outline font rasterized down: strokes land on whole pixels and every glyph keeps a blank row and column for spacing, so a line of Chinese stays legible on the 320×240 panel (Muse's 16 px Unifont captions, which fill their whole cell, ran together). 12 px is also the smallest size that keeps level-one characters readable, and close to 2× ASCII's 14 px. +- **How it's built**: `tools/make-cjk-font.py` downloads the font and Unicode's Unihan database (whose `kTGH` field numbers the 2013 table) pinned by SHA-256, and writes `core/fonts/cjk-12px.bin`: a sorted index of 16-bit codepoints, looked up by binary search, then one packed 1-bit 12×12 bitmap per glyph, 18 bytes. 73,562 bytes in all, committed, plus the font's license files beside it, which the firmware zip carries as `FONTS.txt`. The table is `include_bytes!`, read-only data that stays in flash: it adds about 75 KB to the app image and nothing to RAM or the main stack. +- **Layout**: a Chinese character advances 12 px and is centred on the ASCII line, which at 1× puts the font's own baseline on the ASCII baseline. It stays 12 px next to 2× ASCII (titles, large type). Width is measured per glyph (`font::width`, `font::fit`); text is cut at whole characters. `font::wrap` breaks between any two Chinese characters but never inside an ASCII word unless the word is longer than a row, and lets closing punctuation (，。！？ and the like) hang past the row's end rather than start a row. Curly single quotes next to English letters draw as the ASCII apostrophe. +- **Where it's used**: task card titles and project names, the title under the buddy, the agent line in Pomodoro mode, and the voice overlay. Menus and labels stay ASCII. The daemon still squashes session titles to uppercase ASCII before sending them ([`pet.md`](pet.md)), so Chinese titles reach the cards only once it stops doing that for firmware that can draw them; voice text already arrives as it was spoken. +- **Voice overlay**: text with Chinese in it goes straight to small type (rows 13 px apart, 10 rows); large type, with doubled ASCII beside 12 px Chinese, looked shouty. English-only text keeps large then small type as before. + +![Chinese task titles](images/device-cards-zh.png) ![A Chinese reply](images/voice-reply-zh.png) ![A longer one, mixed with English](images/voice-reply-zh-small.png) + +The pictures come from `cargo run -p vibebuddy-firmware-core --example voice_preview -- `. ## Steps @@ -91,6 +108,99 @@ The review fixes applied before bring-up (see the PR) were in place, so it can't 7. **Change the voice once in the app.** When the write finishes, the box speaks in the new voice. Power-cycle it and check that the voice stays. 8. **Leave it on the UART bridge alone overnight**, with nothing on the USB port, and check the next morning that the screen still animates. This verifies that an output channel without a reader never blocks. +## Microphone + +The box has a microphone, and until 2026-10-13 no firmware of ours had used it. `device.mic.record` is a bring-up diagnostic, like the screenshot: it records a few seconds and sends them back over the cable. It isn't a user feature, and neither app uses it. + +### What we know, and from where + +No ALIENTEK schematic or BSP for this board (ATK-DNESP32S3-BOX V1.1) could be found; the BOX3's wiki (a different board, with different I2S pins) doesn't count. So each fact below was either seen on a box or is a candidate the diagnostic settles. The first recording, on box `30:ED:A0:A4:0D:08` on 2026-10-13, settled the microphone path ([`hardware.md`](hardware.md)). + +| Fact | Value | Source | Confidence | +|---|---|---|---| +| Microphone ADC | The ES8311's own mono ADC; no ES7210 | No chip at 0x40–0x43 in the I2C0 scans of two boxes ([`hardware.md`](hardware.md)); xiaozhi 1.9.4's `atk-dnesp32s3-box` uses `Es8311AudioCodec` for both input and output | Verified on `30:ED:A0:A4:0D:08`: no chip at 0x40–0x43, and the recording is the voice | +| ES8311 address | 0x18 (7-bit) | Seen on boxes | Verified | +| ES7210 addresses checked | 0x40–0x43 (AD1/AD0 straps; esp_codec_dev's `es7210_address_t`) | esp_codec_dev | The diagnostic reports each | +| Microphone data into the ESP32 | GPIO47 (I2S DIN) | xiaozhi 1.9.4 `config.h`, `AUDIO_I2S_GPIO_DIN` | Verified on `30:ED:A0:A4:0D:08` | +| Clocks | Shared with the speaker: BCLK GPIO21, WS GPIO13, no MCLK. The ESP32's receiver runs off its transmitter's clocks (signal loopback), as ESP-IDF's full duplex does | Speaker pins verified on boxes; xiaozhi creates TX and RX on one I2S port with the same pins and `GPIO_NUM_NC` for MCLK | High | +| Format and rate | Standard I2S, 16-bit, two slots, 24 kHz. Not 16 kHz: the ES8311 has one LRCK for ADC and DAC, so input runs at the speaker's rate | xiaozhi asserts input rate == output rate (24000) for this codec | High | +| Which slot is the microphone | Left = ADC, right = DAC reference (silent unless the box is speaking) | esp_codec_dev writes ES8311 register 0x44 = 0x58, "ADCL + DACR"; our sequence already does | Verified on `30:ED:A0:A4:0D:08`: the voice is on the left; the right stays silent while the box is quiet | +| ADC already enabled at boot | Yes: our ES8311 sequence is esp_codec_dev's `ESP_CODEC_DEV_WORK_MODE_BOTH`, the mode the C firmware used, so the ADC is powered and its serial output unmuted (registers 0x0A, 0x0E, 0x14, 0x15, 0x17) | `firmware/main/agent_audio.c`; `core/src/audio.rs` test | Verified in code | +| Microphone gain | Register 0x16, 0–42 dB in 6 dB steps; 30 dB by default (xiaozhi's `AUDIO_CODEC_DEFAULT_MIC_GAIN`). At 30 dB a voice close to the box is only about -31 dBFS, so real use needs more | esp_codec_dev `es8311_set_mic_gain`; the recording | High | +| Boxes on the NS4168 variant | No codec; xiaozhi reads a digital microphone straight from the same DIN with the same clocks | xiaozhi `ATK_NoAudioCodecDuplex` | Untested; the diagnostic still records | + +### Running it + +1. Quit the VibeBuddy app, then flash with `just flash /dev/cu.usbmodemXXXX` (step 2 above). Check the boot report still says `AUDIO READY` and that announcements still play: the speaker's I2S setup changed (the receiver now shares its clocks). +2. Record, making some noise near the box (talk, clap, tap the case) while it records: + + ```bash + uv run --with pyserial python tools/mic-record.py --port /dev/cu.usbmodemXXXX --ms 3000 --out mic.wav + ``` + + `--gain-db` sets the microphone gain. The samples come back at about 11 s per recorded second, because the firmware writes the UART at 115200 baud whichever port you use. +3. Listen to `mic.wav`. The left channel should be your voice. + +The tool judges each slot from the samples, with the DC offset removed and the first 100 ms dropped: `LIVE` (the level moves at least 6 dB and the loudest 100 ms is above -70 dBFS), `FLAT`, `NO SIGNAL`, `SILENT` (all zeros) or `SATURATED`. Its last line, `microphone on this box: WORKS` or `NOT SHOWN`, rests on the left slot, the microphone; it exits 0 only for `WORKS`. + +The box replies, in order: + +``` +MIC PROBE 0x18 ID 8311 ES8311 chip ID (0xFD, 0xFE); NONE if nothing answers +MIC PROBE 0x40 NONE 0x40–0x43: an ES7210 would answer here, with the bytes of 0x3D, 0x3E +... +MIC GAIN 30 DB only with an ES8311 +MIC RECORDING 3000 MS +MIC CAPTURED 3010 MS 288000 BYTES +MIC LEVEL 0 L R one per 100 ms, raw units, full scale 32768 +MIC BEGIN 24000 2 16 288000 +MIC DATA 576 bytes of samples per line +MIC END 288000 +``` + +`MIC UNAVAILABLE` means the board has no input; `MIC BUSY`, that a recording is still going out; `MIC ERROR STALLED`, that the DMA hadn't handed back enough samples a second after the recording should have ended; `MIC ERROR SHORT`, that the buffer filled up with fewer samples than asked for; `MIC ERROR MEMORY`, that the capture (PSRAM) or its DMA descriptors (internal RAM, taken only while recording) didn't fit. + +The DMA writes the whole recording straight into one buffer in PSRAM, so nothing depends on the main loop keeping up. The buffer is 2.5 times the recording: the I2S receiver signals an EOF every 4,094 bytes, and each EOF closes the DMA descriptor it lands in, so the descriptors come back 4080, 14, 4080, 14, …. esp-hal's `read()` sets RXEOF_NUM to 0xFFE, and on the ESP32-S3 that register counts bytes (as ESP-IDF uses it), not samples as the TRM's formula reads. Each descriptor's length says how many bytes it holds, and the samples stay in order. Lining the EOFs up with the descriptors would mean a raw write to RXEOF_NUM after esp-hal's, racing the first EOF; it hasn't been tried. Each recording also prints two `MIC DEBUG` lines, 100 ms in (`T100`) and at the end (`END` or `TIMEOUT`), for when it fails: + +``` +MIC DEBUG T100 DESC 4/179 BYTES 8188 LENS 4080 14 4080 14 0 0 I2S RXCONF … TXCONF … INTRAW … RXCLKM … RXEOF … TXSTATE … DMA ININT … INFIFO … INLINK … INSTATE … INDSCR … INPERI … OUTSTATE … OUTPERI … DIN 200 +``` + +- `DESC done/total` and `BYTES`: DMA descriptors (up to 4,080 bytes each) handed back so far, and the sample bytes in them; at `T100` up to 9,600 (96 KB/s), counted only for descriptors already handed back. `DESC 0` means no sample reached memory. +- `LENS`: the first six descriptors' lengths, which show where the EOFs cut them. +- `I2S …`: I2S0's RX_CONF (0x20), TX_CONF (0x24), INT_RAW (0x0C), RX_CLKM_CONF (0x30), RXEOF_NUM (0x64), TX status (0x6C). RX_CONF bit 2 is RX_START and bit 3 RX_SLAVE_MOD; TX_CONF bit 2 is TX_START and bit 27 SIG_LOOPBACK (field offsets from the esp32s3 0.37 PAC). +- `DMA …`: GDMA channel 1's IN_INT_RAW, INFIFO_STATUS, IN_LINK, IN_STATE, IN_DSCR (the descriptor it is on), IN_PERI_SEL (I2S0 is 3), and OUT_STATE and OUT_PERI_SEL for the speaker's side of the same channel. +- `DIN`: level changes over 2000 quick reads of GPIO47. 0 means the data line doesn't move (or the codec sends exact silence). BCLK and WS can't be checked this way: they are outputs with their input path off, so their input register reads 0 however fast they run. + +### Reading the result + +| What you see | What it means | +|---|---| +| Left `LIVE`, rising when you talk; right silent | The microphone works on this box, as on `30:ED:A0:A4:0D:08`. Record the box's serial number in [`hardware.md`](hardware.md) | +| Both slots `SILENT` (all zeros) | Nothing drives GPIO47: wrong pin, or the ES8311's ADC output isn't on | +| Both slots stuck at one value, or `SATURATED` with no sound | GPIO47 is floating or isn't the codec's data line | +| Left `FLAT` with a level above zero | The ADC runs but the microphone isn't reaching it: try `--gain-db 42` and noise right at the mic hole; then suspect the analog path (MIC bias, which input the ES8311 selects in register 0x14) | +| `LIVE` on the right, not the left | The slots are the other way round from esp_codec_dev's 0x44 setting | +| Any `MIC PROBE 0x4x ID …` | This box has a chip where an ES7210 would be: it differs from the scanned boxes, so stop and look before trusting the ES8311 path | + +Boxes of this board differ (some have a touchscreen, some don't), so run it on each box you care about and say which box in the result. + +### Voice commands + +A voice command records the same input, but for as long as K0 is held (up to 30 s), so it can't use one linear buffer: that would be 7 MB. The DMA writes round a ring instead, 128 descriptors over 520 KB of PSRAM, linked in a circle with owner checks off, so it never reaches an end and never stops, however late the reader. The microphone task reads the descriptors the DMA has handed back every 10 ms, takes the left slot's samples out (the stream runs on across descriptors: frames don't line up with the 4,094-byte EOFs) and queues them for the main loop, which encodes and sends them. An EOF every 4,094 bytes takes two descriptors, so the ring holds 2.7 s of audio before the DMA comes round to what hasn't been read. The ring and the queue are taken from the heap when recording starts and given back when it stops: no statics (LESSONS.md). + +The microphone gain is 42 dB while recording a voice command (the PGA's maximum): at the diagnostic's 30 dB a voice close to the box was about -31 dBFS; 42 dB should bring it to about -19. + +When a recording stops, the box prints one line about it: + +``` +MIC STREAM MS SAMPLES DROPPED BACKLOG OF 128 +``` + +About 24 samples per millisecond means nothing was lost on the way in. `DROPPED` counts samples the main loop's 2 s queue had no room for. A `BACKLOG` near 128 means the DMA lapped the reader and some audio was overwritten. `MIC STREAM MEMORY` and `MIC STREAM I2S` mean the ring couldn't be allocated or the transfer didn't start. + +What to check on the first flash: with the computer offering voice, hold K0 and talk. `VOICE START 1`, then about 21 `VOICE DATA` lines a second with `seq` counting up without gaps (over USB), `VOICE END 1 ` with `samples` about 24,000 per second held, and `MIC STREAM` with about 24 samples per millisecond and a small backlog. The meter should move with your voice and sit near the bottom when quiet. Decode the data lines (24 kHz IMA ADPCM, each line from its own `predictor` and `index`) and listen. Check that announcements still play afterwards, and that a long K0 still gives up a phase once voice is withdrawn. + ## Going back to the C firmware ```bash diff --git a/docs/hardware.md b/docs/hardware.md index 43c7c90f..2e0d55ce 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -69,6 +69,16 @@ A probe firmware, flashed to the app slot only, scanned I2C0 (SDA GPIO48, SCL GP - `0x1E`, `0x50` and `0x6A` were only seen answering; their identities above are guesses from common parts at those addresses, not read from the chips. - Dragon's daily box (`98:88:E0:06:8B:CC`) hasn't been probed. +### Microphone (2026-10-13) + +`device.mic.record` ([`firmware-bringup.md`](firmware-bringup.md#microphone)) recorded 5 s on box `30:ED:A0:A4:0D:08` (the one with the touchscreen): `MIC CAPTURED 5175 MS 480000 BYTES`, CRC correct. + +- The microphone is on the **ES8311's own ADC**. The ES8311 answers at 0x18 with chip ID `8311` (registers 0xFD, 0xFE); nothing answers at 0x40–0x43, so there is **no ES7210** on this box. +- Samples come in on **GPIO47** (I2S DIN), clocked by the speaker's BCLK GPIO21 and WS GPIO13 (24 kHz, 16-bit, two slots), with the ES8311 set up as esp_codec_dev does (register 0x44 = 0x58). +- The **left slot** is the microphone: dragon spoke close to the box and confirmed by ear that the recording is that voice. The right slot is the DAC reference and stays silent while the box doesn't speak (peak 908 raw, about -133 dBFS once the DC offset is removed). +- Levels at the default 30 dB gain, per 250 ms on the left: -31 to -35 dBFS while speaking close, -43 dBFS in a pause. That is quiet; real use needs more gain (up to 42 dB in the ES8311's PGA). +- Other boxes, including dragon's daily box, haven't been recorded. + ## Corroboration from the currently running firmware The hardware runs `xiaozhi` 1.9.4. The source at that pinned version contains an `atk-dnesp32s3-box` board directory that records the ST7789 i80, XL9555, ES8311, and a set of board-level pins. This corroborates the boot log and can be used as a candidate for cross-checking, but that repository is not ALIENTEK's official schematic/BSP and cannot bypass the PCB revision gate to freeze pins directly. @@ -81,5 +91,5 @@ The hardware runs `xiaozhi` 1.9.4. The source at that pinned version contains an - Vendor schematic, BSP, and examples matching the exact PCB revision. - Touch controller wiring (RST, INT) on boxes that have one; whether a box has one is decided by probing `0x2E` (see above). - The chips at `0x1E`, `0x50` and `0x6A`. -- Microphone input chain. +- The microphone on other boxes (verified on `30:ED:A0:A4:0D:08` only, see above). - Buzzer GPIO and active level. diff --git a/docs/images/app-conversation-empty-zh.png b/docs/images/app-conversation-empty-zh.png new file mode 100644 index 00000000..3d145fff Binary files /dev/null and b/docs/images/app-conversation-empty-zh.png differ diff --git a/docs/images/app-conversation-error-zh.png b/docs/images/app-conversation-error-zh.png new file mode 100644 index 00000000..7d42cf52 Binary files /dev/null and b/docs/images/app-conversation-error-zh.png differ diff --git a/docs/images/app-conversation-zh.png b/docs/images/app-conversation-zh.png new file mode 100644 index 00000000..7c74409b Binary files /dev/null and b/docs/images/app-conversation-zh.png differ diff --git a/docs/images/device-cards-zh.png b/docs/images/device-cards-zh.png new file mode 100644 index 00000000..a74c8d52 Binary files /dev/null and b/docs/images/device-cards-zh.png differ diff --git a/docs/images/voice-error.png b/docs/images/voice-error.png new file mode 100644 index 00000000..fa1fb9f3 Binary files /dev/null and b/docs/images/voice-error.png differ diff --git a/docs/images/voice-listening.png b/docs/images/voice-listening.png new file mode 100644 index 00000000..60deb198 Binary files /dev/null and b/docs/images/voice-listening.png differ diff --git a/docs/images/voice-reply-zh-small.png b/docs/images/voice-reply-zh-small.png new file mode 100644 index 00000000..4376de79 Binary files /dev/null and b/docs/images/voice-reply-zh-small.png differ diff --git a/docs/images/voice-reply-zh.png b/docs/images/voice-reply-zh.png new file mode 100644 index 00000000..6aa57ee2 Binary files /dev/null and b/docs/images/voice-reply-zh.png differ diff --git a/docs/images/voice-reply.png b/docs/images/voice-reply.png new file mode 100644 index 00000000..24cb6fc2 Binary files /dev/null and b/docs/images/voice-reply.png differ diff --git a/docs/pomodoro.md b/docs/pomodoro.md index 923d037b..03cc6447 100644 --- a/docs/pomodoro.md +++ b/docs/pomodoro.md @@ -28,7 +28,7 @@ The case has three buttons, K0, K1, and K2 (plus RST). Each button does one thin | Button | Short press | Long press (1 second) | | --- | --- | --- | -| K0 | Pomodoro: start / pause / resume | Abandon the current phase and return to idle; when the break is ready to start, this skips the break | +| K0 | Pomodoro: start / pause / resume | Abandon the current phase and return to idle; when the break is ready to start, this skips the break. While the computer offers voice commands, push-to-talk instead, and abandoning is in the device menu ([`device-menu.md`](device-menu.md#voice-commands)) | | K1 | Toggle between On duty and Pomodoro | Send the buddy to Leisure right now | | K2 | Open the current source (reported to the Mac, behavior unchanged) | Mute toggle | diff --git a/docs/protocol.md b/docs/protocol.md index cc5b8804..7d8eb321 100644 --- a/docs/protocol.md +++ b/docs/protocol.md @@ -104,6 +104,8 @@ These messages are the app operating on the device itself. They don't count as a `device.screenshot` asks for the current screen. The device replies with `SHOT BEGIN 320x240 BACKLIGHT ON|OFF`, then `SHOT` lines of space-separated `rgb565:length` runs (the colour as four hex digits, row by row from the top left), then `SHOT END`; the daemon turns them into a PNG (`POST /v1/device/screenshot`), as does `tools/device-screenshot.py` over the cable. Over the cable the whole frame goes out at once, 16 runs a line. Over the local network or the Relay the device copies the frame when asked and sends one line every 30 ms, up to 180 runs (under 2 KB) a line, so it stays inside the Relay's 50 messages a second ([`relay-protocol.md`](relay-protocol.md)) and waits for room in the box's network buffer rather than drop part of a line; the picture is the screen at the moment of asking, however long the lines take. The idle screen takes one to two seconds; a screen with no two neighbouring pixels alike, about 13 s or more. Older firmware sends screenshots over the cable only, so over the network the request times out. +`device.mic.record` is a bring-up diagnostic for the microphone, over the cable only (over the network it gets `ERROR cable_only`): `ms` is the length, 100 to 5000 (default 3000), and `gain_db` the ES8311 microphone gain, 0 to 42 (default 30). The box reports what answers on I2C (`MIC PROBE`), records, then sends `MIC LEVEL` lines (RMS and peak per 100 ms), `MIC BEGIN`, one `MIC DATA ` line of 576 sample bytes per main loop pass, and `MIC END `. The daemon never sends it; `tools/mic-record.py` does. Lines and their meaning are in [`firmware-bringup.md`](firmware-bringup.md#microphone). + Character packs (format in `firmware-rs/core/src/character_pack.rs`), and the older voice packs (`firmware-rs/core/src/voice_pack.rs`), are written into the `voices` partition over the same serial link, without resetting and without esptool; both connection types behave the same. Stop-and-wait flow control: after each chunk the Mac waits for the device's acknowledgment, and sends nothing further without one. ```json @@ -189,6 +191,41 @@ Once on Wi-Fi the box also answers mDNS (DNS-SD) for `_vibebuddy._tcp` on port 7 USB stays in the clear: the cable is physical presence, as pairing already assumes. +## Voice commands (experimental) + +A voice command (ADR-0017) on a box linked over USB or the local network: the box records while K0 is held and streams the audio over the link; the computer turns it into text, hands it to the resident session and sends the spoken replies back. Over the Relay the box will talk to the voice service itself (ADR-0018); that path isn't described here yet. + +The computer offers voice, and the box only treats a long K0 as push-to-talk while it is offered: + +- `{"version":1,"event":"voice.available","on":true}`: sent after `device.hello` is answered (the daemon sends it on every `DISPLAY READY BUILD`, so a box that rebooted on the same link is told again), and again whenever it changes, when the resident session is switched on and ready (the user's Codex installed and signed in; see [`resident-session.md`](resident-session.md#readiness)) and speech recognition is configured; `"on":false` withdraws it. A box that hasn't been told, or was told `false`, keeps K0 long as it was. A new link starts with voice off until the computer says otherwise. + +One recording is a turn, numbered by the box from 1 after each boot. The box sends diagnostic lines, each under 1024 bytes: + +- `VOICE START `: K0 has been held for the long-press time and the box is recording. Recording starts here, a second after K0 went down, when the screen says LISTENING; nothing from before is kept. +- `VOICE DATA `: the next stretch of audio, `seq` from 0 without gaps. Audio is 24 kHz mono IMA ADPCM, two samples per byte, low nibble first, the same coding the box plays Character packs with (`firmware-rs/core/src/adpcm.rs`). `predictor` (a signed 16-bit sample) and `index` (0–88) are the coder's state before the line's first nibble, so every line decodes on its own and a lost line costs only its own stretch. At most 768 base64 characters per line. +- `VOICE END `: K0 was released, or 30 s passed; `samples` is how many were recorded, always an even number (a last sample without a partner for its byte isn't sent). It counts the samples of lines that were dropped too, so the computer can tell how much is missing. +- `VOICE CANCEL `: the recording was too short (under half a second) or the box gave it up (K2, or an event that needs the user took the screen, or voice was withdrawn); the computer drops the turn. + +`VOICE DATA` comes to about 21 lines and 16 KB a second, and goes only over the native USB port and the local network. The UART bridge carries 11.5 KB/s at 115200 baud and its writes hold up the box's main loop, so data lines never go there; a computer on the bridge would see `START` and `END` with no data, so it shouldn't offer voice there. The Relay doesn't carry them either (see above). A data line that doesn't fit right now (the USB host isn't reading, the network connection's buffer is full) is dropped whole rather than waited for: a gap in `seq`, never a stalled recording. `START`, `END` and `CANCEL` go everywhere the box's other lines go. + +The computer answers with JSON events carrying the same `turn`: + +- `voice.heard` with `text`: what was recognised (cut to fit the line), so the box can show it. +- `voice.reply` with `text`: one spoken reply, sent the moment the resident session says it; a turn may have several. +- `voice.done`: the turn is over; it comes after the last reply's audio `end`. The box leaves what it heard and the replies up for 15 s to be read (K2 closes them sooner), then goes back to what it showed before; with nothing to show, it goes back at once. +- `voice.error` with `reason` (`not_heard` when recognition returned nothing or no audio arrived, `unavailable` when recognition or the resident session failed, `busy` when another turn is still running) and an optional `message`; the turn is over. It may come while the box is still recording (`busy` right after `START`): the box stops recording and sends nothing more for that turn. The box shows `message` if there is one, otherwise a short text of its own, for 6 s. + +`voice.error` is also the name of the box's reply to a failed voice pack write ("Device maintenance" above). The two never meet: that one only goes from the box to the computer, this one only from the computer to the box, and it carries `turn`. + +The box ignores answers for any turn but the one on its screen: after K2 closed it, after a new turn started, or after a minute without a word from the computer, when it closes the overlay on its own. + +The computer also sends each spoken reply's audio, synthesized in the Character's voice, right after its `voice.reply`: + +- `voice.audio` with `reply` (0 for the turn's first `voice.reply`, counting up), `seq` (from 0, without gaps) and `data`: base64 of the next bytes of an MP3 stream (MPEG-2 Layer III, 24 kHz mono, 48 kbps, as the voice service returns it), at most 768 base64 characters. Chunks don't follow MP3 frame boundaries; the box buffers and decodes frame by frame. +- `voice.audio` with `reply`, `seq` and `"end":true` (no `data`): that reply's audio is over. With `"error":"synthesis"` as well, synthesis failed or was cut short: the box plays what it has, keeps the text, and logs the failure instead of passing it off as a whole reply. + +The box plays replies in order, one after another, and starts a reply as soon as it has enough frames, without waiting for its end. If synthesis fails the reply stays text only. While muted the box shows replies without playing them. The box plays audio only for the turn it recorded last (`voice.audio` carries `turn` like the other answers), and stops a turn's replies for good, ignoring the rest of its audio, when the Pomodoro chime is about to play (it wins), when it is muted, when a new K0 push-to-talk starts, or when the overlay closes. The overlay stays up past its read time while a reply is still being spoken. An Agent's news during a voice command (needs input, failed, done) doesn't close the overlay or cut a reply short: the overlay marks needs input and failed in yellow, and the worst of what came is announced once, when the overlay closes (`AUDIO HELD ` when it is held). A missing chunk is skipped (the decoder finds the next frame, so the reply only loses a moment); a chunk arriving after a later one is ignored. + ## Stage 1 error output | Output | Meaning | diff --git a/docs/resident-session.md b/docs/resident-session.md new file mode 100644 index 00000000..31b85dc4 --- /dev/null +++ b/docs/resident-session.md @@ -0,0 +1,186 @@ +# Resident session + +The resident session is the one long-running Agent session every voice command goes to ([ADR-0017](adr/0017-voice-commands-go-to-a-resident-session-that-speaks-as-the-buddy.md); see "Voice commands" in [`CONTEXT.md`](../CONTEXT.md)). A voice command reaches it spoken to a box linked over USB or the local network (below), or typed with `vibebuddy say`. + +It runs on one of two brains: the user's own Codex (a Codex thread, most of this page), or the user's own Claude Code with their own Anthropic API key (a Claude Code session, "On Claude Code" below). Which one is a setting: `codex`, `claude`, or `auto`, the default: Codex when it is ready, else Claude Code when a key is set. Each brain keeps its own session for good; switching brains switches which session voice commands go to and which history the Conversation tab shows. + +It is experimental and off by default. Start `vibebuddyd` with `VIBEBUDDY_RESIDENT_SESSION=1` to turn it on. Voice needs one of the brains on the computer the daemon runs on: the Codex CLI installed and signed in (`codex login`), or Claude Code installed with the user's Anthropic API key entered in the Conversation tab. Without either the box isn't offered voice, and the Conversation tab says what's missing. While it is on, both Apps' settings windows have a Conversation tab that shows it (below); nothing else of it is in the App UI, on the Mac or on Linux. + +## Using it + +```sh +vibebuddy say "What are my agents doing?" # prints the spoken reply(ies); the time taken goes to stderr +vibebuddy resident [n] # prints the thread id and the last n (default 5) commands with their replies +``` + +The thread lives in a Codex home of the daemon's own (below), so the Codex App can't open it; `vibebuddy resident`, or the Apps' Conversation tab, is how to look back at it. + +### The Conversation tab + +Both Apps ask `GET /v1/resident` each time the daemon comes up (only its environment at start decides) and show a Conversation tab, between Device and Advanced, only while it says `enabled`. The tab reads the last 20 exchanges with `GET /v1/resident/history?limit=20` each time it is shown and on Refresh, and lists them oldest first, newest at the bottom: each voice command on the right, the buddy's spoken replies under it on the left. With none yet, it says how to talk to the buddy (hold K0); when the daemon can't be reached it points to Advanced → Restart daemon, and when the resident session can't be read (Codex missing or not logged in) it shows the daemon's reason. The last answer from `GET /v1/resident` is kept while the daemon is down, so an open tab says the daemon isn't running rather than vanishing. + +While the tab is shown it updates by itself: it asks `GET /v1/resident` every 2 seconds and reads the history again only when `revision` moved (or voice just became ready), so a new exchange shows up on its own and the list stays scrolled to the newest. It stops asking when another tab is picked or the window is closed. A daemon older than `revision` gets its history read every 5 seconds instead. When the answer says voice isn't ready, the tab says what's missing in place of the conversation: Codex not installed, with a link to OpenAI's [Codex CLI page](https://learn.chatgpt.com/docs/codex/cli); or not signed in, with `codex login` to run in a terminal (under Automatic it adds that Claude Code can answer instead, with the user's own key); Claude Code not installed, with a link to Anthropic's [setup page](https://code.claude.com/docs/en/setup); or no Anthropic API key yet. It notices the fix within half a minute. + +Above the conversation, while the daemon says `brain_setting`, the tab has the brain setting, "Answered by": Automatic, Codex or Claude Code (`PUT /v1/resident/brain`). Unless it is Codex, a secure field takes the user's Anthropic API key, with Save, and a link to the [Claude Console](https://platform.claude.com/) to make one; once the daemon has a key the field gives way to "Anthropic API key saved" and Remove. The key is never shown again: the daemon only says whether it has one. A line under it says the key is billed to the user's Anthropic account, that a Claude subscription can't be used, and that what they say and their Agents' task titles go to Anthropic. + +![The Conversation tab on the Mac, in Chinese](images/app-conversation-zh.png) + +`vibebuddy` is a second binary of the daemon package (`daemon/src/bin/vibebuddy.rs`). Like every client, it only talks to the daemon's HTTP API, at `VIBEBUDDY_BIND` (default `127.0.0.1:7331`): + +| Endpoint | Does | +| --- | --- | +| `POST /v1/resident/say` `{"text"}` | Runs one turn; answers `{"thread_id","replies":[...],"elapsed_ms"}`. 404 when off, 503 when the brain is missing, not logged in or has no key, 504 after two minutes. | +| `GET /v1/resident` | `{"enabled","thread_id","revision","ready","problem","brain","brain_setting","anthropic_key_set"}`. `thread_id` is the session of the brain voice commands go to now (a Codex thread id or a Claude Code session id). `revision` goes up when a voice command's turn ends (answered or not, from the box or `say`), when the thread changes, and when the brain answering changes; it is kept in memory and starts at 0 with the daemon. `ready` is on and the brain the setting picks ready (Readiness below); `problem` is then `null`, else `codex_missing`, `codex_logged_out`, `claude_missing` or `anthropic_key_missing` (`null` while off). `brain` is `codex` or `claude` while ready, else `null`; `brain_setting` is `auto`, `codex` or `claude`; `anthropic_key_set` says whether a key is stored, never the key. Never starts the app-server or Claude Code. | +| `GET /v1/resident/history?limit=n` | `{"thread_id","exchanges":[{"said","replies":[...]}]}`, oldest first (default 5, at most 50): on Codex read from the thread with `thread/turns/list`, starting the app-server if it isn't running; on Claude Code read from the session's transcript. The daemon keeps no record of its own. | +| `PUT /v1/resident/brain` `{"brain"}` | Sets the brain: `auto`, `codex` or `claude`; answers as `GET /v1/resident`. | +| `PUT /v1/resident/anthropic-key` `{"api_key"}`, `DELETE` | Stores or removes the user's Anthropic API key for Claude Code; answers as `GET /v1/resident`, without the key. 400 for an empty key. A running Claude Code process is stopped, so the next command starts one with the new key. | + +The brain setting and the key are kept in `resident-settings.json` in the config directory (`{"brain","anthropic_api_key"}`), written with mode 0600 and only through these two endpoints, never `PUT /v1/config`: that replaces the whole config, and an App that doesn't know the fields would wipe them (the same reason `speech.json` is a file of its own). Both work while the session is off. + +## Readiness + +Readiness is worked out under the brain setting from two checks, neither of which starts anything: + +- **Claude Code**: `claude` found (`VIBEBUDDY_CLAUDE`, PATH, `~/.local/bin`, `~/.claude/local`, `/opt/homebrew/bin`, `/usr/local/bin`) and a key stored. Free, so it is made every time. `claude_missing` comes first, then `anthropic_key_missing`. Whether the key works is only known at the first turn: a rejected key fails that turn with Claude Code's message. +- **Codex**, below; skipped while the setting is `claude`. + +Under `codex` or `claude` it is that brain's readiness. Under `auto` it is Codex when Codex is ready, else Claude Code when that is ready; when neither is, the problem is Claude Code's if a key is stored (the user means Claude Code) and Codex's otherwise, and nothing is said until Codex has been checked once. Changing the setting or the key checks again at once. + +Voice on Codex needs the user's Codex, so the daemon keeps a readiness check that never starts the app-server or a turn: it looks `codex` up as a start does (`VIBEBUDDY_CODEX`, PATH, the usual install places) and runs `codex -c cli_auth_credentials_store="file" login status` in the resident Codex home (with `auth.json` linked as for a start), which reads the same login `account/read` would and exits 0 only when signed in. The answer stands for 30 seconds: `GET /v1/resident` checks again once it is older, and so does a timer in the daemon every 30 seconds while the session is on. Every start of the app-server is a check too: one that ends in "codex not found" or "not logged in" marks the session not ready, and one that gets through marks it ready. + +The box is offered voice (`voice.available`, below) only while the session is on and ready and a recognizer is configured. When readiness flips, the daemon tells a box that has reported its build `"on":true` or `"on":false` right away (not over the Relay or the UART bridge, where voice is never offered), so a long K0 goes back to giving up the Pomodoro phase while the brain isn't ready (Codex missing or signed out, Claude Code missing or without a key). + +## How it works on Codex + +`daemon/src/app_server.rs` is a newline-delimited JSON-RPC client for `codex app-server` on stdio; `daemon/src/resident.rs` is the session on top of it, and picks the brain. + +- **Its own Codex home.** The app-server runs with `CODEX_HOME` set to `resident-codex/` in the daemon's state directory (mode 0700), never the user's. The ChatGPT App's own app-server takes the writer lock (`thread-writer-locks/.lock`) of every thread in `~/.codex` it lists whenever the lock is free, and keeps it, even for threads nobody opened; a resident thread there could not be resumed while the App runs. Nothing lists this home but the daemon. The home holds no `config.toml`: everything is a `-c` override (below). +- **The user's login, shared.** The user's Codex home is `$CODEX_HOME` if the daemon has it, else `~/.codex`. On every start `resident-codex/auth.json` is made a symlink to its `auth.json` (whatever else is there, a copy or a link elsewhere, is replaced), so there is one login and one set of tokens. Checked in the codex-cli 0.162.0 source (`codex-rs/login/src/auth/storage.rs`, `manager.rs`): the file store reads `auth.json` with a plain open, a refresh first reloads it and keeps a token the other side already refreshed, and a save truncates and rewrites it in place, so all of it goes through the link and the link survives; a rename on the user's side doesn't matter, since the link is by path. `-c cli_auth_credentials_store="file"` pins that store: the keychain store keys its entry by the Codex home's path, so it would never find the user's login, and its save deletes `auth.json`. A login kept in the keychain therefore can't be shared: the session reports "not logged in" and the log says why. Two Codex processes refreshing at the same moment can still race, as the user's CLI and the ChatGPT App already can on the same file. Credentials are never logged. +- **Starting.** On the first voice command, and on the next one after the app-server dies, the daemon runs the `codex` it finds (`VIBEBUDDY_CODEX`, else PATH, else `~/.local/bin`, `/opt/homebrew/bin`, `/usr/local/bin`, since launchd's PATH is bare). It sends `initialize` (`experimentalApi: true`) and `initialized`, then `account/read`: with no account it stops with "not logged in". It never starts a login. +- **One thread for good.** `thread/start` once, with `cwd` set to `resident/` in the daemon's state directory, `sandbox: "read-only"`, `approvalPolicy: "on-request"`, the developer instructions and the dynamic tools. The thread id is kept in `resident.json` beside it, and every later start sends `thread/resume` with the same settings (the tools survive a resume; the instructions are sent again). Only a thread that no longer exists (`no rollout found`) is replaced by a new one, saved, with both ids in the log. That is also what happens once to a thread saved before the session had its own home: it is in `~/.codex`, so the first start in `resident-codex/` begins a new thread; the old one's rollout stays where it is. Should another Codex process ever be the thread's writer, the resume fails with `already has an active writer`, the voice command fails as `unavailable`, and the next one tries again; the thread is never replaced for that, so there stays one resident session. +- **Only our tools.** In its own home none of the user's Codex config loads: not their `config.toml` (model, MCP servers, profiles), `AGENTS.md`, hooks, plugins or skills. Codex's own defaults still give a thread tools, so every start passes `-c features.=false` for plugins, apps (connectors), the shell and exec tools, sub-agents, goals, memories, image generation and viewing, the browser and computer use, sleep, tool suggestions, skill search and shell snapshots; all but `memories` are on by default in an empty home (`codex features list`). Shell snapshots matter beyond tools: at every start Codex runs the user's login shell and saves its whole environment, API keys included, into the Codex home. Skills go too: `skills.include_instructions=false` drops the skill list, which otherwise names the user's own skills (`~/.agents/skills`, read whatever the Codex home), and `skills.bundled.enabled=false` stops Codex installing its bundled ones (`imagegen` and the like) into the home. System and managed config (`/etc/codex`, MDM) still load and may add MCP servers, and an override can't empty `mcp_servers` (it merges), so the daemon first asks `codex mcp list --json`, under the same overrides, which servers would be on, and adds `-c mcp_servers..enabled=false` for each (on this Mac: none). Overrides are `-c` rather than `--disable`, which refuses a feature name another Codex version doesn't know. The model and reasoning effort are set too (see Latency below), and Codex's built-in web search is on, `web_search="live"` (below). The thread's tool list is `speak`, `list_activities`, `search_memories`, the web search and what Codex always adds and has no switch for: `apply_patch` (refused by the read-only sandbox, its approval declined), a clock, and the MCP resource readers (no server is on to read from). The model calls all of these through Codex's code mode, an `exec` tool that runs JavaScript: switching the host off (`features.code_mode_host`) leaves no tool call working, so it stays on. That JavaScript reaches only the thread's tools: checked on 2026-10-14, it has no `fetch`, `require` or `process`, and `import` is refused, so no file, network or process access. +- **Memories, read-only.** The session reads the user's Codex memories (`/memories/`) through the daemon, and Codex in its home never touches them. Codex's own memory feature (`features.memories`) can't do that safely: it gates the read path (injecting `memory_summary.md`) and, at every thread start whatever `generate_memories` says, the memory pipeline, whose consolidation phase rewrites `raw_memories.md` and prunes `rollout_summaries/` from the state database of the Codex home it runs in; in the resident home that database is empty, so on a linked folder it would empty the user's memories, and on a copy it would still run a consolidation agent each time. Its dedicated memory tools also include one that writes a note into the folder. So the feature stays off (and `memories.generate_memories=false`, should it come on), and the daemon does what the read path does: at every start or resume it puts `memory_summary.md`, cut at Codex's own cap of 10,000 bytes, into the developer instructions, and the `search_memories({query})` tool returns the lines of `MEMORY.md`, `raw_memories.md` and `rollout_summaries/` containing any of the query's words, each with the line before and after (at most 8,000 bytes). Both only read. Checked on 2026-10-13: a question answered from the summary, and one whose answer is only in `MEMORY.md`, answered after a `search_memories` call; the memories folder unchanged. +- **Tools.** `speak({text})` is the spoken reply: the daemon hands it on the moment it is called (`Resident::say_streaming`), to the box as `voice.reply`, or collected for `POST /v1/resident/say`. `list_activities({})` returns, as JSON, `{"activities","recently_ended","today"}`: the live Activities the aggregator holds (agent, task-card title, project, `working`/`needs_input`, host, seconds in that state and since the last event, newest first); the last 20 Activities that ended since the daemon started (agent, title, project, `ended`: `done`, `failed`, `stopped` or `expired`, and `ended_s`, seconds ago, newest first); and today's stats (`done`, `asks`, `busy_seconds`), the same numbers as the idle screen. `stopped` is an Activity that ended with no outcome: interrupted or stopped by the user, its session closed, or a CI run cancelled or pushed out; `expired` is one nothing was heard from for too long. Work an agent delegated inside its own session (a Claude Code subagent) is left out of `recently_ended`, as its end is never announced. The ended ones are kept only in memory, so a restart forgets them; today's stats were already kept in `stats.json`. The tool's description says what each part is for ("what just finished", "how did today go"), and the instructions say so too, since a thread started before this keeps its old description. `search_memories({query})` searches the user's Codex memories (above). There are no tools that run commands or write files. Dynamic tools are fixed when a thread starts, so a thread started before `search_memories` existed doesn't have it. +- **Web search.** Codex's built-in web search (`web_search`: `disabled`, `cached`, `indexed` or `live` in codex 0.162.0) is on as `live`, as what it is for (weather, news, prices) is about now, which a cached index may not have yet. Searches run on OpenAI's side, as part of the model's turn; the daemon sees none of it. On `gpt-6-luna` the session looks things up through Codex's own `web.run`, which also has weather and finance lookups. +- **No made-up facts.** The instructions forbid stating anything not seen in a tool result, a search result or the memories: no weather, news, prices or other facts about now, and nothing about the user's world. For those it searches, answers only from what it found, and otherwise says plainly in one sentence that it can't find out. Before this, asked about tomorrow's weather in Singapore, it invented a forecast. Codex's environment context gives the thread today's date and the user's time zone; the instructions point at it for "today" and "tomorrow", after luna once searched for the wrong day's news. +- **Declining.** Every approval or question the session raises is declined: command and file-change approvals (`decline`, or `denied` on the legacy methods), permission requests (nothing granted), MCP elicitations (`decline`) and user-input requests (no answers). `currentTime/read` is answered; anything else gets a JSON-RPC "method not found". +- **Turns.** `turn/start` with the text after a context block, as on Claude Code (below): the user's local date, time, UTC offset and time zone, and who answers in which language. Codex's own clock tool answers in UTC, so without it the time of day came out in UTC. Then the daemon answers tool calls until `turn/completed` for that turn. One turn at a time: a command that arrives during another waits for it, in order, rather than being rejected, since two quick sentences are both meant to be answered. A turn that takes over two minutes is interrupted (`turn/interrupt`). +- **Persona.** The instructions make the buddy the Character it wears, in that Character's language (below). The app-server takes developer instructions only at `thread/start` and `thread/resume` (`turn/start` has no field for them in codex-cli 0.162.0), so a new Character stops the app-server and resumes the same thread with the new ones. + +## Persona and language + +The resident session speaks as the buddy, as the Character it wears (ADR-0017): its instructions carry that Character's persona, the very `characters//persona.md` its lines were written in (embedded at build time, in `daemon/src/persona.rs`), and its one language, `LANGUAGE` in its `voice.env`. The buddy understands whatever language the user speaks, but always replies in its Character's: asked in Chinese, Ada answers in English and Xiaohe in Chinese. The instructions say so whatever earlier replies or the user's Codex memories say about language: one user's memories said the resident session answers in Chinese, and with them a first draft of the rule still had Ada answering a Chinese question in Chinese. The robot, the default Character, has its own `characters/robot/persona.md` and speaks English, the language of its five built-in lines; so do no pack and a pack the daemon doesn't know. + +- **Which Character.** The one the box reports wearing (`VOICES`), else the one last written to it, as for the voice the reply is spoken in. The box reports only the pack's id, so whatever else the App knows it says when it writes a pack, as query parameters of `POST /v1/device/voice-pack`: `lender`, the preset lending a custom Character its voice, persona and lines, and `address`, the form of address its lines carry. The daemon keeps them, with the pack's id, in `character.json` in its state directory, and uses them while the box wears that pack. A custom Character answers in its lender's persona and language, and its spoken replies are in the lender's voice; one written by an App that didn't say who lends it (or by another computer) answers as the robot. +- **Form of address.** When the user picked one for the Character's language, the instructions say the buddy calls the user that now and then, naturally, as the lines do, not in every reply. +- **The rules stay.** Whoever the buddy is, it answers only through `speak` in one or two short spoken sentences, never states a fact it hasn't seen, looks up the Activities with `list_activities`, searches for anything about now, and does no work. +- **A new Character.** The next voice command after the Character changes (the box reports another one, or the App writes one) is answered in the new persona: Codex stops the app-server and resumes the same thread with the new developer instructions; Claude Code, whose `--system-prompt` is per process, is started again with `--resume` on the same session. Neither is ever forked or replaced. That command also carries a context line saying the user switched to another Character and naming it, so the session takes up the new voice rather than carrying on in the one its earlier replies were in. Like the time context, it is left out of the history. Which Character the session last answered as is kept (`persona` in `resident.json` and `resident-claude.json`), so a change while the daemon was down gets the line too. Checked on 2026-10-14 with codex-cli 0.162.0: the same Chinese question answered in English as Ada and in Chinese as Xiaohe, back and forth on one thread, the switch costing about two seconds on the first command. + +## On Claude Code + +`daemon/src/resident_claude.rs` drives the user's own `claude` CLI headless: no Node, no Agent SDK. + +- **Only the user's own API key.** Anthropic doesn't let third-party products use a claude.ai login: its [legal and compliance page](https://code.claude.com/docs/en/legal-and-compliance) says developers building products, including with the Agent SDK, use API key authentication, and "Anthropic does not permit third-party developers to offer Claude.ai login into their own applications, or to route requests through Free, Pro, or Max plan credentials on behalf of their users". So the Claude Code brain authenticates only with the Anthropic API key the user enters, billed to their Anthropic account, and never with a subscription or OAuth login, not even as a fallback: the child's environment loses every `CLAUDE*` and `ANTHROPIC*` variable the daemon has (an OAuth token, `ANTHROPIC_AUTH_TOKEN`, the Bedrock, Vertex and Foundry switches, profiles) and gets `ANTHROPIC_API_KEY`, which `-p` always uses when present ([authentication precedence](https://code.claude.com/docs/en/authentication#authentication-precedence)); its own config dir holds no login, and the macOS Keychain entry is keyed by the config dir. On top of that, the `system/init` line reports `apiKeySource`, and anything but `ANTHROPIC_API_KEY` stops the process and fails the command. The key is never logged, never written outside `resident-settings.json` (the MCP config and the command line don't carry it), and only reaches the child's environment. +- **Its own config dir.** `CLAUDE_CONFIG_DIR` is `resident-claude/` in the daemon's state directory (mode 0700), so none of the user's Claude Code config loads: settings, CLAUDE.md, hooks, plugins, MCP servers, sessions. The session doesn't show up in the user's `claude --resume` or the Claude App. The working directory is the same `resident/` as the Codex thread's. `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` and `DISABLE_AUTOUPDATER=1` keep it from writing memory files or updating the user's Claude Code. +- **One process, one session for good.** One long-lived `claude -p --input-format stream-json --output-format stream-json --verbose`; each voice command is one user message on its stdin, and the turn is over at the `result` line. The first start passes `--session-id` with a new UUID, saved in `resident-claude.json` in the state directory with the Claude Code version; every later start passes `--resume` with it, as long as its transcript (`resident-claude/projects/*/.jsonl`) is there. Only a session whose transcript is gone is replaced, with both ids in the log. A process that exits or doesn't finish a turn in two minutes is stopped, and the next command starts it again with `--resume`; an error `result` (a rejected key, an API error) fails the command and keeps the process. A new key starts a new process. The daemon starts it when it starts, as for Codex. +- **Each command** is two text blocks: a context block with the user's local date, time, UTC offset and time zone, and a reminder of who answers in which language, then what was said (with the context line of a new Character between them, see "Persona and language"). The history leaves the context blocks out. +- **Only our tools.** `--tools WebSearch` leaves Web search as the only built-in tool; `--disallowedTools` denies Bash, Edit, Write, Read, NotebookEdit, WebFetch, Glob, Grep, Agent, Task, Skill and BashOutput by name besides, which removes them from the model's context. `speak` and `list_activities` come from the daemon's MCP server (below), with `--strict-mcp-config` so no other server loads. `--permission-mode dontAsk --permission-prompts none` with `--allowedTools WebSearch,mcp__vibebuddy__speak,mcp__vibebuddy__list_activities`: anything else that would need a permission is refused, never asked. `--disable-slash-commands` turns skills and commands off, and `--settings '{"disableAllHooks":true}'` hooks. `search_memories` isn't offered: Codex memories are Codex's. +- **Why not `--bare`.** Bare mode would skip even more and never reads OAuth or the keychain, but in Claude Code 2.1.296 it gives the model only Bash, Read and Edit and drops WebSearch even when `--tools` names it (checked against a fake Messages API: the request's tool list was empty with `--bare --tools WebSearch`, and `WebSearch` without `--bare`). Without web search the buddy couldn't answer questions about now, so the config dir, the cleaned environment, the init check and the flags do it instead. +- **The MCP server.** Claude Code reaches `speak` and `list_activities` over MCP's Streamable HTTP transport (JSON responses, no server-sent stream) at `http://127.0.0.1:/mcp`, a loopback port of the daemon's own picked at start, behind a random 256-bit bearer token made when the daemon starts. The URL and token are in `resident-claude/mcp.json` (0600), passed with `--mcp-config`. Chosen over a `vibebuddyd` stdio subcommand: `speak` must reach the turn in progress in the daemon at once, and a stdio child would have had to call back into the daemon over HTTP anyway; in-process there is no second process, no binary path to find inside the App bundle, and the daemon's own API (no token, on `VIBEBUDDY_BIND`) stays as it is. Without the token the server answers 401. +- **Spoken replies stream.** A `speak` call is handed to `voice.rs` the moment the MCP request arrives, as on Codex; anything said just before the `result` line is still handed on before the turn ends. +- **The instructions** are the Codex session's (no made-up facts, short spoken replies, `list_activities` for the Agents), with the date taken from the context block, passed as `--system-prompt`: they replace Claude Code's coding-assistant prompt, since the buddy isn't one, and a short prompt is a faster turn. `--system-prompt-snapshot off` has them read again at every request, so a change reaches a resumed session; new instructions (a new Character) start the process again with `--resume`. +- **The model** is `claude-haiku-4-5` (`MODEL` in `resident_claude.rs`), for the same reason Codex runs on luna at low effort: a voice command waits on the answer. +- **Web search** is Claude Code's WebSearch, which runs Anthropic's server-side web search under the user's key; an organization can turn web search off in the Claude Console, and then searches fail. +- **History** is read from the session's transcript: each user message that isn't a tool result, with the text of the `mcp__vibebuddy__speak` calls after it, in time order (Claude Code doesn't always write the lines in order). The transcript format isn't a documented API; the stream-json shape it shares is. + +Checked on 2026-10-14 with Claude Code 2.1.296 against a fake Messages API on this Mac (`ANTHROPIC_BASE_URL` through `VIBEBUDDY_CLAUDE_BASE_URL`, a development-only setting), through `POST /v1/resident/say` on a daemon of its own (temporary home, its own port, no box, no Relay, `CLAUDE_CODE_OAUTH_TOKEN` and `ANTHROPIC_AUTH_TOKEN` set in its environment): the model's tool list was exactly `WebSearch`, `mcp__vibebuddy__list_activities` and `mcp__vibebuddy__speak`; requests carried the key as `x-api-key` and no `Authorization`; calls to Bash, Read, Write, WebFetch and `search_memories` came back "No such tool available" and changed nothing; WebSearch ran; `speak` came back as the reply; the history read back; and after a daemon restart the same session was resumed with its earlier messages. Not yet checked against the real API (no key was at hand): the turn times, and how Haiku follows the instructions. + +## From the box + +The box side of push-to-talk is in [`protocol.md`](protocol.md), "Voice commands (experimental)"; `daemon/src/voice.rs` is the computer side and `daemon/src/speech.rs` the recognizers. + +- **Offered.** Voice is on when the resident session is on and a speech recognizer is configured. The daemon then sends `voice.available` `on: true` every time the box reports its build (after `device.hello` on each new link, and after a reboot). Never over the Relay, where the box will talk to the voice service itself, and never over the UART bridge, which is too slow for the audio and gets none from the box (use the box's native USB port); the log says which kept voice off. Nothing switches either condition at run time yet, so `on: false` is never sent. +- **Recording.** `VOICE DATA` lines are decoded as they arrive, each from the IMA ADPCM state it carries (the same decoder as `firmware-rs/core/src/adpcm.rs`, checked against it in the tests). On `VOICE END` the lines are put in seq order; a lost line becomes silence as long as a line, audio lost at the end is filled up to the sample count `END` gives, and the missing seqs are logged. A turn's audio is kept only in memory and only until it is recognised; `VOICE DATA` lines stay out of the log. +- **A turn.** The audio goes to the recognizer as a 24 kHz 16-bit mono WAV file; the text goes to the box as `voice.heard` and to the resident session as a voice command; each `speak` goes to the box as `voice.reply` as soon as it is called, and its audio follows as `voice.audio` (see "Speech synthesis" below); `voice.done` follows `turn/completed` and the last reply's audio `end`. Text is cut at a character, with an ellipsis, to fit the 1024-byte line. Errors: `not_heard` (no audio arrived, or the recognizer heard nothing), `unavailable` (voice is off, recognition failed, or the resident session failed, with the reason as `message`), `busy` (a turn started while another is still being answered: sent right after its `VOICE START`, so the box stops recording, and the rest of that turn is dropped; unlike `vibebuddy say`, a voice turn doesn't wait). A typed command and a voice turn still queue behind each other in the resident session. +- **Timing.** A 5 s recording, checked end to end on 2026-10-13 (recognition by Workers AI, codex-cli 0.162.0, app-server started cold): `voice.heard` 2.2 s after `VOICE END`, the spoken reply at 10.1 s, `voice.done` at 14.0 s. + +The end-to-end check is an ignored test, `e2e_a_real_recording_through_the_real_recognizer_and_resident_session` in `voice.rs`: it turns the left channel of a WAV file into `VOICE` lines with firmware-core's own voice command code (as do the other voice tests, so they hold the daemon to what the box writes) and feeds them through the real recognizer and the real resident session, in a temporary state directory. Its doc comment says how to run it. + +### Speech recognition + +Behind the `Recognizer` trait. Three backends, picked at start: + +| Backend | Settings | Used | +| --- | --- | --- | +| The user's own recognizer: any OpenAI-compatible `POST {base}/audio/transcriptions` | `VIBEBUDDY_STT_BASE_URL`, `VIBEBUDDY_STT_API_KEY` (optional), `VIBEBUDDY_STT_MODEL` (default `whisper-1`) | When set | +| The voice service in `vibebuddy-cloud`: `POST {base}/v0/voice/transcribe` with the WAV as the body and `?language=` when set, answering `{"text","language"}`. It runs Groq's `whisper-large-v3-turbo` (about 1 s), falling back to Workers AI's | The base spoken replies use (`VIBEBUDDY_TTS_BASE_URL`, below): staging outside a release | Otherwise. A release has no voice service until production's is deployed with accounts, so a release has no recognizer unless the user sets their own | +| Cloudflare Workers AI, `@cf/openai/whisper-large-v3-turbo`, called directly | `CLOUDFLARE_ACCOUNT_ID`, `CLOUDFLARE_AI_TOKEN` | Only when named; for development before the service recognized | + +`VIBEBUDDY_STT=service|openai|cloudflare|off` forces the choice. `VIBEBUDDY_STT_LANGUAGE` (`zh`, `en`, …) tells the recognizer the language; unset, it detects it, and the log names what it detected. + +Each setting is taken from the environment, else from `speech.json` in the config directory (`{"base_url","api_key","model","language"}`, the user's own recognizer), else from a `KEY=VALUE` file named by `VIBEBUDDY_STT_ENV_FILE` (development credentials, so they stay out of the daemon's environment and out of the repository). All are read when the daemon starts. `speech.json` is a file of its own, not part of `config.json`, because `PUT /v1/config` replaces the whole config and an App that doesn't know the field would wipe the key; whoever writes it (an App setting, later) writes it with mode 0600. Credentials are never logged, and errors name only the recognizer's host. + +### Speech synthesis + +Each spoken reply is also synthesized and streamed to the box (`daemon/src/synthesis.rs`, `Speaker` in `voice.rs`), behind the `Synthesizer` trait so the backend can change. There is one backend, the voice service in `vibebuddy-cloud`: `POST {base}/v0/voice/speak` with `{"text","voice"}` (text 1 to 1000 characters, `voice` as `:`: `elevenlabs:` or `volc:`), answering a chunked `audio/mpeg` stream (MPEG-2 Layer III, 24 kHz mono, 48 kbps) that starts as soon as there is audio. v0 takes no credentials, for synthesis or recognition; the account session's header goes in `VoiceService::speak` and `speech::VoiceService` when it does. + +| Setting | Effect | +| --- | --- | +| `VIBEBUDDY_TTS_BASE_URL` unset | Staging (`https://api-staging.korekore.ai`) in every build but a release, as for the Relay; a release has no voice service until production's is deployed, so its replies stay text | +| `VIBEBUDDY_TTS_BASE_URL=` | That service | +| `VIBEBUDDY_TTS_BASE_URL=off` | No synthesis: replies stay text | + +- **Order.** Replies are synthesized one after another, in the order they were said, on a task of their own: a reply's `voice.reply` goes out the moment it is said and never waits for an earlier reply's audio, so the text of reply 1 may come between reply 0's audio pieces. Each reply's audio is forwarded as the bytes arrive, cut into pieces of at most 576 bytes (768 base64 characters), `seq` from 0, then `end`. `voice.done` waits for the last reply's `end`, so the turn is over only once everything has been sent; it is still the last message of the turn. +- **Failures.** A status other than 200, or a stream cut before its end (the service closes the connection when it fails partway), ends that reply's audio at once with `"error":"synthesis"`, keeping what was sent; the box keeps the text. The next reply is still tried. When the resident session fails the turn, the reply being synthesized stops at its next piece, the rest are dropped, and `voice.error` follows. +- **Long replies.** Over 1000 characters (counted in UTF-16, as the service counts), a reply is split at the end of a sentence, else at a comma or space; the pieces are requested one after another and stream as one reply, since MP3 frames stand alone. +- **Voices.** A reply is spoken in its Character's own voice, the one its lines were made with: `synthesis.rs` reads each Character's `characters//voice.env` at build time and sends `elevenlabs:` (our ElevenLabs account; the service uses `eleven_flash_v2_5`, multilingual, so a Chinese reply keeps an English Character's voice). A Doubao Character (the Chinese ones, `ENGINE=volc`) sends `volc:`, which the service speaks through Volcano Engine; it speaks Doubao's 1.0 voices only, so a Character with `VOLC_RESOURCE_ID` set (a 2.0 voice) gets an ElevenLabs stock voice of its `GENDER` (from `voice.env`: Sarah or George) rather than another Character's voice. The robot and no pack get Jessica. The Character is the one the box reports (`VOICES`), else the one last written to it; a custom Character speaks in the voice of the Character lending it its lines, when the App said which (see "Persona and language"). +- **Links.** Only where voice is offered, native USB and the local network: nothing goes over the Relay or the UART bridge. +- **Timing.** Checked on 2026-10-13 against staging, from this computer through the daemon's own client (a new TLS connection each time): a Chinese sentence, first audio after 1.3 to 1.4 s and its `end` after 1.5 to 1.6 s for 4.5 s of speech; an English one, 0.5 to 1.8 s and 0.6 to 2.0 s for 3.8 s. `e2e_spoken_replies_through_the_real_voice_service` in `voice.rs` (ignored) repeats it and saves the reassembled MP3s. + +## Never an Activity + +The user's Codex hooks live in their Codex home, so they don't load for the resident thread. Hooks from system or managed config still could, so the Codex adapter still drops a hook whose session or thread is the resident thread, or whose `cwd` is inside the resident directory, before anything is recorded: no card, announcement, stats, CI watch or K2 target. This works even with the resident session switched off. + +The same goes for Claude Code: its own config dir loads none of the user's hooks (and `disableAllHooks` is set), but managed hooks still could, so the Claude Code adapter drops a hook whose `session_id` is the resident Claude Code session or whose `cwd` is inside the resident directory. + +## Versions + +app-server and `dynamicTools` are experimental, and so is relying on how Codex stores `auth.json` and memories. This was built and checked against codex-cli 0.162.0 (`TESTED_CODEX_VERSION`); another version logs a warning and is still tried. The version last used is written to `resident.json`. + +The Claude Code brain was checked against Claude Code 2.1.296 (`TESTED_CLAUDE_VERSION`), every flag against `claude --help` and the [CLI reference](https://code.claude.com/docs/en/cli-reference), [headless mode](https://code.claude.com/docs/en/headless) and [permissions](https://code.claude.com/docs/en/permissions) pages; another version logs a warning and is still tried. The version last used is written to `resident-claude.json`. + +## Privacy + +`list_activities` sends the task-card titles and project names the daemon holds, with each Activity's agent, state, host app and ages, to the model provider the user's Codex uses, as part of the resident thread. Until now these stayed on the computer and the box. Prompts and replies are not sent: the daemon never holds them. + +It also sends the last ended Activities (the same titles and projects, how each ended and how long ago) and today's stats. + +Web searches go to OpenAI: the resident session's searches are made by OpenAI's model, from words of the voice command (a city, a topic), as part of the resident thread; OpenAI runs them against its search providers. Nothing the daemon holds is searched for unless the model puts it in a query. + +On Claude Code all of this goes to Anthropic instead, under the user's own API key: the voice commands' text with the local time, the instructions, what `list_activities` returns, and the searches, which Anthropic runs. Anthropic's terms for API use apply. The session's transcript stays in `resident-claude/` on this computer, as Claude Code keeps it. The API key is stored in `resident-settings.json` (mode 0600) and is sent only to Claude Code, which sends it only to Anthropic. + +The instructions carry the persona of the Character the buddy wears (`characters//persona.md`, written by us and public in this repository), its language and the user's form of address, so they go to the brain's provider (OpenAI for Codex, Anthropic for Claude Code) with every thread start or resume, as part of the resident session. + +The user's Codex memory summary, and whatever `search_memories` finds, go to the same provider as part of the resident thread, as they already do in the user's own Codex sessions. + +A voice command's audio leaves the computer for the speech recognizer: in development that is Cloudflare Workers AI, under the developer's own account; with the user's own recognizer, only that endpoint (and its key goes nowhere else). The recognised text then goes to the user's Codex model provider as the voice command. The daemon keeps neither the audio nor the text once the turn is over, and logs neither. + +Each spoken reply's text, which may name what the Agents are doing, goes to the voice service to be synthesized (in development, staging), with the voice name; the service passes it on to the speech engine behind it (ElevenLabs). The audio it returns goes only to the box. The daemon logs the voice, timings and byte counts, never the text. + +## Latency + +A voice command waits on recognition, the resident session and synthesis in turn, so the resident session runs on `gpt-6-luna` at `low` reasoning effort (`MODEL` and `REASONING_EFFORT` in `resident.rs`, set with `-c`; the user's own model settings don't load in its Codex home): on 2026-10-13 a turn's first `speak` came 2.7 s in, against 4.3 s on `gpt-6.1-sol` at `high`. + +With web search on, measured on 2026-10-14 through `POST /v1/resident/say` on a daemon of its own (temporary home, its own port, no box, no Relay), each model on a fresh thread, two rounds; seconds from the request to the first `speak`: + +| Question | gpt-6-luna, low | gpt-6.1-sol, low | +| --- | --- | --- | +| No search: "你好，你能做什么" | 1.9 | 5.2 | +| No search, `list_activities`: "现在有几个agent在工作" / "刚刚结束的agent在干什么" | 5.3 (first turn), 4.0 | 6.3, 31.7 | +| "新加坡明天天气怎么样" | 6.7, 5.6 | 14.7, 11.2 | +| "今天有什么重要的科技新闻" | 7.6, 6.4 | 17.1, 26.0 | +| "现在比特币多少钱" | 6.4 | 10.9 | + +Both searched every time and answered from what they found (the weather from a forecast lookup for the right day, the price from a finance lookup, the news from that day's headlines). sol was a little more careful (dates, sources) but two to four times slower; luna's answers were good enough, and once, before the instructions pointed at the date, it searched for the wrong day's news and said it found nothing reliable rather than guess. luna stays. The daemon starts the app-server and resumes the thread when it starts, not on the first command, and the synthesizer keeps one HTTPS agent so its connection to the voice service is reused. + +On Claude Code the session runs on `claude-haiku-4-5`, a small, fast model, for the same reason. Its turn times against the real API are not measured yet; against a fake Messages API a whole turn through the daemon took under 0.1 s once the process was up, so the overhead of the CLI and the MCP round trip is small. diff --git a/firmware-rs/core/Cargo.toml b/firmware-rs/core/Cargo.toml index 60da2a1b..b5f0f26a 100644 --- a/firmware-rs/core/Cargo.toml +++ b/firmware-rs/core/Cargo.toml @@ -16,6 +16,9 @@ fast-clock = [] # The box's pairing key (link.rs). No default features: no std, no OS randomness; the seed comes from the device. ed25519-dalek = { version = "2", default-features = false } libm = "0.2" +# Spoken replies' MP3 (spoken_reply.rs). No `wide` (SIMD the Xtensa core lacks) and no Layer I/II. Pinned: the +# player builds the decoder zeroed on the heap, which relies on this version's layout. +nanomp3-core = { version = "=0.2.0", default-features = false } # Noise sessions on links off the cable (noise.rs, ADR-0015): no std and no getrandom; the device supplies the randomness. clatter = { version = "2.3", default-features = false, features = ["use-25519", "use-chacha20poly1305", "use-sha"] } rand_core = { version = "0.6", default-features = false } diff --git a/firmware-rs/core/examples/voice_preview.rs b/firmware-rs/core/examples/voice_preview.rs new file mode 100644 index 00000000..e6bbf686 --- /dev/null +++ b/firmware-rs/core/examples/voice_preview.rs @@ -0,0 +1,192 @@ +//! Renders a voice command's overlay (docs/device-menu.md, "Voice commands") to PPM on the Mac: the real +//! firmware on a fake board, with a fake microphone and the computer's answers fed in as lines, so the +//! pictures in the docs are what the box shows. Also the Chinese text pictures (docs/firmware-bringup.md, +//! "Fonts"). +//! +//! Usage: cargo run -p vibebuddy-firmware-core --example voice_preview -- + +use std::cell::Cell; +use std::fs; +use std::path::{Path, PathBuf}; + +use vibebuddy_firmware_core::audio::Sound; +use vibebuddy_firmware_core::buttons::Levels; +use vibebuddy_firmware_core::canvas::{FRAME_BYTES, HEIGHT, WIDTH}; +use vibebuddy_firmware_core::display::Screen; +use vibebuddy_firmware_core::firmware::{AudioStatus, Board, Firmware, FrameAction, VolumeError}; +use vibebuddy_firmware_core::storage::{Flash, FlashError}; + +/// No partitions: settings and voice packs aren't what these pictures are about. +struct NoFlash; + +impl Flash for NoFlash { + fn read(&mut self, _offset: u32, _bytes: &mut [u8]) -> Result<(), FlashError> { + Err(FlashError) + } + fn write(&mut self, _offset: u32, _bytes: &[u8]) -> Result<(), FlashError> { + Err(FlashError) + } + fn erase(&mut self, _from: u32, _to: u32) -> Result<(), FlashError> { + Err(FlashError) + } +} + +struct PreviewBoard { + now: Cell, + frame: Vec, + flash: NoFlash, + keys: (bool, bool, bool), + /// The fake microphone: running, and what it has recorded since the last take. + mic: bool, + samples: Vec, +} + +impl Screen for PreviewBoard { + fn frame(&mut self) -> &mut [u8] { + &mut self.frame + } + fn present(&mut self) -> Result<(), ()> { + Ok(()) + } + fn set_backlight(&mut self, _on: bool) -> Result<(), ()> { + Ok(()) + } +} + +impl Board for PreviewBoard { + fn now_ms(&self) -> u32 { + self.now.get() + } + fn write(&mut self, _bytes: &[u8]) {} + fn flash(&mut self) -> &mut dyn Flash { + &mut self.flash + } + fn with_frame_and_output(&mut self, _action: &mut FrameAction) {} + fn init_display(&mut self) -> bool { + true + } + fn init_audio(&mut self, _volume: u32) -> AudioStatus { + Ok("ES8311") + } + fn init_buttons(&mut self) -> Option { + Some(Levels::default()) + } + fn read_buttons(&mut self) -> (bool, Option<(bool, bool)>) { + (self.keys.0, Some((self.keys.1, self.keys.2))) + } + fn play(&mut self, _sound: Sound) -> Result<(), ()> { + Ok(()) + } + fn stop_audio(&mut self) {} + fn audio_busy(&self) -> bool { + false + } + fn set_volume(&mut self, _level: u32) -> Result<(), VolumeError> { + Ok(()) + } + fn voice_start(&mut self) -> Result<(), ()> { + self.mic = true; + Ok(()) + } + fn voice_take(&mut self, samples: &mut Vec) { + samples.append(&mut self.samples); + } + fn voice_stop(&mut self) { + self.mic = false; + } +} + +struct Preview { + directory: PathBuf, + board: PreviewBoard, + firmware: Firmware, +} + +impl Preview { + fn new(directory: &Path) -> Self { + let mut board = PreviewBoard { now: Cell::new(1000), frame: vec![0; FRAME_BYTES], flash: NoFlash, keys: (false, false, false), mic: false, samples: Vec::new() }; + let mut firmware = Firmware::new(1, 1000, b"v0.3.2-11-g0adc18b 2026-10-06 16:34", "0.3.2"); + firmware.boot(&mut board); + let mut preview = Self { directory: directory.to_path_buf(), board, firmware }; + preview.send(r#"{"version":1,"event":"device.heartbeat","build":"0.3.1 v0.3.1-12-ga4a667c 2026-10-06 16:26","hour":14,"day":20261005}"#); + preview.send( + r#"{"version":1,"event":"task.start","title":"CC:MUSE DUAL BOOT","stats":["13 DONE","1 ASKS","46M BUSY"],"tasks":[{"title":"CC:MUSE DUAL BOOT","status":"working","elapsed_s":420,"project":"VIBE-BUDDY"},{"title":"CX:RELEASE NOTES","status":"done","elapsed_s":1500,"project":"VIBE-BUDDY"}]}"#, + ); + preview + } + + fn send(&mut self, line: &str) { + self.firmware.receive(&mut self.board, line.as_bytes()); + self.firmware.receive(&mut self.board, b"\n"); + } + + /// Holds K0 for `hold_ms`; while the microphone runs it hears a tone at `amplitude`. + fn hold_k0(&mut self, hold_ms: u32, amplitude: f32) { + self.board.keys.0 = true; + let mut held = 0; + while held < hold_ms { + self.board.now.set(self.board.now.get() + 10); + held += 10; + if self.board.mic { + let start = self.board.now.get() as f32 * 24.0; + self.board.samples.extend((0..240).map(|index| ((start + index as f32) * 0.06).sin() * amplitude).map(|sample| sample as i16)); + } + self.firmware.poll(&mut self.board); + } + } + + fn release_k0(&mut self) { + self.board.keys.0 = false; + self.board.now.set(self.board.now.get() + 10); + self.firmware.poll(&mut self.board); + } + + fn snapshot(&self, name: &str) { + let mut out = format!("P6\n{WIDTH} {HEIGHT}\n255\n").into_bytes(); + for pair in self.board.frame[..FRAME_BYTES].as_chunks::<2>().0 { + let pixel = u16::from_be_bytes([pair[0], pair[1]]) as u32; + out.push((((pixel >> 11) & 0x1f) * 255 / 31) as u8); + out.push((((pixel >> 5) & 0x3f) * 255 / 63) as u8); + out.push(((pixel & 0x1f) * 255 / 31) as u8); + } + fs::write(self.directory.join(format!("{name}.ppm")), out).expect("write PPM"); + } +} + +fn main() { + let directory = PathBuf::from(std::env::args().nth(1).expect("usage: voice_preview ")); + fs::create_dir_all(&directory).expect("create directory"); + + let mut preview = Preview::new(&directory); + preview.send(r#"{"version":1,"event":"voice.available","on":true}"#); + // Four seconds in (one of them the long press itself), speaking at about -21 dBFS. + preview.hold_k0(4000, 4000.0); + preview.snapshot("voice-listening"); + preview.release_k0(); + preview.snapshot("voice-thinking"); + preview.send(r#"{"version":1,"event":"voice.heard","turn":1,"text":"what is codex doing right now"}"#); + preview.send(r#"{"version":1,"event":"voice.reply","turn":1,"text":"It's writing the release notes. Should be done in a few minutes."}"#); + preview.snapshot("voice-reply"); + + preview.hold_k0(2500, 4000.0); + preview.release_k0(); + preview.send(r#"{"version":1,"event":"voice.error","turn":2,"reason":"not_heard"}"#); + preview.snapshot("voice-error"); + + // Chinese (font.rs): session titles on the task cards, then a voice turn in large and in small type. + let mut preview = Preview::new(&directory); + preview.send( + r#"{"version":1,"event":"task.start","title":"CC:给盒子加中文字体","tasks":[{"title":"CC:给盒子加中文字体","status":"working","elapsed_s":420,"project":"VIBE-BUDDY"},{"title":"CX:修好语音回复的播放","status":"input_required","elapsed_s":95,"project":"VIBE-BUDDY"},{"title":"CC:整理发布说明","status":"done","elapsed_s":1500,"project":"官网"}]}"#, + ); + preview.snapshot("device-cards-zh"); + preview.send(r#"{"version":1,"event":"voice.available","on":true}"#); + preview.hold_k0(2500, 4000.0); + preview.release_k0(); + preview.send(r#"{"version":1,"event":"voice.heard","turn":1,"text":"Codex 现在在忙什么？"}"#); + preview.send(r#"{"version":1,"event":"voice.reply","turn":1,"text":"它在写 v0.5.1 的发布说明，大概还要五分钟。写完我提醒你。"}"#); + preview.snapshot("voice-reply-zh"); + preview.send( + r#"{"version":1,"event":"voice.reply","turn":1,"text":"另外，Claude Code 那边在等你确认：要不要把字体表放进 flash 的只读数据里？它估计会多占七十多 KB，但完全不占内存。你说“可以”，我就让它继续。"}"#, + ); + preview.snapshot("voice-reply-zh-small"); +} diff --git a/firmware-rs/core/fonts/LICENSES/ark-pixel/OFL.txt b/firmware-rs/core/fonts/LICENSES/ark-pixel/OFL.txt new file mode 100644 index 00000000..264873a2 --- /dev/null +++ b/firmware-rs/core/fonts/LICENSES/ark-pixel/OFL.txt @@ -0,0 +1,96 @@ +Ark Pixel Font +https://github.com/TakWolf/ark-pixel-font + +Copyright (c) 2021, TakWolf (https://takwolf.com). + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://openfontlicense.org + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/firmware-rs/core/fonts/LICENSES/cubic-11/OFL.txt b/firmware-rs/core/fonts/LICENSES/cubic-11/OFL.txt new file mode 100644 index 00000000..4e863313 --- /dev/null +++ b/firmware-rs/core/fonts/LICENSES/cubic-11/OFL.txt @@ -0,0 +1,108 @@ +[Cubic 11] +These fonts are free software. +Unlimited permission is granted to use, copy, and distribute them, with or without modification, either commercially or noncommercially. +THESE FONTS ARE PROVIDED "AS IS" WITHOUT WARRANTY. +此字型是免費的。 +無論您是否進行對本字型進行商業或非商業性修改，均可無限制地使用，複製和分發它們。 +本字型的衍生品之授權必須與此字型相同，且不作任何擔保。 +[JF Dot M+H 12] +Copyright(c) 2005 M+ FONTS PROJECT +[M+ BITMAP FONTS] +Copyright (C) 2002-2004 COZ +These fonts are free software. +Unlimited permission is granted to use, copy, and distribute it, with or without modification, either commercially and noncommercially. +THESE FONTS ARE PROVIDED "AS IS" WITHOUT WARRANTY. +これらのフォントはフリー（自由な）ソフトウエアです。 +あらゆる改変の有無に関わらず、また商業的な利用であっても、自由にご利用、複製、再配布することができますが、全て無保証とさせていただきます。 + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/firmware-rs/core/fonts/LICENSES/galmuri/LICENSE.txt b/firmware-rs/core/fonts/LICENSES/galmuri/LICENSE.txt new file mode 100644 index 00000000..ae34182f --- /dev/null +++ b/firmware-rs/core/fonts/LICENSES/galmuri/LICENSE.txt @@ -0,0 +1,93 @@ +Copyright (c) 2019–2025 Lee Minseo (quiple@quiple.dev) + +This font software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://openfontlicense.org + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/firmware-rs/core/fonts/OFL.txt b/firmware-rs/core/fonts/OFL.txt new file mode 100644 index 00000000..0427d1db --- /dev/null +++ b/firmware-rs/core/fonts/OFL.txt @@ -0,0 +1,96 @@ +Fusion Pixel Font +https://github.com/TakWolf/fusion-pixel-font + +Copyright (c) 2022, TakWolf (https://takwolf.com). + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +https://openfontlicense.org + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/firmware-rs/core/fonts/cjk-12px.bin b/firmware-rs/core/fonts/cjk-12px.bin new file mode 100644 index 00000000..fb877d4f Binary files /dev/null and b/firmware-rs/core/fonts/cjk-12px.bin differ diff --git a/firmware-rs/core/src/adpcm.rs b/firmware-rs/core/src/adpcm.rs index 779a3a57..56c0defe 100644 --- a/firmware-rs/core/src/adpcm.rs +++ b/firmware-rs/core/src/adpcm.rs @@ -1,4 +1,6 @@ -//! Decoding a Character pack line: 16 kHz mono IMA ADPCM in, the codec's 24 kHz stereo PCM out. +//! Decoding a Character pack line: 16 kHz mono IMA ADPCM in, the codec's 24 kHz stereo PCM out. And the +//! same coding the other way for voice commands ([`Encoder`], docs/protocol.md): the microphone's 24 kHz +//! samples out as IMA ADPCM, which the computer decodes with the same tables. //! //! Each line is a standalone stream: predictor and step index start at zero, two samples per byte, //! low nibble first. The upsampling is linear interpolation, 2 input samples to 3 output samples; @@ -21,6 +23,17 @@ pub struct Decoder { } impl Decoder { + /// A decoder that picks up a stream at a known state: a voice command line carries the coder's + /// predictor and step index before its first nibble, so each line decodes on its own. + pub fn with_state(predictor: i16, index: u8) -> Self { + Self { predictor: predictor as i32, index: (index as i32).clamp(0, 88) } + } + + /// The predictor (the last sample) and the step index. + pub fn state(&self) -> (i16, u8) { + (self.predictor as i16, self.index as u8) + } + pub fn decode(&mut self, nibble: u8) -> i16 { let step = STEPS[self.index as usize]; let mut diff = step >> 3; @@ -39,6 +52,49 @@ impl Decoder { } } +/// The standard IMA ADPCM encoder (the one `tools/character_pack.py` uses): each nibble is chosen against +/// the decoder's own prediction, so the decoder above reproduces exactly what the encoder tracked. +#[derive(Clone, Copy, Debug, Default)] +pub struct Encoder { + decoder: Decoder, +} + +impl Encoder { + /// The state a decoder needs to start at the next nibble: predictor and step index. + pub fn state(&self) -> (i16, u8) { + self.decoder.state() + } + + pub fn encode(&mut self, sample: i16) -> u8 { + let step = STEPS[self.decoder.index as usize]; + let mut diff = sample as i32 - self.decoder.predictor; + let mut nibble = 0u8; + if diff < 0 { + nibble = 8; + diff = -diff; + } + if diff >= step { + nibble |= 4; + diff -= step; + } + if diff >= step >> 1 { + nibble |= 2; + diff -= step >> 1; + } + if diff >= step >> 2 { + nibble |= 1; + } + self.decoder.decode(nibble); + nibble + } + + /// Two samples to one byte, low nibble first. + pub fn encode_pair(&mut self, first: i16, second: i16) -> u8 { + let low = self.encode(first); + low | self.encode(second) << 4 + } +} + /// Turns one line's ADPCM bytes into 24 kHz stereo PCM, a piece at a time. The caller feeds the /// stored bytes in order; `fill` asks for the next byte through `next_byte`. pub struct LineStream { @@ -122,37 +178,16 @@ mod tests { use super::*; use std::vec::Vec; - /// The same IMA encoder as tools/character_pack.py, so the two can be checked against each other. + /// Packs samples two to a byte, as a Character pack line stores them. fn encode(samples: &[i16]) -> Vec { - let mut decoder = Decoder::default(); - let mut out = Vec::new(); - for pair in samples.chunks(2) { - let mut byte = 0u8; - for (half, &sample) in pair.iter().enumerate() { - let step = STEPS[decoder.index as usize]; - let mut diff = sample as i32 - decoder.predictor; - let mut nibble = 0u8; - if diff < 0 { - nibble = 8; - diff = -diff; - } - if diff >= step { - nibble |= 4; - diff -= step; - } - if diff >= step >> 1 { - nibble |= 2; - diff -= step >> 1; - } - if diff >= step >> 2 { - nibble |= 1; - } - decoder.decode(nibble); - byte |= nibble << (half * 4); - } - out.push(byte); - } - out + let mut encoder = Encoder::default(); + samples + .chunks(2) + .map(|pair| match pair { + [first, second] => encoder.encode_pair(*first, *second), + _ => encoder.encode(pair[0]), + }) + .collect() } fn sine(count: usize) -> Vec { @@ -214,4 +249,65 @@ mod tests { assert_eq!(pieces, whole); assert_eq!(pieces.len(), 452, "301 samples at 16 kHz become ceil(301 × 1.5) at 24 kHz"); } + + /// What a computer does with a voice command line: start a decoder at the line's state and decode + /// every nibble, low first. + fn decode_line(predictor: i16, index: u8, bytes: &[u8]) -> Vec { + let mut decoder = Decoder::with_state(predictor, index); + bytes.iter().flat_map(|&byte| [byte & 15, byte >> 4]).map(|nibble| decoder.decode(nibble)).collect() + } + + #[test] + fn the_encoder_tracks_exactly_what_the_decoder_reproduces() { + let original: Vec = (0..4800).map(|index| (libm::sinf(index as f32 * 0.07) * 9000.0 + libm::sinf(index as f32 * 0.31) * 3000.0) as i16).collect(); + let mut encoder = Encoder::default(); + let mut tracked = Vec::new(); + let bytes: Vec = original + .chunks(2) + .map(|pair| { + let byte = encoder.encode_pair(pair[0], pair[1]); + tracked.push(encoder.state().0); + byte + }) + .collect(); + let decoded = decode_line(0, 0, &bytes); + assert_eq!(decoded.len(), original.len()); + // The encoder's predictor after each byte is the decoder's output for its second sample. + assert_eq!(decoded.iter().skip(1).step_by(2).copied().collect::>(), tracked); + for (index, (&got, &want)) in decoded.iter().zip(&original).enumerate().skip(40) { + assert!((got as i32 - want as i32).abs() < 700, "sample {index}: {got} vs {want}"); + } + } + + #[test] + fn a_stream_cut_into_lines_decodes_line_by_line_from_each_lines_state() { + let original: Vec = (0..3000).map(|index| (libm::sinf(index as f32 * 0.05) * 15000.0) as i16).collect(); + let mut encoder = Encoder::default(); + let mut whole = Vec::new(); + let mut lines = Vec::new(); + for piece in original.chunks(600) { + let (predictor, index) = encoder.state(); + let bytes: Vec = piece.chunks(2).map(|pair| encoder.encode_pair(pair[0], pair[1])).collect(); + whole.extend_from_slice(&bytes); + lines.push((predictor, index, bytes)); + } + let from_start = decode_line(0, 0, &whole); + // Dropping the second line loses only its stretch: the third still decodes exactly. + let third = &lines[2]; + assert_eq!(decode_line(third.0, third.1, &third.2), from_start[1200..1800]); + let joined: Vec = lines.iter().flat_map(|(predictor, index, bytes)| decode_line(*predictor, *index, bytes)).collect(); + assert_eq!(joined, from_start); + } + + #[test] + fn extremes_stay_in_range() { + let mut encoder = Encoder::default(); + let samples = [i16::MAX, i16::MIN, i16::MAX, i16::MIN, 0, 0, i16::MAX, i16::MAX]; + let bytes: Vec = samples.chunks(2).map(|pair| encoder.encode_pair(pair[0], pair[1])).collect(); + let decoded = decode_line(0, 0, &bytes); + assert_eq!(decoded.len(), samples.len()); + let (_, index) = encoder.state(); + assert!(index <= 88); + assert_eq!(Decoder::with_state(-5, 200).state(), (-5, 88), "an index past the table is clamped"); + } } diff --git a/firmware-rs/core/src/buttons.rs b/firmware-rs/core/src/buttons.rs index f9ace952..9cf54ced 100644 --- a/firmware-rs/core/src/buttons.rs +++ b/firmware-rs/core/src/buttons.rs @@ -1,6 +1,7 @@ //! Each of the three keys does one thing, regardless of mode. Short presses fire on //! release; holding past the threshold fires a long press, and the release then no -//! longer counts as a short press. +//! longer counts as a short press. K0's release after a long press is an event of its own, +//! since a voice command records for as long as K0 stays down. //! //! On the board (ATK-DNESP32S3-BOX V1.1) K0 is the BOOT key, wired straight to GPIO0; K1 and //! K2 are P0.4 and P0.3 on the XL9555 expander. All three are active low. Reading the pins is @@ -14,8 +15,11 @@ const LONG_PRESS_MS: u32 = 1000; pub enum ButtonEvent { /// K0: pomodoro start / pause / resume. K0Short, - /// K0 long press: abandon the current phase. + /// K0 long press: push-to-talk while the computer offers voice commands, otherwise abandon the + /// current phase. K0Long, + /// K0 let go after a long press: the end of a push-to-talk recording. + K0LongEnd, /// K1: switch between duty and pomodoro. K1Short, /// K1 long press: send the buddy off to leisure right now. @@ -31,6 +35,8 @@ enum Press { None, Short, Long, + /// Released after a long press. + LongEnd, } #[derive(Clone, Copy)] @@ -65,7 +71,7 @@ impl Button { self.long_fired = false; return Press::None; } - return if self.long_fired { Press::None } else { Press::Short }; + return if self.long_fired { Press::LongEnd } else { Press::Short }; } if pressed && !self.long_fired && now.wrapping_sub(self.pressed_at) as i32 >= LONG_PRESS_MS as i32 { self.long_fired = true; @@ -96,18 +102,19 @@ impl Buttons { /// K0 and the expander are read separately: when the expander read fails only K0 is updated, as in the C firmware. pub fn update(&mut self, k0: bool, expander: Option<(bool, bool)>, now: u32) -> [Option; 3] { - fn emit(press: Press, short: ButtonEvent, long: ButtonEvent) -> Option { + fn emit(press: Press, short: ButtonEvent, long: ButtonEvent, long_end: Option) -> Option { match press { Press::None => None, Press::Short => Some(short), Press::Long => Some(long), + Press::LongEnd => long_end, } } let mut events = [None; 3]; - events[0] = emit(self.k0.update(k0, now), ButtonEvent::K0Short, ButtonEvent::K0Long); + events[0] = emit(self.k0.update(k0, now), ButtonEvent::K0Short, ButtonEvent::K0Long, Some(ButtonEvent::K0LongEnd)); if let Some((k1, k2)) = expander { - events[1] = emit(self.k1.update(k1, now), ButtonEvent::K1Short, ButtonEvent::K1Long); - events[2] = emit(self.k2.update(k2, now), ButtonEvent::K2Short, ButtonEvent::K2Long); + events[1] = emit(self.k1.update(k1, now), ButtonEvent::K1Short, ButtonEvent::K1Long, None); + events[2] = emit(self.k2.update(k2, now), ButtonEvent::K2Short, ButtonEvent::K2Long, None); } events } @@ -138,13 +145,23 @@ mod tests { } #[test] - fn holding_fires_long_once_and_release_is_silent() { + fn holding_fires_long_once_and_release_is_not_a_short_press() { let mut buttons = Buttons::new(Levels::default(), 0); k0(&mut buttons, true, 100); assert_eq!(k0(&mut buttons, true, 1099), None); assert_eq!(k0(&mut buttons, true, 1100), Some(ButtonEvent::K0Long)); assert_eq!(k0(&mut buttons, true, 2500), None); - assert_eq!(k0(&mut buttons, false, 2600), None); + // Push-to-talk needs to know when K0 comes back up. + assert_eq!(k0(&mut buttons, false, 2600), Some(ButtonEvent::K0LongEnd)); + assert_eq!(k0(&mut buttons, false, 2700), None); + } + + #[test] + fn k1_and_k2_say_nothing_when_let_go_after_a_long_press() { + let mut buttons = Buttons::new(Levels::default(), 0); + buttons.update(false, Some((true, true)), 100); + assert_eq!(buttons.update(false, Some((true, true)), 1100), [None, Some(ButtonEvent::K1Long), Some(ButtonEvent::K2Long)]); + assert_eq!(buttons.update(false, Some((false, false)), 1200), [None, None, None]); } #[test] diff --git a/firmware-rs/core/src/canvas.rs b/firmware-rs/core/src/canvas.rs index 6bb51203..1bccf34e 100644 --- a/firmware-rs/core/src/canvas.rs +++ b/firmware-rs/core/src/canvas.rs @@ -1,4 +1,5 @@ -//! A 320×240 RGB565 frame buffer and the most basic drawing: filled rectangles, thick lines, 5×7 bitmap text. +//! A 320×240 RGB565 frame buffer and the most basic drawing: filled rectangles, thick lines, 5×7 bitmap text, +//! and UTF-8 text that adds Chinese from the 12 px font in font.rs. //! //! The buffer is in the byte order the panel wants: high byte first for each pixel. The C firmware stores //! little-endian u16 in memory and has the LCD driver swap bytes on send; here it is stored as it goes on @@ -8,6 +9,8 @@ pub const WIDTH: i32 = 320; pub const HEIGHT: i32 = 240; pub const FRAME_BYTES: usize = (WIDTH * HEIGHT * 2) as usize; +use crate::font::{self, Glyph}; + const LETTER_GLYPHS: [[u8; 7]; 26] = [ [0x0e, 0x11, 0x11, 0x1f, 0x11, 0x11, 0x11], [0x1e, 0x11, 0x11, 0x1e, 0x11, 0x11, 0x1e], @@ -59,6 +62,8 @@ fn glyph_row(character: u8, row: usize) -> u8 { b'0'..=b'9' => DIGIT_GLYPHS[(upper - b'0') as usize][row], b'-' => if row == 3 { 0x1f } else { 0 }, b'.' => if row == 6 { 0x04 } else { 0 }, + b',' => match row { 5 => 0x04, 6 => 0x08, _ => 0 }, + b'\'' => if row < 2 { 0x04 } else { 0 }, b':' => if row == 2 || row == 5 { 0x04 } else { 0 }, b'/' => 1 << if row < 5 { 4 - row } else { 0 }, b'!' => if row < 5 || row == 6 { 0x04 } else { 0 }, @@ -142,21 +147,62 @@ impl<'a> Canvas<'a> { if character == 0 { break; } - for row in 0..7 { - let bits = glyph_row(character, row); - for column in 0..5 { - if bits & (1 << (4 - column)) != 0 { - self.fill_rect( - x + index as i32 * 6 * scale + column * scale, - y + row as i32 * scale, - scale, - scale, - color, - ); + self.draw_ascii(x + index as i32 * 6 * scale, y, character, scale, color); + } + } + + fn draw_ascii(&mut self, x: i32, y: i32, character: u8, scale: i32, color: u16) { + for row in 0..7 { + let bits = glyph_row(character, row); + for column in 0..5 { + if bits & (1 << (4 - column)) != 0 { + self.fill_rect(x + column * scale, y + row as i32 * scale, scale, scale, color); + } + } + } + } + + /// Draws UTF-8 text from (x, y), as many whole characters as fit in `max_width` pixels; stops at NUL. + /// ASCII is the 5×7 font exactly as `draw_text` draws it, with (x, y) its top left; Chinese is the 12 px + /// font, centred on the ASCII line (font.rs). Returns the width drawn. + pub fn draw_str(&mut self, x: i32, y: i32, text: &[u8], scale: i32, color: u16, max_width: i32) -> i32 { + let text = &text[..font::fit(text, scale, max_width)]; + let wide_scale = font::wide_scale(scale); + let top = font::wide_top(y, scale); + let mut left = x; + for (glyph, _) in font::glyphs(text) { + match glyph { + Glyph::Ascii(character) => self.draw_ascii(left, y, character, scale, color), + Glyph::Wide(index) => { + for row in 0..font::CELL { + let bits = font::row(index, row as usize); + for column in 0..font::CELL { + if bits & (0x800 >> column) != 0 { + self.fill_rect(left + column * wide_scale, top + row * wide_scale, wide_scale, wide_scale, color); + } + } + } + } + // A dotted box over the ink area, so it can't pass for 口. + Glyph::Missing => { + for step in (0..11).step_by(2) { + for (column, row) in [(step, 1), (step, 11), (0, 1 + step), (10, 1 + step)] { + self.fill_rect(left + column * wide_scale, top + row * wide_scale, wide_scale, wide_scale, color); + } } } } + left += font::advance(glyph, scale); } + font::width(text, scale) + } + + /// `draw_str` centred across the screen. + pub fn draw_str_centered(&mut self, y: i32, text: &[u8], scale: i32, color: u16) { + // As much as `draw_text_centered` allows, so ASCII lands where it always has. + let max_width = WIDTH / (6 * scale) * 6 * scale - scale; + let text = &text[..font::fit(text, scale, max_width)]; + self.draw_str((WIDTH - font::width(text, scale)) / 2, y, text, scale, color, max_width); } pub fn draw_text_centered(&mut self, y: i32, text: &[u8], scale: i32, color: u16) { @@ -174,3 +220,71 @@ impl<'a> Canvas<'a> { } } } + +#[cfg(test)] +mod tests { + use super::*; + use alloc::vec; + use alloc::vec::Vec; + + fn render(draw: impl FnOnce(&mut Canvas)) -> Vec { + let mut bytes = vec![0; FRAME_BYTES]; + draw(&mut Canvas::new(&mut bytes)); + bytes + } + + /// The lit area as (top, bottom, left, right), inclusive. + fn ink(bytes: &[u8]) -> (i32, i32, i32, i32) { + let lit: Vec<(i32, i32)> = + (0..WIDTH * HEIGHT).filter(|&index| bytes[index as usize * 2] != 0).map(|index| (index / WIDTH, index % WIDTH)).collect(); + let rows = || lit.iter().map(|&(row, _)| row); + let columns = || lit.iter().map(|&(_, column)| column); + (rows().min().unwrap(), rows().max().unwrap(), columns().min().unwrap(), columns().max().unwrap()) + } + + #[test] + fn ascii_draws_the_same_through_either_call() { + let text = b"Ready: 3 TASKS, 12m_ok?\x01~"; + for scale in 1..=3 { + let old = render(|canvas| canvas.draw_text(4, 30, text, scale, 0xffff, 40)); + let new = render(|canvas| { + canvas.draw_str(4, 30, text, scale, 0xffff, 40 * 6 * scale - scale); + }); + assert!(old == new, "scale {scale}"); + let old = render(|canvas| canvas.draw_text_centered(30, text, scale, 0xffff)); + let new = render(|canvas| canvas.draw_str_centered(30, text, scale, 0xffff)); + assert!(old == new, "centred, scale {scale}"); + } + } + + #[test] + fn chinese_sits_on_the_ascii_line() { + // Scale 1: the 5x7 glyph covers rows 100 to 106; the 12 px cell rows 97 to 108, ink from its second. + let (top, bottom, _, _) = ink(&render(|canvas| { + canvas.draw_str(10, 100, "国".as_bytes(), 1, 0xffff, 100); + })); + assert!(top >= 98 && (106..=108).contains(&bottom), "{top}..{bottom}"); + // Scale 2: still 12 px, centred on the 14 px ASCII. + let (top, bottom, left, right) = ink(&render(|canvas| { + canvas.draw_str(10, 100, "国".as_bytes(), 2, 0xffff, 100); + })); + assert!(top >= 102 && bottom <= 112 && left >= 10 && right <= 20, "{top}..{bottom} {left}..{right}"); + } + + #[test] + fn mixed_text_advances_by_each_font_and_stops_at_the_width() { + let whole = render(|canvas| assert_eq!(canvas.draw_str(0, 50, "A你B".as_bytes(), 1, 0xffff, 100), 6 + 12 + 5)); + assert_eq!(ink(&whole).3, 22); + // Whole characters only: 你 doesn't fit in 15 px after A. + let cut = render(|canvas| assert_eq!(canvas.draw_str(0, 50, "A你B".as_bytes(), 1, 0xffff, 15), 5)); + assert_eq!(ink(&cut).3, 4); + } + + #[test] + fn a_character_the_font_lacks_is_a_box() { + let lacking = render(|canvas| { + canvas.draw_str(0, 50, "\u{1F600}".as_bytes(), 1, 0xffff, 100); + }); + assert_eq!(ink(&lacking), (48, 58, 0, 10)); + } +} diff --git a/firmware-rs/core/src/display.rs b/firmware-rs/core/src/display.rs index 38e1eccd..919bab19 100644 --- a/firmware-rs/core/src/display.rs +++ b/firmware-rs/core/src/display.rs @@ -9,6 +9,7 @@ use alloc::boxed::Box; use alloc::vec::Vec; use crate::canvas::{Canvas, FRAME_BYTES, HEIGHT, WIDTH, half_bright}; +use crate::font; use crate::leisure::{self, Skit}; use crate::look::{self, Look}; use crate::pomodoro::{self, Phase, Run}; @@ -131,9 +132,40 @@ pub enum PanelKind { List, /// Facts: a small label, a large value. Facts, + /// A voice command being recorded: a level meter lit to this many of its `METER_STEPS` segments, and + /// the first line's label under it. + Meter(u8), + /// Wrapped text, one row per line's label, at this scale (2 or 1); see [`VOICE_TEXT_LARGE`]. + Text(u8), } -/// A panel over the whole screen (the device menu): what it says is the caller's, drawing it is the display's. +/// How wide a `Text` panel's rows are in pixels: 19 ASCII characters in large type, 38 in small, and 19 +/// Chinese characters in either (font.rs). +pub const VOICE_TEXT_WIDTH: i32 = 228; +const VOICE_TEXT_HEIGHT: i32 = 130; + +/// Row pitch of a `Text` panel: large type, small type, or small type with Chinese in it, whose 12 px +/// glyphs need taller rows than 5×7 ASCII. +fn voice_text_pitch(scale: u8, wide: bool) -> i32 { + match (scale >= 2, wide) { + (true, _) => 18, + (false, false) => 10, + (false, true) => 13, + } +} + +/// Rows a `Text` panel holds: 7 in large type, 13 in small, 10 in small with Chinese. Large type holds no +/// Chinese: the 12 px glyphs stay 12 px, and doubled ASCII next to them looks shouty, so text with Chinese +/// in it goes straight to small type. +pub fn voice_text_rows(scale: u8, wide: bool) -> usize { + if scale >= 2 && wide { + return 0; + } + (VOICE_TEXT_HEIGHT / voice_text_pitch(scale, wide)) as usize +} + +/// A panel over the whole screen (the device menu, a voice command): what it says is the caller's, drawing +/// it is the display's. #[derive(Clone, Debug, PartialEq, Eq)] pub struct Panel { pub kind: PanelKind, @@ -861,12 +893,12 @@ impl Display { canvas.fill_rect(x, y, width, 32, COLOR_MUTED); canvas.fill_rect(x + 2, y + 2, width - 4, 28, card_color); canvas.fill_rect(x + 2, y + 2, 4, 28, color); - canvas.draw_text(x + 12, y + 5, &task.title, 1, COLOR_TEXT, 26); + canvas.draw_str(x + 12, y + 5, &task.title, 1, COLOR_TEXT, 26 * 6 - 1); canvas.draw_text(x + 12, y + 17, short_state_label(task.state), 1, color, 4); // When the first line is a session name, the project name moves to the second line; if the title is the // project name, don't repeat it. if !task.project.is_empty() && !contains(&task.title, &task.project) { - canvas.draw_text(x + 42, y + 17, &task.project, 1, COLOR_MUTED, 16); + canvas.draw_str(x + 42, y + 17, &task.project, 1, COLOR_MUTED, 16 * 6 - 1); } let elapsed = format_elapsed(self.task_elapsed_seconds(task, now_ms)); let elapsed_width = elapsed.len() as i32 * 6 - 1; @@ -902,7 +934,7 @@ impl Display { } let slot = (self.animation_frame / IDLE_ROTATE_FRAMES) as usize % count; let color = if has_title && slot == 0 { COLOR_TEXT } else { COLOR_MUTED }; - canvas.draw_text_centered(195, lines[slot], 2, color); + canvas.draw_str_centered(195, lines[slot], 2, color); } /// Whether the alarm is still ringing. After the end, any change of view means the user acted: starting the @@ -951,7 +983,7 @@ impl Display { canvas.draw_text(PANEL_X, 176, b"AGENT", 1, COLOR_MUTED, 18); canvas.draw_text(PANEL_X, 188, label, 1, status_color, 18); if let Some(task) = self.tasks.first() { - canvas.draw_text(PANEL_X, 200, &task.title, 1, if self.link_lost { COLOR_MUTED } else { COLOR_TEXT }, 18); + canvas.draw_str(PANEL_X, 200, &task.title, 1, if self.link_lost { COLOR_MUTED } else { COLOR_TEXT }, 18 * 6 - 1); } } @@ -1121,7 +1153,7 @@ impl Display { // No rotation while the link is lost: a moving screen looks alive, the exact opposite of NO LINK. self.draw_idle_line(canvas, scene); } else if !self.title.is_empty() { - canvas.draw_text_centered(195, &self.title, 2, COLOR_TEXT); + canvas.draw_str_centered(195, &self.title, 2, COLOR_TEXT); } self.draw_pomodoro_badge(canvas, scene); } @@ -1482,8 +1514,18 @@ fn draw_panel(canvas: &mut Canvas, panel: &Panel) { } canvas.draw_text(120, y - 3, value, 2, tone_color(*tone), 13); } + PanelKind::Meter(_) => canvas.draw_text_centered(150 + index * 18, label, 2, tone_color(*tone)), + PanelKind::Text(scale) => { + let pitch = voice_text_pitch(scale, panel.lines.iter().any(|(label, _, _)| font::has_wide(label))); + // Room for one hanging closing mark past the row's width (font::wrap). + let max_width = VOICE_TEXT_WIDTH + font::CELL; + canvas.draw_str(PANEL_LEFT, 56 + index * pitch, label, if scale >= 2 { 2 } else { 1 }, tone_color(*tone), max_width); + } } } + if let PanelKind::Meter(level) = panel.kind { + draw_meter(canvas, level); + } canvas.fill_rect(PANEL_LEFT, 196, PANEL_RIGHT - PANEL_LEFT, 1, COLOR_MUTED); let mut x = PANEL_LEFT; for (key, action) in &panel.hints { @@ -1495,6 +1537,22 @@ fn draw_panel(canvas: &mut Canvas, panel: &Panel) { } } +/// The voice command's level meter: green for most of the range, then yellow, then orange near the top. +fn draw_meter(canvas: &mut Canvas, level: u8) { + let steps = crate::voice_command::METER_STEPS as i32; + let x = PANEL_LEFT + 5; + for step in 0..steps { + let lit = step < level as i32; + let color = match step * 16 / steps { + _ if !lit => half_bright(COLOR_MUTED), + 0..=9 => COLOR_DONE, + 10..=12 => COLOR_INPUT, + _ => COLOR_WORKING, + }; + canvas.fill_rect(x + step * 14, 92, 12, 40, color); + } +} + /// How the box is linked to its computer (USB wins, then the local network, then the Relay). #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum LinkPath { diff --git a/firmware-rs/core/src/firmware.rs b/firmware-rs/core/src/firmware.rs index 2d1be03a..e4328e46 100644 --- a/firmware-rs/core/src/firmware.rs +++ b/firmware-rs/core/src/firmware.rs @@ -11,7 +11,8 @@ use alloc::vec::Vec; use serde_json::{Map, Value}; use vibebuddy_protocol::{Event, ProtocolError, VERSION}; -use crate::audio::{Occasion, Sound, clamp_volume, VOLUME_DEFAULT}; +use crate::audio::{ES8311_ADDRESS, Occasion, SAMPLE_RATE, Sound, clamp_volume, VOLUME_DEFAULT}; +use crate::mic; use crate::buttons::{ButtonEvent, Buttons, Levels}; use crate::display::{Display, LinkPath, MAX_STATS, MAX_TASKS, Mode, Panel, PanelKind, SNAPSHOT_LINE_BYTES, Scene, Screen, Snapshot, State, TaskInput, Tone}; use crate::leisure::{Leisure, Tier}; @@ -20,13 +21,15 @@ use crate::pomodoro::{Phase, Pomodoro, Run, Transition}; use crate::relay_box; use crate::storage::{Flash, Settings, SettingsStore, find_partition}; use crate::text; -use crate::voice_pack::crc32; +use crate::spoken_reply::ReplyAudio; +use crate::voice_pack::{crc32, decode_base64}; use crate::lines::LinePicker; use crate::lan::{Conn, Lan, Output as LanOutput}; use crate::noise::Sealer; use crate::link::{self, Link, LinkStore, PairError, Wifi}; use crate::look::Look; use crate::voices::{Pick, Voices, VoiceError}; +use crate::voice_command::{self, Out as VoiceOut, VoiceCommand}; pub const MAX_LINE_BYTES: usize = 1024; const LINE_BUFFER_BYTES: usize = MAX_LINE_BYTES + 2; @@ -108,6 +111,70 @@ pub trait Board: Screen { } /// Starts the box again from the top, as after power-on. fn restart(&mut self) {} + + /// One register read on the codec's I2C bus; None when nothing acknowledges. For the microphone + /// diagnostic ([`crate::mic`]). + fn i2c_read(&mut self, _address: u8, _register: u8) -> Option { + None + } + fn i2c_write(&mut self, _address: u8, _register: u8, _value: u8) -> Result<(), ()> { + Err(()) + } + /// Starts capturing `bytes` of I2S input (24 kHz, 16-bit, two slots, little endian). Err where the + /// board has no input. + fn mic_start(&mut self, _bytes: usize) -> Result<(), ()> { + Err(()) + } + /// The capture once all of it is in, or why it stopped short; None while it is still running. + fn mic_take(&mut self) -> Option, &'static str>> { + None + } + + /// Starts the microphone for a voice command ([`crate::voice_command`]): the left slot (the ES8311's ADC) + /// at 24 kHz, without end, until [`Board::voice_stop`]. Err where the board can't record. + fn voice_start(&mut self) -> Result<(), ()> { + Err(()) + } + /// Appends the samples recorded since the last call, without waiting. + fn voice_take(&mut self, _samples: &mut Vec) {} + fn voice_stop(&mut self) {} + /// Writes one line of an audio stream (`VOICE DATA`) to the links that can carry one, and drops it whole + /// where it doesn't fit right now rather than wait: a gap in `seq` is what the computer expects then. + fn write_stream(&mut self, line: &[u8]) { + self.write(line); + } + /// Hands a spoken reply's audio to the player ([`crate::spoken_reply`]), which the speaker plays from while + /// no announcement is playing, through the same output and at the same volume. + fn reply_audio(&mut self, _audio: ReplyAudio) {} + /// A spoken reply is making sound right now. + fn reply_speaking(&self) -> bool { + false + } +} + +/// The codec's registers through the board's I2C, for register sequences in core. +struct BoardI2c<'a, B: Board> { + board: &'a mut B, + address: u8, +} + +impl crate::audio::Registers for BoardI2c<'_, B> { + type Error = (); + fn read(&mut self, register: u8) -> Result { + self.board.i2c_read(self.address, register).ok_or(()) + } + fn write(&mut self, register: u8, value: u8) -> Result<(), ()> { + self.board.i2c_write(self.address, register, value) + } +} + +/// Where the microphone diagnostic is. +enum MicState { + Idle, + /// Capturing since this time. + Recording { since: u32 }, + /// Streaming the capture out as `MIC DATA` lines, one per poll. + Sending { capture: Vec, sent: usize }, } /// What the box is linked to without the Relay. @@ -179,6 +246,17 @@ pub struct Firmware { reported_lights_out: bool, reported_hour: i32, reported_look: alloc::string::String, + mic: MicState, + /// Voice commands: whether the computer offers them, the turn and its overlay. + voice: VoiceCommand, + /// The microphone is running for a voice command. + voice_mic: bool, + /// The turn whose spoken replies are played; None once they were stopped (an announcement, mute, K2, the + /// overlay closing), and the rest of that turn's audio is then ignored. + reply_turn: Option, + /// An Agent's news that came while the voice overlay was up: it doesn't take the screen from a conversation + /// the user is having, the overlay marks it, and it is announced once the overlay closes. The worst one wins. + voice_held: Option, } /// Borrows the Firmware's pomodoro and leisure director to build the context for one frame. @@ -290,6 +368,11 @@ impl Firmware { reported_lights_out: false, reported_hour: -1, reported_look: alloc::string::String::from("ROBOT"), + mic: MicState::Idle, + voice: VoiceCommand::new(), + voice_mic: false, + reply_turn: None, + voice_held: None, } } @@ -590,9 +673,11 @@ impl Firmware { } self.continue_voice_begin(board); self.send_snapshot(board, now); + self.tend_mic(board, now); + self.tend_voice(board, now); if self.menu.expire(now) { Self::write_literal(board, "MENU CLOSED\n"); - self.show_menu(board); + self.show_overlay(board); } self.tick_buttons(board); let transition = self.pomodoro.tick(board.now_ms()); @@ -621,7 +706,89 @@ impl Firmware { } } + /// `device.mic.record`: report what answers on I2C where a microphone ADC could be, set the ES8311's + /// microphone gain if there is one, and start capturing; `tend_mic` reports and sends the result. + fn start_mic(&mut self, board: &mut B, fields: &BTreeMap) { + if !matches!(self.mic, MicState::Idle) || self.voice_mic { + return Self::write_literal(board, "MIC BUSY\n"); + } + let ms = number(fields, "ms").map_or(mic::DEFAULT_MS, |ms| (ms.max(0.0) as u32).clamp(mic::MIN_MS, mic::MAX_MS)); + let gain = number(fields, "gain_db").map_or(mic::DEFAULT_GAIN_DB, |db| (db.max(0.0) as u32).min(mic::MAX_GAIN_DB)); + let mut es8311 = false; + for (address, [high, low]) in mic::PROBES { + let id = board.i2c_read(address, high).zip(board.i2c_read(address, low)); + es8311 |= address == ES8311_ADDRESS && id.is_some(); + let line = match id { + Some((high, low)) => text!(32, "MIC PROBE 0x{:02X} ID {:02X}{:02X}\n", address, high, low), + None => text!(32, "MIC PROBE 0x{:02X} NONE\n", address), + }; + board.write(line.as_bytes()); + } + if es8311 { + let line = match mic::set_es8311_mic_gain(&mut BoardI2c { board: &mut *board, address: ES8311_ADDRESS }, gain) { + Ok(()) => text!(24, "MIC GAIN {} DB\n", mic::mic_gain_register(gain) as u32 * 6), + Err(()) => text!(24, "MIC GAIN ERROR\n"), + }; + board.write(line.as_bytes()); + } + match board.mic_start(mic::capture_bytes(ms)) { + Ok(()) => { + self.mic = MicState::Recording { since: board.now_ms() }; + board.write(text!(32, "MIC RECORDING {} MS\n", ms).as_bytes()); + } + Err(()) => Self::write_literal(board, "MIC UNAVAILABLE\n"), + } + } + + fn tend_mic(&mut self, board: &mut B, now: u32) { + match &mut self.mic { + MicState::Idle => {} + MicState::Recording { since } => { + let since = *since; + match board.mic_take() { + None => {} + Some(Err(reason)) => { + Self::write_value_line(board, "MIC ERROR ", reason.as_bytes()); + self.mic = MicState::Idle; + } + Some(Ok(capture)) => { + // Much longer than asked for means samples were lost on the way in. + let line = text!(48, "MIC CAPTURED {} MS {} BYTES\n", now.wrapping_sub(since), capture.len()); + board.write(line.as_bytes()); + for (index, [left, right]) in mic::levels(&capture).iter().enumerate() { + let at = index as u32 * mic::LEVEL_WINDOW_MS; + let line = text!(64, "MIC LEVEL {} L {} {} R {} {}\n", at, left.rms, left.peak, right.rms, right.peak); + board.write(line.as_bytes()); + } + let line = text!(48, "MIC BEGIN {} {} 16 {}\n", SAMPLE_RATE, mic::CHANNELS, capture.len()); + board.write(line.as_bytes()); + self.mic = MicState::Sending { capture, sent: 0 }; + } + } + } + // One line per pass, so the main loop keeps running while a minute of lines goes out over the UART. + MicState::Sending { capture, sent } if *sent < capture.len() => { + let end = (*sent + mic::DATA_CHUNK_BYTES).min(capture.len()); + let mut line = alloc::format!("MIC DATA {} ", *sent / mic::DATA_CHUNK_BYTES); + line.push_str(&mic::encode_base64(&capture[*sent..end])); + // One write, so the newline can't be dropped apart from its line. + line.push('\n'); + board.write(line.as_bytes()); + *sent = end; + } + MicState::Sending { capture, .. } => { + let line = text!(48, "MIC END {} {}\n", capture.len(), crc32(0, capture)); + board.write(line.as_bytes()); + self.mic = MicState::Idle; + } + } + } + fn set_link_lost(&mut self, board: &mut B, lost: bool) { + if lost && !self.link_lost { + // Whoever offered voice is gone; the next link offers it again if it can. + self.set_voice_available(board, false); + } self.link_lost = lost; let now = board.now_ms(); self.display.set_link_lost(board, &scene!(self, now), lost); @@ -666,6 +833,10 @@ impl Firmware { } }, }; + // An announcement or the Pomodoro chime wins over a spoken reply: the reply stops and the rest of its turn + // stays text only. The announcement is the box's own news and is short; queued behind a reply it + // could come half a minute late, and two voices taking turns would be one too many. + self.stop_reply(board); let queued = self.audio_ready && board.play(first).is_ok() && then.is_none_or(|sound| board.play(sound).is_ok()); if queued { Self::write_value_line(board, "AUDIO QUEUED ", label.as_bytes()); @@ -724,6 +895,23 @@ impl Firmware { /// its own job: leisure hides nothing that needs a look first. fn on_button(&mut self, board: &mut B, event: ButtonEvent) { let now = board.now_ms(); + // Letting go of K0 after a long press only ends a recording. + if event == ButtonEvent::K0LongEnd { + if self.voice.listening() { + self.end_recording(board, now); + } + return; + } + // The voice command's overlay: while recording only K2 (cancel) counts; afterwards K2 closes it, a + // long K0 starts the next command, and any other key closes it and then does its own job. + if self.voice.is_open() { + match event { + ButtonEvent::K2Short => return self.close_voice(board), + _ if self.voice.listening() => return, + ButtonEvent::K0Long if self.voice.available() => {} + _ => self.close_voice(board), + } + } if self.menu.is_open() { self.leisure.note_activity(now); let action = self.menu.key(event, self.menu_context(now), now); @@ -753,14 +941,20 @@ impl Firmware { ButtonEvent::K1Long => { self.menu.open(self.menu_context(now), now); Self::write_literal(board, "MENU OPEN\n"); - self.show_menu(board); + self.show_overlay(board); return; } - ButtonEvent::K0Short | ButtonEvent::K0Long => {} + ButtonEvent::K0Short | ButtonEvent::K0Long | ButtonEvent::K0LongEnd => {} } + // While the computer offers voice commands, a long K0 is push-to-talk; giving up a phase is then in + // the menu (ADR-0017). if event == ButtonEvent::K0Long { - self.stop_phase(board); + if self.voice.available() { + self.start_recording(board, now); + } else { + self.stop_phase(board); + } return; } let before = self.pomodoro.view(now); @@ -792,6 +986,10 @@ impl Firmware { /// K2 long, or MUTE in the menu. fn toggle_mute(&mut self, board: &mut B) { self.muted = !self.muted; + // Muting silences a reply at once; unmuting doesn't bring back the rest of that turn. + if self.muted { + self.stop_reply(board); + } let now = board.now_ms(); let muted = self.muted; self.display.set_muted(board, &scene!(self, now), muted); @@ -829,7 +1027,7 @@ impl Firmware { MenuAction::Close => Self::write_literal(board, "MENU CLOSED\n"), MenuAction::StopPhase => { Self::write_literal(board, "MENU CLOSED\n"); - self.show_menu(board); + self.show_overlay(board); self.stop_phase(board); } // The preview goes through announce, so it is silent when muted, like the app's. @@ -841,29 +1039,213 @@ impl Firmware { MenuAction::ForgetComputers => self.forget_computers(board), MenuAction::NewIdentity => { Self::write_literal(board, "MENU CLOSED\n"); - self.show_menu(board); + self.show_overlay(board); return self.new_identity(board); } } - self.show_menu(board); + self.show_overlay(board); + } + + /// Something that needs the user takes the screen back from the menu and from a voice command (a + /// recording is given up). + fn close_overlays(&mut self, board: &mut B) { + self.close_menu(board); + if self.voice.is_open() { + self.close_voice(board); + } } - /// Something that needs the user takes the screen back from the menu. fn close_menu(&mut self, board: &mut B) { if self.menu.is_open() { self.menu.close(); Self::write_literal(board, "MENU CLOSED\n"); - self.show_menu(board); + self.show_overlay(board); + } + } + + /// An Agent's news during a voice command waits for the overlay to close; the worst of what waits is said. + fn hold_for_voice(&mut self, board: &mut B, occasion: Occasion) { + let rank = |occasion: Occasion| match occasion.fallback().unwrap_or(occasion) { + Occasion::Failed => 2, + Occasion::InputRequired => 1, + _ => 0, + }; + if self.voice_held.is_none_or(|held| rank(occasion) > rank(held)) { + self.voice_held = Some(occasion); } + Self::write_value_line(board, "AUDIO HELD ", occasion_label(occasion).as_bytes()); + self.show_overlay(board); } - /// Hands the display the panel for what the menu shows now, or none when it's closed. - fn show_menu(&mut self, board: &mut B) { + /// Once the voice overlay is gone, says what was held back during it. + fn release_held(&mut self, board: &mut B) { + if !self.voice.is_open() + && let Some(occasion) = self.voice_held.take() + { + self.announce(board, occasion); + } + } + + /// Hands the display the panel for what the menu shows now, or the voice command's overlay, or none. + fn show_overlay(&mut self, board: &mut B) { let now = board.now_ms(); - let panel = self.menu.view().map(|view| self.menu_panel(view, now)); + let panel = match self.menu.view() { + Some(view) => Some(self.menu_panel(view, now)), + None => self.voice.panel(now).map(|mut panel| { + let waiting: Option<&[u8]> = match self.voice_held.map(|held| held.fallback().unwrap_or(held)) { + Some(Occasion::Failed) => Some(b"AN AGENT FAILED"), + Some(Occasion::InputRequired) => Some(b"AN AGENT NEEDS YOU"), + _ => None, + }; + if let Some(waiting) = waiting { + panel.lines.push((waiting.to_vec(), Vec::new(), Tone::Warn)); + } + panel + }), + }; self.display.set_panel(board, &scene!(self, now), panel); } + fn write_voice(board: &mut B, out: impl IntoIterator) { + for out in out { + match out { + VoiceOut::Line(line) => board.write(line.as_bytes()), + VoiceOut::Data(line) => board.write_stream(line.as_bytes()), + } + } + } + + /// `voice.available`, a new link or a lost one. + fn set_voice_available(&mut self, board: &mut B, on: bool) { + let out = self.voice.set_available(on); + Self::write_voice(board, out); + self.stop_voice_mic(board); + if !on { + self.stop_reply(board); + } + self.show_overlay(board); + } + + /// K0 held while voice is offered: the microphone at voice gain, any sound stopped so the box doesn't + /// hear itself (a spoken reply too: a new command interrupts the last one's answer), and a new turn. + fn start_recording(&mut self, board: &mut B, now: u32) { + self.voice.cancel(); + self.stop_voice_mic(board); + self.stop_reply(board); + // A box without the ES8311 (the NS4168 variant) has no gain to set; its microphone, if any, is untested. + let _ = mic::set_es8311_mic_gain(&mut BoardI2c { board: &mut *board, address: ES8311_ADDRESS }, voice_command::GAIN_DB); + let started = matches!(self.mic, MicState::Idle) && board.voice_start().is_ok(); + if started { + self.voice_mic = true; + board.stop_audio(); + let start = self.voice.start(now); + Self::write_voice(board, [start]); + self.reply_turn = Some(self.voice.turn()); + } else { + self.voice.no_microphone(now); + } + self.show_overlay(board); + } + + fn take_voice_samples(&mut self, board: &mut B) { + let mut samples = Vec::new(); + board.voice_take(&mut samples); + let out = self.voice.push(&samples); + Self::write_voice(board, out); + } + + /// K0 let go, or the 30 s are up: the last samples, then `VOICE END` (or `VOICE CANCEL` if it was short). + fn end_recording(&mut self, board: &mut B, now: u32) { + self.take_voice_samples(board); + let out = self.voice.release(now); + Self::write_voice(board, out); + self.stop_voice_mic(board); + self.show_overlay(board); + } + + /// K2, or something that needs the user: the overlay closes, and a recording is given up. + fn close_voice(&mut self, board: &mut B) { + let out = self.voice.cancel(); + Self::write_voice(board, out); + self.stop_voice_mic(board); + self.stop_reply(board); + self.show_overlay(board); + self.release_held(board); + } + + /// Stops the turn's spoken replies, and ignores the rest of its audio. + fn stop_reply(&mut self, board: &mut B) { + if self.reply_turn.take().is_some() { + board.reply_audio(ReplyAudio::Stop); + } + } + + /// `voice.audio`: a chunk of a spoken reply's MP3, or its end, for the turn being played. While muted the + /// replies are only shown. A chunk that doesn't decode is a chunk lost: the next one's `seq` gives it away. + fn on_voice_audio(&mut self, board: &mut B, fields: &BTreeMap) { + let field = |key| number(fields, key).filter(|value| *value >= 0.0).map(|value| value as u32); + let (Some(turn), Some(reply), Some(seq)) = (field("turn"), field("reply"), field("seq")) else { return }; + if self.reply_turn != Some(turn) { + return; + } + if fields.get("end") == Some(&Value::Bool(true)) { + // Synthesis failed or was cut short: what came plays, and the log says it isn't the whole reply. + if fields.get("error").is_some() { + let line = text!(48, "VOICE AUDIO CUT SHORT {} {}\n", turn, reply); + board.write(line.as_bytes()); + } + if !self.muted { + board.reply_audio(ReplyAudio::End { reply, seq }); + } + return; + } + let Some(data) = string(fields, "data").filter(|_| !self.muted) else { return }; + let mut bytes = alloc::vec![0u8; data.len() / 4 * 3 + 3]; + let Some(count) = decode_base64(data.as_bytes(), &mut bytes) else { return }; + bytes.truncate(count); + board.reply_audio(ReplyAudio::Chunk { reply, seq, data: bytes }); + } + + fn stop_voice_mic(&mut self, board: &mut B) { + if self.voice_mic && !self.voice.listening() { + board.voice_stop(); + self.voice_mic = false; + } + } + + /// Each pass: stream what the microphone has, end the recording at 30 s, move the meter, and close the + /// overlay when its time is up. + fn tend_voice(&mut self, board: &mut B, now: u32) { + if self.voice.listening() { + self.take_voice_samples(board); + if self.voice.due(now) { + self.end_recording(board, now); + } + self.show_overlay(board); + } else if self.voice.waiting() && self.menu.view().is_none() { + self.show_overlay(board); + } + // The text stays up while a reply is still being spoken; when it goes, so does what is left to say. + if !board.reply_speaking() && self.voice.expire(now) { + self.stop_reply(board); + self.show_overlay(board); + } + // However the overlay went (expired, K2, a lost link, voice withdrawn), what was held back is said now. + self.release_held(board); + self.stop_voice_mic(board); + } + + /// `voice.heard`, `voice.reply`, `voice.done` and `voice.error` for a turn. + fn on_voice_turn(&mut self, board: &mut B, fields: &BTreeMap, event: &str) { + let Some(turn) = number(fields, "turn").filter(|turn| *turn >= 0.0) else { return }; + let text = string(fields, if event == "voice.error" { "message" } else { "text" }); + let now = board.now_ms(); + if self.voice.event(event, turn as u32, text, string(fields, "reason"), now) { + self.stop_voice_mic(board); + self.show_overlay(board); + } + } + /// What the menu draws for a view. pub fn menu_panel(&self, view: View, now: u32) -> Panel { let line = |label: &[u8], value: &[u8], tone| (label.to_vec(), value.to_vec(), tone); @@ -1036,7 +1418,7 @@ impl Firmware { }; Self::report_pomodoro(board, if focus_ended { "FOCUS END" } else { "BREAK END" }); // The alarm waits for K0, which the menu would take. - self.close_menu(board); + self.close_overlays(board); if focus_ended { self.save_tally(board); } @@ -1059,7 +1441,7 @@ impl Firmware { fn tend_leisure(&mut self, board: &mut B) { let now = board.now_ms(); let pomodoro = self.pomodoro.view(now); - if !self.display.agent_idle() || pomodoro.run == Run::Running || self.link_lost || self.menu.is_open() { + if !self.display.agent_idle() || pomodoro.run == Run::Running || self.link_lost || self.menu.is_open() || self.voice.is_open() { self.leisure.note_activity(now); } @@ -1156,8 +1538,13 @@ impl Firmware { occasion = Some(special); } + let talking = self.voice.is_open(); if matches!(state, State::InputRequired | State::Failed) { - self.close_menu(board); + if talking { + self.close_menu(board); + } else { + self.close_overlays(board); + } } let shown = self.display.show_tasks(board, &scene!(self, now), state, title.map(str::as_bytes), &tasks); if shown.is_err() { @@ -1166,7 +1553,9 @@ impl Firmware { Self::write_value_line(board, "DISPLAY STATE ", state_label.as_bytes()); } if let Some(occasion) = occasion { - if self.end_speaks(occasion, now) { + if talking { + self.hold_for_voice(board, occasion); + } else if self.end_speaks(occasion, now) { self.announce(board, occasion); } else { Self::write_value_line(board, "AUDIO MERGED ", occasion_label(occasion).as_bytes()); @@ -1269,7 +1658,11 @@ impl Firmware { // The Mac asks when it first connects: mode, firmware build and voice are only reported // at boot or on change, and the daemon restarts more often than the device, so without // asking it would never know. - "device.hello" => self.announce_state(board), + // A new link starts with voice commands off; the computer offers them after this if it can. + "device.hello" => { + self.set_voice_available(board, false); + self.announce_state(board); + } // The daily greeting or a welcome back: a line and nothing else. It isn't agent activity, // so it neither changes the screen nor wakes leisure, and only those may be said this way. "buddy.say" => { @@ -1294,11 +1687,15 @@ impl Firmware { None => self.announce_volume(board), } // The app's slider can move while the menu shows the volume. - self.show_menu(board); + self.show_overlay(board); if fields.get("preview") == Some(&Value::Bool(true)) { self.announce(board, Occasion::Done); } } + // Voice commands (docs/protocol.md): the computer offers them, then answers each turn. + "voice.available" => self.set_voice_available(board, fields.get("on") == Some(&Value::Bool(true))), + "voice.heard" | "voice.reply" | "voice.done" | "voice.error" => self.on_voice_turn(board, fields, event), + "voice.audio" => self.on_voice_audio(board, fields), _ if event.starts_with("voice.") => self.handle_voice_event(board, fields, event), // Pairing is the computer at the other end of the cable telling the box who it is (ADR-0012). // Only over the cable: whoever is at the box decides who may link and which network it joins. @@ -1314,6 +1711,10 @@ impl Firmware { "device.relay.set" => self.handle_relay(board, fields), // The radio belongs to the device layer, which reports what it hears as `WIFI SEEN` lines. "device.wifi.scan" => self.scan_requested = true, + // Microphone bring-up (docs/firmware-bringup.md): a diagnostic like the screenshot, so not agent + // activity. Over the cable only: a recording is hundreds of lines. + "device.mic.record" if self.remote => Self::write_literal(board, "ERROR cable_only\n"), + "device.mic.record" => self.start_mic(board, fields), // Link self-test: send the received string's length and CRC back to the Mac to check for corrupted serial bytes. "device.echo" => { if let Some(data) = string(fields, "data") { @@ -1530,6 +1931,7 @@ impl Firmware { // stalled, and DMA replays whatever is in its buffer over and over, so it must be // silence first. The wait continues in poll. board.stop_audio(); + self.stop_reply(board); self.pending_voice_begin = Some((size as u32, board.now_ms())); self.continue_voice_begin(board); } diff --git a/firmware-rs/core/src/fixtures/spoken_reply.mp3 b/firmware-rs/core/src/fixtures/spoken_reply.mp3 new file mode 100644 index 00000000..1f7ac928 Binary files /dev/null and b/firmware-rs/core/src/fixtures/spoken_reply.mp3 differ diff --git a/firmware-rs/core/src/font.rs b/firmware-rs/core/src/font.rs new file mode 100644 index 00000000..629f5678 --- /dev/null +++ b/firmware-rs/core/src/font.rs @@ -0,0 +1,343 @@ +//! Text beyond ASCII: Chinese from a 12 px pixel font, next to the 5×7 ASCII font in canvas.rs. +//! +//! The glyphs are the 3,500 characters of level one of the Table of General Standard Chinese Characters +//! (通用规范汉字表 一级) plus CJK punctuation and the full-width forms, from Fusion Pixel Font 12 px +//! (SIL OFL 1.1, fonts/OFL.txt). `tools/make-cjk-font.py` builds the table: a sorted index of codepoints +//! and one packed 12×12 bitmap per character, 18 bytes each. It is included as read-only data, so it +//! stays in flash and costs no RAM. +//! +//! ASCII keeps the 5×7 font, so English looks as it always has. A Chinese character is one 12 px cell at +//! scales 1 and 2 (two cells at 4), vertically centred on the ASCII line; at scale 1 that puts the font's +//! own baseline exactly on the ASCII baseline. A character the table lacks draws as a dotted box. + +use alloc::vec::Vec; + +static TABLE: &[u8] = include_bytes!("../fonts/cjk-12px.bin"); + +/// A wide glyph's cell: 12 pixels square, ink in rows 1 to 11 and columns 0 to 10. +pub const CELL: i32 = 12; +const GLYPH_BYTES: usize = 18; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Glyph { + /// A byte for the 5×7 font, which draws its own placeholder for bytes it has no glyph for. + Ascii(u8), + /// An index into the table. + Wide(usize), + /// Not in the table, or not UTF-8: a box the width of a wide glyph. + Missing, +} + +fn count() -> usize { + u16::from_le_bytes([TABLE[0], TABLE[1]]) as usize +} + +fn codepoint(index: usize) -> u32 { + u16::from_le_bytes([TABLE[2 + index * 2], TABLE[3 + index * 2]]) as u32 +} + +/// The table index of a character, by binary search. +pub fn lookup(character: char) -> Option { + let wanted = character as u32; + let (mut low, mut high) = (0, count()); + while low < high { + let middle = (low + high) / 2; + match codepoint(middle).cmp(&wanted) { + core::cmp::Ordering::Less => low = middle + 1, + core::cmp::Ordering::Greater => high = middle, + core::cmp::Ordering::Equal => return Some(middle), + } + } + None +} + +/// One row of a wide glyph, 12 bits, the leftmost pixel in bit 11. +pub fn row(index: usize, row: usize) -> u16 { + let bitmap = &TABLE[2 + count() * 2 + index * GLYPH_BYTES..][..GLYPH_BYTES]; + let at = row * 3 / 2; + let (first, second) = (bitmap[at] as u16, bitmap[at + 1] as u16); + if row.is_multiple_of(2) { first << 4 | second >> 4 } else { (first & 0x0f) << 8 | second } +} + +/// A wide glyph's pixel size at a text scale: one pixel up to scale 3, so Chinese stays 12 px next to both +/// small and large ASCII. +pub fn wide_scale(scale: i32) -> i32 { + (scale / 2).max(1) +} + +/// Where a wide glyph's cell starts for ASCII drawn at `y`: centred on the 7-row ASCII glyph. +pub fn wide_top(y: i32, scale: i32) -> i32 { + y + (7 * scale - CELL * wide_scale(scale)).div_euclid(2) +} + +pub fn advance(glyph: Glyph, scale: i32) -> i32 { + match glyph { + Glyph::Ascii(_) => 6 * scale, + _ => CELL * wide_scale(scale), + } +} + +/// The blank column at the right of every glyph, which a measured width leaves off the last one. +fn gap(glyph: Glyph, scale: i32) -> i32 { + match glyph { + Glyph::Ascii(_) => scale, + _ => wide_scale(scale), + } +} + +/// The glyphs of UTF-8 text, each with its length in bytes; stops at NUL like the 5×7 drawing does. +pub fn glyphs(text: &[u8]) -> Glyphs<'_> { + Glyphs { text, at: 0, previous_alphanumeric: false } +} + +pub struct Glyphs<'a> { + text: &'a [u8], + at: usize, + previous_alphanumeric: bool, +} + +impl Iterator for Glyphs<'_> { + type Item = (Glyph, usize); + + fn next(&mut self) -> Option { + let byte = *self.text.get(self.at)?; + if byte == 0 { + return None; + } + let (glyph, length) = if byte.is_ascii() { + (Glyph::Ascii(byte), 1) + } else { + let length = match byte { + 0xc2..=0xdf => 2, + 0xe0..=0xef => 3, + 0xf0..=0xf4 => 4, + _ => 0, + }; + let character = self.text.get(self.at..self.at + length).and_then(|bytes| core::str::from_utf8(bytes).ok()?.chars().next()); + match character { + Some(character) => { + let next_alphanumeric = self.text.get(self.at + length).is_some_and(u8::is_ascii_alphanumeric); + (self.wide(character, next_alphanumeric), length) + } + None => { + // One box for a broken sequence, not one per byte. + let mut length = 1; + while self.text.get(self.at + length).is_some_and(|&byte| byte & 0xc0 == 0x80) { + length += 1; + } + (Glyph::Missing, length) + } + } + }; + self.at += length; + self.previous_alphanumeric = byte.is_ascii_alphanumeric(); + Some((glyph, length)) + } +} + +impl Glyphs<'_> { + fn wide(&self, character: char, next_alphanumeric: bool) -> Glyph { + // Curly single quotes against English letters are apostrophes ("it’s"): the ASCII one fits there, + // a full-width quote would leave a gap. + if matches!(character, '\u{2018}' | '\u{2019}') && (self.previous_alphanumeric || next_alphanumeric) { + return Glyph::Ascii(b'\''); + } + lookup(character).map_or(Glyph::Missing, Glyph::Wide) + } +} + +/// Pixel width of text as drawn at `scale`: up to the last glyph's ink, without its trailing gap. +pub fn width(text: &[u8], scale: i32) -> i32 { + let mut width = 0; + let mut last_gap = 0; + for (glyph, _) in glyphs(text) { + width += advance(glyph, scale); + last_gap = gap(glyph, scale); + } + width - last_gap +} + +/// How many bytes of `text` fit in `max_width` pixels, whole characters only. +pub fn fit(text: &[u8], scale: i32, max_width: i32) -> usize { + let mut used = 0; + let mut bytes = 0; + for (glyph, length) in glyphs(text) { + if used + advance(glyph, scale) - gap(glyph, scale) > max_width { + break; + } + used += advance(glyph, scale); + bytes += length; + } + bytes +} + +/// Whether any of the text draws in wide glyphs, which need taller rows than ASCII alone. +pub fn has_wide(text: &[u8]) -> bool { + glyphs(text).any(|(glyph, _)| !matches!(glyph, Glyph::Ascii(_))) +} + +/// Punctuation that must not start a row: rather than move it down alone, it hangs past the row's end. +fn closing(text: &[u8]) -> bool { + core::str::from_utf8(text) + .is_ok_and(|text| matches!(text, "，" | "。" | "、" | "；" | "：" | "！" | "？" | "）" | "》" | "〉" | "」" | "』" | "】" | "〕" | "”" | "’" | "…")) +} + +/// Wraps text to rows of at most `max_width` pixels at `scale`. Breaks at spaces, and between any two +/// Chinese characters or a Chinese character and a word; an ASCII word is only cut when it is longer than +/// a row. Runs of whitespace become one space. Closing Chinese punctuation that doesn't fit hangs one +/// glyph past `max_width` instead of starting a row. +pub fn wrap(text: &str, max_width: i32, scale: i32) -> Vec> { + let bytes = text.as_bytes(); + let mut rows: Vec> = Vec::new(); + let mut row: Vec = Vec::new(); + let mut at = 0; + let mut space = false; + let place = |unit: &[u8], space: bool, rows: &mut Vec>, row: &mut Vec| { + let mut unit = unit; + if !row.is_empty() { + let mut candidate = row.clone(); + if space { + candidate.push(b' '); + } + candidate.extend_from_slice(unit); + if width(&candidate, scale) <= max_width || (!space && closing(unit)) { + *row = candidate; + return; + } + rows.push(core::mem::take(row)); + } + while width(unit, scale) > max_width { + let first = glyphs(unit).next().map_or(unit.len(), |(_, length)| length); + let cut = fit(unit, scale, max_width).max(first); + rows.push(unit[..cut].to_vec()); + unit = &unit[cut..]; + } + *row = unit.to_vec(); + }; + let mut iterator = glyphs(bytes).peekable(); + while let Some((glyph, length)) = iterator.next() { + let start = at; + at += length; + match glyph { + Glyph::Ascii(byte) if byte.is_ascii_whitespace() => { + space = true; + continue; + } + Glyph::Ascii(_) => { + // A word: ASCII up to the next space or wide glyph. + while let Some(&(Glyph::Ascii(byte), length)) = iterator.peek() { + if byte.is_ascii_whitespace() { + break; + } + at += length; + iterator.next(); + } + } + _ => {} + } + place(&bytes[start..at], space, &mut rows, &mut row); + space = false; + } + if !row.is_empty() { + rows.push(row); + } + rows +} + +#[cfg(test)] +mod tests { + use super::*; + use alloc::string::String; + use alloc::vec; + + fn rows(text: &str, max_width: i32, scale: i32) -> Vec { + wrap(text, max_width, scale).into_iter().map(|row| String::from_utf8(row).unwrap()).collect() + } + + #[test] + fn the_table_holds_level_one_and_punctuation_in_order() { + assert!(count() > 3500); + assert_eq!(TABLE.len(), 2 + count() * (2 + GLYPH_BYTES)); + assert!((1..count()).all(|index| codepoint(index - 1) < codepoint(index))); + for character in "你好世界的一了是我龟鼠，。、！？：；“”‘’（）《》…—".chars() { + assert!(lookup(character).is_some(), "{character}"); + } + // Level two and beyond, and other scripts, are not in it. + assert_eq!(lookup('龘'), None); + assert_eq!(lookup('é'), None); + assert_eq!(lookup('A'), None); + } + + #[test] + fn a_glyph_keeps_its_spacing_row_and_column_blank() { + let index = lookup('国').unwrap(); + assert_eq!(row(index, 0), 0, "the top row is the line spacing"); + assert!((0..12).all(|line| row(index, line) & 1 == 0), "the right column is the character spacing"); + assert!((1..12).filter(|&line| row(index, line) != 0).count() >= 9); + // 一 is a single stroke across the middle. + let one = lookup('一').unwrap(); + assert_eq!((0..12).filter(|&line| row(one, line) != 0).count(), 1); + } + + #[test] + fn utf8_turns_into_glyphs_with_their_byte_lengths() { + let found: Vec<(Glyph, usize)> = glyphs("a你\u{1F600}".as_bytes()).collect(); + assert_eq!(found, [(Glyph::Ascii(b'a'), 1), (Glyph::Wide(lookup('你').unwrap()), 3), (Glyph::Missing, 4)]); + // A broken sequence is one box; NUL ends the text. + assert_eq!(glyphs(&[0xe4, 0xbd, b'x', 0, b'y']).collect::>(), [(Glyph::Missing, 2), (Glyph::Ascii(b'x'), 1)]); + assert_eq!(glyphs(&[0xff]).collect::>(), [(Glyph::Missing, 1)]); + } + + #[test] + fn curly_quotes_by_english_letters_are_apostrophes() { + assert_eq!(glyphs("it’s".as_bytes()).nth(2), Some((Glyph::Ascii(b'\''), 3))); + assert_eq!(glyphs("‘ok".as_bytes()).next(), Some((Glyph::Ascii(b'\''), 3))); + assert_eq!(glyphs("说‘好’".as_bytes()).nth(1), Some((Glyph::Wide(lookup('‘').unwrap()), 3))); + } + + #[test] + fn mixed_text_measures_each_font_by_its_own_advance() { + assert_eq!(width(b"", 1), 0); + assert_eq!(width(b"ABC", 1), 17, "the 5x7 font: 6 px a character, less the last gap"); + assert_eq!(width(b"ABC", 2), 34); + assert_eq!(width("你好".as_bytes(), 1), 23); + assert_eq!(width("你好".as_bytes(), 2), 23, "12 px next to large ASCII too"); + assert_eq!(width("OK了".as_bytes(), 1), 12 + 11); + assert_eq!(width("\u{1F600}".as_bytes(), 1), 11, "a missing character is as wide as a Chinese one"); + assert_eq!(fit("你好吗".as_bytes(), 1, 30), 6); + assert_eq!(fit("你好吗".as_bytes(), 1, 35), 9); + assert_eq!(fit(b"ABCDE", 1, 17), 3); + assert!(has_wide("ok，".as_bytes())); + assert!(!has_wide("it’s ok".as_bytes())); + } + + #[test] + fn ascii_wraps_at_spaces_and_cuts_only_long_words() { + let ascii = |text: &str, columns: i32| rows(text, columns * 6 - 1, 1); + assert_eq!(ascii("a bb ccc", 4), ["a bb", "ccc"]); + assert_eq!(ascii("abcdefghij", 4), ["abcd", "efgh", "ij"]); + assert_eq!(ascii("x abcdefghij y", 4), ["x", "abcd", "efgh", "ij y"]); + assert_eq!(ascii(" spaced\nout ", 20), ["spaced out"]); + assert_eq!(ascii("it’s", 10), ["it’s"]); + assert!(wrap("", 100, 1).is_empty()); + } + + #[test] + fn chinese_breaks_between_any_two_characters_but_keeps_words_whole() { + // Four characters to a row: 4 * 12 - 1. + assert_eq!(rows("一二三四五六七八九", 47, 1), ["一二三四", "五六七八", "九"]); + assert_eq!(rows("打开Codex看看", 47, 1), ["打开", "Codex看", "看"]); + assert_eq!(rows("你好 world", 47, 1), ["你好", "world"]); + // Closing punctuation hangs past the end rather than starting a row. + assert_eq!(rows("一二三四。五", 47, 1), ["一二三四。", "五"]); + assert_eq!(rows("一二三四 。", 47, 1), ["一二三四", "。"], "unless a space stands before it"); + // A missing character still takes a cell and wraps like one. + assert_eq!(rows("一二三\u{1F600}四", 47, 1), ["一二三\u{1F600}", "四"]); + } + + #[test] + fn a_row_narrower_than_one_glyph_still_moves_on() { + assert_eq!(rows("你好", 5, 1), ["你", "好"]); + assert_eq!(wrap("ab", 3, 1), vec![b"a".to_vec(), b"b".to_vec()]); + } +} diff --git a/firmware-rs/core/src/lib.rs b/firmware-rs/core/src/lib.rs index 33acb46e..9a6913b8 100644 --- a/firmware-rs/core/src/lib.rs +++ b/firmware-rs/core/src/lib.rs @@ -19,6 +19,7 @@ pub mod canvas; pub mod character_pack; pub mod display; pub mod firmware; +pub mod font; pub mod lan; pub mod lcd; pub mod leisure; @@ -26,11 +27,14 @@ pub mod lines; pub mod link; pub mod look; pub mod menu; +pub mod mic; pub mod noise; pub mod ota; pub mod pomodoro; pub mod relay_box; +pub mod spoken_reply; pub mod storage; +pub mod voice_command; pub mod voice_pack; pub mod voices; diff --git a/firmware-rs/core/src/mic.rs b/firmware-rs/core/src/mic.rs new file mode 100644 index 00000000..075b10a2 --- /dev/null +++ b/firmware-rs/core/src/mic.rs @@ -0,0 +1,159 @@ +//! Microphone bring-up diagnostic (`device.mic.record`, docs/firmware-bringup.md): record a few seconds of +//! I2S input, then report levels and stream the raw samples back as lines. Not a user feature. +//! +//! What the board has is settled on the device, not here: the command first reports which chips answer on +//! I2C (the ES8311 at 0x18, and anything at 0x40-0x43, where an ES7210 would sit), then records whatever +//! arrives on the I2S data-in pin. The input shares the speaker's clocks (one LRCK on the ES8311), so it +//! runs at the speaker's rate: 24 kHz, 16-bit, two slots. With esp_codec_dev's ES8311 setup (register +//! 0x44 = 0x58, "ADCL + DACR") the left slot carries the ADC and the right slot the DAC reference, as a +//! recording on one box showed (docs/hardware.md, 2026-10-13); the per-slot levels show it on others. + +use alloc::string::String; +use alloc::vec::Vec; + +use crate::audio::{ES8311_ADDRESS, Registers, SAMPLE_RATE}; + +/// Two 16-bit slots per frame. +pub const CHANNELS: u32 = 2; +pub const FRAME_BYTES: usize = 4; +pub const DEFAULT_MS: u32 = 3000; +/// At most 5 s: 480 KB of samples (the device's DMA buffer is 2.5 times that, in PSRAM), and about a +/// minute to send over the UART. +pub const MAX_MS: u32 = 5000; +pub const MIN_MS: u32 = 100; +/// xiaozhi's AUDIO_CODEC_DEFAULT_MIC_GAIN for this board. +pub const DEFAULT_GAIN_DB: u32 = 30; +pub const MAX_GAIN_DB: u32 = 42; +/// Raw bytes per `MIC DATA` line: 768 base64 characters, a line under 800 bytes like the protocol's 1 KB lines. +pub const DATA_CHUNK_BYTES: usize = 576; +/// One `MIC LEVEL` line per this much audio. +pub const LEVEL_WINDOW_MS: u32 = 100; + +/// Addresses whose chip ID the command reports, with the two ID registers read at each: the ES8311's +/// 0xFD/0xFE, and the ES7210's four strap addresses (AD1/AD0) with 0x3D/0x3E, the ID registers its +/// datasheet is reported to use (not confirmed from a primary source; the raw bytes are printed as read). +pub const PROBES: [(u8, [u8; 2]); 5] = [ + (ES8311_ADDRESS, [0xFD, 0xFE]), + (0x40, [0x3D, 0x3E]), + (0x41, [0x3D, 0x3E]), + (0x42, [0x3D, 0x3E]), + (0x43, [0x3D, 0x3E]), +]; + +/// Bytes captured in `ms` of input. +pub fn capture_bytes(ms: u32) -> usize { + (SAMPLE_RATE / 1000 * ms) as usize * FRAME_BYTES +} + +/// esp_codec_dev's `es8311_set_mic_gain`: the PGA gain in 6 dB steps, 0 to 42 dB, rounded down, written +/// as the whole of register 0x16. +pub fn mic_gain_register(db: u32) -> u8 { + (db.min(MAX_GAIN_DB) / 6) as u8 +} + +pub fn set_es8311_mic_gain(codec: &mut R, db: u32) -> Result<(), R::Error> { + codec.write(0x16, mic_gain_register(db)) +} + +/// RMS and peak of one slot over one window, in raw sample units (full scale is 32768). +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct Level { + pub rms: u32, + pub peak: u32, +} + +/// Levels of both slots for each `LEVEL_WINDOW_MS` of a capture of interleaved little-endian frames. +pub fn levels(capture: &[u8]) -> Vec<[Level; 2]> { + let window = (SAMPLE_RATE / 1000 * LEVEL_WINDOW_MS) as usize * FRAME_BYTES; + capture + .chunks(window) + .map(|piece| { + let mut squares = [0u64; 2]; + let mut peaks = [0u32; 2]; + let frames = (piece.len() / FRAME_BYTES).max(1) as u64; + for frame in piece.as_chunks::().0 { + for slot in 0..2 { + let sample = i16::from_le_bytes([frame[slot * 2], frame[slot * 2 + 1]]) as i32; + squares[slot] += (sample * sample) as u64; + peaks[slot] = peaks[slot].max(sample.unsigned_abs()); + } + } + core::array::from_fn(|slot| Level { rms: libm::sqrt(squares[slot] as f64 / frames as f64) as u32, peak: peaks[slot] }) + }) + .collect() +} + +/// Standard base64 with `=` padding, the counterpart of [`crate::voice_pack::decode_base64`]. +pub fn encode_base64(bytes: &[u8]) -> String { + const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; + let mut out = String::with_capacity(bytes.len().div_ceil(3) * 4); + for chunk in bytes.chunks(3) { + let bits = (chunk[0] as u32) << 16 | (*chunk.get(1).unwrap_or(&0) as u32) << 8 | *chunk.get(2).unwrap_or(&0) as u32; + for index in 0..4 { + out.push(if index <= chunk.len() { ALPHABET[(bits >> (18 - 6 * index) & 63) as usize] as char } else { '=' }); + } + } + out +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::voice_pack::decode_base64; + use std::vec; + + #[test] + fn gain_matches_esp_codec_dev() { + // es8311_set_mic_gain: <6 dB → 0 dB (0), ..., 30 dB → 5, ≥42 dB → 7. + assert_eq!(mic_gain_register(0), 0); + assert_eq!(mic_gain_register(5), 0); + assert_eq!(mic_gain_register(6), 1); + assert_eq!(mic_gain_register(30), 5); + assert_eq!(mic_gain_register(41), 6); + assert_eq!(mic_gain_register(42), 7); + assert_eq!(mic_gain_register(60), 7); + } + + #[test] + fn a_capture_is_24_khz_stereo_16_bit() { + assert_eq!(capture_bytes(1000), 96_000); + assert_eq!(capture_bytes(MAX_MS), 480_000); + } + + #[test] + fn levels_are_per_slot_and_per_window() { + // 100 ms of a full-scale square wave on the left, silence on the right, then 50 ms of a quieter left. + let mut capture = Vec::new(); + for index in 0..2400 { + let left: i16 = if index % 2 == 0 { 32767 } else { -32767 }; + capture.extend_from_slice(&left.to_le_bytes()); + capture.extend_from_slice(&0i16.to_le_bytes()); + } + for _ in 0..1200 { + capture.extend_from_slice(&(-100i16).to_le_bytes()); + capture.extend_from_slice(&7i16.to_le_bytes()); + } + assert_eq!( + levels(&capture), + vec![ + [Level { rms: 32767, peak: 32767 }, Level { rms: 0, peak: 0 }], + [Level { rms: 100, peak: 100 }, Level { rms: 7, peak: 7 }], + ] + ); + // i16::MIN has no positive counterpart; its peak is still reported, not wrapped. + assert_eq!(levels(&i16::MIN.to_le_bytes().repeat(2))[0][0].peak, 32768); + } + + #[test] + fn base64_round_trips() { + for length in 0..10 { + let bytes: Vec = (0..length).map(|index| (index * 37 + 250) as u8).collect(); + let encoded = encode_base64(&bytes); + let mut out = [0u8; 16]; + assert_eq!(decode_base64(encoded.as_bytes(), &mut out), Some(length)); + assert_eq!(&out[..length], &bytes[..]); + } + assert_eq!(encode_base64(b"foob"), "Zm9vYg=="); + assert_eq!(encode_base64(&[0u8; DATA_CHUNK_BYTES]).len(), 768); + } +} diff --git a/firmware-rs/core/src/spoken_reply.rs b/firmware-rs/core/src/spoken_reply.rs new file mode 100644 index 00000000..a5401a39 --- /dev/null +++ b/firmware-rs/core/src/spoken_reply.rs @@ -0,0 +1,467 @@ +//! A spoken reply's audio (docs/protocol.md, "Voice commands (experimental)"): the computer sends each reply +//! of a voice command's turn as an MP3 stream in numbered `voice.audio` chunks, and the box plays the replies +//! one after another through the same speaker path as announcements. +//! +//! [`ReplyPlayer`] takes the chunks as they come and hands out the codec's 24 kHz stereo PCM a piece at a time, +//! like [`crate::adpcm::LineStream`] does for a Character's lines. It holds no hardware: the firmware feeds it +//! ([`ReplyAudio`], through [`crate::firmware::Board::reply_audio`]) and the device's audio task pulls from it +//! when no announcement is playing. +//! +//! - Each reply's bytes are kept on the heap while it waits and plays, and let go of as they are decoded; the +//! decoder (about 22 KB) is allocated when a reply starts and freed once nothing is left to play. +//! - A reply starts once [`START_MS`] of it is buffered, or once all of it is in, and plays without waiting for +//! its end. If it runs dry it goes quiet until that much is buffered again, rather than stutter. +//! - Replies play in their order. One whose audio never comes (synthesis failed) is simply not there: the next +//! one plays. +//! - A missing chunk is skipped: the decoder finds the next frame, so a lost line costs a moment of the reply, +//! not the rest of it. A chunk that comes late (behind one already taken) is ignored. A reply too long to hold +//! or in a sample rate the speaker doesn't run at drops its audio, and the next reply follows. + +use alloc::boxed::Box; +use alloc::collections::VecDeque; +use alloc::vec::Vec; + +use nanomp3_core::{Decoder, MAX_SAMPLES_PER_FRAME}; + +/// The speaker's rate; the voice service's MP3 is 24 kHz mono, 48 kbps. +pub const SAMPLE_RATE: u32 = 24_000; +/// 48 kbps is 6 bytes a millisecond. +const BYTES_PER_MS: usize = 6; +/// A reply starts, and resumes after running dry, once this much of it is buffered. +pub const START_MS: usize = 300; +const START_BYTES: usize = START_MS * BYTES_PER_MS; +/// While more is coming, a frame is only decoded with this much past its start: the decoder trusts a frame once +/// it sees the next one's header, and skips a frame cut short as junk. The largest MPEG-2 Layer III frame at +/// 24 kHz (160 kbps) is 481 bytes. +const LOOKAHEAD: usize = 1024; +/// The most of one reply held at a time, about 43 s at 48 kbps; past it the reply is dropped. +pub const MAX_REPLY_BYTES: usize = 256 * 1024; +/// Decoded bytes are let go of in steps of this. +const COMPACT_BYTES: usize = 4096; + +/// What the firmware hands the player. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum ReplyAudio { + /// The next bytes of a reply's MP3 stream. + Chunk { reply: u32, seq: u32, data: Vec }, + /// The reply's audio is over; `seq` follows its last chunk. + End { reply: u32, seq: u32 }, + /// Stop playing and forget every reply. + Stop, +} + +struct Reply { + index: u32, + next_seq: u32, + bytes: Vec, + /// Decoded up to here. + read: usize, + ended: bool, + dropped: bool, +} + +impl Reply { + fn ahead(&self) -> usize { + self.bytes.len() - self.read + } + + fn drop_audio(&mut self) { + self.dropped = true; + self.bytes = Vec::new(); + self.read = 0; + } +} + +pub struct ReplyPlayer { + /// Replies with audio, in their order; the first is the one playing or next to. + replies: VecDeque, + /// Replies numbered below this are over (played, dropped, or passed by because a later one started); their + /// chunks are ignored. + floor: u32, + /// The playing reply's decoder; each reply is its own stream. + decoder: Option>, + /// The last decoded frame, and how much of it is handed out. + pcm: Vec, + pcm_read: usize, + pcm_len: usize, + /// The first reply had enough buffered to start and hasn't run dry since. + sounding: bool, +} + +impl Default for ReplyPlayer { + fn default() -> Self { + Self::new() + } +} + +impl ReplyPlayer { + pub const fn new() -> Self { + Self { replies: VecDeque::new(), floor: 0, decoder: None, pcm: Vec::new(), pcm_read: 0, pcm_len: 0, sounding: false } + } + + pub fn apply(&mut self, audio: ReplyAudio) { + match audio { + ReplyAudio::Chunk { reply, seq, data } => self.chunk(reply, seq, Some(&data)), + ReplyAudio::End { reply, seq } => self.chunk(reply, seq, None), + // Everything goes, the memory too. + ReplyAudio::Stop => *self = Self::new(), + } + } + + /// Something is still to be played: a reply buffered or playing. + pub fn pending(&self) -> bool { + !self.replies.is_empty() || self.pcm_read < self.pcm_len + } + + /// A reply is making sound right now (not waiting for its first bytes or after running dry). + pub fn speaking(&self) -> bool { + self.sounding || self.pcm_read < self.pcm_len + } + + /// A chunk of a reply, or with no data its end. + fn chunk(&mut self, index: u32, seq: u32, data: Option<&[u8]>) { + if index < self.floor { + return; + } + let at = match self.replies.iter().position(|reply| reply.index >= index) { + Some(at) if self.replies[at].index == index => at, + found => { + let at = found.unwrap_or(self.replies.len()); + let reply = Reply { index, next_seq: 0, bytes: Vec::new(), read: 0, ended: false, dropped: false }; + self.replies.insert(at, reply); + at + } + }; + let reply = &mut self.replies[at]; + if reply.dropped || reply.ended { + return; + } + if seq < reply.next_seq { + return; + } + reply.next_seq = seq + 1; + let Some(data) = data else { + reply.ended = true; + return; + }; + if reply.bytes.len() + data.len() > MAX_REPLY_BYTES || reply.bytes.try_reserve(data.len()).is_err() { + return self.drop_reply(at); + } + reply.bytes.extend_from_slice(data); + } + + fn drop_reply(&mut self, at: usize) { + self.replies[at].drop_audio(); + // The one playing stops at once, with what it had decoded. + if at == 0 { + self.pcm_len = 0; + self.pcm_read = 0; + } + } + + /// Fills `out` with the next 24 kHz stereo 16-bit PCM and returns how many bytes (a multiple of 4) were + /// written; fewer than asked, or none, while there is nothing to play right now. The caller pads with silence. + pub fn fill(&mut self, out: &mut [u8]) -> usize { + let mut written = 0; + while written + 4 <= out.len() { + if self.pcm_read == self.pcm_len && !self.decode_next() { + break; + } + let [low, high] = self.pcm[self.pcm_read].to_le_bytes(); + out[written..written + 4].copy_from_slice(&[low, high, low, high]); + self.pcm_read += 1; + written += 4; + } + written + } + + /// Decodes the next frame of the first reply; false when there is nothing to play right now. + fn decode_next(&mut self) -> bool { + loop { + let Some(reply) = self.replies.front_mut() else { + // Nothing left: the decoder and its buffers go back to the heap. + self.decoder = None; + self.pcm = Vec::new(); + self.sounding = false; + return false; + }; + if reply.dropped || (reply.ended && reply.ahead() == 0) { + self.floor = self.floor.max(reply.index + 1); + self.replies.pop_front(); + self.decoder = None; + self.sounding = false; + continue; + } + if !self.sounding { + if reply.ahead() < START_BYTES && !reply.ended { + return false; + } + self.sounding = true; + // Once a reply starts, an earlier one that shows up late is passed by. + self.floor = self.floor.max(reply.index); + } + if reply.ahead() < LOOKAHEAD && !reply.ended { + // Ran dry: quiet until it has enough to start again. + self.sounding = false; + return false; + } + if self.decoder.is_none() { + self.decoder = new_decoder(); + } + let Some(decoder) = self.decoder.as_mut() else { + reply.drop_audio(); + continue; + }; + if self.pcm.is_empty() { + self.pcm = alloc::vec![0; MAX_SAMPLES_PER_FRAME]; + } + let (used, result) = decoder.decode(&reply.bytes[reply.read..], &mut self.pcm); + // Nothing consumed only happens on an empty input; with the reply over, the rest is junk. + reply.read += if used == 0 { reply.ahead() } else { used }; + if reply.read >= COMPACT_BYTES { + reply.bytes.drain(..reply.read); + reply.read = 0; + } + match result { + Ok(frame) if frame.sample_rate != SAMPLE_RATE => reply.drop_audio(), + Ok(frame) if frame.samples_produced > 0 => { + // Mono as the service sends it; of a stereo stream, the left channel. + let channels = frame.channels.num() as usize; + for sample in 1..frame.samples_produced { + self.pcm[sample] = self.pcm[sample * channels]; + } + self.pcm_read = 0; + self.pcm_len = frame.samples_produced; + return true; + } + // The first frames after a start lean on a bit reservoir that isn't there; a corrupt frame is skipped. + _ => {} + } + } + } +} + +/// The decoder, built straight on the heap: 22 KB must not pass through the stack on its way there. +fn new_decoder() -> Option> { + let layout = core::alloc::Layout::new::(); + // SAFETY: `Decoder::new()` is all zero bytes in nanomp3-core 0.2.0 (pinned with `=` for this): zeroed + // float arrays, zero counters and a `false`. The test `a_zeroed_decoder_decodes_like_a_new_one` holds it + // to that. A non-null pointer from `alloc_zeroed` with Decoder's layout is what `Box::from_raw` takes. + unsafe { + let raw = alloc::alloc::alloc_zeroed(layout).cast::(); + (!raw.is_null()).then(|| Box::from_raw(raw)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::vec; + + const FIXTURE: &[u8] = include_bytes!("fixtures/spoken_reply.mp3"); + + /// The fixture decoded in one go with a decoder on the stack: the reference, as stereo bytes. + fn one_shot(mp3: &[u8], mut decoder: Decoder) -> Vec { + let mut pcm = [0i16; MAX_SAMPLES_PER_FRAME]; + let mut out = Vec::new(); + let mut rest = mp3; + while !rest.is_empty() { + let (used, result) = decoder.decode(rest, &mut pcm); + rest = &rest[used.max(1).min(rest.len())..]; + if let Ok(frame) = result { + assert_eq!((frame.sample_rate, frame.channels.num()), (24_000, 1)); + for &sample in &pcm[..frame.samples_produced] { + out.extend_from_slice(&sample.to_le_bytes()); + out.extend_from_slice(&sample.to_le_bytes()); + } + } + } + out + } + + fn chunks(mp3: &[u8], sizes: &[usize]) -> Vec> { + let mut out = Vec::new(); + let mut at = 0; + for size in sizes.iter().cycle() { + if at == mp3.len() { + break; + } + let end = (at + size).min(mp3.len()); + out.push(mp3[at..end].to_vec()); + at = end; + } + out + } + + fn send(player: &mut ReplyPlayer, reply: u32, pieces: &[Vec]) { + for (seq, data) in pieces.iter().enumerate() { + player.apply(ReplyAudio::Chunk { reply, seq: seq as u32, data: data.clone() }); + } + player.apply(ReplyAudio::End { reply, seq: pieces.len() as u32 }); + } + + /// Everything the player has to give, pulled in pieces the size the audio task uses. + fn drain(player: &mut ReplyPlayer, piece: usize) -> Vec { + let mut out = Vec::new(); + let mut buffer = vec![0u8; piece]; + loop { + let count = player.fill(&mut buffer); + out.extend_from_slice(&buffer[..count]); + if count == 0 { + return out; + } + } + } + + #[test] + fn the_fixture_is_the_services_format() { + let pcm = one_shot(FIXTURE, Decoder::new()); + // Stereo 16-bit bytes at 24 kHz: a few seconds of speech. + let seconds = pcm.len() as f32 / 4.0 / 24_000.0; + assert!((3.0..6.0).contains(&seconds), "{seconds} s"); + } + + #[test] + fn a_zeroed_decoder_decodes_like_a_new_one() { + let zeroed = new_decoder().unwrap(); + assert_eq!(one_shot(FIXTURE, (*zeroed).clone()), one_shot(FIXTURE, Decoder::new())); + } + + #[test] + fn odd_chunks_pulled_as_they_come_give_the_same_pcm_as_one_go() { + let expected = one_shot(FIXTURE, Decoder::new()); + for (sizes, piece) in [(&[1usize, 7, 333, 577][..], 1000usize), (&[576], 4), (&[97, 1500], 1004), (&[FIXTURE.len()], 3000)] { + let mut player = ReplyPlayer::new(); + let pieces = chunks(FIXTURE, sizes); + let mut out = Vec::new(); + let mut buffer = vec![0u8; piece]; + // The audio task pulls between chunks, so the player runs dry and waits along the way. + for (seq, data) in pieces.iter().enumerate() { + player.apply(ReplyAudio::Chunk { reply: 0, seq: seq as u32, data: data.clone() }); + let count = player.fill(&mut buffer); + out.extend_from_slice(&buffer[..count]); + } + player.apply(ReplyAudio::End { reply: 0, seq: pieces.len() as u32 }); + out.extend(drain(&mut player, piece)); + assert!(out == expected, "chunks {sizes:?}, pulled {piece} at a time: {} bytes, expected {}", out.len(), expected.len()); + assert!(!player.pending() && player.decoder.is_none() && player.pcm.is_empty(), "everything is let go of"); + } + } + + #[test] + fn a_reply_starts_with_300_ms_buffered_without_waiting_for_its_end() { + let mut player = ReplyPlayer::new(); + let mut buffer = [0u8; 1000]; + player.apply(ReplyAudio::Chunk { reply: 0, seq: 0, data: FIXTURE[..START_BYTES - 1].to_vec() }); + assert_eq!(player.fill(&mut buffer), 0, "not yet"); + assert!(!player.speaking() && player.pending()); + player.apply(ReplyAudio::Chunk { reply: 0, seq: 1, data: FIXTURE[START_BYTES - 1..START_BYTES].to_vec() }); + assert_eq!(player.fill(&mut buffer), 1000); + assert!(player.speaking()); + // It plays until less than a frame's lookahead is left, then waits quietly instead of stuttering. + let played = drain(&mut player, 1000).len() + 1000; + assert!(played < one_shot(&FIXTURE[..START_BYTES], Decoder::new()).len() && !player.speaking(), "{played}"); + player.apply(ReplyAudio::Chunk { reply: 0, seq: 2, data: FIXTURE[START_BYTES..START_BYTES + 100].to_vec() }); + assert_eq!(player.fill(&mut buffer), 0, "a little more isn't enough to start again"); + // A short reply that ends below the threshold plays at once. + let mut short = ReplyPlayer::new(); + send(&mut short, 0, &[FIXTURE[..600].to_vec()]); + assert!(short.fill(&mut buffer) > 0); + } + + #[test] + fn replies_play_in_order_whatever_order_their_audio_comes_in() { + let first = &FIXTURE[..FIXTURE.len() / 2]; + let expected_first = one_shot(first, Decoder::new()); + let expected_whole = one_shot(FIXTURE, Decoder::new()); + let mut player = ReplyPlayer::new(); + // Reply 1 comes in whole before reply 0's end, but waits its turn. + player.apply(ReplyAudio::Chunk { reply: 0, seq: 0, data: first[..100].to_vec() }); + send(&mut player, 1, &chunks(FIXTURE, &[768])); + player.apply(ReplyAudio::Chunk { reply: 0, seq: 1, data: first[100..].to_vec() }); + player.apply(ReplyAudio::End { reply: 0, seq: 2 }); + let out = drain(&mut player, 1000); + assert!(out[..expected_first.len()] == expected_first[..], "reply 0 first"); + assert!(out[expected_first.len()..] == expected_whole[..], "then reply 1, from a fresh decoder"); + } + + #[test] + fn a_reply_without_audio_is_passed_by_and_a_late_earlier_one_is_ignored() { + let mut player = ReplyPlayer::new(); + // Reply 0 never got audio; reply 2 starts, so reply 1 turning up afterwards is too late. + send(&mut player, 2, &chunks(FIXTURE, &[768])); + let mut buffer = [0u8; 1000]; + assert!(player.fill(&mut buffer) > 0); + send(&mut player, 1, &chunks(FIXTURE, &[768])); + let out = drain(&mut player, 1000).len() + 1000; + assert_eq!(out, one_shot(FIXTURE, Decoder::new()).len()); + } + + #[test] + fn a_missing_chunk_is_skipped_and_a_late_one_ignored() { + let pieces = chunks(FIXTURE, &[768]); + let expected = one_shot(FIXTURE, Decoder::new()); + // A chunk lost in the middle: the reply plays on past it, only a little shorter, then the next one whole. + let mut player = ReplyPlayer::new(); + for (seq, data) in pieces.iter().enumerate() { + if seq != 3 { + player.apply(ReplyAudio::Chunk { reply: 0, seq: seq as u32, data: data.clone() }); + } + } + player.apply(ReplyAudio::End { reply: 0, seq: pieces.len() as u32 }); + send(&mut player, 1, &pieces); + let played = drain(&mut player, 1000); + assert!(played.ends_with(&expected), "the next reply plays whole"); + let first = played.len() - expected.len(); + assert!(first > expected.len() * 3 / 4 && first < expected.len(), "{first} of {}", expected.len()); + + // A chunk that comes after a later one was taken is ignored; the reply carries on. + let mut player = ReplyPlayer::new(); + for (seq, data) in pieces.iter().enumerate() { + if seq == 5 { + continue; + } + player.apply(ReplyAudio::Chunk { reply: 0, seq: seq as u32, data: data.clone() }); + if seq == 6 { + player.apply(ReplyAudio::Chunk { reply: 0, seq: 5, data: pieces[5].clone() }); + } + } + player.apply(ReplyAudio::End { reply: 0, seq: pieces.len() as u32 }); + let played = drain(&mut player, 1000).len(); + assert!(played > expected.len() * 3 / 4 && played < expected.len(), "{played}"); + } + + #[test] + fn a_reply_cut_short_plays_what_it_has() { + // Synthesis stopped partway: the end arrives in order after fewer bytes. + let part = &FIXTURE[..FIXTURE.len() / 3]; + let mut player = ReplyPlayer::new(); + send(&mut player, 0, &chunks(part, &[768])); + assert!(drain(&mut player, 1000) == one_shot(part, Decoder::new())); + } + + #[test] + fn stop_forgets_everything_and_garbage_is_not_played() { + let mut player = ReplyPlayer::new(); + send(&mut player, 0, &chunks(FIXTURE, &[768])); + let mut buffer = [0u8; 1000]; + assert_eq!(player.fill(&mut buffer), 1000); + player.apply(ReplyAudio::Stop); + assert!(!player.pending() && player.decoder.is_none()); + assert_eq!(player.fill(&mut buffer), 0); + // After a stop, a new turn's replies count from 0 again. + send(&mut player, 0, &[vec![0x55; 3000]]); + assert_eq!(drain(&mut player, 1000).len(), 0, "no frames, no sound"); + assert!(!player.pending()); + } + + #[test] + fn a_reply_too_long_to_hold_is_dropped() { + let mut player = ReplyPlayer::new(); + let piece = vec![0u8; 64 * 1024]; + for seq in 0..5 { + player.apply(ReplyAudio::Chunk { reply: 0, seq, data: piece.clone() }); + } + assert!(player.replies[0].dropped && player.replies[0].bytes.capacity() == 0); + } +} diff --git a/firmware-rs/core/src/text.rs b/firmware-rs/core/src/text.rs index e0755c2b..fdee1769 100644 --- a/firmware-rs/core/src/text.rs +++ b/firmware-rs/core/src/text.rs @@ -55,8 +55,25 @@ macro_rules! text { }}; } -/// Copies bytes truncated to at most `limit` bytes, matching C's strncpy into a fixed-size array. +/// Copies bytes truncated to at most `limit` bytes, matching C's strncpy into a fixed-size array. A cut never +/// falls inside a UTF-8 character: the partial character goes too. pub fn truncated(bytes: &[u8], limit: usize) -> alloc::vec::Vec { - let end = bytes.iter().take(limit).position(|&byte| byte == 0).unwrap_or(bytes.len().min(limit)); + let mut end = bytes.iter().take(limit).position(|&byte| byte == 0).unwrap_or(bytes.len().min(limit)); + while end > 0 && bytes.get(end).is_some_and(|&byte| byte & 0xc0 == 0x80) { + end -= 1; + } bytes[..end].to_vec() } + +#[cfg(test)] +mod tests { + use super::truncated; + + #[test] + fn truncation_keeps_whole_characters() { + assert_eq!(truncated(b"abcdef", 4), b"abcd"); + assert_eq!(truncated(b"ab\0cd", 4), b"ab"); + assert_eq!(truncated("你好".as_bytes(), 4), "你".as_bytes()); + assert_eq!(truncated("你好".as_bytes(), 6), "你好".as_bytes()); + } +} diff --git a/firmware-rs/core/src/voice_command.rs b/firmware-rs/core/src/voice_command.rs new file mode 100644 index 00000000..2a025cab --- /dev/null +++ b/firmware-rs/core/src/voice_command.rs @@ -0,0 +1,720 @@ +//! Voice commands on the box (ADR-0017; the contract is docs/protocol.md, "Voice commands (experimental)"): +//! while the computer offers voice, holding K0 records, and the recording streams out as `VOICE` lines; the +//! computer answers with what it heard and the resident session's spoken replies, which the box shows on an +//! overlay. Like the device menu, the overlay is not a mode: the mode underneath keeps running. +//! +//! This module is the turn and the overlay, with no hardware: it takes the microphone's samples, encodes them, +//! frames the lines and says what the overlay shows. `firmware.rs` feeds it keys, samples and events, writes +//! its lines and starts and stops the microphone. + +use alloc::string::String; +use alloc::vec::Vec; + +use crate::adpcm::Encoder; +use crate::display::{Panel, PanelKind, Tone, VOICE_TEXT_WIDTH, voice_text_rows}; +use crate::font; +use crate::mic::encode_base64; + +/// The microphone runs at the speaker's rate (docs/firmware-bringup.md, "Microphone"). +pub const SAMPLE_RATE: u32 = 24_000; +/// Samples per `VOICE DATA` line: 576 bytes of ADPCM, 768 base64 characters, the protocol's limit. A line +/// is 48 ms of audio, so about 21 lines a second, 16 KB/s of text. +pub const SAMPLES_PER_LINE: usize = 1152; +/// A recording stops on its own after this much audio. +pub const MAX_SAMPLES: u32 = SAMPLE_RATE * 30; +/// If the samples stop coming, the recording still ends this long after it started. +const MAX_RECORDING_MS: u32 = 31_000; +/// Shorter than this is a slip of the finger: the turn is cancelled. +pub const MIN_RECORDING_MS: u32 = 500; +/// The ES8311's microphone gain while recording a voice command. At the 30 dB the bring-up diagnostic +/// defaults to, a voice close to the box was only about -31 dBFS; 42 dB (the most the PGA has) brings it +/// to about -19. +pub const GAIN_DB: u32 = 42; +/// With nothing from the computer for this long after the recording, the overlay closes. +pub const QUIET_CLOSE_MS: u32 = 60_000; +/// After `voice.done`, the replies stay up this long to be read (or until K2). +pub const READ_MS: u32 = 15_000; +/// An error stays up this long. +pub const ERROR_MS: u32 = 6_000; +/// Segments in the level meter. +pub const METER_STEPS: u8 = 16; +/// The meter shows the level of each 100 ms. +const LEVEL_WINDOW: u32 = SAMPLE_RATE / 10; +/// The meter's range: silence at the bottom, a voice up close near the top. +const METER_FLOOR_DB: f32 = -60.0; +const METER_TOP_DB: f32 = -10.0; +/// Replies kept for the overlay; older ones scroll away anyway. +const MAX_REPLIES: usize = 8; + +/// A line for the computer. Data lines may only go where an audio stream fits (docs/protocol.md): the +/// device layer drops them, whole, rather than hold up the main loop. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum Out { + Line(String), + Data(String), +} + +/// Why a turn ended badly, from `voice.error`'s `reason`, or the box's own. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Reason { + NotHeard, + Unavailable, + Busy, + /// A reason this firmware doesn't know. + Other, + /// The box couldn't start its microphone. + NoMicrophone, +} + +impl Reason { + fn from_name(name: &str) -> Self { + match name { + "not_heard" => Self::NotHeard, + "unavailable" => Self::Unavailable, + "busy" => Self::Busy, + _ => Self::Other, + } + } + + fn title(self) -> &'static [u8] { + match self { + Self::NotHeard => b"NOT HEARD", + Self::Unavailable => b"UNAVAILABLE", + Self::Busy => b"BUSY", + Self::Other => b"VOICE ERROR", + Self::NoMicrophone => b"NO MICROPHONE", + } + } + + fn text(self) -> &'static str { + match self { + Self::NotHeard => "Didn't catch that. Try again, a bit closer.", + Self::Unavailable => "Voice isn't working on the computer right now.", + Self::Busy => "Still on the last one.", + Self::Other => "Something went wrong.", + Self::NoMicrophone => "This box can't record.", + } + } +} + +/// The recording of one turn: the encoder, the line being filled and the level meter. +struct Recording { + turn: u32, + started: u32, + /// Samples encoded so far, including those in lines that were dropped on the way out. + samples: u32, + seq: u32, + encoder: Encoder, + /// The coder's state before the first nibble of the line being filled. + line_state: (i16, u8), + bytes: Vec, + /// A sample waiting for the next one: a byte holds two. + held: Option, + /// The meter window: sum and sum of squares (the mean is taken out, the ADC has a DC offset). + sum: i64, + squares: u64, + count: u32, + level: u8, +} + +impl Recording { + fn new(turn: u32, now: u32) -> Self { + Self { + turn, + started: now, + samples: 0, + seq: 0, + encoder: Encoder::default(), + line_state: (0, 0), + bytes: Vec::with_capacity(SAMPLES_PER_LINE / 2), + held: None, + sum: 0, + squares: 0, + count: 0, + level: 0, + } + } + + fn full(&self) -> bool { + self.samples >= MAX_SAMPLES + } + + fn push(&mut self, samples: &[i16], out: &mut Vec) { + for &sample in samples { + self.measure(sample); + if self.full() { + continue; + } + let Some(first) = self.held.take() else { + self.held = Some(sample); + continue; + }; + if self.bytes.is_empty() { + self.line_state = self.encoder.state(); + } + self.bytes.push(self.encoder.encode_pair(first, sample)); + self.samples += 2; + if self.bytes.len() * 2 == SAMPLES_PER_LINE { + out.push(self.data_line()); + } + } + } + + fn measure(&mut self, sample: i16) { + self.sum += sample as i64; + self.squares += (sample as i64 * sample as i64) as u64; + self.count += 1; + if self.count == LEVEL_WINDOW { + let mean = self.sum as f64 / self.count as f64; + let power = (self.squares as f64 / self.count as f64 - mean * mean).max(0.0); + self.level = meter_level(libm::sqrt(power) as f32); + (self.sum, self.squares, self.count) = (0, 0, 0); + } + } + + fn data_line(&mut self) -> Out { + let (predictor, index) = self.line_state; + let mut line = alloc::format!("VOICE DATA {} {} {} {} ", self.turn, self.seq, predictor, index); + line.push_str(&encode_base64(&self.bytes)); + line.push('\n'); + self.seq += 1; + self.bytes.clear(); + Out::Data(line) + } + + /// The last, shorter line and `VOICE END`. A sample left without a partner isn't sent. + fn finish(&mut self, out: &mut Vec) { + if !self.bytes.is_empty() { + out.push(self.data_line()); + } + out.push(Out::Line(alloc::format!("VOICE END {} {}\n", self.turn, self.samples))); + } +} + +/// An RMS in sample units to meter segments: -60 dBFS and below is none, -10 dBFS and above all. +pub fn meter_level(rms: f32) -> u8 { + if rms < 1.0 { + return 0; + } + let db = 20.0 * libm::log10f(rms / 32768.0); + let steps = (db - METER_FLOOR_DB) / (METER_TOP_DB - METER_FLOOR_DB) * METER_STEPS as f32; + (steps.max(0.0) as u8).min(METER_STEPS) +} + +enum Stage { + Idle, + Listening(Recording), + /// The recording went out; waiting for the computer. Quiet since the last thing it said. + Thinking { turn: u32, heard: Option, replies: Vec, quiet_since: u32 }, + /// `voice.done`: the replies stay up to be read. + Done { heard: Option, replies: Vec, until: u32 }, + Failed { reason: Reason, message: Option, until: u32 }, +} + +pub struct VoiceCommand { + available: bool, + /// The last turn's number; turns count from 1 after each boot. + turn: u32, + stage: Stage, +} + +impl Default for VoiceCommand { + fn default() -> Self { + Self::new() + } +} + +impl VoiceCommand { + pub const fn new() -> Self { + Self { available: false, turn: 0, stage: Stage::Idle } + } + + /// The computer offers voice: K0 long is push-to-talk. + pub fn available(&self) -> bool { + self.available + } + + /// `voice.available`, or a new link (which starts with voice off). Withdrawn mid-recording, the turn is + /// given up. + pub fn set_available(&mut self, on: bool) -> Option { + self.available = on; + if on { None } else { self.cancel() } + } + + /// The last turn's number. + pub fn turn(&self) -> u32 { + self.turn + } + + pub fn listening(&self) -> bool { + matches!(self.stage, Stage::Listening(_)) + } + + /// The overlay is up. + /// Waiting for the computer's first reply: the THINKING panel animates, so it is redrawn each pass. + pub fn waiting(&self) -> bool { + matches!(&self.stage, Stage::Thinking { replies, .. } if replies.is_empty()) + } + + pub fn is_open(&self) -> bool { + !matches!(self.stage, Stage::Idle) + } + + /// K0 held long enough, with the microphone running: a new turn. A turn still waiting for its replies is + /// left to the computer, which answers the new one `busy` if it must. + pub fn start(&mut self, now: u32) -> Out { + self.turn += 1; + self.stage = Stage::Listening(Recording::new(self.turn, now)); + Out::Line(alloc::format!("VOICE START {}\n", self.turn)) + } + + /// The microphone couldn't start: no turn, just the overlay saying so. + pub fn no_microphone(&mut self, now: u32) { + self.fail(Reason::NoMicrophone, None, now); + } + + /// The microphone's samples since the last call: `VOICE DATA` lines for each full line. Past 30 s of + /// audio they only move the meter; [`Self::due`] then ends the recording. + pub fn push(&mut self, samples: &[i16]) -> Vec { + let mut out = Vec::new(); + if let Stage::Listening(recording) = &mut self.stage { + recording.push(samples, &mut out); + } + out + } + + /// The recording has run its 30 s (or the samples stopped coming): end it as if K0 were let go. + pub fn due(&self, now: u32) -> bool { + match &self.stage { + Stage::Listening(recording) => recording.full() || now.wrapping_sub(recording.started) >= MAX_RECORDING_MS, + _ => false, + } + } + + /// K0 let go (or the 30 s are up): `VOICE END`, or `VOICE CANCEL` for a press under half a second, which + /// closes the overlay too. The caller has already handed over the last samples. + pub fn release(&mut self, now: u32) -> Vec { + let Stage::Listening(recording) = &mut self.stage else { return Vec::new() }; + let turn = recording.turn; + if now.wrapping_sub(recording.started) < MIN_RECORDING_MS { + self.stage = Stage::Idle; + return alloc::vec![Out::Line(alloc::format!("VOICE CANCEL {turn}\n"))]; + } + let mut out = Vec::new(); + recording.finish(&mut out); + self.stage = Stage::Thinking { turn, heard: None, replies: Vec::new(), quiet_since: now }; + out + } + + /// K2, or something that needs the user taking the screen: a recording is given up with `VOICE CANCEL`; + /// anything else just closes, and the rest of that turn's events are ignored. + pub fn cancel(&mut self) -> Option { + let stage = core::mem::replace(&mut self.stage, Stage::Idle); + match stage { + Stage::Listening(recording) => Some(Out::Line(alloc::format!("VOICE CANCEL {}\n", recording.turn))), + _ => None, + } + } + + /// The turn this event is about, if it is the one on the overlay. + fn current(&self, turn: u32) -> bool { + match &self.stage { + Stage::Listening(recording) => recording.turn == turn, + Stage::Thinking { turn: current, .. } => *current == turn, + _ => false, + } + } + + /// `voice.heard`, `voice.reply`, `voice.done` or `voice.error` from the computer. Events for any turn but + /// the one on the overlay are ignored. True when the overlay changed. + pub fn event(&mut self, event: &str, turn: u32, text: Option<&str>, reason: Option<&str>, now: u32) -> bool { + if !self.current(turn) { + return false; + } + match event { + "voice.error" => { + let message = text.filter(|text| !text.is_empty()).map(String::from); + self.fail(Reason::from_name(reason.unwrap_or("")), message, now); + true + } + // Only a recording that has ended can be heard or answered; the computer is ahead of itself. + _ if self.listening() => false, + "voice.done" => { + let Stage::Thinking { heard, replies, .. } = core::mem::replace(&mut self.stage, Stage::Idle) else { return false }; + // With nothing to read, there's nothing to wait for. + if heard.is_some() || !replies.is_empty() { + self.stage = Stage::Done { heard, replies, until: now.wrapping_add(READ_MS) }; + } + true + } + "voice.heard" | "voice.reply" => { + let Stage::Thinking { heard, replies, quiet_since, .. } = &mut self.stage else { return false }; + *quiet_since = now; + let Some(text) = text.filter(|text| !text.trim().is_empty()) else { return false }; + if event == "voice.heard" { + *heard = Some(String::from(text.trim())); + } else { + if replies.len() == MAX_REPLIES { + replies.remove(0); + } + replies.push(String::from(text.trim())); + } + true + } + _ => false, + } + } + + fn fail(&mut self, reason: Reason, message: Option, now: u32) { + self.stage = Stage::Failed { reason, message, until: now.wrapping_add(ERROR_MS) }; + } + + /// Closes the overlay once its time is up: an error or the replies read, or the computer quiet for a + /// minute. True if it just closed. + pub fn expire(&mut self, now: u32) -> bool { + let over = match &self.stage { + Stage::Thinking { quiet_since, .. } => now.wrapping_sub(*quiet_since) >= QUIET_CLOSE_MS, + Stage::Done { until, .. } | Stage::Failed { until, .. } => now.wrapping_sub(*until) as i32 >= 0, + Stage::Idle | Stage::Listening(_) => false, + }; + if over { + self.stage = Stage::Idle; + } + over + } + + /// How often THINKING gains a dot while it waits. +const THINKING_DOT_MS: u32 = 400; + +/// What the overlay shows now; None when it is closed. + pub fn panel(&self, now: u32) -> Option { + let close = alloc::vec![(&b"K2"[..], &b"CLOSE"[..])]; + match &self.stage { + Stage::Idle => None, + Stage::Listening(recording) => { + let elapsed_s = now.wrapping_sub(recording.started) / 1000; + let left = crate::text!(16, "{} S LEFT", 30u32.saturating_sub(elapsed_s)); + Some(Panel { + kind: PanelKind::Meter(recording.level), + title: b"LISTENING".to_vec(), + lines: alloc::vec![(left.as_bytes().to_vec(), Vec::new(), Tone::Dim)], + selected: None, + hints: alloc::vec![(b"K0", b"LET GO"), (b"K2", b"CANCEL")], + }) + } + Stage::Thinking { turn: _, heard, replies, quiet_since } => { + if !replies.is_empty() { + return Some(text_panel(b"REPLY", heard.as_deref(), replies, close)); + } + // Recognition and the resident session take seconds: the dots and the count keep the wait visibly + // alive. + let waited = now.wrapping_sub(*quiet_since); + let mut title = b"THINKING".to_vec(); + title.extend(core::iter::repeat_n(b'.', (waited / Self::THINKING_DOT_MS % 4) as usize)); + let mut panel = text_panel(&title, heard.as_deref(), replies, close); + panel.lines.push((crate::text!(8, "{} S", waited / 1000).as_bytes().to_vec(), Vec::new(), Tone::Dim)); + Some(panel) + } + Stage::Done { heard, replies, .. } => Some(text_panel(b"REPLY", heard.as_deref(), replies, close)), + Stage::Failed { reason, message, .. } => { + let text = [String::from(message.as_deref().unwrap_or(reason.text()))]; + Some(text_panel(reason.title(), None, &text, close)) + } + } + } +} + +/// The heard text dimmed, then the replies, wrapped to the panel: large type while it fits, small type +/// after that, and past what small type holds, the oldest rows give way. +fn text_panel(title: &[u8], heard: Option<&str>, replies: &[String], hints: Vec<(&'static [u8], &'static [u8])>) -> Panel { + let layout = |scale: u8| { + let mut lines: Vec<(Vec, Vec, Tone)> = Vec::new(); + if let Some(heard) = heard { + lines.extend(font::wrap(heard, VOICE_TEXT_WIDTH, scale as i32).into_iter().map(|row| (row, Vec::new(), Tone::Dim))); + } + for reply in replies { + lines.extend(font::wrap(reply, VOICE_TEXT_WIDTH, scale as i32).into_iter().map(|row| (row, Vec::new(), Tone::Plain))); + } + let rows = voice_text_rows(scale, lines.iter().any(|(row, _, _)| font::has_wide(row))); + let fits = lines.len() <= rows; + if !fits { + lines.drain(..lines.len() - rows); + } + (lines, fits) + }; + let (lines, scale) = match layout(2) { + (lines, true) => (lines, 2), + _ => (layout(1).0, 1), + }; + Panel { kind: PanelKind::Text(scale), title: title.to_vec(), lines, selected: None, hints } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::adpcm::Decoder; + use crate::voice_pack::decode_base64; + use std::string::ToString; + + fn lines(out: &[Out]) -> Vec { + out.iter() + .map(|out| match out { + Out::Line(line) | Out::Data(line) => line.clone(), + }) + .collect() + } + + fn tone(level: i16, count: usize) -> Vec { + (0..count).map(|index| if index % 2 == 0 { level } else { -level }).collect() + } + + #[test] + fn thinking_counts_dots_and_seconds_while_it_waits() { + let mut voice = VoiceCommand::new(); + voice.set_available(true); + voice.start(0); + voice.push(&tone(3000, SAMPLES_PER_LINE)); + voice.release(1000); + assert!(voice.waiting()); + let at = |now| voice.panel(now).unwrap(); + assert_eq!(at(1000).title, b"THINKING"); + assert_eq!(at(1000 + 2 * VoiceCommand::THINKING_DOT_MS).title, b"THINKING.."); + let panel = at(4500); + assert_eq!(panel.lines.last().unwrap().0, b"3 S"); + } + + #[test] + fn a_turn_streams_numbered_lines_and_ends_with_the_sample_count() { + let mut voice = VoiceCommand::new(); + voice.set_available(true); + assert_eq!(voice.start(1000), Out::Line("VOICE START 1\n".into())); + // 2.5 lines' worth arrives in uneven pieces. + let samples = tone(3000, SAMPLES_PER_LINE * 5 / 2); + let mut out = Vec::new(); + for piece in samples.chunks(700) { + out.extend(voice.push(piece)); + } + assert_eq!(out.len(), 2); + assert!(matches!(out[0], Out::Data(_))); + out.extend(voice.release(2000)); + let lines = lines(&out); + assert_eq!(lines.len(), 4); + for (seq, line) in lines[..3].iter().enumerate() { + assert!(line.starts_with(&std::format!("VOICE DATA 1 {seq} ")), "{line}"); + assert!(line.len() < 1024 && line.ends_with('\n')); + } + assert_eq!(lines[3], "VOICE END 1 2880\n"); + assert!(!voice.listening()); + assert!(voice.is_open(), "the overlay waits for the computer"); + } + + #[test] + fn a_line_holds_768_base64_characters_and_decodes_on_its_own() { + let mut voice = VoiceCommand::new(); + voice.start(0); + let samples: Vec = (0..SAMPLES_PER_LINE * 3).map(|index| (libm::sinf(index as f32 * 0.04) * 8000.0) as i16).collect(); + let out = lines(&voice.push(&samples)); + assert_eq!(out.len(), 3); + let mut decoded_lines = Vec::new(); + for line in &out { + let fields: Vec<&str> = line.trim_end().split(' ').collect(); + assert_eq!(fields.len(), 7, "VOICE DATA turn seq predictor index base64"); + assert_eq!(fields[6].len(), 768); + let (predictor, index): (i16, u8) = (fields[4].parse().unwrap(), fields[5].parse().unwrap()); + let mut bytes = [0u8; 576]; + assert_eq!(decode_base64(fields[6].as_bytes(), &mut bytes), Some(576)); + let mut decoder = Decoder::with_state(predictor, index); + decoded_lines.push(bytes.iter().flat_map(|&byte| [byte & 15, byte >> 4]).map(|nibble| decoder.decode(nibble)).collect::>()); + } + // Each line decoded from its own header continues the previous exactly: the third alone matches + // the third stretch of the stream decoded from the start. + let mut decoder = Decoder::default(); + let mut encoder = Encoder::default(); + let whole: Vec = samples + .chunks(2) + .flat_map(|pair| { + let byte = encoder.encode_pair(pair[0], pair[1]); + [decoder.decode(byte & 15), decoder.decode(byte >> 4)] + }) + .collect(); + assert_eq!(decoded_lines[2], whole[SAMPLES_PER_LINE * 2..]); + let longest = "VOICE DATA 4294967295 4294967295 -32768 88 ".len() + 768 + 1; + assert!(longest < 1024, "the longest possible line is {longest} bytes"); + } + + #[test] + fn under_half_a_second_is_cancelled_and_closes() { + let mut voice = VoiceCommand::new(); + voice.start(1000); + voice.push(&tone(100, 4000)); + assert_eq!(lines(&voice.release(1499)), ["VOICE CANCEL 1\n"]); + assert!(!voice.is_open()); + // Turns keep counting. + assert_eq!(voice.start(2000), Out::Line("VOICE START 2\n".into())); + assert_eq!(lines(&voice.release(2500))[0], "VOICE END 2 0\n"); + } + + #[test] + fn thirty_seconds_is_the_most_a_turn_records() { + let mut voice = VoiceCommand::new(); + voice.start(0); + let second = tone(500, SAMPLE_RATE as usize); + let mut data = 0; + for at in 1..=31 { + data += voice.push(&second).len(); + if voice.due(at * 1000) { + break; + } + } + assert!(voice.due(30_000)); + let out = lines(&voice.release(30_000)); + assert_eq!(out.last().unwrap(), &std::format!("VOICE END 1 {MAX_SAMPLES}\n")); + assert_eq!(data + out.len() - 1, (MAX_SAMPLES as usize).div_ceil(SAMPLES_PER_LINE)); + } + + #[test] + fn a_recording_whose_samples_stop_still_ends() { + let mut voice = VoiceCommand::new(); + voice.start(0); + assert!(!voice.due(30_999)); + assert!(voice.due(31_000)); + } + + #[test] + fn an_odd_sample_is_held_for_its_partner() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.push(&[1]); + voice.push(&[2, 3]); + assert_eq!(lines(&voice.release(1000)).last().unwrap(), "VOICE END 1 2\n", "the third sample has no partner"); + } + + #[test] + fn withdrawing_voice_or_k2_gives_up_a_recording() { + let mut voice = VoiceCommand::new(); + voice.set_available(true); + voice.start(0); + assert_eq!(voice.set_available(false), Some(Out::Line("VOICE CANCEL 1\n".into()))); + assert!(!voice.available() && !voice.is_open()); + voice.set_available(true); + voice.start(0); + assert_eq!(voice.cancel(), Some(Out::Line("VOICE CANCEL 2\n".into()))); + voice.start(0); + voice.release(1000); + assert_eq!(voice.cancel(), None, "after the recording, closing says nothing"); + assert!(!voice.event("voice.reply", 3, Some("late"), None, 2000), "a closed turn's events are ignored"); + } + + #[test] + fn heard_then_replies_then_done_then_read_time() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.release(1000); + assert_eq!(voice.panel(1000).unwrap().title, b"THINKING"); + assert!(!voice.event("voice.heard", 7, Some("other turn"), None, 1100)); + assert!(voice.event("voice.heard", 1, Some("what is codex doing"), None, 1200)); + assert!(voice.event("voice.reply", 1, Some("It's writing the tests."), None, 1300)); + assert!(voice.event("voice.reply", 1, Some("Almost done."), None, 1400)); + let panel = voice.panel(1400).unwrap(); + assert_eq!(panel.title, b"REPLY"); + assert_eq!(panel.kind, PanelKind::Text(2)); + let rows: Vec<(String, Tone)> = panel.lines.iter().map(|(row, _, tone)| (String::from_utf8(row.clone()).unwrap(), *tone)).collect(); + assert_eq!( + rows, + [ + ("what is codex doing".to_string(), Tone::Dim), + ("It's writing the".to_string(), Tone::Plain), + ("tests.".to_string(), Tone::Plain), + ("Almost done.".to_string(), Tone::Plain), + ] + ); + assert!(voice.event("voice.done", 1, None, None, 2000)); + assert!(!voice.expire(2000 + READ_MS - 1)); + assert_eq!(voice.panel(2000).unwrap().title, b"REPLY"); + assert!(voice.expire(2000 + READ_MS)); + assert!(voice.panel(2000 + READ_MS).is_none()); + } + + #[test] + fn done_with_nothing_to_read_closes_at_once() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.release(1000); + assert!(voice.event("voice.done", 1, None, None, 1500)); + assert!(!voice.is_open()); + } + + #[test] + fn a_quiet_computer_closes_the_overlay_after_a_minute() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.release(1000); + voice.event("voice.heard", 1, Some("hello"), None, 20_000); + assert!(!voice.expire(20_000 + QUIET_CLOSE_MS - 1), "each event restarts the wait"); + assert!(voice.expire(20_000 + QUIET_CLOSE_MS)); + } + + #[test] + fn errors_show_a_short_message_and_go() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.release(1000); + assert!(voice.event("voice.error", 1, None, Some("not_heard"), 3000)); + let panel = voice.panel(3000).unwrap(); + assert_eq!(panel.title, b"NOT HEARD"); + assert_eq!(panel.lines[0].0, b"Didn't catch that."); + assert!(voice.expire(3000 + ERROR_MS)); + + // The computer's own message wins; and `busy` can come while still recording, which ends it. + voice.start(10_000); + assert!(voice.event("voice.error", 2, Some("Another turn is running"), Some("busy"), 10_100)); + assert!(!voice.listening()); + let panel = voice.panel(10_100).unwrap(); + assert_eq!((panel.title.as_slice(), panel.lines[0].0.as_slice()), (&b"BUSY"[..], &b"Another turn is"[..])); + assert!(!voice.event("voice.reply", 2, Some("x"), None, 10_200), "the turn is over"); + } + + #[test] + fn the_overlay_says_listening_with_a_meter_and_time_left() { + let mut voice = VoiceCommand::new(); + voice.start(1000); + assert_eq!(voice.panel(1000).unwrap().kind, PanelKind::Meter(0)); + // 100 ms of a loud tone on a DC offset: the offset doesn't count. + let loud: Vec = tone(8000, LEVEL_WINDOW as usize).iter().map(|sample| sample + 2000).collect(); + voice.push(&loud); + let panel = voice.panel(4500).unwrap(); + assert_eq!(panel.title, b"LISTENING"); + assert_eq!(panel.kind, PanelKind::Meter(meter_level(8000.0))); + assert_eq!(panel.lines[0].0, b"27 S LEFT"); + assert_eq!(panel.hints, [(&b"K0"[..], &b"LET GO"[..]), (&b"K2"[..], &b"CANCEL"[..])]); + } + + #[test] + fn the_meter_spans_silence_to_a_voice_up_close() { + assert_eq!(meter_level(0.0), 0); + assert_eq!(meter_level(30.0), 0, "-61 dBFS"); + // A close voice at 42 dB, about -19 dBFS, is high on the meter but not pinned. + let close = meter_level(32768.0 * libm::powf(10.0, -19.0 / 20.0)); + assert!((12..METER_STEPS).contains(&close), "{close}"); + assert_eq!(meter_level(32767.0), METER_STEPS); + } + + #[test] + fn long_text_moves_to_small_type_and_keeps_the_newest() { + let mut voice = VoiceCommand::new(); + voice.start(0); + voice.release(1000); + voice.event("voice.heard", 1, Some("tell me everything"), None, 1000); + let long = "word ".repeat(40); + voice.event("voice.reply", 1, Some(&long), None, 1000); + let panel = voice.panel(1000).unwrap(); + assert_eq!(panel.kind, PanelKind::Text(1)); + for _ in 0..5 { + voice.event("voice.reply", 1, Some(&long), None, 1000); + } + voice.event("voice.reply", 1, Some("the end"), None, 1000); + let panel = voice.panel(1000).unwrap(); + assert_eq!(panel.lines.len(), voice_text_rows(1, false)); + assert_eq!(panel.lines.last().unwrap().0, b"the end"); + } +} diff --git a/firmware-rs/core/tests/firmware.rs b/firmware-rs/core/tests/firmware.rs index 8ec76d1e..31906e2f 100644 --- a/firmware-rs/core/tests/firmware.rs +++ b/firmware-rs/core/tests/firmware.rs @@ -10,6 +10,7 @@ use vibebuddy_firmware_core::canvas::FRAME_BYTES; use vibebuddy_firmware_core::display::Screen; use vibebuddy_firmware_core::firmware::{AudioStatus, Board, Firmware, FrameAction, VolumeError}; use vibebuddy_firmware_core::noise::Sealer; +use vibebuddy_firmware_core::spoken_reply::{ReplyAudio, ReplyPlayer}; use vibebuddy_firmware_core::storage::{Flash, FlashError}; use vibebuddy_firmware_core::voice_pack; @@ -89,6 +90,19 @@ struct FakeBoard { peer_full: bool, /// Restarts asked for. restarts: usize, + /// Registers of the chips on I2C, by (address, register), and what was written. + i2c: std::collections::BTreeMap<(u8, u8), u8>, + i2c_writes: Vec<(u8, u8, u8)>, + /// The board has I2S input; and the size of the capture asked for, until it is taken. + mic_input: bool, + mic_capture: Option, + /// The voice command microphone is running; samples waiting to be taken; `VOICE DATA` lines written. + voice_running: bool, + voice_samples: Vec, + stream: Vec, + /// What the spoken reply player was handed, and whether it says a reply is sounding. + reply_audio: Vec, + speaking: bool, } impl FakeBoard { @@ -114,6 +128,15 @@ impl FakeBoard { relay_peer: false, peer_full: false, restarts: 0, + i2c: [((0x18, 0xFD), 0x83), ((0x18, 0xFE), 0x11), ((0x18, 0x16), 0x24)].into(), + i2c_writes: Vec::new(), + mic_input: true, + mic_capture: None, + voice_running: false, + voice_samples: Vec::new(), + stream: Vec::new(), + reply_audio: Vec::new(), + speaking: false, } } @@ -207,6 +230,46 @@ impl Board for FakeBoard { self.codec_volume = Some(level); Ok(()) } + fn i2c_read(&mut self, address: u8, register: u8) -> Option { + self.i2c.get(&(address, register)).copied() + } + fn i2c_write(&mut self, address: u8, register: u8, value: u8) -> Result<(), ()> { + self.i2c.keys().any(|&(chip, _)| chip == address).then_some(()).ok_or(())?; + self.i2c.insert((address, register), value); + self.i2c_writes.push((address, register, value)); + Ok(()) + } + fn mic_start(&mut self, bytes: usize) -> Result<(), ()> { + self.mic_input.then_some(()).ok_or(())?; + self.mic_capture = Some(bytes); + Ok(()) + } + fn voice_start(&mut self) -> Result<(), ()> { + self.mic_input.then_some(()).ok_or(())?; + self.voice_running = true; + Ok(()) + } + fn voice_take(&mut self, samples: &mut Vec) { + samples.append(&mut self.voice_samples); + } + fn voice_stop(&mut self) { + self.voice_running = false; + self.voice_samples.clear(); + } + fn write_stream(&mut self, line: &[u8]) { + self.stream.push(String::from_utf8(line.to_vec()).unwrap()); + } + fn reply_audio(&mut self, audio: ReplyAudio) { + self.reply_audio.push(audio); + } + fn reply_speaking(&self) -> bool { + self.speaking + } + /// The left slot counts up, the right slot is silent. + fn mic_take(&mut self) -> Option, &'static str>> { + let bytes = self.mic_capture.take()?; + Some(Ok((0..bytes / 4).flat_map(|frame| [(frame % 256) as u8, 0, 0, 0]).collect())) + } } /// The key the box makes from the entropy `Firmware::new(7, …)` derives. @@ -1241,3 +1304,446 @@ mod relay { assert!(!relay.has_peer()); } } + +/// Polls until `MIC END` and returns every line. +fn poll_mic(firmware: &mut Firmware, board: &mut FakeBoard) -> Vec { + let mut lines = Vec::new(); + for _ in 0..2_000 { + board.advance(10); + firmware.poll(board); + lines.extend(board.take_lines()); + if lines.last().is_some_and(|line| line.starts_with("MIC END")) { + return lines; + } + } + panic!("no MIC END; got {} lines", lines.len()); +} + +#[test] +fn a_mic_recording_reports_probes_levels_and_the_samples() { + let (mut firmware, mut board, _) = booted(blank_flash()); + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record","ms":200}"#); + assert_eq!( + lines, + [ + "MIC PROBE 0x18 ID 8311", + "MIC PROBE 0x40 NONE", + "MIC PROBE 0x41 NONE", + "MIC PROBE 0x42 NONE", + "MIC PROBE 0x43 NONE", + "MIC GAIN 30 DB", + "MIC RECORDING 200 MS", + ] + ); + // es8311_set_mic_gain(30 dB) writes 5 to register 0x16. + assert_eq!(board.i2c_writes, [(0x18, 0x16, 5)]); + assert_eq!(send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record"}"#), ["MIC BUSY"]); + + let lines = poll_mic(&mut firmware, &mut board); + assert_eq!(lines[0], "MIC CAPTURED 10 MS 19200 BYTES"); + assert_eq!(&lines[1..3], ["MIC LEVEL 0 L 144 255 R 0 0", "MIC LEVEL 100 L 147 255 R 0 0"]); + assert_eq!(lines[3], "MIC BEGIN 24000 2 16 19200"); + let data: Vec<&String> = lines.iter().filter(|line| line.starts_with("MIC DATA ")).collect(); + assert_eq!(data.len(), 19200usize.div_ceil(576)); + let mut samples = Vec::new(); + for (seq, line) in data.iter().enumerate() { + assert!(line.len() < 800, "a data line stays well under the 1 KB line limit"); + let mut parts = line.splitn(4, ' ').skip(2); + assert_eq!(parts.next().unwrap(), seq.to_string()); + let mut out = vec![0u8; 576]; + let length = voice_pack::decode_base64(parts.next().unwrap().as_bytes(), &mut out).unwrap(); + samples.extend_from_slice(&out[..length]); + } + let expected: Vec = (0..4800).flat_map(|frame| [(frame % 256) as u8, 0, 0, 0]).collect(); + assert_eq!(samples, expected); + assert_eq!(lines.last().unwrap(), &format!("MIC END 19200 {}", voice_pack::crc32(0, &expected))); + + // Done: another recording may start. + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record","ms":100}"#); + assert_eq!(lines.last().map(String::as_str), Some("MIC RECORDING 100 MS")); +} + +#[test] +fn a_mic_recording_is_clamped_and_reports_what_is_missing() { + let (mut firmware, mut board, _) = booted(blank_flash()); + board.i2c.clear(); + board.i2c.insert((0x41, 0x3D), 0x72); + board.i2c.insert((0x41, 0x3E), 0x10); + board.mic_input = false; + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record","ms":60000,"gain_db":99}"#); + assert_eq!( + lines, + [ + "MIC PROBE 0x18 NONE", + "MIC PROBE 0x40 NONE", + "MIC PROBE 0x41 ID 7210", + "MIC PROBE 0x42 NONE", + "MIC PROBE 0x43 NONE", + "MIC UNAVAILABLE", + ] + ); + // No ES8311: no gain write. + assert!(board.i2c_writes.is_empty()); + + board.mic_input = true; + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record","ms":60000}"#); + assert_eq!(lines.last().map(String::as_str), Some("MIC RECORDING 5000 MS")); + assert_eq!(board.mic_capture, Some(480_000)); +} + +const VOICE_ON: &str = r#"{"version":1,"event":"voice.available","on":true}"#; + +/// Holds K0 for `hold_ms`, with the microphone delivering 24 samples a millisecond while it runs; returns +/// the lines printed (the `VOICE DATA` lines are in `board.stream`). +fn hold_k0_talking(firmware: &mut Firmware, board: &mut FakeBoard, hold_ms: u32) -> Vec { + board.advance(100); + board.k0 = true; + firmware.poll(board); + let mut held = 0; + while held < hold_ms { + board.advance(10); + held += 10; + if board.voice_running { + board.voice_samples.extend((0..240).map(|index| if index % 2 == 0 { 4000 } else { -4000 })); + } + firmware.poll(board); + } + board.k0 = false; + firmware.poll(board); + board.take_lines() +} + +#[test] +fn without_voice_a_long_k0_still_gives_up_the_phase() { + let (mut firmware, mut board, _) = booted(blank_flash()); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 50), ["POMODORO FOCUS START", "MODE POMODORO"]); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 1100), ["POMODORO STOPPED"]); + assert!(!board.voice_running); + // Offered and then withdrawn, it's the same. + send(&mut firmware, &mut board, VOICE_ON); + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.available","on":false}"#); + press(&mut firmware, &mut board, Key::K0, 50); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 1100), ["POMODORO STOPPED"]); +} + +#[test] +fn with_voice_a_long_k0_records_a_turn_and_the_phase_keeps_running() { + let (mut firmware, mut board, _) = booted(blank_flash()); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 50), ["POMODORO FOCUS START", "MODE POMODORO"]); + assert!(send(&mut firmware, &mut board, VOICE_ON).is_empty()); + let lines = hold_k0_talking(&mut firmware, &mut board, 2000); + // Recording starts when the long press fires, a second in, and runs until K0 comes up. + assert_eq!(lines.first().map(String::as_str), Some("VOICE START 1")); + assert!(!lines.iter().any(|line| line.starts_with("POMODORO")), "{lines:?}"); + let end = lines.iter().find(|line| line.starts_with("VOICE END 1 ")).expect("VOICE END"); + let samples: usize = end.rsplit(' ').next().unwrap().parse().unwrap(); + assert!((22_000..=24_500).contains(&samples), "about a second at 24 kHz: {samples}"); + assert!(!board.voice_running, "the microphone stops with the recording"); + // Every data line went to the stream, numbered without gaps, the last one shorter. + assert_eq!(board.stream.len(), samples.div_ceil(1152)); + for (seq, line) in board.stream.iter().enumerate() { + assert!(line.starts_with(&format!("VOICE DATA 1 {seq} ")) && line.ends_with('\n') && line.len() < 1024, "{line}"); + } + // The microphone ran at voice gain: 42 dB is register 0x16 = 7. + assert!(board.i2c_writes.contains(&(0x18, 0x16, 7))); + // The overlay counts as the user being there. + assert_eq!(press(&mut firmware, &mut board, Key::K2, 50), Vec::::new(), "K2 closes the overlay"); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 50), ["POMODORO PAUSED"], "the phase was never given up"); +} + +#[test] +fn a_tap_past_the_long_press_is_cancelled() { + let (mut firmware, mut board, _) = booted(blank_flash()); + send(&mut firmware, &mut board, VOICE_ON); + let lines = hold_k0_talking(&mut firmware, &mut board, 1300); + assert_eq!(lines, ["VOICE START 1", "VOICE CANCEL 1"]); + assert!(!board.voice_running); +} + +#[test] +fn the_computers_answers_fill_the_overlay_and_k2_closes_it() { + let (mut firmware, mut board, _) = booted(blank_flash()); + send(&mut firmware, &mut board, VOICE_ON); + hold_k0_talking(&mut firmware, &mut board, 1800); + let before = board.presents; + assert!(send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.heard","turn":1,"text":"what is codex doing"}"#).is_empty()); + assert!(send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.reply","turn":1,"text":"Writing tests."}"#).is_empty()); + assert!(board.presents >= before + 2, "each answer redraws the overlay"); + // Another turn's answers change nothing. + let before = board.presents; + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.reply","turn":9,"text":"nope"}"#); + assert_eq!(board.presents, before); + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.done","turn":1}"#); + // K2 closes it rather than asking the computer to open a source; the next K2 does that again. + assert!(press(&mut firmware, &mut board, Key::K2, 50).is_empty()); + assert_eq!(press(&mut firmware, &mut board, Key::K2, 50), [r#"{"version":1,"event":"button","button":"K2","action":"press"}"#]); +} + +#[test] +fn a_needs_input_event_waits_for_a_voice_command_and_is_said_after_it() { + let (mut firmware, mut board, _) = booted(blank_flash()); + send(&mut firmware, &mut board, VOICE_ON); + board.advance(100); + board.k0 = true; + firmware.poll(&mut board); + board.advance(1100); + firmware.poll(&mut board); + assert_eq!(board.take_lines(), ["VOICE START 1"]); + board.played.clear(); + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"agent.input_required","title":"CC:ASK"}"#); + assert!(!lines.iter().any(|line| line.starts_with("VOICE CANCEL")), "the recording goes on: {lines:?}"); + assert!(lines.contains(&"AUDIO HELD INPUT_REQUIRED".to_owned()), "{lines:?}"); + assert!(board.voice_running); + assert!(board.played.is_empty(), "nothing is said over the conversation"); + // The same Agent asking again is still one thing held. + send(&mut firmware, &mut board, r#"{"version":1,"event":"agent.input_required","title":"CC:ASK"}"#); + board.advance(500); + board.k0 = false; + firmware.poll(&mut board); + assert!(board.take_lines().iter().any(|line| line.starts_with("VOICE END 1"))); + assert!(board.played.is_empty()); + // K2 closes the overlay, and the held news is said once. + let lines = press(&mut firmware, &mut board, Key::K2, 50); + assert!(lines.contains(&"AUDIO QUEUED INPUT_REQUIRED".to_owned()), "{lines:?}"); + assert_eq!(board.played.len(), 1, "{:?}", board.played); +} + +#[test] +fn a_new_link_starts_with_voice_off() { + let (mut firmware, mut board, _) = booted(blank_flash()); + send(&mut firmware, &mut board, VOICE_ON); + send(&mut firmware, &mut board, r#"{"version":1,"event":"device.hello"}"#); + press(&mut firmware, &mut board, Key::K0, 50); + assert_eq!(press(&mut firmware, &mut board, Key::K0, 1100), ["POMODORO STOPPED"]); + // A link that goes quiet takes voice with it too. + send(&mut firmware, &mut board, VOICE_ON); + board.advance(16_000); + firmware.poll(&mut board); + board.take_lines(); + assert!(hold_k0_talking(&mut firmware, &mut board, 1500).iter().all(|line| !line.starts_with("VOICE"))); +} + +#[test] +fn a_box_that_cannot_record_says_so_and_sends_nothing() { + let (mut firmware, mut board, _) = booted(blank_flash()); + board.mic_input = false; + send(&mut firmware, &mut board, VOICE_ON); + let before = board.presents; + let lines = hold_k0_talking(&mut firmware, &mut board, 1500); + assert!(lines.iter().all(|line| !line.starts_with("VOICE") && !line.starts_with("POMODORO")), "{lines:?}"); + assert!(board.presents > before, "the overlay says there's no microphone"); +} + +#[test] +fn the_diagnostic_and_a_voice_command_share_the_microphone() { + let (mut firmware, mut board, _) = booted(blank_flash()); + send(&mut firmware, &mut board, VOICE_ON); + board.advance(100); + board.k0 = true; + firmware.poll(&mut board); + board.advance(1100); + firmware.poll(&mut board); + board.take_lines(); + let lines = send(&mut firmware, &mut board, r#"{"version":1,"event":"device.mic.record"}"#); + assert_eq!(lines, ["MIC BUSY"]); +} + +const REPLY_MP3: &[u8] = include_bytes!("../src/fixtures/spoken_reply.mp3"); +const K2_PRESS: &str = r#"{"version":1,"event":"button","button":"K2","action":"press"}"#; + +/// A turn recorded and answered with a reply's text: the box is waiting on turn 1's audio. +fn answered_turn(firmware: &mut Firmware, board: &mut FakeBoard) { + send(firmware, board, VOICE_ON); + hold_k0_talking(firmware, board, 1800); + send(firmware, board, r#"{"version":1,"event":"voice.reply","turn":1,"text":"Writing tests."}"#); + board.reply_audio.clear(); +} + +/// A reply's MP3 as the computer sends it: 576 bytes (768 base64 characters) a line, then the end. +fn audio_lines(turn: u32, reply: u32, mp3: &[u8], end: &str) -> Vec { + let mut lines: Vec = mp3 + .chunks(576) + .enumerate() + .map(|(seq, piece)| format!(r#"{{"version":1,"event":"voice.audio","turn":{turn},"reply":{reply},"seq":{seq},"data":"{}"}}"#, encode_base64(piece))) + .collect(); + lines.push(format!(r#"{{"version":1,"event":"voice.audio","turn":{turn},"reply":{reply},"seq":{},"end":true{end}}}"#, mp3.len().div_ceil(576))); + lines +} + +/// What a player makes of what the board was handed. +fn pcm_of(audio: &[ReplyAudio]) -> Vec { + let mut player = ReplyPlayer::new(); + for audio in audio { + player.apply(audio.clone()); + } + let mut out = Vec::new(); + let mut buffer = [0u8; 1000]; + loop { + let count = player.fill(&mut buffer); + if count == 0 { + return out; + } + out.extend_from_slice(&buffer[..count]); + } +} + +#[test] +fn a_spoken_replys_audio_reaches_the_player_whole() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + for line in audio_lines(1, 0, REPLY_MP3, "") { + assert!(line.len() < 1024); + assert!(send(&mut firmware, &mut board, &line).is_empty(), "{line}"); + } + let chunks = REPLY_MP3.len().div_ceil(576); + assert_eq!(board.reply_audio.len(), chunks + 1); + assert_eq!(board.reply_audio.last(), Some(&ReplyAudio::End { reply: 0, seq: chunks as u32 })); + let received: Vec = board + .reply_audio + .iter() + .flat_map(|audio| match audio { + ReplyAudio::Chunk { data, .. } => data.clone(), + _ => Vec::new(), + }) + .collect(); + assert!(received == REPLY_MP3, "the bytes arrive intact"); + assert!(pcm_of(&board.reply_audio).len() > 24_000 * 4 * 3, "seconds of speech"); + // Another turn's audio, or audio without a turn, goes nowhere. + board.reply_audio.clear(); + send(&mut firmware, &mut board, &audio_lines(7, 0, &REPLY_MP3[..576], "")[0]); + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.audio","reply":0,"seq":0,"data":"AAAA"}"#); + assert!(board.reply_audio.is_empty()); +} + +#[test] +fn a_reply_cut_short_by_synthesis_plays_what_came_and_says_so() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + let mut printed = Vec::new(); + for line in audio_lines(1, 0, &REPLY_MP3[..576 * 5], r#","error":"synthesis""#) { + printed.extend(send(&mut firmware, &mut board, &line)); + } + assert_eq!(printed, ["VOICE AUDIO CUT SHORT 1 0"]); + assert_eq!(board.reply_audio.last(), Some(&ReplyAudio::End { reply: 0, seq: 5 })); + assert!(!pcm_of(&board.reply_audio).is_empty(), "what came still plays"); + // The text stays up to be read. + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.done","turn":1}"#); + assert!(press(&mut firmware, &mut board, Key::K2, 50).is_empty(), "K2 closes the overlay"); +} + +#[test] +fn while_muted_replies_are_shown_and_not_played() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + let lines = audio_lines(1, 0, &REPLY_MP3[..576 * 4], ""); + send(&mut firmware, &mut board, &lines[0]); + assert_eq!(board.reply_audio.len(), 1); + // Muting mid-reply (K2 long): the reply stops at once, and the rest of the turn stays silent. + board.reply_audio.clear(); + assert_eq!(press(&mut firmware, &mut board, Key::K2, 1100), ["MUTE ON"]); + assert_eq!(board.reply_audio, [ReplyAudio::Stop]); + for line in &lines[1..] { + send(&mut firmware, &mut board, line); + } + assert_eq!(board.reply_audio, [ReplyAudio::Stop], "nothing more reaches the player"); + + // Muted before the turn: nothing reaches the player at all, and the text still shows. + let (mut firmware, mut board, _) = booted(blank_flash()); + assert_eq!(press(&mut firmware, &mut board, Key::K2, 1100), ["MUTE ON"]); + answered_turn(&mut firmware, &mut board); + for line in audio_lines(1, 0, &REPLY_MP3[..576 * 4], r#","error":"synthesis""#) { + send(&mut firmware, &mut board, &line); + } + assert!(board.reply_audio.is_empty()); + assert!(press(&mut firmware, &mut board, Key::K2, 50).is_empty(), "the overlay is up"); +} + +#[test] +fn an_agents_news_during_a_reply_waits_and_the_reply_plays_on() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + let lines = audio_lines(1, 0, &REPLY_MP3[..576 * 6], ""); + for line in &lines[..3] { + send(&mut firmware, &mut board, line); + } + board.reply_audio.clear(); + board.played.clear(); + let out = send( + &mut firmware, + &mut board, + r#"{"version":1,"event":"task.done","title":"CC:VIBE","tasks":[{"title":"CC:VIBE","status":"done","elapsed_s":3}]}"#, + ); + assert!(out.contains(&"AUDIO HELD DONE".to_owned()), "{out:?}"); + assert!(!board.reply_audio.contains(&ReplyAudio::Stop), "the reply isn't cut off"); + assert!(board.played.is_empty(), "nothing is said over the reply"); + for line in &lines[3..] { + send(&mut firmware, &mut board, line); + } + assert!(board.reply_audio.iter().any(|audio| matches!(audio, ReplyAudio::Chunk { .. })), "the reply's audio still plays"); + // Once the overlay closes, the news is said. + press(&mut firmware, &mut board, Key::K2, 50); + assert_eq!(board.played, [Sound::Builtin(Prompt::Done)]); +} + +#[test] +fn the_pomodoro_chime_stops_a_reply_too() { + let (mut firmware, mut board, _) = booted(blank_flash()); + press(&mut firmware, &mut board, Key::K0, 50); + answered_turn(&mut firmware, &mut board); + send(&mut firmware, &mut board, &audio_lines(1, 0, &REPLY_MP3[..576], "")[0]); + board.reply_audio.clear(); + board.advance(25 * 60 * 1000); + firmware.poll(&mut board); + assert!(board.take_lines().contains(&"POMODORO FOCUS END".to_owned())); + assert_eq!(board.reply_audio, [ReplyAudio::Stop]); +} + +#[test] +fn a_new_push_to_talk_stops_the_reply_and_plays_the_new_turns() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + send(&mut firmware, &mut board, &audio_lines(1, 0, &REPLY_MP3[..576], "")[0]); + board.reply_audio.clear(); + board.advance(100); + board.k0 = true; + firmware.poll(&mut board); + board.advance(1100); + firmware.poll(&mut board); + assert!(board.voice_running); + assert_eq!(board.reply_audio, [ReplyAudio::Stop], "the reply stops before the box listens"); + board.advance(1000); + board.k0 = false; + firmware.poll(&mut board); + board.take_lines(); + send(&mut firmware, &mut board, &audio_lines(1, 1, &REPLY_MP3[..576], "")[0]); + assert_eq!(board.reply_audio, [ReplyAudio::Stop], "the old turn's audio is ignored"); + send(&mut firmware, &mut board, &audio_lines(2, 0, &REPLY_MP3[..576], "")[0]); + assert_eq!(board.reply_audio.len(), 2, "the new turn's plays"); +} + +#[test] +fn the_text_stays_up_while_a_reply_is_spoken_and_k2_stops_it() { + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + send(&mut firmware, &mut board, &audio_lines(1, 0, &REPLY_MP3[..576], "")[0]); + send(&mut firmware, &mut board, r#"{"version":1,"event":"voice.done","turn":1}"#); + board.speaking = true; + // The link stays up past the 15 s read time. + board.advance(10_000); + send(&mut firmware, &mut board, r#"{"version":1,"event":"device.heartbeat"}"#); + board.advance(6_000); + firmware.poll(&mut board); + assert_eq!(board.reply_audio.last(), Some(&ReplyAudio::Chunk { reply: 0, seq: 0, data: REPLY_MP3[..576].to_vec() }), "past the read time, still speaking"); + board.speaking = false; + board.advance(10); + firmware.poll(&mut board); + assert_eq!(board.reply_audio.last(), Some(&ReplyAudio::Stop), "the overlay closes, and what was left to say goes with it"); + assert_eq!(press(&mut firmware, &mut board, Key::K2, 50), [K2_PRESS]); + + let (mut firmware, mut board, _) = booted(blank_flash()); + answered_turn(&mut firmware, &mut board); + send(&mut firmware, &mut board, &audio_lines(1, 0, &REPLY_MP3[..576], "")[0]); + board.reply_audio.clear(); + assert!(press(&mut firmware, &mut board, Key::K2, 50).is_empty()); + assert_eq!(board.reply_audio, [ReplyAudio::Stop]); +} diff --git a/firmware-rs/device/Cargo.lock b/firmware-rs/device/Cargo.lock index bdff8552..e58458ae 100644 --- a/firmware-rs/device/Cargo.lock +++ b/firmware-rs/device/Cargo.lock @@ -2256,6 +2256,10 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" +[[package]] +name = "nanomp3-core" +version = "0.2.0" + [[package]] name = "nb" version = "0.1.3" @@ -3319,6 +3323,7 @@ dependencies = [ "clatter", "ed25519-dalek", "libm", + "nanomp3-core", "rand_core 0.6.4", "serde_json", "vibebuddy-protocol", diff --git a/firmware-rs/device/Cargo.toml b/firmware-rs/device/Cargo.toml index ea9aef74..a91be2a1 100644 --- a/firmware-rs/device/Cargo.toml +++ b/firmware-rs/device/Cargo.toml @@ -72,3 +72,7 @@ codegen-units = 1 debug = 2 lto = "fat" opt-level = "s" + +# The MP3 decoder for spoken replies, patched for the Xtensa backend (see its Cargo.toml). +[patch.crates-io] +nanomp3-core = { path = "../vendor/nanomp3-core" } diff --git a/firmware-rs/device/src/audio.rs b/firmware-rs/device/src/audio.rs index d7384a80..b89f25e2 100644 --- a/firmware-rs/device/src/audio.rs +++ b/firmware-rs/device/src/audio.rs @@ -12,12 +12,14 @@ use core::sync::atomic::{AtomicBool, Ordering}; use embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex; use embassy_sync::channel::Channel; +use embassy_sync::mutex::Mutex; use embassy_time::Timer; use esp_hal::Async; use esp_hal::dma::DmaTxStreamBuf; use esp_hal::i2s::master::I2sTx; use vibebuddy_firmware_core::adpcm::LineStream; use vibebuddy_firmware_core::audio::{Chime, Codec, Prompt, Sound}; +use vibebuddy_firmware_core::spoken_reply::ReplyPlayer; use crate::storage; @@ -35,6 +37,11 @@ pub static QUEUE: Channel = Channel::ne pub static PLAYING: AtomicBool = AtomicBool::new(false); /// The main loop asks for an immediate stop: drop what is playing; the requester clears the queue. pub static STOP: AtomicBool = AtomicBool::new(false); +/// Spoken replies (core's spoken_reply.rs): the main loop hands in their MP3, and this task decodes and plays +/// them whenever no line is queued or playing, so an announcement always goes first. Both sides run on the +/// same executor and neither holds the lock across an await, so `try_lock` always gets it. The player holds +/// only empty containers here; its buffers and decoder live on the heap, and only while there is a reply. +pub static REPLIES: Mutex = Mutex::new(ReplyPlayer::new()); static INPUT_REQUIRED: &[u8] = include_bytes!("../../../firmware/main/assets/input_required.pcm"); static DONE: &[u8] = include_bytes!("../../../firmware/main/assets/done.pcm"); @@ -206,6 +213,9 @@ pub async fn audio_task(tx: I2sTx<'static, Async>, buffer: DmaTxStreamBuf) { if cursor.remaining() == 0 { current = None; } + } else if let Ok(mut replies) = REPLIES.try_lock() { + // A reply's frames are decoded here, on this task's poll, about 25 ms of audio at a time. + filled = replies.fill(&mut chunk); } chunk[filled..].fill(0); transfer.push(&chunk); diff --git a/firmware-rs/device/src/main.rs b/firmware-rs/device/src/main.rs index 949f6226..dacd3c7c 100644 --- a/firmware-rs/device/src/main.rs +++ b/firmware-rs/device/src/main.rs @@ -7,7 +7,9 @@ //! codec (0x18, absent on the NS4168 variant). //! - XL9555 P0: bit7 LCD backlight, bit5 amplifier enable, bit4 K1, bit3 K2 (keys active low). //! - K0: BOOT key, GPIO0, active low. -//! - I2S0: BCLK GPIO21, WS GPIO13, DOUT GPIO14, no MCLK. +//! - I2S0: BCLK GPIO21, WS GPIO13, DOUT GPIO14, no MCLK. DIN GPIO47 (the ES8311's ADC, left slot; +//! verified on one box, docs/hardware.md; read while a voice command records, and by the +//! `device.mic.record` diagnostic). //! - UART0: TX GPIO43, RX GPIO44, to the CH343 bridge; the chip's own USB Serial/JTAG as well. #![no_std] #![no_main] @@ -17,6 +19,7 @@ extern crate alloc; mod audio; mod lcd; +mod mic; mod net; mod relay; mod serial; @@ -49,6 +52,7 @@ use vibebuddy_firmware_core::display::Screen; use vibebuddy_firmware_core::firmware::{AudioStatus, Board, Firmware, FrameAction, VolumeError}; use vibebuddy_firmware_core::noise::{self, SealedLines, Sealer}; use vibebuddy_firmware_core::relay_box::{self, RelayBox}; +use vibebuddy_firmware_core::spoken_reply::ReplyAudio; use vibebuddy_firmware_core::storage::Flash; use crate::audio::{PLAYING, PlayCommand, QUEUE, STOP, STREAM_BYTES, STREAM_CHUNK}; @@ -63,9 +67,18 @@ esp_bootloader_esp_idf::esp_app_desc!(); /// the box freezes until power is cut. #[unsafe(no_mangle)] fn custom_halt() -> ! { + // SAFETY: one word written just before the reset; nothing else runs. + unsafe { PANICKED = PANIC_MARK }; esp_hal::system::software_reset() } +/// Set by a panic just before its reset, so the next boot can tell it from a restart asked for: both are +/// software resets to the chip. RTC fast memory keeps it across the reset; after power-on it holds noise, +/// hence a mark rather than a flag. +#[esp_hal::ram(unstable(rtc_fast, persistent))] +static mut PANICKED: u32 = 0; +const PANIC_MARK: u32 = 0x5041_4e43; + const XL9555_ADDRESS: u8 = 0x20; const XL9555_INPUT_PORT0: u8 = 0x00; const XL9555_OUTPUT_PORT0: u8 = 0x02; @@ -149,6 +162,22 @@ struct DeviceBoard { } impl DeviceBoard { + /// To the computer linked over the network: a line with no room is dropped whole, before it is sealed + /// (noise::SealedLines). + fn write_lan(&mut self, bytes: &[u8]) { + if let Some((conn, lines)) = &mut self.lan_peer { + let conn = *conn; + lines.lines( + bytes, + |length| net::room(conn) > length, + |line| { + net::send(conn, line.as_bytes()); + net::send(conn, b"\n"); + }, + ); + } + } + fn expander(&mut self) -> Expander<'_> { Expander { i2c: &mut self.i2c } } @@ -179,18 +208,7 @@ impl Board for DeviceBoard { fn write(&mut self, bytes: &[u8]) { self.transport.write(bytes); - // A line with no room is dropped whole, before it is sealed (noise::SealedLines). - if let Some((conn, lines)) = &mut self.lan_peer { - let conn = *conn; - lines.lines( - bytes, - |length| net::room(conn) > length, - |line| { - net::send(conn, line.as_bytes()); - net::send(conn, b"\n"); - }, - ); - } + self.write_lan(bytes); if let Some(lines) = &mut self.relay_peer { lines.lines( bytes, @@ -202,6 +220,14 @@ impl Board for DeviceBoard { } } + /// `VOICE DATA`, 16 KB/s of it: USB and the local network only. The UART carries 11.5 KB/s at 115200 + /// baud and its writes block, so the stream would stall the main loop there; the Relay's 50 messages a + /// second are for the box's other output (a voice command over the Relay is ADR-0018's). + fn write_stream(&mut self, line: &[u8]) { + self.transport.write_usb(line); + self.write_lan(line); + } + fn lan_send(&mut self, conn: u32, bytes: &[u8]) { net::send(conn, bytes); } @@ -314,6 +340,52 @@ impl Board for DeviceBoard { } codec::set_es8311_volume(&mut self.codec(), level).map_err(|_| VolumeError::Failed) } + + fn i2c_read(&mut self, address: u8, register: u8) -> Option { + let mut value = [0u8]; + self.i2c.write_read(address, &[register], &mut value).ok().map(|()| value[0]) + } + + fn i2c_write(&mut self, address: u8, register: u8, value: u8) -> Result<(), ()> { + self.i2c.write(address, &[register, value]).map_err(|_| ()) + } + + fn mic_start(&mut self, bytes: usize) -> Result<(), ()> { + mic::DONE.reset(); + mic::COMMAND.signal(mic::Command::Record(bytes)); + Ok(()) + } + + fn mic_take(&mut self) -> Option, &'static str>> { + while let Ok(note) = mic::NOTES.try_receive() { + self.write(note.as_bytes()); + self.write(b"\n"); + } + mic::DONE.try_take() + } + + fn voice_start(&mut self) -> Result<(), ()> { + mic::start_stream(); + Ok(()) + } + + fn voice_take(&mut self, samples: &mut alloc::vec::Vec) { + mic::take_samples(samples); + } + + fn voice_stop(&mut self) { + mic::stop_stream(); + } + + fn reply_audio(&mut self, audio: ReplyAudio) { + if let Ok(mut replies) = audio::REPLIES.try_lock() { + replies.apply(audio); + } + } + + fn reply_speaking(&self) -> bool { + audio::REPLIES.try_lock().is_ok_and(|replies| replies.speaking()) + } } #[allow(clippy::large_stack_frames, reason = "main has to hold a lot of peripherals and buffers anyway")] @@ -378,20 +450,25 @@ async fn main(spawner: Spawner) -> ! { let lcd = Lcd::new(bus, parameters, frame); // Start the I2S stream first (all zeros): as in the C firmware, the clock runs before the codec is - // configured. + // configured. The receiver shares the transmitter's BCLK and WS (signal loopback), as ESP-IDF sets up a + // full-duplex channel pair on the same pins; it only runs while a voice command records and during the + // microphone diagnostic. let i2s = I2s::new( peripherals.I2S0, peripherals.DMA_CH1, TdmConfig::new_tdm_philips() .with_sample_rate(Rate::from_hz(SAMPLE_RATE)) .with_data_format(DataFormat::Data16Channel16) - .with_channels(Channels::STEREO), + .with_channels(Channels::STEREO) + .with_signal_loopback(true), ) .expect("I2S0 config") .into_async(); let i2s_tx = i2s.i2s_tx.with_bclk(peripherals.GPIO21).with_ws(peripherals.GPIO13).with_dout(peripherals.GPIO14).build(); let stream: DmaTxStreamBuf = dma_tx_stream_buffer!(STREAM_BYTES, STREAM_CHUNK); spawner.spawn(audio::audio_task(i2s_tx, stream).expect("audio task")); + let i2s_rx = i2s.i2s_rx.with_din(peripherals.GPIO47).build(); + spawner.spawn(mic::mic_task(i2s_rx).expect("mic task")); let k0 = Input::new(peripherals.GPIO0, InputConfig::default().with_pull(Pull::Up)); @@ -427,6 +504,7 @@ async fn main(spawner: Spawner) -> ! { // Booting got this far, so an image installed over the air keeps its slot. storage::confirm_running_image(); net::CREDENTIALS.signal(wifi_credentials(&firmware)); + report_reset(&mut board); report_memory(&mut board); if let Some(stack) = stack { if let Some(link) = firmware.link() { @@ -437,13 +515,24 @@ async fn main(spawner: Spawner) -> ! { let mut relay_box = RelayBox::new(); let mut input = [0u8; 256]; + // UART and USB feed one line parser, so UART bytes are held here until their line is whole: a stray byte on + // the bridge must not land in the middle of a line arriving over USB. + let mut uart_line: alloc::vec::Vec = alloc::vec::Vec::new(); loop { loop { let count = serial::take_uart(&mut input); if count == 0 { break; } - firmware.receive(&mut board, &input[..count]); + for &byte in &input[..count] { + if uart_line.len() < 2048 { + uart_line.push(byte); + } + if byte == b'\n' { + firmware.receive(&mut board, &uart_line); + uart_line.clear(); + } + } } loop { let count = serial::take_usb(&mut input); @@ -511,6 +600,11 @@ async fn main(spawner: Spawner) -> ! { net::CREDENTIALS.signal(wifi_credentials(&firmware)); } firmware.poll(&mut board); + // The microphone's own report on a stream that just ended. + while let Ok(note) = mic::NOTES.try_receive() { + board.write(note.as_bytes()); + board.write(b"\n"); + } // Before restarting, close the Relay socket and the network connections so the computers and the // Relay see the box go at once, rather than after their silence timeouts; a new identity has a new // box-id, so nothing would replace the old one's socket (#122). The wait also lets the last line out. @@ -530,6 +624,16 @@ fn wifi_credentials(firmware: &Firmware) -> Option<(alloc::string::String, alloc } /// `MEMORY INTERNAL / PSRAM /` in bytes: what the heaps have left (#108). +/// Why the box last started, so a box that restarted by itself shows in the daemon's log: a watchdog, a brownout or +/// a panic. +fn report_reset(board: &mut DeviceBoard) { + // SAFETY: read and cleared once at boot, before anything can panic again. + let panicked = unsafe { core::mem::replace(&mut *core::ptr::addr_of_mut!(PANICKED), 0) } == PANIC_MARK; + let reason = esp_hal::system::reset_reason(); + let line = alloc::format!("RESET {reason:?}{}\n", if panicked { " PANIC" } else { "" }); + board.write(line.as_bytes()); +} + fn report_memory(board: &mut DeviceBoard) { let stats = esp_alloc::HEAP.stats(); let internal = esp_alloc::HEAP.free_caps(esp_alloc::MemoryCapability::Internal.into()); diff --git a/firmware-rs/device/src/mic.rs b/firmware-rs/device/src/mic.rs new file mode 100644 index 00000000..dcf0803c --- /dev/null +++ b/firmware-rs/device/src/mic.rs @@ -0,0 +1,478 @@ +//! Microphone capture: I2S0's receiver, clocked by the transmitter's BCLK and WS (signal loopback, as ESP-IDF +//! does for full duplex), reads GPIO47 by DMA. Two users: +//! +//! - The bring-up diagnostic (`device.mic.record`, firmware-core's `mic`) captures into one linear buffer +//! in PSRAM, as long as the whole capture. +//! - A voice command (firmware-core's `voice_command`) streams for as long as K0 is held, up to 30 s: a +//! linear buffer for that would be 7 MB, so the DMA writes round a ring instead ([`Ring`], below), and +//! this task moves the left slot's samples out of it into a queue the main loop takes from. +//! +//! Nothing here is a static in internal RAM beyond the task itself: every such static comes out of the +//! main task's stack (LESSONS.md). The descriptors (12 bytes per 4 KB) come from the internal heap for the +//! capture and go back after. +//! +//! The I2S receiver ends a DMA descriptor early at every EOF it signals. esp-hal's `rx_start` writes +//! RXEOF_NUM = 0xFFE (its `usize::MAX - 1` cut to the 12-bit field), and on the box the EOFs came every +//! 4094 bytes: descriptor lengths 4080, 14, 4080, 14, … (2026-10-13). So the ESP32-S3 counts RXEOF_NUM +//! in bytes, as ESP-IDF's driver uses it (it writes the DMA buffer size in bytes), not in samples as the +//! TRM's formula reads. Each 4094 bytes of samples take two 4080-byte descriptors, about twice their size +//! in buffer. The samples stay in order and each descriptor's length says how many it holds, so the +//! buffer is 2.5 times the capture plus two descriptors, and the capture ends once the descriptors +//! handed back hold enough. +//! +//! Setting RXEOF_NUM to 4080 would line the EOFs up with the descriptors and drop the overhead, but +//! esp-hal writes it inside `read()` after resetting the receiver, so changing it means a raw register +//! write racing the first EOF. That hasn't been tried on a box; the PSRAM is cheap (1.2 MB for 5 s). +//! +//! A ring that had to be relinked by this task before the DMA reached its end stalled for good on the box +//! (`STALLED`); one linear buffer doesn't depend on the main loop. Each capture also reports `MIC DEBUG` +//! lines 100 ms in and at the end: the descriptors and bytes so far, the first descriptor lengths, the I2S +//! and DMA registers, and how often the data pin changed level. + +use alloc::collections::VecDeque; +use alloc::string::String; +use alloc::vec::Vec; +use core::alloc::{GlobalAlloc, Layout}; +use core::cell::RefCell; +use core::fmt::Write; + +use critical_section::Mutex; + +use embassy_sync::blocking_mutex::raw::CriticalSectionRawMutex; +use embassy_sync::channel::Channel; +use embassy_sync::signal::Signal; +use embassy_time::{Instant, Timer}; +use esp_alloc::{HEAP, MemoryCapability}; +use esp_hal::Async; +use esp_hal::dma::aligned::{DmaAlignedMut, DmaAlignedRef}; +use esp_hal::dma::{BurstConfig, DmaDescriptor, DmaRxBuf, DmaRxBuffer, Owner, Preparation}; +use esp_hal::i2s::master::I2sRx; + +/// PSRAM receive buffers must be aligned to the external burst size (16 bytes by default); 64 covers any. +const ALIGN: usize = 64; +/// Bytes one descriptor covers at least (esp-hal uses 4096 minus the burst alignment); spares go unused. +const DESCRIPTOR_BYTES: usize = 4096 - ALIGN; +/// The capture takes its own length plus this long before it counts as stalled. +const SLACK_MS: u64 = 1000; + +/// Descriptors in the voice command's ring: 520 KB of PSRAM. An EOF every 4094 bytes takes two descriptors +/// (one full, one cut short), so the ring holds 64 × 4094 bytes, 2.7 s of audio, before the DMA comes round +/// to what this task hasn't read yet. +const RING_DESCRIPTORS: usize = 128; +/// Samples the queue to the main loop holds (2 s); past that, new ones are dropped and counted. +const QUEUE_SAMPLES: usize = 48_000; +/// How often the streaming task looks at the ring. +const STREAM_POLL_MS: u64 = 10; + +pub enum Command { + /// Capture this many bytes for the diagnostic. + Record(usize), + /// Stream the left slot into [`SAMPLES`] until [`Command::Stop`]. While streaming, a second `Stream` + /// just empties the queue for the new turn. + Stream, + Stop, +} + +pub static COMMAND: Signal = Signal::new(); +/// The left slot's samples while streaming, for the main loop. Its storage is on the heap (PSRAM), taken +/// when a stream starts and given back when it stops; the static itself is a few words. +static SAMPLES: Mutex>> = Mutex::new(RefCell::new(VecDeque::new())); +/// The capture, or why it stopped short. +pub static DONE: Signal, &'static str>> = Signal::new(); +/// `MIC DEBUG` lines for the main loop to print. +pub static NOTES: Channel = Channel::new(); + +/// A block from the heap with the given capabilities, freed on drop. +struct Block { + pointer: *mut u8, + layout: Layout, +} + +impl Block { + fn new(capability: MemoryCapability, size: usize) -> Option { + let layout = Layout::from_size_align(size, ALIGN).ok()?; + // SAFETY: size is non-zero (callers ask for at least one descriptor or frame). + let pointer = unsafe { HEAP.alloc_caps(capability.into(), layout) }; + if pointer.is_null() { + return None; + } + // SAFETY: just allocated with this size; zeroed before any reference to it is made. + unsafe { pointer.write_bytes(0, size) }; + Some(Self { pointer, layout }) + } +} + +impl Drop for Block { + fn drop(&mut self) { + // SAFETY: allocated in `new` with this layout. + unsafe { HEAP.dealloc(self.pointer, self.layout) }; + } +} + +/// Reads a 32-bit peripheral register. +fn register(address: u32) -> u32 { + // SAFETY: a read of a fixed ESP32-S3 peripheral address, with no side effects for these registers. + unsafe { core::ptr::read_volatile(address as *const u32) } +} + +const I2S0: u32 = 0x6000_F000; +/// GDMA channel 1 (I2S0's DMA_CH1 in main.rs); channels are 0xC0 apart. +const GDMA_CH1: u32 = 0x6003_F000 + 0xC0; +const GPIO_IN1: u32 = 0x6000_4040; + +/// How often GPIO47 (DIN) changed level over 2000 quick reads. Only DIN: BCLK and WS are outputs whose +/// input path is off, so their input register reads a constant 0 however fast they run. +fn din_activity() -> u32 { + let sample = || register(GPIO_IN1) >> 15 & 1; + let mut last = sample(); + let mut changes = 0; + for _ in 0..2000 { + let now = sample(); + changes += (now != last) as u32; + last = now; + } + changes +} + +/// The descriptors the DMA has handed back, in chain order, and the sample bytes they hold. +fn received(first: *const DmaDescriptor, linked: usize) -> (usize, usize) { + let mut bytes = 0; + for index in 0..linked { + // SAFETY: the descriptors live for the capture; the DMA may write them, so read them volatile. + let flags = unsafe { core::ptr::read_volatile(first.add(index)).flags }; + if flags.owner() { + return (index, bytes); + } + bytes += flags.length() as usize; + } + (linked, bytes) +} + +/// One line of the receive path's state: descriptors handed back and the bytes they hold, the lengths of +/// the first six, then I2S0 RX_CONF, TX_CONF, INT_RAW, RX_CLKM_CONF, RXEOF_NUM, TX state, and GDMA channel +/// 1's IN_INT_RAW, INFIFO_STATUS, IN_LINK, IN_STATE, IN_DSCR, IN_PERI_SEL, OUT_STATE, OUT_PERI_SEL, then +/// the data pin's level changes. +fn note(stage: &str, first: *const DmaDescriptor, linked: usize) { + let (done, bytes) = received(first, linked); + let mut line = String::new(); + let _ = write!(line, "MIC DEBUG {stage} DESC {done}/{linked} BYTES {bytes} LENS"); + for index in 0..linked.min(6) { + // SAFETY: as in `received`. + let _ = write!(line, " {}", unsafe { core::ptr::read_volatile(first.add(index)).flags.length() }); + } + let din = din_activity(); + let _ = write!( + line, + " I2S RXCONF {:08X} TXCONF {:08X} INTRAW {:08X} RXCLKM {:08X} RXEOF {:08X} TXSTATE {:08X} \ + DMA ININT {:08X} INFIFO {:08X} INLINK {:08X} INSTATE {:08X} INDSCR {:08X} INPERI {:08X} OUTSTATE {:08X} OUTPERI {:08X} \ + DIN {din}", + register(I2S0 + 0x20), + register(I2S0 + 0x24), + register(I2S0 + 0x0C), + register(I2S0 + 0x30), + register(I2S0 + 0x64), + register(I2S0 + 0x6C), + register(GDMA_CH1 + 0x08), + register(GDMA_CH1 + 0x18), + register(GDMA_CH1 + 0x20), + register(GDMA_CH1 + 0x24), + register(GDMA_CH1 + 0x30), + register(GDMA_CH1 + 0x48), + register(GDMA_CH1 + 0x84), + register(GDMA_CH1 + 0xA8), + ); + let _ = NOTES.try_send(line); +} + +#[embassy_executor::task] +pub async fn mic_task(mut rx: I2sRx<'static, Async>) { + loop { + match COMMAND.wait().await { + Command::Record(wanted) => { + let (result, idle) = capture(rx, wanted).await; + rx = idle; + DONE.signal(result); + } + Command::Stream => rx = stream(rx).await, + Command::Stop => {} + } + } +} + +/// Starts streaming for a voice command; an empty queue either way. +pub fn start_stream() { + critical_section::with(|cs| SAMPLES.borrow_ref_mut(cs).clear()); + COMMAND.signal(Command::Stream); +} + +/// Stops streaming and gives the queue's memory back. +pub fn stop_stream() { + COMMAND.signal(Command::Stop); + critical_section::with(|cs| *SAMPLES.borrow_ref_mut(cs) = VecDeque::new()); +} + +/// Moves the samples streamed so far to `out`. +pub fn take_samples(out: &mut Vec) { + critical_section::with(|cs| out.extend(SAMPLES.borrow_ref_mut(cs).drain(..))); +} + +/// The voice command's DMA ring: descriptors in internal RAM linked in a circle, each over its own +/// `DESCRIPTOR_BYTES` of one PSRAM block. The DMA never reaches an end, so it never stops however late +/// this task is (a linear chain the main loop had to relink stalled for good, LESSONS.md); owner checks +/// are off, so if the task falls a whole ring behind, the DMA writes over what wasn't read rather than +/// stopping, and [`Reader`] counts it. +struct Ring { + descriptors: Block, + data: Block, +} + +impl Ring { + fn new() -> Option { + let descriptors = Block::new(MemoryCapability::Internal, RING_DESCRIPTORS * size_of::())?; + let data = Block::new(MemoryCapability::External, RING_DESCRIPTORS * DESCRIPTOR_BYTES)?; + Some(Self { descriptors, data }) + } + + fn descriptor(&self, index: usize) -> *mut DmaDescriptor { + // SAFETY: within the block, which holds RING_DESCRIPTORS descriptors. + unsafe { (self.descriptors.pointer as *mut DmaDescriptor).add(index) } + } +} + +// SAFETY: both blocks live as long as the ring, which the transfer holds until it is stopped; the +// descriptors point only into the data block and at each other. +unsafe impl DmaRxBuffer for Ring { + type View = Ring; + type Final = Ring; + + fn prepare(&mut self) -> Preparation { + for index in 0..RING_DESCRIPTORS { + let next = self.descriptor((index + 1) % RING_DESCRIPTORS); + // SAFETY: ours, and the DMA hasn't started. + let descriptor = unsafe { &mut *self.descriptor(index) }; + // SAFETY: within the data block. + descriptor.buffer = unsafe { self.data.pointer.add(index * DESCRIPTOR_BYTES) }; + descriptor.set_size(DESCRIPTOR_BYTES); + descriptor.reset_for_rx(); + descriptor.next = next; + } + // What the CPU wrote while zeroing must not be written back over what the DMA writes. + // SAFETY: the block is ours, 64-byte aligned and a multiple of 64 bytes, so no other data shares + // its cache lines. + unsafe { DmaAlignedRef::new_unchecked(core::slice::from_raw_parts(self.data.pointer, RING_DESCRIPTORS * DESCRIPTOR_BYTES)) }.invalidate(); + Preparation { + start: self.descriptor(0), + accesses_psram: true, + burst_transfer: BurstConfig::default(), + check_owner: Some(false), + auto_write_back: true, + } + } + + fn into_view(self) -> Ring { + self + } + + fn from_view(view: Ring) -> Ring { + view + } +} + +/// Reads the ring in order: each descriptor the DMA has handed back (owner CPU) holds `length` bytes of the +/// interleaved stream, which runs on across descriptors (frames don't line up with the 4094-byte EOFs). +/// The left slot is bytes 0 and 1 of each 4-byte frame. +struct Reader { + next: usize, + /// Position in the current frame, and the low byte of a left sample split across descriptors. + offset: usize, + low: u8, + samples: u32, + dropped: u32, + /// The most descriptors found waiting at one look; near the ring's size means the DMA lapped us. + backlog: usize, +} + +impl Reader { + fn new() -> Self { + Self { next: 0, offset: 0, low: 0, samples: 0, dropped: 0, backlog: 0 } + } + + /// `first` is the ring's first descriptor; the transfer holds the ring itself. + fn drain(&mut self, first: *mut DmaDescriptor) { + let mut found = 0; + let mut pending: Vec = Vec::new(); + for _ in 0..RING_DESCRIPTORS { + // SAFETY: within the ring's descriptors, which live until the transfer is stopped. + let pointer = unsafe { first.add(self.next) }; + // SAFETY: the DMA writes descriptors; read them volatile. Internal RAM isn't cached on the S3. + let descriptor = unsafe { core::ptr::read_volatile(pointer) }; + if descriptor.owner() == Owner::Dma { + break; + } + found += 1; + let length = descriptor.len(); + // SAFETY: this descriptor's slice of the data block, which the DMA has finished with. + let bytes = unsafe { core::slice::from_raw_parts(descriptor.buffer as *const u8, length) }; + // The cache may hold what was there a lap ago. The slice is 64-byte aligned and whole lines + // are invalidated, all within this descriptor's DESCRIPTOR_BYTES. + // SAFETY: as above; nothing else uses these lines. + unsafe { DmaAlignedRef::new_unchecked(core::slice::from_raw_parts(descriptor.buffer as *const u8, DESCRIPTOR_BYTES)) }.invalidate(); + for &byte in bytes { + match self.offset { + 0 => self.low = byte, + 1 => pending.push(i16::from_le_bytes([self.low, byte])), + _ => {} + } + self.offset = (self.offset + 1) % 4; + } + let mut handed_back = descriptor; + handed_back.reset_for_rx(); + // SAFETY: as above. + unsafe { core::ptr::write_volatile(&raw mut (*pointer).flags, handed_back.flags) }; + self.next = (self.next + 1) % RING_DESCRIPTORS; + } + self.backlog = self.backlog.max(found); + if pending.is_empty() { + return; + } + self.samples += pending.len() as u32; + critical_section::with(|cs| { + let mut queue = SAMPLES.borrow_ref_mut(cs); + if queue.capacity() == 0 { + queue.reserve_exact(QUEUE_SAMPLES); + } + let room = QUEUE_SAMPLES.saturating_sub(queue.len()).min(pending.len()); + queue.extend(&pending[..room]); + self.dropped += (pending.len() - room) as u32; + }); + } +} + +/// Streams until told to stop, then reports `MIC STREAM MS SAMPLES DROPPED BACKLOG +/// OF `: about 24 samples a millisecond means nothing was lost on the +/// way in, and a backlog near the ring's size means the DMA lapped the reader. +async fn stream(rx: I2sRx<'static, Async>) -> I2sRx<'static, Async> { + let Some(ring) = Ring::new() else { + let _ = NOTES.try_send(String::from("MIC STREAM MEMORY")); + return rx; + }; + let first = ring.descriptor(0); + let transfer = match rx.read(ring) { + Ok(transfer) => transfer, + Err((_, rx, _)) => { + let _ = NOTES.try_send(String::from("MIC STREAM I2S")); + return rx; + } + }; + let started = Instant::now(); + let mut reader = Reader::new(); + loop { + Timer::after_millis(STREAM_POLL_MS).await; + match COMMAND.try_take() { + Some(Command::Stream) => { + critical_section::with(|cs| SAMPLES.borrow_ref_mut(cs).clear()); + reader.dropped = 0; + } + Some(Command::Stop) => break, + // The firmware doesn't start the diagnostic while a voice command records; if it did, it would + // run once the stream is over. + Some(Command::Record(bytes)) => { + COMMAND.signal(Command::Record(bytes)); + break; + } + None => {} + } + reader.drain(first); + } + let (rx, ring) = transfer.stop(); + drop(ring); + let mut line = String::new(); + let _ = write!( + line, + "MIC STREAM {} MS {} SAMPLES {} DROPPED BACKLOG {} OF {}", + started.elapsed().as_millis(), + reader.samples, + reader.dropped, + reader.backlog, + RING_DESCRIPTORS + ); + let _ = NOTES.try_send(line); + rx +} + +async fn capture(rx: I2sRx<'static, Async>, wanted: usize) -> (Result, &'static str>, I2sRx<'static, Async>) { + // Room for the descriptors an EOF cuts short (see the top of this file). + let size = (wanted * 5 / 2 + 2 * DESCRIPTOR_BYTES).next_multiple_of(ALIGN); + let count = size.div_ceil(DESCRIPTOR_BYTES); + let (Some(data), Some(descriptors)) = (Block::new(MemoryCapability::External, size), Block::new(MemoryCapability::Internal, count * size_of::())) + else { + return (Err("MEMORY"), rx); + }; + let first = descriptors.pointer as *mut DmaDescriptor; + // SAFETY: both blocks are ours alone, zeroed (a zeroed descriptor is `DmaDescriptor::EMPTY`), and outlive + // the buffer, which is dropped below before them. + let buffer = unsafe { + let descriptor_slice = core::slice::from_raw_parts_mut(first, count); + let data_slice = core::slice::from_raw_parts_mut(data.pointer, size); + match (DmaAlignedMut::new(descriptor_slice), DmaAlignedMut::new(data_slice)) { + (Ok(descriptor_slice), Ok(data_slice)) => DmaRxBuf::new(descriptor_slice, data_slice).ok(), + _ => None, + } + }; + let Some(buffer) = buffer else { return (Err("DMA BUFFER"), rx) }; + // How many descriptors esp-hal linked. + let (mut linked, mut last) = (1, first); + // SAFETY: esp-hal just linked these within the block; the DMA hasn't started. + while let Some(next) = unsafe { (*last).next.as_mut() } { + last = next as *mut DmaDescriptor; + linked += 1; + } + let transfer = match rx.read(buffer) { + Ok(transfer) => transfer, + Err((_, rx, buffer)) => { + drop(buffer); + return (Err("I2S"), rx); + } + }; + + let started = Instant::now(); + let deadline = (wanted / 96) as u64 + SLACK_MS; + let mut noted = false; + let finished = loop { + Timer::after_millis(20).await; + if !noted && started.elapsed().as_millis() >= 100 { + note("T100", first, linked); + noted = true; + } + let (done, bytes) = received(first, linked); + if bytes >= wanted || done == linked { + break true; + } + if started.elapsed().as_millis() > deadline { + break false; + } + }; + note(if finished { "END" } else { "TIMEOUT" }, first, linked); + let (rx, buffer) = transfer.stop(); + let result = if finished { + let mut capture = Vec::new(); + if capture.try_reserve_exact(wanted).is_ok() { + // The buffer's own read invalidates the PSRAM cache for what the DMA wrote. + let mut received = 0; + for piece in buffer.received_data() { + let take = piece.len().min(wanted - received); + capture.extend_from_slice(&piece[..take]); + received += take; + } + if capture.len() == wanted { Ok(capture) } else { Err("SHORT") } + } else { + Err("MEMORY") + } + } else { + Err("STALLED") + }; + drop(buffer); + drop(descriptors); + drop(data); + (result, rx) +} diff --git a/firmware-rs/device/src/serial.rs b/firmware-rs/device/src/serial.rs index 926cbc46..3c9f506e 100644 --- a/firmware-rs/device/src/serial.rs +++ b/firmware-rs/device/src/serial.rs @@ -151,7 +151,8 @@ impl Transport { } } - fn write_usb(&mut self, data: &[u8]) { + /// USB alone, for a stream the UART can't keep up with. + pub fn write_usb(&mut self, data: &[u8]) { for &byte in data { if self.usb.write_byte_nb(byte).is_ok() { self.usb_stalled = false; diff --git a/firmware-rs/vendor/nanomp3-core/Cargo.toml b/firmware-rs/vendor/nanomp3-core/Cargo.toml new file mode 100644 index 00000000..eb011b8d --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/Cargo.toml @@ -0,0 +1,24 @@ +# nanomp3-core 0.2.0 from crates.io (https://github.com/robbie01/nanomp3), MIT OR Apache-2.0, with one change so +# the Xtensa LLVM backend in the esp toolchain can compile it at opt-level "s": `l3_reorder` is `#[inline(never)]` +# (src/minimp3.rs, marked "VibeBuddy"). Without it the firmware build stops at "Cannot select: i32 = +# Constant<4608>" in mp3dec_decode_frame. Its tests and test MP3 are left out. Both workspaces use this copy +# through [patch.crates-io]; drop it once the backend is fixed. +[package] +edition = "2021" +rust-version = "1.89" +name = "nanomp3-core" +version = "0.2.0" +authors = ["Robert B. Langer "] +description = "The no_std, allocation-free MPEG audio decoder behind nanomp3 (a bit-exact, safe Rust port of minimp3)" +readme = "README.md" +license = "MIT OR Apache-2.0" +repository = "https://github.com/robbie01/nanomp3" +publish = false + +[features] +default = ["wide", "layer12"] +layer12 = [] +wide = ["dep:wide"] + +[dependencies] +wide = { version = "1", default-features = false, optional = true } diff --git a/firmware-rs/vendor/nanomp3-core/LICENSE-APACHE b/firmware-rs/vendor/nanomp3-core/LICENSE-APACHE new file mode 100644 index 00000000..fef8c29f --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/LICENSE-APACHE @@ -0,0 +1,202 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + diff --git a/firmware-rs/vendor/nanomp3-core/LICENSE-MIT b/firmware-rs/vendor/nanomp3-core/LICENSE-MIT new file mode 100644 index 00000000..e0e6608f --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/LICENSE-MIT @@ -0,0 +1,25 @@ +Copyright (c) 2025 Robert B. Langer + +Permission is hereby granted, free of charge, to any +person obtaining a copy of this software and associated +documentation files (the "Software"), to deal in the +Software without restriction, including without +limitation the rights to use, copy, modify, merge, +publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software +is furnished to do so, subject to the following +conditions: + +The above copyright notice and this permission notice +shall be included in all copies or substantial portions +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF +ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED +TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A +PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT +SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY +CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION +OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR +IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER +DEALINGS IN THE SOFTWARE. diff --git a/firmware-rs/vendor/nanomp3-core/README.md b/firmware-rs/vendor/nanomp3-core/README.md new file mode 100644 index 00000000..417af2f2 --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/README.md @@ -0,0 +1,5 @@ +# nanomp3-core + +The frame decoder behind [nanomp3](https://crates.io/crates/nanomp3): a `no_std`, allocation-free, `#![forbid(unsafe_code)]` Rust port of [minimp3](https://github.com/lieff/minimp3) whose output is bit-identical to upstream's. + +Most users want `nanomp3`, which re-exports everything here and adds tag skipping, VBR tag parsing (duration, gapless playback), frame iteration, and a sample-accurate seekable reader. Use this crate directly if you only need the frame decoder. diff --git a/firmware-rs/vendor/nanomp3-core/src/lib.rs b/firmware-rs/vendor/nanomp3-core/src/lib.rs new file mode 100644 index 00000000..33b22eb1 --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/src/lib.rs @@ -0,0 +1,227 @@ +//! The MPEG audio frame decoder behind [nanomp3](https://docs.rs/nanomp3): a +//! safe Rust port of [minimp3](https://github.com/lieff/minimp3) whose output +//! is bit-identical to upstream's. +//! +//! It's `no_std`, allocates nothing and does no buffering: [`Decoder::decode`] +//! finds and decodes one frame at a time from the bytes you give it. Because a +//! frame only counts once the frames after it are seen to be consistent, keep +//! several frames of input available (16 KiB is plenty) and refill as it +//! drains. +//! +//! ``` +//! # let data: &[u8] = &[]; +//! use nanomp3_core::{DecodeError, Decoder, MAX_SAMPLES_PER_FRAME}; +//! +//! let mut decoder = Decoder::new(); +//! let mut pcm = [0f32; MAX_SAMPLES_PER_FRAME]; // or i16, u8, i32, ... +//! let mut mp3 = data; +//! while !mp3.is_empty() { +//! let (consumed, result) = decoder.decode(mp3, &mut pcm); +//! mp3 = &mp3[consumed..]; +//! match result { +//! Ok(info) => { +//! let samples = &pcm[..info.samples_produced * info.channels.num() as usize]; +//! // ... interleaved samples at info.sample_rate +//! } +//! // Normal while starting up or after a seek; later frames decode. +//! Err(DecodeError::ReservoirUnavailable(_)) => {} +//! Err(e) => eprintln!("skipped: {e}"), +//! } +//! } +//! ``` +//! +//! Most users want the `nanomp3` crate, which re-exports this one and adds a +//! seekable reader with its own buffering, gapless playback, and tag handling. +//! +//! # Features +//! +//! - `wide` (default): explicit SIMD via the `wide` crate. Without it, the +//! same kernels run on plain arrays and rely on LLVM's auto-vectorizer. +//! - `layer12` (default): decode MPEG Layer I and II frames too. Without it +//! they are reported as [`DecodeError::UnsupportedLayer`], like minimp3 +//! built with `MINIMP3_ONLY_MP3`. + +#![no_std] +#![forbid(unsafe_code)] + +mod minimp3; + +pub use minimp3::Sample; + +#[doc(hidden)] +pub mod __private { + //! Internals shared with the `nanomp3` crate, which pins an exact version + //! of this one. Not public API: no semver guarantees. + pub use crate::minimp3::{l3_side_info, mp3d_find_frame as find_frame, FrameInfo, Header}; + use crate::{minimp3, Decoder, Sample}; + + /// `mp3dec_decode_frame` with C's `info` semantics: when no frame is + /// found, only `frame_bytes` is written. + pub fn decode_frame( + dec: &mut Decoder, + mp3: &[u8], + pcm: Option<&mut [S]>, + info: &mut FrameInfo, + ) -> usize { + minimp3::mp3dec_decode_frame(&mut dec.dec, mp3, pcm, info, dec.minimp3_compat) + } + + /// `mp3dec_init`. + pub fn init(dec: &mut Decoder) { + minimp3::mp3dec_init(&mut dec.dec); + } +} + +#[cfg(test)] +mod tests; + +/// The minimum length of the PCM output buffer, in samples. +pub const MAX_SAMPLES_PER_FRAME: usize = 1152*2; + +/// The core MP3 decoder, with no internal buffering. +/// +/// The decoder is about 22 KiB: it holds the filterbank state between frames +/// and the working memory for decoding one. Nothing is allocated. +#[derive(Clone)] +pub struct Decoder { + dec: minimp3::Mp3Dec, + minimp3_compat: bool, +} + + +/// The channel formats that may be encoded in an MP3 frame. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)] +pub enum Channels { + Mono = 1, + Stereo +} + +impl Channels { + /// Returns the corresponding number of channels for `self`. + pub fn num(self) -> u8 { + self as u8 + } +} + +/// Information about a frame found by [`Decoder::decode`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub struct FrameInfo { + /// The number of PCM samples produced per channel (0 when the frame + /// produced no audio). + pub samples_produced: usize, + /// The number of channels in this frame. + pub channels: Channels, + /// Sample rate of this frame, in Hz. + pub sample_rate: u32, + /// The frame's bitrate, in kilobits per second (0 for free-format streams). + pub bitrate: u32, + /// The MPEG layer: 1, 2 or 3 (MP1, MP2 or MP3). + pub layer: u8, +} + +/// Why [`Decoder::decode`] produced no audio. +/// +/// Every variant except `NoFrame` carries the frame's header information. +/// `NoFrame` at the end of the input and `ReservoirUnavailable` right after +/// starting or seeking are part of normal decoding. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[non_exhaustive] +pub enum DecodeError { + /// The input holds no complete frame, so the consumed bytes (all of the + /// input, unless it was empty) were skipped as tags or junk. At the end of + /// a stream this means there's nothing left; mid-stream, the input was + /// cut short or corrupt. + NoFrame, + /// The frame uses bit-reservoir data from earlier frames that weren't + /// decoded, as happens for the first frame or two after starting in the + /// middle of a stream or seeking. Later frames decode normally. + ReservoirUnavailable(FrameInfo), + /// The frame is corrupt and was skipped. + Corrupt(FrameInfo), + /// The frame is Layer I or II and the `layer12` feature is disabled. + UnsupportedLayer(FrameInfo), +} + +impl DecodeError { + /// The header information of the skipped frame, if there was one. + pub fn frame_info(&self) -> Option<&FrameInfo> { + match self { + DecodeError::NoFrame => None, + DecodeError::ReservoirUnavailable(info) + | DecodeError::Corrupt(info) + | DecodeError::UnsupportedLayer(info) => Some(info), + } + } +} + +impl core::fmt::Display for DecodeError { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.write_str(match self { + DecodeError::NoFrame => "no complete MPEG audio frame in the input", + DecodeError::ReservoirUnavailable(_) => "frame depends on bit reservoir data that wasn't decoded", + DecodeError::Corrupt(_) => "corrupt frame", + DecodeError::UnsupportedLayer(_) => "Layer I/II frame (the layer12 feature is disabled)", + }) + } +} + +impl core::error::Error for DecodeError {} + +impl Decoder { + /// Instantiates a `Decoder`. + pub const fn new() -> Self { + Self { dec: minimp3::Mp3Dec::new(), minimp3_compat: false } + } + + /// Instantiates a `Decoder` whose output is bit-identical to minimp3's, + /// including quirks nanomp3 otherwise corrects. Currently that only + /// affects `i16` output, where minimp3 rounds samples in (−1.5, −0.5] to + /// 0 instead of −1 (see [`Sample`]). + pub const fn new_minimp3_compat() -> Self { + Self { dec: minimp3::Mp3Dec::new(), minimp3_compat: true } + } + + /// Decodes the next frame of `mp3` into `pcm`. Returns how many bytes of + /// `mp3` were consumed (always advance past them, whatever the result) and + /// either the decoded frame's information or why no audio came out. + /// + /// `mp3` should hold several frames' worth of data (16 KiB is plenty): + /// frames are only recognized once the frames after them are seen, and a + /// frame cut off at the end of the input is skipped as junk. + /// + /// Samples are interleaved. `pcm` can hold `f32` samples (nominally in + /// `[-1.0, 1.0]`, not clipped) or signed or unsigned 8-, 16- or 32-bit + /// integers; see [`Sample`] for the conversions. + /// + /// # Panics + /// + /// Panics if `pcm` is less than [`MAX_SAMPLES_PER_FRAME`] long. + pub fn decode(&mut self, mp3: &[u8], pcm: &mut [S]) -> (usize, Result) { + assert!(pcm.len() >= MAX_SAMPLES_PER_FRAME, "pcm buffer too small"); + + let mut info = minimp3::FrameInfo::default(); + let samples = minimp3::mp3dec_decode_frame(&mut self.dec, mp3, Some(pcm), &mut info, self.minimp3_compat); + let frame = FrameInfo { + samples_produced: samples, + channels: if info.channels == 1 { Channels::Mono } else { Channels::Stereo }, + sample_rate: info.hz, + bitrate: info.bitrate_kbps, + layer: info.layer, + }; + let result = match info.status { + minimp3::Status::Decoded => Ok(frame), + minimp3::Status::NoFrame | minimp3::Status::HeaderOnly => Err(DecodeError::NoFrame), + minimp3::Status::Corrupt => Err(DecodeError::Corrupt(frame)), + minimp3::Status::ReservoirUnavailable => Err(DecodeError::ReservoirUnavailable(frame)), + minimp3::Status::UnsupportedLayer => Err(DecodeError::UnsupportedLayer(frame)), + }; + (info.frame_bytes, result) + } +} + +impl Default for Decoder { + fn default() -> Self { + Self::new() + } +} \ No newline at end of file diff --git a/firmware-rs/vendor/nanomp3-core/src/minimp3.rs b/firmware-rs/vendor/nanomp3-core/src/minimp3.rs new file mode 100644 index 00000000..b0a9fa5b --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/src/minimp3.rs @@ -0,0 +1,1525 @@ +//! Safe Rust port of the Layer III decoder from [minimp3](https://github.com/lieff/minimp3). +//! +//! The code deliberately mirrors the structure of `minimp3.h` (function names +//! are snake_case versions of the C ones) so the two can be read side by side. +//! Every floating-point expression is evaluated in the same order as in C, which +//! keeps the output bit-identical to upstream built with `MINIMP3_ONLY_MP3`, +//! `MINIMP3_FLOAT_OUTPUT` and `MINIMP3_NO_SIMD`; `parity/` checks this. + +// Float literals are spelled exactly as in minimp3.h. Rounding them or swapping +// in `core::f32::consts` could change results and break bit-exactness. +#![allow(clippy::excessive_precision, clippy::approx_constant)] +// Index loops mirror the C and keep parallel arrays visibly in step. +#![allow(clippy::needless_range_loop)] + +mod lanes; +#[cfg(feature = "layer12")] +mod layer12; +mod tables; +use lanes::{Lanes, F4}; +use tables::*; + +const HDR_SIZE: usize = 4; +const MAX_FREE_FORMAT_FRAME_SIZE: usize = 2304; +const MAX_FRAME_SYNC_MATCHES: usize = 10; +const MAX_BITRESERVOIR_BYTES: usize = 511; +const MAX_L3_FRAME_PAYLOAD_BYTES: usize = MAX_FREE_FORMAT_FRAME_SIZE; +const SHORT_BLOCK_TYPE: u8 = 2; +const STOP_BLOCK_TYPE: u8 = 3; +/// `MAX_SCFI` in C: `(255 + BITS_DEQUANTIZER_OUT*4 - 210 + 3) & ~3` with `BITS_DEQUANTIZER_OUT = -1`. +const MAX_SCFI: i32 = (255 - 4 - 210 + 3) & !3; + +/// A PCM output sample format: `f32`, `i8`, `u8`, `i16`, `u16`, `i32` or `u32`. +/// +/// The decoder synthesizes floating-point samples at 16-bit scale (full scale +/// is ±32768) and converts them as follows: +/// +/// | type | conversion | +/// |---|---| +/// | `f32` | divided by 32768, so nominally in `[-1.0, 1.0]`; not clipped | +/// | `i16` | rounded half away from zero and clipped | +/// | `i8` | divided by 256, rounded half away from zero and clipped | +/// | `i32` | multiplied by 65536, rounded half away from zero and clipped | +/// | `u8`, `u16`, `u32` | as the signed type, offset by half the range (sign bit flipped) | +/// +/// Integer formats carry at most about 24 significant bits, since that's the +/// precision of the synthesis. +/// +/// With [`Decoder::new_minimp3_compat`](crate::Decoder::new_minimp3_compat), +/// `i16` uses minimp3's exact rounding instead, which maps samples in +/// (−1.5, −0.5] to 0 rather than −1. `f32` output is identical to minimp3's +/// either way; the other formats have no minimp3 counterpart. +pub trait Sample: Copy + sealed::Sealed { + #[doc(hidden)] + fn from_synth(x: f32, minimp3_compat: bool) -> Self; +} + +mod sealed { + pub trait Sealed {} + impl Sealed for f32 {} + impl Sealed for i8 {} + impl Sealed for u8 {} + impl Sealed for i16 {} + impl Sealed for u16 {} + impl Sealed for i32 {} + impl Sealed for u32 {} +} + +/// Rounds half away from zero, saturating to the `i32` range (NaN becomes 0). +/// `f32::round` isn't available in `core`, and the usual `(x + 0.5) as i32` +/// misrounds 0.49999997 (the sum rounds up to 1.0). +#[inline(always)] +fn round_i32(x: f32) -> i32 { + let t = x as i32; // truncates toward zero and saturates + let frac = x - t as f32; // exact: |x| < 2^24 or x is integral + if frac >= 0.5 { + t.saturating_add(1) + } else if frac <= -0.5 { + t.saturating_sub(1) + } else { + t + } +} + +impl Sample for f32 { + #[inline(always)] + fn from_synth(x: f32, _: bool) -> f32 { + x * (1.0 / 32768.0) + } +} + +impl Sample for i16 { + #[inline(always)] + fn from_synth(x: f32, minimp3_compat: bool) -> i16 { + if minimp3_compat { + // minimp3's mp3d_scale_pcm, quirk included. + if x >= 32766.5 { + return 32767; + } + if x <= -32767.5 { + return -32768; + } + let s = (x + 0.5) as i16; + s - (s < 0) as i16 + } else { + round_i32(x).clamp(i16::MIN.into(), i16::MAX.into()) as i16 + } + } +} + +impl Sample for i8 { + #[inline(always)] + fn from_synth(x: f32, _: bool) -> i8 { + round_i32(x * (1.0 / 256.0)).clamp(i8::MIN.into(), i8::MAX.into()) as i8 + } +} + +impl Sample for i32 { + #[inline(always)] + fn from_synth(x: f32, _: bool) -> i32 { + round_i32(x * 65536.0) + } +} + +impl Sample for u8 { + #[inline(always)] + fn from_synth(x: f32, c: bool) -> u8 { + i8::from_synth(x, c) as u8 ^ 0x80 + } +} + +impl Sample for u16 { + #[inline(always)] + fn from_synth(x: f32, _: bool) -> u16 { + i16::from_synth(x, false) as u16 ^ 0x8000 + } +} + +impl Sample for u32 { + #[inline(always)] + fn from_synth(x: f32, c: bool) -> u32 { + i32::from_synth(x, c) as u32 ^ 0x8000_0000 + } +} + +/// Information about a decoded frame (`mp3dec_frame_info_t`). +#[derive(Copy, Clone, Default, Debug)] +pub struct FrameInfo { + pub frame_bytes: usize, + pub frame_offset: usize, + pub channels: u8, + pub hz: u32, + pub layer: u8, + pub bitrate_kbps: u32, + /// Why the last call produced the samples it did. Not part of C's struct. + pub status: Status, +} + +/// The outcome of `mp3dec_decode_frame`, which C only conveys as "0 samples". +#[derive(Copy, Clone, Default, Debug, PartialEq, Eq)] +pub enum Status { + /// Samples were decoded. + Decoded, + /// No frame was found; `frame_bytes` bytes of junk were skipped. + #[default] + NoFrame, + /// Only the header was parsed (`pcm == None`). + HeaderOnly, + /// The frame's side info or bitstream is invalid; the decoder resyncs. + Corrupt, + /// The frame needs bit-reservoir bytes from frames that weren't decoded. + ReservoirUnavailable, + /// A Layer I/II frame, and the `layer12` feature is off. + UnsupportedLayer, +} + +/// Persistent decoder state (`mp3dec_t`). +#[derive(Clone)] +struct State { + mdct_overlap: [[f32; 9 * 32]; 2], + qmf_state: [f32; 15 * 2 * 32], + reserv: i32, + free_format_bytes: usize, + header: [u8; 4], + reserv_buf: [u8; MAX_BITRESERVOIR_BYTES], +} + +impl State { + const fn new() -> Self { + Self { + mdct_overlap: [[0.; 9 * 32]; 2], + qmf_state: [0.; 15 * 2 * 32], + reserv: 0, + free_format_bytes: 0, + header: [0; 4], + reserv_buf: [0; MAX_BITRESERVOIR_BYTES], + } + } +} + +/// Per-frame working memory (`mp3dec_scratch_t`). Unlike C, it lives in the +/// decoder rather than on the stack so it doesn't have to be re-initialized for +/// every frame; only the parts C might read before writing are cleared. +#[derive(Clone)] +struct Scratch { + grbuf: [[f32; 576]; 2], + scf: [f32; 40], + syn: [[f32; 4]; SYN_ROWS], + ist_pos: [[u8; 39]; 2], +} + +/// The decoder: persistent state plus scratch memory. +#[derive(Clone)] +pub struct Mp3Dec { + st: State, + scratch: Scratch, + maindata: [u8; MAX_BITRESERVOIR_BYTES + MAX_L3_FRAME_PAYLOAD_BYTES], +} + +impl Mp3Dec { + pub const fn new() -> Self { + Self { + st: State::new(), + scratch: Scratch { + grbuf: [[0.; 576]; 2], + scf: [0.; 40], + syn: [[0.; 4]; SYN_ROWS], + ist_pos: [[0; 39]; 2], + }, + maindata: [0; MAX_BITRESERVOIR_BYTES + MAX_L3_FRAME_PAYLOAD_BYTES], + } + } +} + +#[derive(Copy, Clone, Default)] +struct GrInfo { + sfbtab: &'static [u8], + part_23_length: u16, + big_values: u16, + scalefac_compress: u16, + global_gain: u8, + block_type: u8, + mixed_block_flag: u8, + n_long_sfb: u8, + n_short_sfb: u8, + table_select: [u8; 3], + region_count: [u8; 3], + subblock_gain: [u8; 3], + preflag: u8, + scalefac_scale: u8, + count1_table: u8, + scfsi: u8, +} + +/// Bit reader (`bs_t`). Positions are signed like in C: the limit of a +/// malformed tiny frame can be negative, and `pos` keeps advancing past the +/// limit so callers can detect overruns. +struct Bs<'a> { + buf: &'a [u8], + pos: i32, + limit: i32, +} + +impl<'a> Bs<'a> { + fn new(buf: &'a [u8], bytes: i32) -> Self { + Self { buf, pos: 0, limit: bytes * 8 } + } + + fn get_bits(&mut self, n: u32) -> u32 { + let s = (self.pos & 7) as u32; + let mut p = (self.pos >> 3) as usize; + self.pos += n as i32; + if self.pos > self.limit { + return 0; + } + // In bounds: every byte read lies below `limit`, and `limit` never + // exceeds the buffer. + let mut shl = (n + s) as i32; + let mut next = u32::from(self.buf[p] & (255 >> s)); + p += 1; + let mut cache = 0; + loop { + shl -= 8; + if shl <= 0 { + break; + } + cache |= next << shl; + next = u32::from(self.buf[p]); + p += 1; + } + cache | (next >> -shl) + } +} + +/// A 4-byte frame header. +#[derive(Copy, Clone)] +pub struct Header(pub [u8; 4]); + +impl Header { + pub fn read(buf: &[u8]) -> Self { + Self([buf[0], buf[1], buf[2], buf[3]]) + } + pub fn is_mono(self) -> bool { + self.0[3] & 0xC0 == 0xC0 + } + fn is_ms_stereo(self) -> bool { + self.0[3] & 0xE0 == 0x60 + } + fn is_free_format(self) -> bool { + self.0[2] & 0xF0 == 0 + } + pub fn is_crc(self) -> bool { + self.0[1] & 1 == 0 + } + fn test_padding(self) -> bool { + self.0[2] & 0x2 != 0 + } + fn test_mpeg1(self) -> bool { + self.0[1] & 0x8 != 0 + } + fn test_not_mpeg25(self) -> bool { + self.0[1] & 0x10 != 0 + } + fn test_i_stereo(self) -> bool { + self.0[3] & 0x10 != 0 + } + fn test_ms_stereo(self) -> bool { + self.0[3] & 0x20 != 0 + } + pub fn get_layer(self) -> u8 { + (self.0[1] >> 1) & 3 + } + fn get_bitrate(self) -> u8 { + self.0[2] >> 4 + } + fn get_sample_rate(self) -> u8 { + (self.0[2] >> 2) & 3 + } + fn get_my_sample_rate(self) -> usize { + let h1 = self.0[1]; + (self.get_sample_rate() + (((h1 >> 3) & 1) + ((h1 >> 4) & 1)) * 3) as usize + } + fn is_frame_576(self) -> bool { + self.0[1] & 14 == 2 + } + fn is_layer_1(self) -> bool { + self.0[1] & 6 == 6 + } + + pub fn valid(self) -> bool { + let h = self.0; + h[0] == 0xff + && (h[1] & 0xF0 == 0xf0 || h[1] & 0xFE == 0xe2) + && self.get_layer() != 0 + && self.get_bitrate() != 15 + && self.get_sample_rate() != 3 + } + + /// `hdr_compare(self, other)`: is `other` a valid header for the same stream? + pub fn compare(self, other: Header) -> bool { + let (h1, h2) = (self.0, other.0); + other.valid() + && (h1[1] ^ h2[1]) & 0xFE == 0 + && (h1[2] ^ h2[2]) & 0x0C == 0 + && self.is_free_format() == other.is_free_format() + } + + pub fn bitrate_kbps(self) -> u32 { + 2 * u32::from( + HDR_BITRATE_KBPS_HALFRATE[self.test_mpeg1() as usize][self.get_layer() as usize - 1] + [self.get_bitrate() as usize], + ) + } + + pub fn sample_rate_hz(self) -> u32 { + HDR_SAMPLE_RATE_HZ_G_HZ[self.get_sample_rate() as usize] + >> !self.test_mpeg1() as u32 + >> !self.test_not_mpeg25() as u32 + } + + pub fn frame_samples(self) -> u32 { + if self.is_layer_1() { + 384 + } else { + 1152 >> self.is_frame_576() as u32 + } + } + + pub fn frame_bytes(self, free_format_size: usize) -> usize { + let mut frame_bytes = + (self.frame_samples() * self.bitrate_kbps() * 125 / self.sample_rate_hz()) as usize; + if self.is_layer_1() { + frame_bytes &= !3; // slot align + } + if frame_bytes != 0 { + frame_bytes + } else { + free_format_size + } + } + + pub fn padding(self) -> usize { + if self.test_padding() { + if self.is_layer_1() { + 4 + } else { + 1 + } + } else { + 0 + } + } +} + +fn l3_read_side_info(bs: &mut Bs, gr: &mut [GrInfo; 4], hdr: Header) -> i32 { + let mut scfsi: u32 = 0; + let mut part_23_sum: i32 = 0; + let mut sr_idx = hdr.get_my_sample_rate(); + sr_idx -= (sr_idx != 0) as usize; + let mut gr_count = if hdr.is_mono() { 1 } else { 2 }; + let main_data_begin; + if hdr.test_mpeg1() { + gr_count *= 2; + main_data_begin = bs.get_bits(9) as i32; + scfsi = bs.get_bits(7 + gr_count); + } else { + main_data_begin = (bs.get_bits(8 + gr_count) >> gr_count) as i32; + } + + for gr in &mut gr[..gr_count as usize] { + if hdr.is_mono() { + scfsi <<= 4; + } + gr.part_23_length = bs.get_bits(12) as u16; + part_23_sum += i32::from(gr.part_23_length); + gr.big_values = bs.get_bits(9) as u16; + if gr.big_values > 288 { + return -1; + } + gr.global_gain = bs.get_bits(8) as u8; + gr.scalefac_compress = bs.get_bits(if hdr.test_mpeg1() { 4 } else { 9 }) as u16; + gr.sfbtab = &L3_READ_SIDE_INFO_G_SCF_LONG[sr_idx]; + gr.n_long_sfb = 22; + gr.n_short_sfb = 0; + let tables; + if bs.get_bits(1) != 0 { + gr.block_type = bs.get_bits(2) as u8; + if gr.block_type == 0 { + return -1; + } + gr.mixed_block_flag = bs.get_bits(1) as u8; + gr.region_count[0] = 7; + gr.region_count[1] = 255; + if gr.block_type == SHORT_BLOCK_TYPE { + scfsi &= 0x0F0F; + if gr.mixed_block_flag == 0 { + gr.region_count[0] = 8; + gr.sfbtab = &L3_READ_SIDE_INFO_G_SCF_SHORT[sr_idx]; + gr.n_long_sfb = 0; + gr.n_short_sfb = 39; + } else { + gr.sfbtab = &L3_READ_SIDE_INFO_G_SCF_MIXED[sr_idx]; + gr.n_long_sfb = if hdr.test_mpeg1() { 8 } else { 6 }; + gr.n_short_sfb = 30; + } + } + tables = bs.get_bits(10) << 5; + gr.subblock_gain[0] = bs.get_bits(3) as u8; + gr.subblock_gain[1] = bs.get_bits(3) as u8; + gr.subblock_gain[2] = bs.get_bits(3) as u8; + } else { + gr.block_type = 0; + gr.mixed_block_flag = 0; + tables = bs.get_bits(15); + gr.region_count[0] = bs.get_bits(4) as u8; + gr.region_count[1] = bs.get_bits(3) as u8; + gr.region_count[2] = 255; + } + gr.table_select[0] = (tables >> 10) as u8; + gr.table_select[1] = ((tables >> 5) & 31) as u8; + gr.table_select[2] = (tables & 31) as u8; + gr.preflag = if hdr.test_mpeg1() { + bs.get_bits(1) as u8 + } else { + (gr.scalefac_compress >= 500) as u8 + }; + gr.scalefac_scale = bs.get_bits(1) as u8; + gr.count1_table = bs.get_bits(1) as u8; + gr.scfsi = ((scfsi >> 12) & 15) as u8; + scfsi <<= 4; + } + + if part_23_sum + bs.pos > bs.limit + main_data_begin * 8 { + return -1; + } + main_data_begin +} + +fn l3_read_scalefactors( + scf: &mut [u8; 40], + ist_pos: &mut [u8; 39], + scf_size: &[u8; 4], + scf_count: &[u8], + bitbuf: &mut Bs, + mut scfsi: i32, +) { + let mut pos = 0; + for i in 0..4 { + let cnt = scf_count[i] as usize; + if cnt == 0 { + break; + } + let scf = &mut scf[pos..pos + cnt]; + let ist_pos = &mut ist_pos[pos..pos + cnt]; + if scfsi & 8 != 0 { + scf.copy_from_slice(ist_pos); + } else { + let bits = u32::from(scf_size[i]); + if bits == 0 { + scf.fill(0); + ist_pos.fill(0); + } else { + let max_scf = if scfsi < 0 { (1 << bits) - 1 } else { -1 }; + for (scf, ist_pos) in scf.iter_mut().zip(ist_pos) { + let s = bitbuf.get_bits(bits) as i32; + *ist_pos = if s == max_scf { -1i32 as u8 } else { s as u8 }; + *scf = s as u8; + } + } + } + pos += cnt; + scfsi *= 2; + } + scf[pos..pos + 3].fill(0); +} + +fn l3_ldexp_q2(mut y: f32, mut exp_q2: i32) -> f32 { + loop { + let e = (30 * 4).min(exp_q2); + y *= L3_LDEXP_Q2_G_EXPFRAC[(e & 3) as usize] * ((1 << 30 >> (e >> 2)) as f32); + exp_q2 -= e; + if exp_q2 <= 0 { + return y; + } + } +} + +fn l3_decode_scalefactors( + hdr: Header, + ist_pos: &mut [u8; 39], + bs: &mut Bs, + gr: &GrInfo, + scf: &mut [f32; 40], + ch: usize, +) { + let mut scf_partition: &[u8] = &L3_DECODE_SCALEFACTORS_G_SCM_PARTITIONS + [(gr.n_short_sfb != 0) as usize + (gr.n_long_sfb == 0) as usize]; + let mut scf_size = [0u8; 4]; + let mut iscf = [0u8; 40]; + let scf_shift = u32::from(gr.scalefac_scale) + 1; + let mut scfsi = i32::from(gr.scfsi); + + if hdr.test_mpeg1() { + let part = L3_DECODE_SCALEFACTORS_G_SCFC_DECODE[gr.scalefac_compress as usize]; + scf_size[0] = part >> 2; + scf_size[1] = scf_size[0]; + scf_size[2] = part & 3; + scf_size[3] = scf_size[2]; + } else { + let ist = (hdr.test_i_stereo() && ch != 0) as usize; + let mut sfc = i32::from(gr.scalefac_compress) >> ist; + let mut k = ist * 3 * 4; + while sfc >= 0 { + let mut modprod = 1; + for i in (0..4).rev() { + let m = i32::from(L3_DECODE_SCALEFACTORS_G_MOD[k + i]); + scf_size[i] = (sfc / modprod % m) as u8; + modprod *= m; + } + sfc -= modprod; + k += 4; + } + scf_partition = &scf_partition[k..]; + scfsi = -16; + } + l3_read_scalefactors(&mut iscf, ist_pos, &scf_size, scf_partition, bs, scfsi); + + let n_long = gr.n_long_sfb as usize; + let n_short = gr.n_short_sfb as usize; + if n_short != 0 { + let sh = 3 - scf_shift; + for i in (0..n_short).step_by(3) { + for j in 0..3 { + let v = &mut iscf[n_long + i + j]; + *v = v.wrapping_add(gr.subblock_gain[j] << sh); + } + } + } else if gr.preflag != 0 { + for (v, &p) in iscf[11..21].iter_mut().zip(&L3_DECODE_SCALEFACTORS_G_PREAMP) { + *v = v.wrapping_add(p); + } + } + + let gain_exp = i32::from(gr.global_gain) - 4 - 210 - if hdr.is_ms_stereo() { 2 } else { 0 }; + let gain = l3_ldexp_q2((1 << (MAX_SCFI / 4)) as f32, MAX_SCFI - gain_exp); + for (scf, &iscf) in scf.iter_mut().zip(&iscf).take(n_long + n_short) { + *scf = l3_ldexp_q2(gain, i32::from(iscf) << scf_shift); + } +} + +fn l3_pow_43(mut x: i32) -> f32 { + let mut mult = 256; + if x < 129 { + return G_POW43[(16 + x) as usize]; + } + if x < 1024 { + mult = 16; + x <<= 3; + } + let sign = (2 * x) & 64; + let frac = ((x & 63) - sign) as f32 / ((x & !63) + sign) as f32; + G_POW43[(16 + ((x + sign) >> 6)) as usize] + * (1.0 + frac * ((4.0 / 3.0) + frac * (2.0 / 9.0))) + * mult as f32 +} + +/// The Huffman decoder's 32-bit look-ahead cache. Bytes past the end of the +/// buffer read as zero: a corrupt `big_values` can make the decoder run past +/// the granule's data, which C tolerates by reading neighbouring memory. +struct BitCache<'a> { + buf: &'a [u8], + next: usize, + cache: u32, + sh: i32, +} + +impl BitCache<'_> { + #[inline(always)] + fn byte(&self, i: usize) -> u32 { + self.buf.get(i).copied().map_or(0, u32::from) + } + + #[inline(always)] + fn peek(&self, n: i32) -> u32 { + self.cache >> (32 - n) + } + + #[inline(always)] + fn flush(&mut self, n: i32) { + self.cache <<= n; + self.sh += n; + } + + #[inline(always)] + fn check(&mut self) { + while self.sh >= 0 { + self.cache |= self.byte(self.next) << self.sh; + self.next += 1; + self.sh -= 8; + } + } + + /// Current bit position (`BSPOS` in C). + fn pos(&self) -> i32 { + self.next as i32 * 8 - 24 + self.sh + } +} + +/// Decodes Huffman pairs for one scalefactor band of the big-values region. +#[inline(always)] +fn l3_huffman_pairs( + out: &mut [f32], + br: &mut BitCache, + codebook: &[i16], + linbits: i32, + one: f32, +) { + for pair in out.as_chunks_mut::<2>().0 { + let mut w = 5; + let mut leaf = i32::from(codebook[br.peek(w) as usize]); + while leaf < 0 { + br.flush(w); + w = leaf & 7; + leaf = i32::from(codebook[br.peek(w).wrapping_sub((leaf >> 3) as u32) as usize]); + } + br.flush(leaf >> 8); + + for v in pair { + let mut lsb = leaf & 0x0F; + if LINBITS && lsb == 15 { + lsb += br.peek(linbits) as i32; + br.flush(linbits); + br.check(); + *v = one * l3_pow_43(lsb) * if (br.cache as i32) < 0 { -1.0 } else { 1.0 }; + } else { + *v = G_POW43[(16 + lsb) as usize - 16 * (br.cache >> 31) as usize] * one; + } + br.flush((lsb != 0) as i32); + leaf >>= 4; + } + br.check(); + } +} + +fn l3_huffman(dst: &mut [f32; 576], bs: &mut Bs, gr_info: &GrInfo, scf: &[f32; 40], layer3gr_limit: i32) { + let mut one = 0.0f32; + let mut ireg = 0; + let mut big_val_cnt = i32::from(gr_info.big_values); + let sfbtab = gr_info.sfbtab; + let mut sfb = 0; + let mut scf_idx = 0; + let mut d = 0; + + let start = (bs.pos / 8) as usize; + let mut br = BitCache { buf: bs.buf, next: start + 4, cache: 0, sh: (bs.pos & 7) - 8 }; + br.cache = (((br.byte(start) * 256 + br.byte(start + 1)) * 256 + br.byte(start + 2)) * 256 + + br.byte(start + 3)) + << (bs.pos & 7); + + while big_val_cnt > 0 { + let tab_num = gr_info.table_select[ireg] as usize; + let mut sfb_cnt = i32::from(gr_info.region_count[ireg]); + ireg += 1; + let codebook = &L3_HUFFMAN_TABS[L3_HUFFMAN_TABINDEX[tab_num] as usize..]; + let linbits = i32::from(L3_HUFFMAN_G_LINBITS[tab_num]); + loop { + let np = i32::from(sfbtab[sfb]) / 2; + sfb += 1; + let pairs_to_decode = big_val_cnt.min(np) as usize; + one = scf[scf_idx]; + scf_idx += 1; + // C's do/while always decodes at least one pair; np is never 0 here + // because big_values <= 288 and every table covers 288 pairs. + let out = &mut dst[d..d + 2 * pairs_to_decode.max(1)]; + if linbits != 0 { + l3_huffman_pairs::(out, &mut br, codebook, linbits, one); + } else { + l3_huffman_pairs::(out, &mut br, codebook, linbits, one); + } + d += out.len(); + big_val_cnt -= np; + if big_val_cnt <= 0 { + break; + } + sfb_cnt -= 1; + if sfb_cnt < 0 { + break; + } + } + } + + let codebook_count1: &[u8] = + if gr_info.count1_table != 0 { &L3_HUFFMAN_TAB33 } else { &L3_HUFFMAN_TAB32 }; + let mut np = 1 - big_val_cnt; + 'count1: loop { + let mut leaf = i32::from(codebook_count1[br.peek(4) as usize]); + if leaf & 8 == 0 { + leaf = i32::from( + codebook_count1[(leaf >> 3) as usize + (br.cache << 4 >> (32 - (leaf & 3))) as usize], + ); + } + br.flush(leaf & 7); + if br.pos() > layer3gr_limit { + break; + } + for half in 0..2 { + np -= 1; + if np == 0 { + np = i32::from(sfbtab[sfb]) / 2; + sfb += 1; + if np == 0 { + break 'count1; + } + one = scf[scf_idx]; + scf_idx += 1; + } + for s in 2 * half..2 * half + 2 { + if leaf & (128 >> s) != 0 { + dst[d + s] = if (br.cache as i32) < 0 { -one } else { one }; + br.flush(1); + } + } + } + br.check(); + d += 4; + } + bs.pos = layer3gr_limit; +} + +fn l3_midside_stereo(buf: &mut [f32; 1152], start: usize, n: usize) { + let (left, right) = buf.split_at_mut(576); + for (l, r) in left[start..start + n].iter_mut().zip(&mut right[start..start + n]) { + let a = *l; + let b = *r; + *l = a + b; + *r = a - b; + } +} + +fn l3_intensity_stereo_band(buf: &mut [f32; 1152], start: usize, n: usize, kl: f32, kr: f32) { + let (left, right) = buf.split_at_mut(576); + for (l, r) in left[start..start + n].iter_mut().zip(&mut right[start..start + n]) { + *r = *l * kr; + *l *= kl; + } +} + +fn l3_stereo_top_band(right: &[f32], sfb: &[u8], nbands: usize, max_band: &mut [i32; 3]) { + *max_band = [-1; 3]; + let mut off = 0; + for i in 0..nbands { + let n = sfb[i] as usize; + let band = &right[off..off + n]; + if band.as_chunks::<2>().0.iter().any(|p| p[0] != 0.0 || p[1] != 0.0) { + max_band[i % 3] = i as i32; + } + off += n; + } +} + +fn l3_stereo_process( + buf: &mut [f32; 1152], + ist_pos: &[u8; 39], + sfb: &[u8], + hdr: Header, + max_band: &[i32; 3], + mpeg2_sh: u32, +) { + let max_pos = if hdr.test_mpeg1() { 7 } else { 64 }; + let mut off = 0; + for (i, &n) in sfb.iter().take_while(|&&n| n != 0).enumerate() { + let n = n as usize; + let ipos = u32::from(ist_pos[i]); + if i as i32 > max_band[i % 3] && ipos < max_pos { + let s = if hdr.test_ms_stereo() { 1.41421356 } else { 1.0 }; + let (kl, kr); + if hdr.test_mpeg1() { + kl = L3_STEREO_PROCESS_G_PAN[2 * ipos as usize]; + kr = L3_STEREO_PROCESS_G_PAN[2 * ipos as usize + 1]; + } else { + let k = l3_ldexp_q2(1.0, (((ipos + 1) >> 1) << mpeg2_sh) as i32); + (kl, kr) = if ipos & 1 != 0 { (k, 1.0) } else { (1.0, k) }; + } + l3_intensity_stereo_band(buf, off, n, kl * s, kr * s); + } else if hdr.test_ms_stereo() { + l3_midside_stereo(buf, off, n); + } + off += n; + } +} + +fn l3_intensity_stereo(buf: &mut [f32; 1152], ist_pos: &mut [u8; 39], gr: &[GrInfo], hdr: Header) { + let mut max_band = [0i32; 3]; + let n_sfb = gr[0].n_long_sfb as usize + gr[0].n_short_sfb as usize; + let max_blocks = if gr[0].n_short_sfb != 0 { 3 } else { 1 }; + + l3_stereo_top_band(&buf[576..], gr[0].sfbtab, n_sfb, &mut max_band); + if gr[0].n_long_sfb != 0 { + max_band = [max_band[0].max(max_band[1]).max(max_band[2]); 3]; + } + for i in 0..max_blocks { + let default_pos = if hdr.test_mpeg1() { 3 } else { 0 }; + let itop = n_sfb - max_blocks + i; + let prev = itop - max_blocks; + ist_pos[itop] = if max_band[i] >= prev as i32 { default_pos } else { ist_pos[prev] }; + } + l3_stereo_process(buf, ist_pos, gr[0].sfbtab, hdr, &max_band, u32::from(gr[1].scalefac_compress & 1)); +} + +// VibeBuddy: inlined into mp3dec_decode_frame, this loop comes out of Loop Strength Reduction addressed from +// the decoder's base plus 4608, which the Xtensa backend can't select ("Cannot select: i32 = Constant<4608>"). +#[inline(never)] +fn l3_reorder(grbuf: &mut [f32], scratch: &mut [f32], sfb: &[u8]) { + let mut src = 0; + let mut dst = 0; + for &len in sfb.iter().step_by(3).take_while(|&&len| len != 0) { + let len = len as usize; + for i in 0..len { + scratch[dst] = grbuf[src + i]; + scratch[dst + 1] = grbuf[src + i + len]; + scratch[dst + 2] = grbuf[src + i + 2 * len]; + dst += 3; + } + src += 3 * len; + } + grbuf[..dst].copy_from_slice(&scratch[..dst]); +} + +fn l3_antialias(grbuf: &mut [f32; 576], nbands: usize) { + let aa = |k: usize, i: usize| F4::load(&L3_ANTIALIAS_G_AA[k][i..]); + for b in 0..nbands { + let (lo, hi) = grbuf[18 * b..18 * b + 36].split_at_mut(18); + for i in [0, 4] { + // hi[i..i+4] against lo[17-i..=14-i] (reversed). + let u = F4::load(&hi[i..]); + let d = F4::load(&lo[14 - i..]).rev(); + (u * aa(0, i) - d * aa(1, i)).store(&mut hi[i..]); + (u * aa(1, i) + d * aa(0, i)).rev().store(&mut lo[14 - i..]); + } + } +} + +#[inline(always)] +fn l3_dct3_9(y: &mut [T; 9]) { + let c = T::splat; + let mut s0 = y[0]; + let mut s2 = y[2]; + let mut s4 = y[4]; + let mut s6 = y[6]; + let mut s8 = y[8]; + let mut t0 = s0 + s6 * c(0.5); + s0 -= s6; + let mut t4 = (s4 + s2) * c(0.93969262); + let mut t2 = (s8 + s2) * c(0.76604444); + s6 = (s4 - s8) * c(0.17364818); + s4 += s8 - s2; + + s2 = s0 - s4 * c(0.5); + y[4] = s4 + s0; + s8 = t0 - t2 + s6; + s0 = t0 - t4 + t2; + s4 = t0 + t4 - s6; + + let mut s1 = y[1]; + let mut s3 = y[3]; + let mut s5 = y[5]; + let mut s7 = y[7]; + + s3 = s3 * c(0.86602540); + t0 = (s5 + s1) * c(0.98480775); + t4 = (s5 - s7) * c(0.34202014); + t2 = (s1 + s7) * c(0.64278761); + s1 = (s1 - s5 - s7) * c(0.86602540); + + s5 = t0 - s3 - t2; + s7 = t4 - s3 - t0; + s3 = t4 + s3 - t2; + + y[0] = s4 - s7; + y[1] = s2 + s1; + y[2] = s0 - s3; + y[3] = s8 + s5; + y[5] = s8 - s5; + y[6] = s0 + s3; + y[7] = s2 - s1; + y[8] = s4 + s7; +} + +/// IMDCT-36 of `T::N` adjacent bands at once (one band per lane). +#[inline(always)] +fn l3_imdct36_bands(grbuf: &mut [f32], overlap: &mut [f32], window: &[f32; 18]) { + let c = T::splat; + let g = |grbuf: &[f32], o: usize| T::gather(&grbuf[o..], 18); + let mut co = [c(0.0); 9]; + let mut si = [c(0.0); 9]; + co[0] = -g(grbuf, 0); + si[0] = g(grbuf, 17); + for i in 0..4 { + si[8 - 2 * i] = g(grbuf, 4 * i + 1) - g(grbuf, 4 * i + 2); + co[1 + 2 * i] = g(grbuf, 4 * i + 1) + g(grbuf, 4 * i + 2); + si[7 - 2 * i] = g(grbuf, 4 * i + 4) - g(grbuf, 4 * i + 3); + co[2 + 2 * i] = -(g(grbuf, 4 * i + 3) + g(grbuf, 4 * i + 4)); + } + l3_dct3_9(&mut co); + l3_dct3_9(&mut si); + + si[1] = -si[1]; + si[3] = -si[3]; + si[5] = -si[5]; + si[7] = -si[7]; + + for i in 0..9 { + let ovl = T::gather(&overlap[i..], 9); + let sum = co[i] * c(L3_IMDCT36_G_TWID9[9 + i]) + si[i] * c(L3_IMDCT36_G_TWID9[i]); + (co[i] * c(L3_IMDCT36_G_TWID9[i]) - si[i] * c(L3_IMDCT36_G_TWID9[9 + i])).scatter(&mut overlap[i..], 9); + (ovl * c(window[i]) - sum * c(window[9 + i])).scatter(&mut grbuf[i..], 18); + (ovl * c(window[9 + i]) + sum * c(window[i])).scatter(&mut grbuf[17 - i..], 18); + } +} + +fn l3_imdct36(grbuf: &mut [f32], overlap: &mut [f32], window: &[f32; 18], nbands: usize) { + let mut j = 0; + while j + 4 <= nbands { + l3_imdct36_bands::(&mut grbuf[18 * j..], &mut overlap[9 * j..], window); + j += 4; + } + while j < nbands { + l3_imdct36_bands::(&mut grbuf[18 * j..], &mut overlap[9 * j..], window); + j += 1; + } +} + +fn l3_idct3(x0: f32, x1: f32, x2: f32, dst: &mut [f32; 3]) { + let m1 = x1 * 0.86602540; + let a1 = x0 - x2 * 0.5; + dst[1] = x0 + x2; + dst[0] = a1 + m1; + dst[2] = a1 - m1; +} + +fn l3_imdct12(x: &[f32], dst: &mut [f32], overlap: &mut [f32]) { + let mut co = [0f32; 3]; + let mut si = [0f32; 3]; + l3_idct3(-x[0], x[6] + x[3], x[12] + x[9], &mut co); + l3_idct3(x[15], x[12] - x[9], x[6] - x[3], &mut si); + si[1] = -si[1]; + + for i in 0..3 { + let ovl = overlap[i]; + let sum = co[i] * L3_IMDCT12_G_TWID3[3 + i] + si[i] * L3_IMDCT12_G_TWID3[i]; + overlap[i] = co[i] * L3_IMDCT12_G_TWID3[i] - si[i] * L3_IMDCT12_G_TWID3[3 + i]; + dst[i] = ovl * L3_IMDCT12_G_TWID3[2 - i] - sum * L3_IMDCT12_G_TWID3[5 - i]; + dst[5 - i] = ovl * L3_IMDCT12_G_TWID3[5 - i] + sum * L3_IMDCT12_G_TWID3[2 - i]; + } +} + +fn l3_imdct_short(grbuf: &mut [f32], overlap: &mut [f32], nbands: usize) { + for (grbuf, overlap) in grbuf.as_chunks_mut::<18>().0.iter_mut().zip(overlap.as_chunks_mut::<9>().0).take(nbands) { + let mut tmp = [0f32; 18]; + tmp.copy_from_slice(grbuf); + grbuf[..6].copy_from_slice(&overlap[..6]); + let (ov_out, ov_state) = overlap.split_at_mut(6); + l3_imdct12(&tmp, &mut grbuf[6..12], ov_state); + l3_imdct12(&tmp[1..], &mut grbuf[12..18], ov_state); + l3_imdct12(&tmp[2..], ov_out, ov_state); + } +} + +fn l3_change_sign(grbuf: &mut [f32; 576]) { + for band in grbuf.as_chunks_mut::<18>().0.iter_mut().skip(1).step_by(2) { + for x in band.iter_mut().skip(1).step_by(2) { + *x = -*x; + } + } +} + +fn l3_imdct_gr(grbuf: &mut [f32; 576], overlap: &mut [f32; 288], block_type: u8, n_long_bands: usize) { + let (long_buf, rest_buf) = grbuf.split_at_mut(18 * n_long_bands); + let (long_ovl, rest_ovl) = overlap.split_at_mut(9 * n_long_bands); + if n_long_bands != 0 { + l3_imdct36(long_buf, long_ovl, &L3_IMDCT_GR_G_MDCT_WINDOW[0], n_long_bands); + } + if block_type == SHORT_BLOCK_TYPE { + l3_imdct_short(rest_buf, rest_ovl, 32 - n_long_bands); + } else { + let window = &L3_IMDCT_GR_G_MDCT_WINDOW[(block_type == STOP_BLOCK_TYPE) as usize]; + l3_imdct36(rest_buf, rest_ovl, window, 32 - n_long_bands); + } +} + +fn l3_save_reservoir(h: &mut State, bs: &Bs) { + let mut pos = ((bs.pos + 7) as u32 / 8) as i32; + let mut remains = (bs.limit as u32 / 8).wrapping_sub(pos as u32) as i32; + if remains > MAX_BITRESERVOIR_BYTES as i32 { + pos += remains - MAX_BITRESERVOIR_BYTES as i32; + remains = MAX_BITRESERVOIR_BYTES as i32; + } + if remains > 0 { + let (pos, remains) = (pos as usize, remains as usize); + h.reserv_buf[..remains].copy_from_slice(&bs.buf[pos..pos + remains]); + } + h.reserv = remains; +} + +fn l3_restore_reservoir<'a>( + h: &State, + bs: &Bs, + maindata: &'a mut [u8; MAX_BITRESERVOIR_BYTES + MAX_L3_FRAME_PAYLOAD_BYTES], + main_data_begin: i32, +) -> (Bs<'a>, bool) { + let frame_bytes = ((bs.limit - bs.pos) / 8) as usize; + let bytes_have = h.reserv.min(main_data_begin) as usize; + let reserv_off = (h.reserv - main_data_begin).max(0) as usize; + maindata[..bytes_have].copy_from_slice(&h.reserv_buf[reserv_off..reserv_off + bytes_have]); + let frame_start = (bs.pos / 8) as usize; + maindata[bytes_have..bytes_have + frame_bytes] + .copy_from_slice(&bs.buf[frame_start..frame_start + frame_bytes]); + // Bounding the reader to the valid bytes makes Huffman over-reads on corrupt + // streams see zeros without having to clear the buffer. + let len = bytes_have + frame_bytes; + (Bs::new(&maindata[..len], len as i32), h.reserv >= main_data_begin) +} + +fn l3_decode(h: &mut State, s: &mut Scratch, bs: &mut Bs, gr_info: &[GrInfo], nch: usize) { + let hdr = Header(h.header); + for ch in 0..nch { + let layer3gr_limit = bs.pos + i32::from(gr_info[ch].part_23_length); + l3_decode_scalefactors(hdr, &mut s.ist_pos[ch], bs, &gr_info[ch], &mut s.scf, ch); + l3_huffman(&mut s.grbuf[ch], bs, &gr_info[ch], &s.scf, layer3gr_limit); + } + + let stereo: &mut [f32; 1152] = s.grbuf.as_flattened_mut().try_into().unwrap(); + if hdr.test_i_stereo() { + l3_intensity_stereo(stereo, &mut s.ist_pos[1], gr_info, hdr); + } else if hdr.is_ms_stereo() { + l3_midside_stereo(stereo, 0, 576); + } + + for ch in 0..nch { + let gr = &gr_info[ch]; + let mut aa_bands = 31; + let n_long_bands: usize = + (if gr.mixed_block_flag != 0 { 2 } else { 0 }) << (hdr.get_my_sample_rate() == 2) as u32; + + if gr.n_short_sfb != 0 { + aa_bands = n_long_bands.saturating_sub(1); + l3_reorder( + &mut s.grbuf[ch][n_long_bands * 18..], + s.syn.as_flattened_mut(), + &gr.sfbtab[gr.n_long_sfb as usize..], + ); + } + + l3_antialias(&mut s.grbuf[ch], aa_bands); + l3_imdct_gr(&mut s.grbuf[ch], &mut h.mdct_overlap[ch], gr.block_type, n_long_bands); + l3_change_sign(&mut s.grbuf[ch]); + } +} + +/// DCT-II over columns `k..k + T::N` of the 18x32 subband matrix. +#[inline(always)] +fn mp3d_dct_ii_cols(grbuf: &mut [f32; 576], k: usize) { + let c = T::splat; + let mut t = [[c(0.0); 8]; 4]; + for i in 0..8 { + let x0 = T::load(&grbuf[k + i * 18..]); + let x1 = T::load(&grbuf[k + (15 - i) * 18..]); + let x2 = T::load(&grbuf[k + (16 + i) * 18..]); + let x3 = T::load(&grbuf[k + (31 - i) * 18..]); + let t0 = x0 + x3; + let t1 = x1 + x2; + let t2 = (x1 - x2) * c(MP3D_DCT_II_G_SEC[3 * i]); + let t3 = (x0 - x3) * c(MP3D_DCT_II_G_SEC[3 * i + 1]); + t[0][i] = t0 + t1; + t[1][i] = (t0 - t1) * c(MP3D_DCT_II_G_SEC[3 * i + 2]); + t[2][i] = t3 + t2; + t[3][i] = (t3 - t2) * c(MP3D_DCT_II_G_SEC[3 * i + 2]); + } + for x in &mut t { + let [mut x0, mut x1, mut x2, mut x3, mut x4, mut x5, mut x6, mut x7] = *x; + let mut xt = x0 - x7; + x0 += x7; + x7 = x1 - x6; + x1 += x6; + x6 = x2 - x5; + x2 += x5; + x5 = x3 - x4; + x3 += x4; + x4 = x0 - x3; + x0 += x3; + x3 = x1 - x2; + x1 += x2; + x[0] = x0 + x1; + x[4] = (x0 - x1) * c(0.70710677); + x5 += x6; + x6 = (x6 + x7) * c(0.70710677); + x7 += xt; + x3 = (x3 + x4) * c(0.70710677); + x5 -= x7 * c(0.198912367); // rotate by PI/8 + x7 += x5 * c(0.382683432); + x5 -= x7 * c(0.198912367); + x0 = xt - x6; + xt += x6; + x[1] = (xt + x7) * c(0.50979561); + x[2] = (x4 + x3) * c(0.54119611); + x[3] = (x0 - x5) * c(0.60134488); + x[5] = (x0 + x5) * c(0.89997619); + x[6] = (x4 - x3) * c(1.30656302); + x[7] = (xt - x7) * c(2.56291556); + } + for i in 0..7 { + let y = k + i * 4 * 18; + t[0][i].store(&mut grbuf[y..]); + (t[2][i] + t[3][i] + t[3][i + 1]).store(&mut grbuf[y + 18..]); + (t[1][i] + t[1][i + 1]).store(&mut grbuf[y + 2 * 18..]); + (t[2][i + 1] + t[3][i] + t[3][i + 1]).store(&mut grbuf[y + 3 * 18..]); + } + let y = k + 7 * 4 * 18; + t[0][7].store(&mut grbuf[y..]); + (t[2][7] + t[3][7]).store(&mut grbuf[y + 18..]); + t[1][7].store(&mut grbuf[y + 2 * 18..]); + t[3][7].store(&mut grbuf[y + 3 * 18..]); +} + +fn mp3d_dct_ii(grbuf: &mut [f32; 576], n: usize) { + let mut k = 0; + while k + 4 <= n { + mp3d_dct_ii_cols::(grbuf, k); + k += 4; + } + while k < n { + mp3d_dct_ii_cols::(grbuf, k); + k += 1; + } +} + +/// `mp3d_synth_pair`: the two samples C writes to `pcm[0]` and `pcm[16*nch]`, +/// read from `lane` (and `lane + 2`) of every 16th row starting at `row`. +#[inline(always)] +fn mp3d_synth_pair(z: &[[f32; 4]; SYNTH_ROWS], row: usize, lane: usize) -> (f32, f32) { + let x = |m: usize| z[row + 16 * m][lane]; + let mut a; + a = (x(14) - x(0)) * 29.0; + a += (x(1) + x(13)) * 213.0; + a += (x(12) - x(2)) * 459.0; + a += (x(3) + x(11)) * 2037.0; + a += (x(10) - x(4)) * 5153.0; + a += (x(5) + x(9)) * 6574.0; + a += (x(8) - x(6)) * 37489.0; + a += x(7) * 75038.0; + let first = a; + + let x = |m: usize| z[row + 16 * m][lane + 2]; + a = x(14) * 104.0; + a += x(12) * 1567.0; + a += x(10) * 9727.0; + a += x(8) * 64019.0; + a += x(6) * -9975.0; + a += x(4) * -45.0; + a += x(2) * 146.0; + a += x(0) * -5.0; + (first, a) +} + +// The polyphase filterbank history (`lins` in C) is viewed as rows of 4 floats: +// [left even, right even, left odd, right odd]. In C terms, row `r` lane `l` is +// `lins[4*r + l]`, and `zlin = lins + 15*64` starts at row `ZLIN`. +const ZLIN: usize = 15 * 16; +/// Rows of `lins` touched by one call to `mp3d_synth`. +const SYNTH_ROWS: usize = ZLIN + 2 * 16; + +fn mp3d_synth( + xl: &[f32], + xr: &[f32], + dst: &mut [S], + lins: &mut [[f32; 4]; SYNTH_ROWS], + compat: bool, +) { + let xl: &[f32; 560] = xl[..560].try_into().unwrap(); + let xr: &[f32; 560] = xr[..560].try_into().unwrap(); + let dst = &mut dst[..64 * NCH]; + let r = NCH - 1; + + lins[ZLIN + 15] = [xl[18 * 16], xr[18 * 16], xl[0], xr[0]]; + lins[ZLIN + 31] = [xl[1 + 18 * 16], xr[1 + 18 * 16], xl[1], xr[1]]; + + // For mono, C also computes "right" samples into the same slots and then + // overwrites them with the left ones; skipping them gives identical output. + let mut pair = |at: usize, row: usize, lane: usize| { + let (a, b) = mp3d_synth_pair(lins, row, lane); + dst[at] = S::from_synth(a, compat); + dst[at + 16 * NCH] = S::from_synth(b, compat); + }; + if NCH == 2 { + pair(r, 15, 1); + pair(r + 32 * NCH, 31, 1); + } + pair(0, 15, 0); + pair(32 * NCH, 31, 0); + + for i in (0..15).rev() { + lins[ZLIN + i] = [xl[18 * (31 - i)], xr[18 * (31 - i)], xl[1 + 18 * (31 - i)], xr[1 + 18 * (31 - i)]]; + lins[ZLIN + 16 + i][0] = xl[1 + 18 * (1 + i)]; + lins[ZLIN + 16 + i][1] = xr[1 + 18 * (1 + i)]; + lins[ZLIN - 16 + i][2] = xl[18 * (1 + i)]; + lins[ZLIN - 16 + i][3] = xr[18 * (1 + i)]; + + let w: &[f32; 16] = MP3D_SYNTH_G_WIN[(14 - i) * 16..][..16].try_into().unwrap(); + let vz = |k: usize| F4::load(&lins[ZLIN + i - 16 * k]); + let vy = |k: usize| F4::load(&lins[i + 16 * k]); + let w0 = |k: usize| F4::splat(w[2 * k]); + let w1 = |k: usize| F4::splat(w[2 * k + 1]); + let mut b = vz(0) * w1(0) + vy(0) * w0(0); + let mut a = vz(0) * w0(0) - vy(0) * w1(0); + for k in [1, 3, 5, 7] { + b += vz(k) * w1(k) + vy(k) * w0(k); + a += vy(k) * w1(k) - vz(k) * w0(k); + if k < 7 { + let k = k + 1; + b += vz(k) * w1(k) + vy(k) * w0(k); + a += vz(k) * w0(k) - vy(k) * w1(k); + } + } + let (a, b) = (a.to_array(), b.to_array()); + + if NCH == 2 { + dst[r + (15 - i) * NCH] = S::from_synth(a[1], compat); + dst[r + (17 + i) * NCH] = S::from_synth(b[1], compat); + } + dst[(15 - i) * NCH] = S::from_synth(a[0], compat); + dst[(17 + i) * NCH] = S::from_synth(b[0], compat); + if NCH == 2 { + dst[r + (47 - i) * NCH] = S::from_synth(a[3], compat); + dst[r + (49 + i) * NCH] = S::from_synth(b[3], compat); + } + dst[(47 - i) * NCH] = S::from_synth(a[2], compat); + dst[(49 + i) * NCH] = S::from_synth(b[2], compat); + } +} + +fn mp3d_synth_all( + grbuf: &[f32], + nbands: usize, + pcm: &mut [S], + lins: &mut [[f32; 4]; SYN_ROWS], + compat: bool, +) { + for i in (0..nbands).step_by(2) { + mp3d_synth::( + &grbuf[i..], + &grbuf[576 * (NCH - 1) + i..], + &mut pcm[32 * NCH * i..], + (&mut lins[16 * i..16 * i + SYNTH_ROWS]).try_into().unwrap(), + compat, + ); + } +} + +/// Rows in the whole synthesis buffer (`syn` in C: `(18 + 15) * 2 * 32` floats). +const SYN_ROWS: usize = (18 + 15) * 2 * 32 / 4; + +fn mp3d_synth_granule( + qmf_state: &mut [f32; 960], + grbuf: &mut [[f32; 576]; 2], + nbands: usize, + nch: usize, + pcm: &mut [S], + lins: &mut [[f32; 4]; SYN_ROWS], + compat: bool, +) { + for ch in &mut grbuf[..nch] { + mp3d_dct_ii(ch, nbands); + } + + lins.as_flattened_mut()[..15 * 64].copy_from_slice(qmf_state); + + let grbuf = grbuf.as_flattened(); + if nch == 1 { + mp3d_synth_all::(grbuf, nbands, pcm, lins, compat); + } else { + mp3d_synth_all::(grbuf, nbands, pcm, lins, compat); + } + + let tail = &lins.as_flattened()[nbands * 64..nbands * 64 + 15 * 64]; + if nch == 1 { + // Standard (not MINIMP3_NONSTANDARD_BUT_LOGICAL) behavior: a mono frame + // only advances the left channel's filterbank history, so a later + // switch to stereo starts the right channel from its old state. + for (q, l) in qmf_state.as_chunks_mut::<2>().0.iter_mut().zip(tail.as_chunks::<2>().0) { + q[0] = l[0]; + } + } else { + qmf_state.copy_from_slice(tail); + } +} + +fn mp3d_match_frame(hdr: &[u8], frame_bytes: usize) -> bool { + let first = Header::read(hdr); + let mut i = 0; + for nmatch in 0..MAX_FRAME_SYNC_MATCHES { + let h = Header::read(&hdr[i..]); + i += h.frame_bytes(frame_bytes) + h.padding(); + if i + HDR_SIZE > hdr.len() { + return nmatch > 0; + } + if !first.compare(Header::read(&hdr[i..])) { + return false; + } + } + true +} + +/// Returns `(offset, frame_bytes)`; `frame_bytes` is 0 when no frame was found. +pub fn mp3d_find_frame(mp3: &[u8], free_format_bytes: &mut usize) -> (usize, usize) { + let mp3_bytes = mp3.len(); + let mut i = 0; + while i + HDR_SIZE < mp3_bytes { + let buf = &mp3[i..]; + let h = Header::read(buf); + if h.valid() { + let mut frame_bytes = h.frame_bytes(*free_format_bytes); + let mut frame_and_padding = frame_bytes + h.padding(); + + let mut k = HDR_SIZE; + while frame_bytes == 0 && k < MAX_FREE_FORMAT_FRAME_SIZE && i + 2 * k + HDR_SIZE < mp3_bytes { + let next = Header::read(&buf[k..]); + if h.compare(next) { + let fb = k - h.padding(); + let nextfb = fb + next.padding(); + if i + k + nextfb + HDR_SIZE <= mp3_bytes && h.compare(Header::read(&buf[k + nextfb..])) { + frame_and_padding = k; + frame_bytes = fb; + *free_format_bytes = fb; + } + } + k += 1; + } + + if (frame_bytes != 0 && i + frame_and_padding <= mp3_bytes && mp3d_match_frame(buf, frame_bytes)) + || (i == 0 && frame_and_padding == mp3_bytes) + { + return (i, frame_and_padding); + } + *free_format_bytes = 0; + } + i += 1; + } + (mp3_bytes, 0) +} + +/// `mp3dec_init`: forces a full resynchronization (and state reset) on the +/// next decode. +pub fn mp3dec_init(dec: &mut Mp3Dec) { + dec.st.header[0] = 0; +} + +/// Parses the Layer III side info of `frame` (header included) as a frame of +/// `frame_size` bytes, like minimp3_ex does for VBR tags and seeking. Returns +/// the bit reader's `(pos, limit)` afterwards, relative to the end of the +/// header, or `None` if the side info is invalid. +pub fn l3_side_info(frame: &[u8], frame_size: usize) -> Option<(i32, i32)> { + let hdr = Header::read(frame); + let body = &frame[HDR_SIZE..]; + // C trusts frame_size; clamp it so a truncated frame can't be over-read. + let mut bs = Bs::new(body, (frame_size as i32 - HDR_SIZE as i32).min(body.len() as i32)); + if hdr.is_crc() { + bs.get_bits(16); + } + let mut gr_info = [GrInfo::default(); 4]; + (l3_read_side_info(&mut bs, &mut gr_info, hdr) >= 0).then_some((bs.pos, bs.limit)) +} + +/// `mp3dec_decode_frame`. With `pcm == None`, only parses the frame header and +/// returns the number of samples the frame would produce (per channel). +/// `compat` selects minimp3's exact sample conversion (see [`Sample`]). +pub fn mp3dec_decode_frame( + dec: &mut Mp3Dec, + mp3: &[u8], + pcm: Option<&mut [S]>, + info: &mut FrameInfo, + compat: bool, +) -> usize { + let Mp3Dec { st: dec, scratch, maindata } = dec; + let mp3_bytes = mp3.len(); + let mut i = 0; + let mut frame_size = 0; + + if mp3_bytes > 4 && dec.header[0] == 0xff && Header(dec.header).compare(Header::read(mp3)) { + let h = Header::read(mp3); + frame_size = h.frame_bytes(dec.free_format_bytes) + h.padding(); + if frame_size != mp3_bytes + && (frame_size + HDR_SIZE > mp3_bytes || !h.compare(Header::read(&mp3[frame_size..]))) + { + frame_size = 0; + } + } + if frame_size == 0 { + *dec = State::new(); + (i, frame_size) = mp3d_find_frame(mp3, &mut dec.free_format_bytes); + if frame_size == 0 || i + frame_size > mp3_bytes { + info.frame_bytes = i; + info.status = Status::NoFrame; + return 0; + } + } + + let hdr = Header::read(&mp3[i..]); + dec.header = hdr.0; + info.frame_bytes = i + frame_size; + info.frame_offset = i; + info.channels = if hdr.is_mono() { 1 } else { 2 }; + info.hz = hdr.sample_rate_hz(); + info.layer = 4 - hdr.get_layer(); + info.bitrate_kbps = hdr.bitrate_kbps(); + + let Some(pcm) = pcm else { + info.status = Status::HeaderOnly; + return hdr.frame_samples() as usize; + }; + + let frame = mp3.get(i + HDR_SIZE..i + frame_size).unwrap_or(&[]); + let mut bs_frame = Bs::new(frame, frame_size as i32 - HDR_SIZE as i32); + if hdr.is_crc() { + bs_frame.get_bits(16); + } + + if info.layer != 3 { + #[cfg(feature = "layer12")] + { + let n = layer12::decode_frame(dec, scratch, hdr, &mut bs_frame, pcm, info.channels as usize, compat); + info.status = if n == 0 { Status::Corrupt } else { Status::Decoded }; + return n; + } + #[cfg(not(feature = "layer12"))] + { + info.status = Status::UnsupportedLayer; + return 0; + } + } + + let nch = info.channels as usize; + let mut gr_info = [GrInfo::default(); 4]; + let main_data_begin = l3_read_side_info(&mut bs_frame, &mut gr_info, hdr); + if main_data_begin < 0 || bs_frame.pos > bs_frame.limit { + dec.header[0] = 0; // mp3dec_init + info.status = Status::Corrupt; + return 0; + } + + let (mut bs, success) = l3_restore_reservoir(dec, &bs_frame, maindata, main_data_begin); + if success { + // C leaves these uninitialized and corrupt streams can read them before + // writing; clearing them keeps results deterministic. + scratch.scf = [0.; 40]; + scratch.ist_pos = [[0; 39]; 2]; + let granules = if hdr.test_mpeg1() { 2 } else { 1 }; + for (igr, pcm) in pcm.chunks_exact_mut(576 * nch).take(granules).enumerate() { + scratch.grbuf = [[0.; 576]; 2]; + l3_decode(dec, scratch, &mut bs, &gr_info[igr * nch..], nch); + mp3d_synth_granule(&mut dec.qmf_state, &mut scratch.grbuf, 18, nch, pcm, &mut scratch.syn, compat); + } + } + l3_save_reservoir(dec, &bs); + info.status = if success { Status::Decoded } else { Status::ReservoirUnavailable }; + success as usize * Header(dec.header).frame_samples() as usize +} diff --git a/firmware-rs/vendor/nanomp3-core/src/minimp3/lanes.rs b/firmware-rs/vendor/nanomp3-core/src/minimp3/lanes.rs new file mode 100644 index 00000000..7c9419d6 --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/src/minimp3/lanes.rs @@ -0,0 +1,178 @@ +//! A minimal abstraction over "one float" and "four floats" so each DSP kernel +//! is written once and instantiated for both a SIMD body and a scalar tail. +//! +//! Every operation acts lane-wise with exactly the scalar semantics (no fused +//! multiply-add, no reassociation), so vectorized kernels produce bit-identical +//! results to the scalar reference. + +use core::ops::{Add, AddAssign, Mul, Neg, Sub, SubAssign}; + +pub(crate) trait Lanes: + Copy + + Add + + Sub + + Mul + + Neg + + AddAssign + + SubAssign +{ + fn splat(x: f32) -> Self; + /// Loads the first `N` floats of `s`. + fn load(s: &[f32]) -> Self; + /// Stores into the first `N` floats of `s`. + fn store(self, s: &mut [f32]); + /// Loads `s[0], s[stride], s[2*stride], ...`. + fn gather(s: &[f32], stride: usize) -> Self; + /// Stores to `s[0], s[stride], s[2*stride], ...`. + fn scatter(self, s: &mut [f32], stride: usize); + /// Reverses the lane order. + fn rev(self) -> Self; +} + +impl Lanes for f32 { + #[inline(always)] + fn splat(x: f32) -> Self { + x + } + #[inline(always)] + fn load(s: &[f32]) -> Self { + s[0] + } + #[inline(always)] + fn store(self, s: &mut [f32]) { + s[0] = self; + } + #[inline(always)] + fn gather(s: &[f32], _stride: usize) -> Self { + s[0] + } + #[inline(always)] + fn scatter(self, s: &mut [f32], _stride: usize) { + s[0] = self; + } + #[inline(always)] + fn rev(self) -> Self { + self + } +} + +pub(crate) use f32x4::F4; + +#[cfg(feature = "wide")] +mod f32x4 { + pub(crate) type F4 = wide::f32x4; + + impl super::Lanes for F4 { + #[inline(always)] + fn splat(x: f32) -> Self { + F4::splat(x) + } + #[inline(always)] + fn load(s: &[f32]) -> Self { + F4::new(s[..4].try_into().unwrap()) + } + #[inline(always)] + fn store(self, s: &mut [f32]) { + s[..4].copy_from_slice(&self.to_array()); + } + #[inline(always)] + fn gather(s: &[f32], stride: usize) -> Self { + F4::new([s[0], s[stride], s[2 * stride], s[3 * stride]]) + } + #[inline(always)] + fn scatter(self, s: &mut [f32], stride: usize) { + let a = self.to_array(); + s[3 * stride] = a[3]; + s[2 * stride] = a[2]; + s[stride] = a[1]; + s[0] = a[0]; + } + #[inline(always)] + fn rev(self) -> Self { + let [a, b, c, d] = self.to_array(); + F4::new([d, c, b, a]) + } + } +} + +#[cfg(not(feature = "wide"))] +mod f32x4 { + use core::ops::{Add, AddAssign, Mul, Neg, Sub, SubAssign}; + + /// Four floats; LLVM turns these element-wise loops into SIMD on its own. + #[derive(Copy, Clone)] + #[repr(align(16))] + pub(crate) struct F4([f32; 4]); + + impl F4 { + #[inline(always)] + pub(crate) fn to_array(self) -> [f32; 4] { + self.0 + } + } + + macro_rules! binop { + ($trait:ident, $f:ident, $op:tt, $atrait:ident, $af:ident) => { + impl $trait for F4 { + type Output = F4; + #[inline(always)] + fn $f(self, o: F4) -> F4 { + F4(core::array::from_fn(|i| self.0[i] $op o.0[i])) + } + } + impl $atrait for F4 { + #[inline(always)] + fn $af(&mut self, o: F4) { + *self = *self $op o; + } + } + }; + } + binop!(Add, add, +, AddAssign, add_assign); + binop!(Sub, sub, -, SubAssign, sub_assign); + impl Mul for F4 { + type Output = F4; + #[inline(always)] + fn mul(self, o: F4) -> F4 { + F4(core::array::from_fn(|i| self.0[i] * o.0[i])) + } + } + impl Neg for F4 { + type Output = F4; + #[inline(always)] + fn neg(self) -> F4 { + F4(self.0.map(|x| -x)) + } + } + + impl super::Lanes for F4 { + #[inline(always)] + fn splat(x: f32) -> Self { + F4([x; 4]) + } + #[inline(always)] + fn load(s: &[f32]) -> Self { + F4(s[..4].try_into().unwrap()) + } + #[inline(always)] + fn store(self, s: &mut [f32]) { + s[..4].copy_from_slice(&self.0); + } + #[inline(always)] + fn gather(s: &[f32], stride: usize) -> Self { + F4([s[0], s[stride], s[2 * stride], s[3 * stride]]) + } + #[inline(always)] + fn scatter(self, s: &mut [f32], stride: usize) { + s[3 * stride] = self.0[3]; + s[2 * stride] = self.0[2]; + s[stride] = self.0[1]; + s[0] = self.0[0]; + } + #[inline(always)] + fn rev(self) -> Self { + let [a, b, c, d] = self.0; + F4([d, c, b, a]) + } + } +} diff --git a/firmware-rs/vendor/nanomp3-core/src/minimp3/layer12.rs b/firmware-rs/vendor/nanomp3-core/src/minimp3/layer12.rs new file mode 100644 index 00000000..2e18dfd6 --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/src/minimp3/layer12.rs @@ -0,0 +1,244 @@ +//! MPEG Layer I and Layer II decoding (the `#ifndef MINIMP3_ONLY_MP3` parts of +//! minimp3.h). Enabled by the `layer12` feature. + +use super::{mp3d_synth_granule, Bs, Header, Sample, Scratch, State}; + +const MODE_MONO: u8 = 3; +const MODE_JOINT_STEREO: u8 = 1; + +struct ScaleInfo { + scf: [f32; 3 * 64], + total_bands: usize, + stereo_bands: usize, + bitalloc: [u8; 64], + scfcod: [u8; 64], +} + +struct SubbandAlloc { + tab_offset: u8, + code_tab_width: u8, + band_count: u8, +} + +const fn alloc(tab_offset: u8, code_tab_width: u8, band_count: u8) -> SubbandAlloc { + SubbandAlloc { tab_offset, code_tab_width, band_count } +} + +static G_ALLOC_L1: [SubbandAlloc; 1] = [alloc(76, 4, 32)]; +static G_ALLOC_L2M2: [SubbandAlloc; 3] = [alloc(60, 4, 4), alloc(44, 3, 7), alloc(44, 2, 19)]; +static G_ALLOC_L2M1: [SubbandAlloc; 4] = [alloc(0, 4, 3), alloc(16, 4, 8), alloc(32, 3, 12), alloc(40, 2, 7)]; +static G_ALLOC_L2M1_LOWRATE: [SubbandAlloc; 2] = [alloc(44, 4, 2), alloc(44, 3, 10)]; + +#[rustfmt::skip] +static G_BITALLOC_CODE_TAB: [u8; 92] = [ + 0,17, 3, 4, 5,6,7, 8,9,10,11,12,13,14,15,16, + 0,17,18, 3,19,4,5, 6,7, 8, 9,10,11,12,13,16, + 0,17,18, 3,19,4,5,16, + 0,17,18,16, + 0,17,18,19, 4,5,6, 7,8, 9,10,11,12,13,14,15, + 0,17,18, 3,19,4,5, 6,7, 8, 9,10,11,12,13,14, + 0, 2, 3, 4, 5,6,7, 8,9,10,11,12,13,14,15,16, +]; + +/// `DQ(x)` from C: three dequantizer steps per bit allocation, divided by `x`. +const fn dq(x: f32) -> [f32; 3] { + [9.53674316e-07 / x, 7.56931807e-07 / x, 6.00777173e-07 / x] +} + +static G_DEQ_L12: [[f32; 3]; 18] = [ + dq(3.), + dq(7.), + dq(15.), + dq(31.), + dq(63.), + dq(127.), + dq(255.), + dq(511.), + dq(1023.), + dq(2047.), + dq(4095.), + dq(8191.), + dq(16383.), + dq(32767.), + dq(65535.), + dq(3.), + dq(5.), + dq(9.), +]; + +fn l12_subband_alloc_table(hdr: Header, sci: &mut ScaleInfo) -> &'static [SubbandAlloc] { + let mode = hdr.0[3] >> 6; + let stereo_bands = if mode == MODE_MONO { + 0 + } else if mode == MODE_JOINT_STEREO { + ((((hdr.0[3] >> 4) & 3) as usize) << 2) + 4 + } else { + 32 + }; + + let (alloc, nbands): (&'static [SubbandAlloc], usize) = if hdr.is_layer_1() { + (&G_ALLOC_L1, 32) + } else if !hdr.test_mpeg1() { + (&G_ALLOC_L2M2, 30) + } else { + let sample_rate_idx = hdr.get_sample_rate(); + let mut kbps = hdr.bitrate_kbps() >> (mode != MODE_MONO) as u32; + if kbps == 0 { + kbps = 192; // free-format + } + if kbps < 56 { + (&G_ALLOC_L2M1_LOWRATE, if sample_rate_idx == 2 { 12 } else { 8 }) + } else if kbps >= 96 && sample_rate_idx != 1 { + (&G_ALLOC_L2M1, 30) + } else { + (&G_ALLOC_L2M1, 27) + } + }; + + sci.total_bands = nbands; + sci.stereo_bands = stereo_bands.min(nbands); + alloc +} + +fn l12_read_scalefactors(bs: &mut Bs, sci: &mut ScaleInfo, bands: usize) { + for i in 0..bands { + let mut s = 0.0; + let ba = sci.bitalloc[i] as usize; + let mask = if ba != 0 { 4 + ((19 >> sci.scfcod[i]) & 3) } else { 0 }; + for (j, m) in [4, 2, 1].into_iter().enumerate() { + if mask & m != 0 { + let b = bs.get_bits(6) as usize; + s = G_DEQ_L12[ba - 2][b % 3] * (1 << 21 >> (b / 3)) as f32; + } + sci.scf[3 * i + j] = s; + } + } +} + +fn l12_read_scale_info(hdr: Header, bs: &mut Bs, sci: &mut ScaleInfo) { + let mut subband_alloc = l12_subband_alloc_table(hdr, sci).iter(); + let mut k = 0; + let mut ba_bits = 0; + let mut ba_code_tab: &[u8] = &G_BITALLOC_CODE_TAB; + + for i in 0..sci.total_bands { + if i == k { + let a = subband_alloc.next().unwrap(); + k += a.band_count as usize; + ba_bits = u32::from(a.code_tab_width); + ba_code_tab = &G_BITALLOC_CODE_TAB[a.tab_offset as usize..]; + } + let mut ba = ba_code_tab[bs.get_bits(ba_bits) as usize]; + sci.bitalloc[2 * i] = ba; + if i < sci.stereo_bands { + ba = ba_code_tab[bs.get_bits(ba_bits) as usize]; + } + sci.bitalloc[2 * i + 1] = if sci.stereo_bands != 0 { ba } else { 0 }; + } + + for i in 0..2 * sci.total_bands { + sci.scfcod[i] = if sci.bitalloc[i] == 0 { + 6 + } else if hdr.is_layer_1() { + 2 + } else { + bs.get_bits(2) as u8 + }; + } + + l12_read_scalefactors(bs, sci, sci.total_bands * 2); + + for i in sci.stereo_bands..sci.total_bands { + sci.bitalloc[2 * i + 1] = 0; + } +} + +/// Dequantizes one group of samples into `grbuf[base..]`; returns how many +/// samples per subband it produced. +fn l12_dequantize_granule( + grbuf: &mut [f32; 1152], + base: usize, + bs: &mut Bs, + sci: &ScaleInfo, + group_size: usize, +) -> usize { + for j in 0..4 { + for i in 0..2 * sci.total_bands { + // C walks `dst` alternately +576 and -558: left band, right band, + // next left band, ... + let dst = &mut grbuf[base + group_size * j + (i / 2) * 18 + (i % 2) * 576..][..group_size]; + let ba = u32::from(sci.bitalloc[i]); + if ba == 0 { + continue; + } + if ba < 17 { + let half = (1 << (ba - 1)) - 1; + for d in dst { + *d = (bs.get_bits(ba) as i32 - half) as f32; + } + } else { + let m = (2 << (ba - 17)) + 1; // 3, 5, 9 + let mut code = bs.get_bits(m + 2 - (m >> 3)); // 5, 7, 10 bits + for d in dst { + *d = (code % m).wrapping_sub(m / 2) as i32 as f32; + code /= m; + } + } + } + } + group_size * 4 +} + +fn l12_apply_scf_384(sci: &ScaleInfo, igr: usize, grbuf: &mut [f32; 1152]) { + let (left, right) = grbuf.split_at_mut(576); + let (from, to) = (sci.stereo_bands * 18, sci.total_bands * 18); + right[from..to].copy_from_slice(&left[from..to]); + for b in 0..sci.total_bands { + let scf = &sci.scf[6 * b + igr..]; + for k in 0..12 { + left[18 * b + k] *= scf[0]; + right[18 * b + k] *= scf[3]; + } + } +} + +/// The Layer I/II branch of `mp3dec_decode_frame`. +pub(super) fn decode_frame( + st: &mut State, + scratch: &mut Scratch, + hdr: Header, + bs: &mut Bs, + pcm: &mut [S], + nch: usize, + compat: bool, +) -> usize { + let mut sci = ScaleInfo { + scf: [0.; 3 * 64], + total_bands: 0, + stereo_bands: 0, + bitalloc: [0; 64], + scfcod: [0; 64], + }; + l12_read_scale_info(hdr, bs, &mut sci); + + let group_size = if hdr.is_layer_1() { 1 } else { 3 }; + scratch.grbuf = [[0.; 576]; 2]; + let mut i = 0; + let mut out = 0; + for igr in 0..3 { + let grbuf: &mut [f32; 1152] = scratch.grbuf.as_flattened_mut().try_into().unwrap(); + i += l12_dequantize_granule(grbuf, i, bs, &sci, group_size); + if i == 12 { + i = 0; + l12_apply_scf_384(&sci, igr, grbuf); + mp3d_synth_granule(&mut st.qmf_state, &mut scratch.grbuf, 12, nch, &mut pcm[out..], &mut scratch.syn, compat); + scratch.grbuf = [[0.; 576]; 2]; + out += 384 * nch; + } + if bs.pos > bs.limit { + st.header[0] = 0; // mp3dec_init + return 0; + } + } + hdr.frame_samples() as usize +} diff --git a/firmware-rs/vendor/nanomp3-core/src/minimp3/tables.rs b/firmware-rs/vendor/nanomp3-core/src/minimp3/tables.rs new file mode 100644 index 00000000..e67d10f4 --- /dev/null +++ b/firmware-rs/vendor/nanomp3-core/src/minimp3/tables.rs @@ -0,0 +1,3972 @@ +pub static HDR_BITRATE_KBPS_HALFRATE: [[[u8; 15]; 3]; 2] = [ + [ + [ + 0, + 4, + 8, + 12, + 16, + 20, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + ], + [ + 0, + 4, + 8, + 12, + 16, + 20, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + ], + [ + 0, + 16, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 112, + 128, + ], + ], + [ + [ + 0, + 16, + 20, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 80, + 96, + 112, + 128, + 160, + ], + [ + 0, + 16, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 80, + 96, + 112, + 128, + 160, + 192, + ], + [ + 0, + 16, + 32, + 48, + 64, + 80, + 96, + 112, + 128, + 144, + 160, + 176, + 192, + 208, + 224, + ], + ], +]; + +pub static L3_READ_SIDE_INFO_G_SCF_LONG: [[u8; 23]; 8] = [ + [ + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 10, + 12, + 14, + 16, + 20, + 24, + 28, + 32, + 38, + 46, + 52, + 60, + 68, + 58, + 54, + 0, + ], + [ + 12, + 12, + 12, + 12, + 12, + 12, + 16, + 20, + 24, + 28, + 32, + 40, + 48, + 56, + 64, + 76, + 90, + 2, + 2, + 2, + 2, + 2, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 10, + 12, + 14, + 16, + 20, + 24, + 28, + 32, + 38, + 46, + 52, + 60, + 68, + 58, + 54, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 10, + 12, + 14, + 16, + 18, + 22, + 26, + 32, + 38, + 46, + 54, + 62, + 70, + 76, + 36, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 10, + 12, + 14, + 16, + 20, + 24, + 28, + 32, + 38, + 46, + 52, + 60, + 68, + 58, + 54, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 8, + 8, + 10, + 12, + 16, + 20, + 24, + 28, + 34, + 42, + 50, + 54, + 76, + 158, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 10, + 12, + 16, + 18, + 22, + 28, + 34, + 40, + 46, + 54, + 54, + 192, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 8, + 10, + 12, + 16, + 20, + 24, + 30, + 38, + 46, + 56, + 68, + 84, + 102, + 26, + 0, + ], +]; + +pub static L3_READ_SIDE_INFO_G_SCF_SHORT: [[u8; 40]; 8] = [ + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 30, + 30, + 30, + 40, + 40, + 40, + 18, + 18, + 18, + 0, + ], + [ + 8, + 8, + 8, + 8, + 8, + 8, + 8, + 8, + 8, + 12, + 12, + 12, + 16, + 16, + 16, + 20, + 20, + 20, + 24, + 24, + 24, + 28, + 28, + 28, + 36, + 36, + 36, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 26, + 26, + 26, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 14, + 14, + 14, + 18, + 18, + 18, + 26, + 26, + 26, + 32, + 32, + 32, + 42, + 42, + 42, + 18, + 18, + 18, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 32, + 32, + 32, + 44, + 44, + 44, + 12, + 12, + 12, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 30, + 30, + 30, + 40, + 40, + 40, + 18, + 18, + 18, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 22, + 22, + 22, + 30, + 30, + 30, + 56, + 56, + 56, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 6, + 6, + 6, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 16, + 16, + 16, + 20, + 20, + 20, + 26, + 26, + 26, + 66, + 66, + 66, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 12, + 12, + 12, + 16, + 16, + 16, + 20, + 20, + 20, + 26, + 26, + 26, + 34, + 34, + 34, + 42, + 42, + 42, + 12, + 12, + 12, + 0, + ], +]; + +pub static L3_READ_SIDE_INFO_G_SCF_MIXED: [[u8; 40]; 8] = [ + [ + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 30, + 30, + 30, + 40, + 40, + 40, + 18, + 18, + 18, + 0, + 0, + 0, + 0, + ], + [ + 12, + 12, + 12, + 4, + 4, + 4, + 8, + 8, + 8, + 12, + 12, + 12, + 16, + 16, + 16, + 20, + 20, + 20, + 24, + 24, + 24, + 28, + 28, + 28, + 36, + 36, + 36, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 2, + 26, + 26, + 26, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 14, + 14, + 14, + 18, + 18, + 18, + 26, + 26, + 26, + 32, + 32, + 32, + 42, + 42, + 42, + 18, + 18, + 18, + 0, + 0, + 0, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 32, + 32, + 32, + 44, + 44, + 44, + 12, + 12, + 12, + 0, + 0, + 0, + 0, + ], + [ + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 24, + 24, + 24, + 30, + 30, + 30, + 40, + 40, + 40, + 18, + 18, + 18, + 0, + 0, + 0, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 18, + 18, + 18, + 22, + 22, + 22, + 30, + 30, + 30, + 56, + 56, + 56, + 0, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 4, + 4, + 4, + 6, + 6, + 6, + 6, + 6, + 6, + 10, + 10, + 10, + 12, + 12, + 12, + 14, + 14, + 14, + 16, + 16, + 16, + 20, + 20, + 20, + 26, + 26, + 26, + 66, + 66, + 66, + 0, + 0, + ], + [ + 4, + 4, + 4, + 4, + 4, + 4, + 6, + 6, + 4, + 4, + 4, + 6, + 6, + 6, + 8, + 8, + 8, + 12, + 12, + 12, + 16, + 16, + 16, + 20, + 20, + 20, + 26, + 26, + 26, + 34, + 34, + 34, + 42, + 42, + 42, + 12, + 12, + 12, + 0, + 0, + ], +]; + +pub static L3_DECODE_SCALEFACTORS_G_SCM_PARTITIONS: [[u8; 28]; 3] = [ + [ + 6, + 5, + 5, + 5, + 6, + 5, + 5, + 5, + 6, + 5, + 7, + 3, + 11, + 10, + 0, + 0, + 7, + 7, + 7, + 0, + 6, + 6, + 6, + 3, + 8, + 8, + 5, + 0, + ], + [ + 8, + 9, + 6, + 12, + 6, + 9, + 9, + 9, + 6, + 9, + 12, + 6, + 15, + 18, + 0, + 0, + 6, + 15, + 12, + 0, + 6, + 12, + 9, + 6, + 6, + 18, + 9, + 0, + ], + [ + 9, + 9, + 6, + 12, + 9, + 9, + 9, + 9, + 9, + 9, + 12, + 6, + 18, + 18, + 0, + 0, + 12, + 12, + 12, + 0, + 12, + 9, + 9, + 6, + 15, + 12, + 9, + 0, + ], +]; + +pub static L3_DECODE_SCALEFACTORS_G_SCFC_DECODE: [u8; 16] = [ + 0, + 1, + 2, + 3, + 12, + 5, + 6, + 7, + 9, + 10, + 11, + 13, + 14, + 15, + 18, + 19, +]; + +pub static L3_DECODE_SCALEFACTORS_G_MOD: [u8; 24] = [ + 5, + 5, + 4, + 4, + 5, + 5, + 4, + 1, + 4, + 3, + 1, + 1, + 5, + 6, + 6, + 1, + 4, + 4, + 4, + 1, + 4, + 3, + 1, + 1, +]; + +pub static L3_DECODE_SCALEFACTORS_G_PREAMP: [u8; 10] = [ + 1, + 1, + 1, + 1, + 2, + 2, + 3, + 3, + 3, + 2, +]; + +pub static G_POW43: [f32; 145] = [ + 0f32, + -1f32, + -2.519842f32, + -4.326749f32, + -6.349604f32, + -8.549880f32, + -10.902724f32, + -13.390518f32, + -16.000000f32, + -18.720754f32, + -21.544347f32, + -24.463781f32, + -27.473142f32, + -30.567351f32, + -33.741992f32, + -36.993181f32, + 0f32, + 1f32, + 2.519842f32, + 4.326749f32, + 6.349604f32, + 8.549880f32, + 10.902724f32, + 13.390518f32, + 16.000000f32, + 18.720754f32, + 21.544347f32, + 24.463781f32, + 27.473142f32, + 30.567351f32, + 33.741992f32, + 36.993181f32, + 40.317474f32, + 43.711787f32, + 47.173345f32, + 50.699631f32, + 54.288352f32, + 57.937408f32, + 61.644865f32, + 65.408941f32, + 69.227979f32, + 73.100443f32, + 77.024898f32, + 81.000000f32, + 85.024491f32, + 89.097188f32, + 93.216975f32, + 97.382800f32, + 101.593667f32, + 105.848633f32, + 110.146801f32, + 114.487321f32, + 118.869381f32, + 123.292209f32, + 127.755065f32, + 132.257246f32, + 136.798076f32, + 141.376907f32, + 145.993119f32, + 150.646117f32, + 155.335327f32, + 160.060199f32, + 164.820202f32, + 169.614826f32, + 174.443577f32, + 179.305980f32, + 184.201575f32, + 189.129918f32, + 194.090580f32, + 199.083145f32, + 204.107210f32, + 209.162385f32, + 214.248292f32, + 219.364564f32, + 224.510845f32, + 229.686789f32, + 234.892058f32, + 240.126328f32, + 245.389280f32, + 250.680604f32, + 256.000000f32, + 261.347174f32, + 266.721841f32, + 272.123723f32, + 277.552547f32, + 283.008049f32, + 288.489971f32, + 293.998060f32, + 299.532071f32, + 305.091761f32, + 310.676898f32, + 316.287249f32, + 321.922592f32, + 327.582707f32, + 333.267377f32, + 338.976394f32, + 344.709550f32, + 350.466646f32, + 356.247482f32, + 362.051866f32, + 367.879608f32, + 373.730522f32, + 379.604427f32, + 385.501143f32, + 391.420496f32, + 397.362314f32, + 403.326427f32, + 409.312672f32, + 415.320884f32, + 421.350905f32, + 427.402579f32, + 433.475750f32, + 439.570269f32, + 445.685987f32, + 451.822757f32, + 457.980436f32, + 464.158883f32, + 470.357960f32, + 476.577530f32, + 482.817459f32, + 489.077615f32, + 495.357868f32, + 501.658090f32, + 507.978156f32, + 514.317941f32, + 520.677324f32, + 527.056184f32, + 533.454404f32, + 539.871867f32, + 546.308458f32, + 552.764065f32, + 559.238575f32, + 565.731879f32, + 572.243870f32, + 578.774440f32, + 585.323483f32, + 591.890898f32, + 598.476581f32, + 605.080431f32, + 611.702349f32, + 618.342238f32, + 625.000000f32, + 631.675540f32, + 638.368763f32, + 645.079578f32, +]; + +pub static L3_HUFFMAN_TABS: [i16; 2164] = [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 513, + 513, + 513, + 513, + 513, + 513, + 513, + 513, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + -255, + 1313, + 1298, + 1282, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 290, + 288, + -255, + 1313, + 1298, + 1282, + 769, + 769, + 769, + 769, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 528, + 528, + 528, + 528, + 528, + 528, + 528, + 528, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + 290, + 288, + -253, + -318, + -351, + -367, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 819, + 818, + 547, + 547, + 275, + 275, + 275, + 275, + 561, + 560, + 515, + 546, + 289, + 274, + 288, + 258, + -254, + -287, + 1329, + 1299, + 1314, + 1312, + 1057, + 1057, + 1042, + 1042, + 1026, + 1026, + 784, + 784, + 784, + 784, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 769, + 769, + 769, + 769, + 768, + 768, + 768, + 768, + 563, + 560, + 306, + 306, + 291, + 259, + -252, + -413, + -477, + -542, + 1298, + -575, + 1041, + 1041, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + -383, + -399, + 1107, + 1092, + 1106, + 1061, + 849, + 849, + 789, + 789, + 1104, + 1091, + 773, + 773, + 1076, + 1075, + 341, + 340, + 325, + 309, + 834, + 804, + 577, + 577, + 532, + 532, + 516, + 516, + 832, + 818, + 803, + 816, + 561, + 561, + 531, + 531, + 515, + 546, + 289, + 289, + 288, + 258, + -252, + -429, + -493, + -559, + 1057, + 1057, + 1042, + 1042, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 529, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + -382, + 1077, + -415, + 1106, + 1061, + 1104, + 849, + 849, + 789, + 789, + 1091, + 1076, + 1029, + 1075, + 834, + 834, + 597, + 581, + 340, + 340, + 339, + 324, + 804, + 833, + 532, + 532, + 832, + 772, + 818, + 803, + 817, + 787, + 816, + 771, + 290, + 290, + 290, + 290, + 288, + 258, + -253, + -349, + -414, + -447, + -463, + 1329, + 1299, + -479, + 1314, + 1312, + 1057, + 1057, + 1042, + 1042, + 1026, + 1026, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 768, + 768, + 768, + 768, + -319, + 851, + 821, + -335, + 836, + 850, + 805, + 849, + 341, + 340, + 325, + 336, + 533, + 533, + 579, + 579, + 564, + 564, + 773, + 832, + 578, + 548, + 563, + 516, + 321, + 276, + 306, + 291, + 304, + 259, + -251, + -572, + -733, + -830, + -863, + -879, + 1041, + 1041, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + -511, + -527, + -543, + 1396, + 1351, + 1381, + 1366, + 1395, + 1335, + 1380, + -559, + 1334, + 1138, + 1138, + 1063, + 1063, + 1350, + 1392, + 1031, + 1031, + 1062, + 1062, + 1364, + 1363, + 1120, + 1120, + 1333, + 1348, + 881, + 881, + 881, + 881, + 375, + 374, + 359, + 373, + 343, + 358, + 341, + 325, + 791, + 791, + 1123, + 1122, + -703, + 1105, + 1045, + -719, + 865, + 865, + 790, + 790, + 774, + 774, + 1104, + 1029, + 338, + 293, + 323, + 308, + -799, + -815, + 833, + 788, + 772, + 818, + 803, + 816, + 322, + 292, + 307, + 320, + 561, + 531, + 515, + 546, + 289, + 274, + 288, + 258, + -251, + -525, + -605, + -685, + -765, + -831, + -846, + 1298, + 1057, + 1057, + 1312, + 1282, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + 512, + 1399, + 1398, + 1383, + 1367, + 1382, + 1396, + 1351, + -511, + 1381, + 1366, + 1139, + 1139, + 1079, + 1079, + 1124, + 1124, + 1364, + 1349, + 1363, + 1333, + 882, + 882, + 882, + 882, + 807, + 807, + 807, + 807, + 1094, + 1094, + 1136, + 1136, + 373, + 341, + 535, + 535, + 881, + 775, + 867, + 822, + 774, + -591, + 324, + 338, + -671, + 849, + 550, + 550, + 866, + 864, + 609, + 609, + 293, + 336, + 534, + 534, + 789, + 835, + 773, + -751, + 834, + 804, + 308, + 307, + 833, + 788, + 832, + 772, + 562, + 562, + 547, + 547, + 305, + 275, + 560, + 515, + 290, + 290, + -252, + -397, + -477, + -557, + -622, + -653, + -719, + -735, + -750, + 1329, + 1299, + 1314, + 1057, + 1057, + 1042, + 1042, + 1312, + 1282, + 1024, + 1024, + 785, + 785, + 785, + 785, + 784, + 784, + 784, + 784, + 769, + 769, + 769, + 769, + -383, + 1127, + 1141, + 1111, + 1126, + 1140, + 1095, + 1110, + 869, + 869, + 883, + 883, + 1079, + 1109, + 882, + 882, + 375, + 374, + 807, + 868, + 838, + 881, + 791, + -463, + 867, + 822, + 368, + 263, + 852, + 837, + 836, + -543, + 610, + 610, + 550, + 550, + 352, + 336, + 534, + 534, + 865, + 774, + 851, + 821, + 850, + 805, + 593, + 533, + 579, + 564, + 773, + 832, + 578, + 578, + 548, + 548, + 577, + 577, + 307, + 276, + 306, + 291, + 516, + 560, + 259, + 259, + -250, + -2107, + -2507, + -2764, + -2909, + -2974, + -3007, + -3023, + 1041, + 1041, + 1040, + 1040, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + -767, + -1052, + -1213, + -1277, + -1358, + -1405, + -1469, + -1535, + -1550, + -1582, + -1614, + -1647, + -1662, + -1694, + -1726, + -1759, + -1774, + -1807, + -1822, + -1854, + -1886, + 1565, + -1919, + -1935, + -1951, + -1967, + 1731, + 1730, + 1580, + 1717, + -1983, + 1729, + 1564, + -1999, + 1548, + -2015, + -2031, + 1715, + 1595, + -2047, + 1714, + -2063, + 1610, + -2079, + 1609, + -2095, + 1323, + 1323, + 1457, + 1457, + 1307, + 1307, + 1712, + 1547, + 1641, + 1700, + 1699, + 1594, + 1685, + 1625, + 1442, + 1442, + 1322, + 1322, + -780, + -973, + -910, + 1279, + 1278, + 1277, + 1262, + 1276, + 1261, + 1275, + 1215, + 1260, + 1229, + -959, + 974, + 974, + 989, + 989, + -943, + 735, + 478, + 478, + 495, + 463, + 506, + 414, + -1039, + 1003, + 958, + 1017, + 927, + 942, + 987, + 957, + 431, + 476, + 1272, + 1167, + 1228, + -1183, + 1256, + -1199, + 895, + 895, + 941, + 941, + 1242, + 1227, + 1212, + 1135, + 1014, + 1014, + 490, + 489, + 503, + 487, + 910, + 1013, + 985, + 925, + 863, + 894, + 970, + 955, + 1012, + 847, + -1343, + 831, + 755, + 755, + 984, + 909, + 428, + 366, + 754, + 559, + -1391, + 752, + 486, + 457, + 924, + 997, + 698, + 698, + 983, + 893, + 740, + 740, + 908, + 877, + 739, + 739, + 667, + 667, + 953, + 938, + 497, + 287, + 271, + 271, + 683, + 606, + 590, + 712, + 726, + 574, + 302, + 302, + 738, + 736, + 481, + 286, + 526, + 725, + 605, + 711, + 636, + 724, + 696, + 651, + 589, + 681, + 666, + 710, + 364, + 467, + 573, + 695, + 466, + 466, + 301, + 465, + 379, + 379, + 709, + 604, + 665, + 679, + 316, + 316, + 634, + 633, + 436, + 436, + 464, + 269, + 424, + 394, + 452, + 332, + 438, + 363, + 347, + 408, + 393, + 448, + 331, + 422, + 362, + 407, + 392, + 421, + 346, + 406, + 391, + 376, + 375, + 359, + 1441, + 1306, + -2367, + 1290, + -2383, + 1337, + -2399, + -2415, + 1426, + 1321, + -2431, + 1411, + 1336, + -2447, + -2463, + -2479, + 1169, + 1169, + 1049, + 1049, + 1424, + 1289, + 1412, + 1352, + 1319, + -2495, + 1154, + 1154, + 1064, + 1064, + 1153, + 1153, + 416, + 390, + 360, + 404, + 403, + 389, + 344, + 374, + 373, + 343, + 358, + 372, + 327, + 357, + 342, + 311, + 356, + 326, + 1395, + 1394, + 1137, + 1137, + 1047, + 1047, + 1365, + 1392, + 1287, + 1379, + 1334, + 1364, + 1349, + 1378, + 1318, + 1363, + 792, + 792, + 792, + 792, + 1152, + 1152, + 1032, + 1032, + 1121, + 1121, + 1046, + 1046, + 1120, + 1120, + 1030, + 1030, + -2895, + 1106, + 1061, + 1104, + 849, + 849, + 789, + 789, + 1091, + 1076, + 1029, + 1090, + 1060, + 1075, + 833, + 833, + 309, + 324, + 532, + 532, + 832, + 772, + 818, + 803, + 561, + 561, + 531, + 560, + 515, + 546, + 289, + 274, + 288, + 258, + -250, + -1179, + -1579, + -1836, + -1996, + -2124, + -2253, + -2333, + -2413, + -2477, + -2542, + -2574, + -2607, + -2622, + -2655, + 1314, + 1313, + 1298, + 1312, + 1282, + 785, + 785, + 785, + 785, + 1040, + 1040, + 1025, + 1025, + 768, + 768, + 768, + 768, + -766, + -798, + -830, + -862, + -895, + -911, + -927, + -943, + -959, + -975, + -991, + -1007, + -1023, + -1039, + -1055, + -1070, + 1724, + 1647, + -1103, + -1119, + 1631, + 1767, + 1662, + 1738, + 1708, + 1723, + -1135, + 1780, + 1615, + 1779, + 1599, + 1677, + 1646, + 1778, + 1583, + -1151, + 1777, + 1567, + 1737, + 1692, + 1765, + 1722, + 1707, + 1630, + 1751, + 1661, + 1764, + 1614, + 1736, + 1676, + 1763, + 1750, + 1645, + 1598, + 1721, + 1691, + 1762, + 1706, + 1582, + 1761, + 1566, + -1167, + 1749, + 1629, + 767, + 766, + 751, + 765, + 494, + 494, + 735, + 764, + 719, + 749, + 734, + 763, + 447, + 447, + 748, + 718, + 477, + 506, + 431, + 491, + 446, + 476, + 461, + 505, + 415, + 430, + 475, + 445, + 504, + 399, + 460, + 489, + 414, + 503, + 383, + 474, + 429, + 459, + 502, + 502, + 746, + 752, + 488, + 398, + 501, + 473, + 413, + 472, + 486, + 271, + 480, + 270, + -1439, + -1455, + 1357, + -1471, + -1487, + -1503, + 1341, + 1325, + -1519, + 1489, + 1463, + 1403, + 1309, + -1535, + 1372, + 1448, + 1418, + 1476, + 1356, + 1462, + 1387, + -1551, + 1475, + 1340, + 1447, + 1402, + 1386, + -1567, + 1068, + 1068, + 1474, + 1461, + 455, + 380, + 468, + 440, + 395, + 425, + 410, + 454, + 364, + 467, + 466, + 464, + 453, + 269, + 409, + 448, + 268, + 432, + 1371, + 1473, + 1432, + 1417, + 1308, + 1460, + 1355, + 1446, + 1459, + 1431, + 1083, + 1083, + 1401, + 1416, + 1458, + 1445, + 1067, + 1067, + 1370, + 1457, + 1051, + 1051, + 1291, + 1430, + 1385, + 1444, + 1354, + 1415, + 1400, + 1443, + 1082, + 1082, + 1173, + 1113, + 1186, + 1066, + 1185, + 1050, + -1967, + 1158, + 1128, + 1172, + 1097, + 1171, + 1081, + -1983, + 1157, + 1112, + 416, + 266, + 375, + 400, + 1170, + 1142, + 1127, + 1065, + 793, + 793, + 1169, + 1033, + 1156, + 1096, + 1141, + 1111, + 1155, + 1080, + 1126, + 1140, + 898, + 898, + 808, + 808, + 897, + 897, + 792, + 792, + 1095, + 1152, + 1032, + 1125, + 1110, + 1139, + 1079, + 1124, + 882, + 807, + 838, + 881, + 853, + 791, + -2319, + 867, + 368, + 263, + 822, + 852, + 837, + 866, + 806, + 865, + -2399, + 851, + 352, + 262, + 534, + 534, + 821, + 836, + 594, + 594, + 549, + 549, + 593, + 593, + 533, + 533, + 848, + 773, + 579, + 579, + 564, + 578, + 548, + 563, + 276, + 276, + 577, + 576, + 306, + 291, + 516, + 560, + 305, + 305, + 275, + 259, + -251, + -892, + -2058, + -2620, + -2828, + -2957, + -3023, + -3039, + 1041, + 1041, + 1040, + 1040, + 769, + 769, + 769, + 769, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + 256, + -511, + -527, + -543, + -559, + 1530, + -575, + -591, + 1528, + 1527, + 1407, + 1526, + 1391, + 1023, + 1023, + 1023, + 1023, + 1525, + 1375, + 1268, + 1268, + 1103, + 1103, + 1087, + 1087, + 1039, + 1039, + 1523, + -604, + 815, + 815, + 815, + 815, + 510, + 495, + 509, + 479, + 508, + 463, + 507, + 447, + 431, + 505, + 415, + 399, + -734, + -782, + 1262, + -815, + 1259, + 1244, + -831, + 1258, + 1228, + -847, + -863, + 1196, + -879, + 1253, + 987, + 987, + 748, + -767, + 493, + 493, + 462, + 477, + 414, + 414, + 686, + 669, + 478, + 446, + 461, + 445, + 474, + 429, + 487, + 458, + 412, + 471, + 1266, + 1264, + 1009, + 1009, + 799, + 799, + -1019, + -1276, + -1452, + -1581, + -1677, + -1757, + -1821, + -1886, + -1933, + -1997, + 1257, + 1257, + 1483, + 1468, + 1512, + 1422, + 1497, + 1406, + 1467, + 1496, + 1421, + 1510, + 1134, + 1134, + 1225, + 1225, + 1466, + 1451, + 1374, + 1405, + 1252, + 1252, + 1358, + 1480, + 1164, + 1164, + 1251, + 1251, + 1238, + 1238, + 1389, + 1465, + -1407, + 1054, + 1101, + -1423, + 1207, + -1439, + 830, + 830, + 1248, + 1038, + 1237, + 1117, + 1223, + 1148, + 1236, + 1208, + 411, + 426, + 395, + 410, + 379, + 269, + 1193, + 1222, + 1132, + 1235, + 1221, + 1116, + 976, + 976, + 1192, + 1162, + 1177, + 1220, + 1131, + 1191, + 963, + 963, + -1647, + 961, + 780, + -1663, + 558, + 558, + 994, + 993, + 437, + 408, + 393, + 407, + 829, + 978, + 813, + 797, + 947, + -1743, + 721, + 721, + 377, + 392, + 844, + 950, + 828, + 890, + 706, + 706, + 812, + 859, + 796, + 960, + 948, + 843, + 934, + 874, + 571, + 571, + -1919, + 690, + 555, + 689, + 421, + 346, + 539, + 539, + 944, + 779, + 918, + 873, + 932, + 842, + 903, + 888, + 570, + 570, + 931, + 917, + 674, + 674, + -2575, + 1562, + -2591, + 1609, + -2607, + 1654, + 1322, + 1322, + 1441, + 1441, + 1696, + 1546, + 1683, + 1593, + 1669, + 1624, + 1426, + 1426, + 1321, + 1321, + 1639, + 1680, + 1425, + 1425, + 1305, + 1305, + 1545, + 1668, + 1608, + 1623, + 1667, + 1592, + 1638, + 1666, + 1320, + 1320, + 1652, + 1607, + 1409, + 1409, + 1304, + 1304, + 1288, + 1288, + 1664, + 1637, + 1395, + 1395, + 1335, + 1335, + 1622, + 1636, + 1394, + 1394, + 1319, + 1319, + 1606, + 1621, + 1392, + 1392, + 1137, + 1137, + 1137, + 1137, + 345, + 390, + 360, + 375, + 404, + 373, + 1047, + -2751, + -2767, + -2783, + 1062, + 1121, + 1046, + -2799, + 1077, + -2815, + 1106, + 1061, + 789, + 789, + 1105, + 1104, + 263, + 355, + 310, + 340, + 325, + 354, + 352, + 262, + 339, + 324, + 1091, + 1076, + 1029, + 1090, + 1060, + 1075, + 833, + 833, + 788, + 788, + 1088, + 1028, + 818, + 818, + 803, + 803, + 561, + 561, + 531, + 531, + 816, + 771, + 546, + 546, + 289, + 274, + 288, + 258, + -253, + -317, + -381, + -446, + -478, + -509, + 1279, + 1279, + -811, + -1179, + -1451, + -1756, + -1900, + -2028, + -2189, + -2253, + -2333, + -2414, + -2445, + -2511, + -2526, + 1313, + 1298, + -2559, + 1041, + 1041, + 1040, + 1040, + 1025, + 1025, + 1024, + 1024, + 1022, + 1007, + 1021, + 991, + 1020, + 975, + 1019, + 959, + 687, + 687, + 1018, + 1017, + 671, + 671, + 655, + 655, + 1016, + 1015, + 639, + 639, + 758, + 758, + 623, + 623, + 757, + 607, + 756, + 591, + 755, + 575, + 754, + 559, + 543, + 543, + 1009, + 783, + -575, + -621, + -685, + -749, + 496, + -590, + 750, + 749, + 734, + 748, + 974, + 989, + 1003, + 958, + 988, + 973, + 1002, + 942, + 987, + 957, + 972, + 1001, + 926, + 986, + 941, + 971, + 956, + 1000, + 910, + 985, + 925, + 999, + 894, + 970, + -1071, + -1087, + -1102, + 1390, + -1135, + 1436, + 1509, + 1451, + 1374, + -1151, + 1405, + 1358, + 1480, + 1420, + -1167, + 1507, + 1494, + 1389, + 1342, + 1465, + 1435, + 1450, + 1326, + 1505, + 1310, + 1493, + 1373, + 1479, + 1404, + 1492, + 1464, + 1419, + 428, + 443, + 472, + 397, + 736, + 526, + 464, + 464, + 486, + 457, + 442, + 471, + 484, + 482, + 1357, + 1449, + 1434, + 1478, + 1388, + 1491, + 1341, + 1490, + 1325, + 1489, + 1463, + 1403, + 1309, + 1477, + 1372, + 1448, + 1418, + 1433, + 1476, + 1356, + 1462, + 1387, + -1439, + 1475, + 1340, + 1447, + 1402, + 1474, + 1324, + 1461, + 1371, + 1473, + 269, + 448, + 1432, + 1417, + 1308, + 1460, + -1711, + 1459, + -1727, + 1441, + 1099, + 1099, + 1446, + 1386, + 1431, + 1401, + -1743, + 1289, + 1083, + 1083, + 1160, + 1160, + 1458, + 1445, + 1067, + 1067, + 1370, + 1457, + 1307, + 1430, + 1129, + 1129, + 1098, + 1098, + 268, + 432, + 267, + 416, + 266, + 400, + -1887, + 1144, + 1187, + 1082, + 1173, + 1113, + 1186, + 1066, + 1050, + 1158, + 1128, + 1143, + 1172, + 1097, + 1171, + 1081, + 420, + 391, + 1157, + 1112, + 1170, + 1142, + 1127, + 1065, + 1169, + 1049, + 1156, + 1096, + 1141, + 1111, + 1155, + 1080, + 1126, + 1154, + 1064, + 1153, + 1140, + 1095, + 1048, + -2159, + 1125, + 1110, + 1137, + -2175, + 823, + 823, + 1139, + 1138, + 807, + 807, + 384, + 264, + 368, + 263, + 868, + 838, + 853, + 791, + 867, + 822, + 852, + 837, + 866, + 806, + 865, + 790, + -2319, + 851, + 821, + 836, + 352, + 262, + 850, + 805, + 849, + -2399, + 533, + 533, + 835, + 820, + 336, + 261, + 578, + 548, + 563, + 577, + 532, + 532, + 832, + 772, + 562, + 562, + 547, + 547, + 305, + 275, + 560, + 515, + 290, + 290, + 288, + 258, +]; + +pub static L3_HUFFMAN_TAB32: [u8; 28] = [ + 130, + 162, + 193, + 209, + 44, + 28, + 76, + 140, + 9, + 9, + 9, + 9, + 9, + 9, + 9, + 9, + 190, + 254, + 222, + 238, + 126, + 94, + 157, + 157, + 109, + 61, + 173, + 205, +]; + +pub static L3_HUFFMAN_TAB33: [u8; 16] = [ + 252, + 236, + 220, + 204, + 188, + 172, + 156, + 140, + 124, + 108, + 92, + 76, + 60, + 44, + 28, + 12, +]; + +pub static L3_HUFFMAN_TABINDEX: [i16; 32] = [ + 0, + 32, + 64, + 98, + 0, + 132, + 180, + 218, + 292, + 364, + 426, + 538, + 648, + 746, + 0, + 1126, + 1460, + 1460, + 1460, + 1460, + 1460, + 1460, + 1460, + 1460, + 1842, + 1842, + 1842, + 1842, + 1842, + 1842, + 1842, + 1842, +]; + +pub static L3_HUFFMAN_G_LINBITS: [u8; 32] = [ + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 1, + 2, + 3, + 4, + 6, + 8, + 10, + 13, + 4, + 5, + 6, + 7, + 8, + 9, + 11, + 13, +]; + +pub static L3_STEREO_PROCESS_G_PAN: [f32; 14] = [ + 0f32, + 1f32, + 0.21132487f32, + 0.78867513f32, + 0.36602540f32, + 0.63397460f32, + 0.5f32, + 0.5f32, + 0.63397460f32, + 0.36602540f32, + 0.78867513f32, + 0.21132487f32, + 1f32, + 0f32, +]; + +pub static L3_ANTIALIAS_G_AA: [[f32; 8]; 2] = [ + [ + 0.85749293f32, + 0.88174200f32, + 0.94962865f32, + 0.98331459f32, + 0.99551782f32, + 0.99916056f32, + 0.99989920f32, + 0.99999316f32, + ], + [ + 0.51449576f32, + 0.47173197f32, + 0.31337745f32, + 0.18191320f32, + 0.09457419f32, + 0.04096558f32, + 0.01419856f32, + 0.00369997f32, + ], +]; + +pub static L3_IMDCT36_G_TWID9: [f32; 18] = [ + 0.73727734f32, + 0.79335334f32, + 0.84339145f32, + 0.88701083f32, + 0.92387953f32, + 0.95371695f32, + 0.97629601f32, + 0.99144486f32, + 0.99904822f32, + 0.67559021f32, + 0.60876143f32, + 0.53729961f32, + 0.46174861f32, + 0.38268343f32, + 0.30070580f32, + 0.21643961f32, + 0.13052619f32, + 0.04361938f32, +]; + +pub static L3_IMDCT_GR_G_MDCT_WINDOW: [[f32; 18]; 2] = [ + [ + 0.99904822f32, + 0.99144486f32, + 0.97629601f32, + 0.95371695f32, + 0.92387953f32, + 0.88701083f32, + 0.84339145f32, + 0.79335334f32, + 0.73727734f32, + 0.04361938f32, + 0.13052619f32, + 0.21643961f32, + 0.30070580f32, + 0.38268343f32, + 0.46174861f32, + 0.53729961f32, + 0.60876143f32, + 0.67559021f32, + ], + [ + 1f32, + 1f32, + 1f32, + 1f32, + 1f32, + 1f32, + 0.99144486f32, + 0.92387953f32, + 0.79335334f32, + 0f32, + 0f32, + 0f32, + 0f32, + 0f32, + 0f32, + 0.13052619f32, + 0.38268343f32, + 0.60876143f32, + ], +]; + +pub static MP3D_DCT_II_G_SEC: [f32; 24] = [ + 10.19000816f32, + 0.50060302f32, + 0.50241929f32, + 3.40760851f32, + 0.50547093f32, + 0.52249861f32, + 2.05778098f32, + 0.51544732f32, + 0.56694406f32, + 1.48416460f32, + 0.53104258f32, + 0.64682180f32, + 1.16943991f32, + 0.55310392f32, + 0.78815460f32, + 0.97256821f32, + 0.58293498f32, + 1.06067765f32, + 0.83934963f32, + 0.62250412f32, + 1.72244716f32, + 0.74453628f32, + 0.67480832f32, + 5.10114861f32, +]; + +pub static MP3D_SYNTH_G_WIN: [f32; 240] = [ + -1f32, + 26f32, + -31f32, + 208f32, + 218f32, + 401f32, + -519f32, + 2063f32, + 2000f32, + 4788f32, + -5517f32, + 7134f32, + 5959f32, + 35640f32, + -39336f32, + 74992f32, + -1f32, + 24f32, + -35f32, + 202f32, + 222f32, + 347f32, + -581f32, + 2080f32, + 1952f32, + 4425f32, + -5879f32, + 7640f32, + 5288f32, + 33791f32, + -41176f32, + 74856f32, + -1f32, + 21f32, + -38f32, + 196f32, + 225f32, + 294f32, + -645f32, + 2087f32, + 1893f32, + 4063f32, + -6237f32, + 8092f32, + 4561f32, + 31947f32, + -43006f32, + 74630f32, + -1f32, + 19f32, + -41f32, + 190f32, + 227f32, + 244f32, + -711f32, + 2085f32, + 1822f32, + 3705f32, + -6589f32, + 8492f32, + 3776f32, + 30112f32, + -44821f32, + 74313f32, + -1f32, + 17f32, + -45f32, + 183f32, + 228f32, + 197f32, + -779f32, + 2075f32, + 1739f32, + 3351f32, + -6935f32, + 8840f32, + 2935f32, + 28289f32, + -46617f32, + 73908f32, + -1f32, + 16f32, + -49f32, + 176f32, + 228f32, + 153f32, + -848f32, + 2057f32, + 1644f32, + 3004f32, + -7271f32, + 9139f32, + 2037f32, + 26482f32, + -48390f32, + 73415f32, + -2f32, + 14f32, + -53f32, + 169f32, + 227f32, + 111f32, + -919f32, + 2032f32, + 1535f32, + 2663f32, + -7597f32, + 9389f32, + 1082f32, + 24694f32, + -50137f32, + 72835f32, + -2f32, + 13f32, + -58f32, + 161f32, + 224f32, + 72f32, + -991f32, + 2001f32, + 1414f32, + 2330f32, + -7910f32, + 9592f32, + 70f32, + 22929f32, + -51853f32, + 72169f32, + -2f32, + 11f32, + -63f32, + 154f32, + 221f32, + 36f32, + -1064f32, + 1962f32, + 1280f32, + 2006f32, + -8209f32, + 9750f32, + -998f32, + 21189f32, + -53534f32, + 71420f32, + -2f32, + 10f32, + -68f32, + 147f32, + 215f32, + 2f32, + -1137f32, + 1919f32, + 1131f32, + 1692f32, + -8491f32, + 9863f32, + -2122f32, + 19478f32, + -55178f32, + 70590f32, + -3f32, + 9f32, + -73f32, + 139f32, + 208f32, + -29f32, + -1210f32, + 1870f32, + 970f32, + 1388f32, + -8755f32, + 9935f32, + -3300f32, + 17799f32, + -56778f32, + 69679f32, + -3f32, + 8f32, + -79f32, + 132f32, + 200f32, + -57f32, + -1283f32, + 1817f32, + 794f32, + 1095f32, + -8998f32, + 9966f32, + -4533f32, + 16155f32, + -58333f32, + 68692f32, + -4f32, + 7f32, + -85f32, + 125f32, + 189f32, + -83f32, + -1356f32, + 1759f32, + 605f32, + 814f32, + -9219f32, + 9959f32, + -5818f32, + 14548f32, + -59838f32, + 67629f32, + -4f32, + 7f32, + -91f32, + 117f32, + 177f32, + -106f32, + -1428f32, + 1698f32, + 402f32, + 545f32, + -9416f32, + 9916f32, + -7154f32, + 12980f32, + -61289f32, + 66494f32, + -5f32, + 6f32, + -97f32, + 111f32, + 163f32, + -127f32, + -1498f32, + 1634f32, + 185f32, + 288f32, + -9585f32, + 9838f32, + -8540f32, + 11455f32, + -62684f32, + 65290f32, +]; + +pub static L3_LDEXP_Q2_G_EXPFRAC: [f32; 4] = [ + 9.31322575e-10f32, + 7.83145814e-10f32, + 6.58544508e-10f32, + 5.53767716e-10f32, +]; + +pub static L3_IMDCT12_G_TWID3: [f32; 6] = [ + 0.79335334f32, + 0.92387953f32, + 0.99144486f32, + 0.60876143f32, + 0.38268343f32, + 0.13052619f32, +]; + +pub static HDR_SAMPLE_RATE_HZ_G_HZ: [u32; 3] = [ + 44100, + 48000, + 32000, +]; \ No newline at end of file diff --git a/tools/make-character.sh b/tools/make-character.sh index be584011..ca60ace4 100755 --- a/tools/make-character.sh +++ b/tools/make-character.sh @@ -3,7 +3,8 @@ # # Reads characters//voice.env and characters//lines.tsv: # - voice.env sets ENGINE (volc or elevenlabs) and VOICE (the engine's voice id), and for a Doubao -# 2.0 voice VOLC_RESOURCE_ID=seed-tts-2.0. The API keys stay in your environment, never in the repo: +# 2.0 voice VOLC_RESOURCE_ID=seed-tts-2.0, and GENDER (female or male), which picks a stock voice for spoken +# replies when the voice service can't speak the Character's own. The API keys stay in your environment, never in the repo: # VOLC_API_KEY or ELEVENLABS_API_KEY (a paid plan; the free one has no commercial license). # - lines.tsv has one line per row: occasiontext. Blank rows and rows starting with # are skipped. # A line with {address} is said with each form of address of the Character's LANGUAGE (voice.env) diff --git a/tools/make-cjk-font.py b/tools/make-cjk-font.py new file mode 100755 index 00000000..55442d30 --- /dev/null +++ b/tools/make-cjk-font.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Build the firmware's Chinese glyph table, firmware-rs/core/fonts/cjk-12px.bin. + +The box draws ASCII with its own 5x7 font (canvas.rs); everything else it can draw comes from this table: +the 3,500 characters of level one of the Table of General Standard Chinese Characters (通用规范汉字表 一级, +2013), the CJK punctuation and the full-width forms, rasterized by Fusion Pixel Font 12 px (SIL OFL 1.1). + +Both inputs are pinned by URL and SHA-256, so the output is the same on every machine: + - Fusion Pixel Font, 12 px monospaced, BDF, its zh_Hans glyph shapes. Fusion is Ark Pixel Font with the + characters Ark Pixel hasn't drawn yet at 12 px filled in from Cubic 11 (both OFL). + - Unicode's Unihan database, whose kTGH field gives each character's number in the 2013 table; + numbers 1 to 3500 are level one. + +Downloads are cached in ~/.cache/vibebuddy-fonts. Pass --fusion and --unihan to use local copies. + +Output layout (little-endian), read by firmware-rs/core/src/font.rs: + u16 count + u16 codepoint[count], ascending (all in the Basic Multilingual Plane) + 18-byte bitmap[count]: the 12x12 cell, 12 rows of 12 pixels, row-major, most significant bit first. + The cell's baseline is under row 9 (ascent 10, descent 2). + +The font's license files (OFL.txt, LICENSES/) are written next to the table. + +Usage: tools/make-cjk-font.py [--fusion ZIP] [--unihan ZIP] [--output PATH] +""" +import argparse +import hashlib +import io +import struct +import sys +import urllib.request +import zipfile +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +OUTPUT = ROOT / "firmware-rs/core/fonts/cjk-12px.bin" +CACHE = Path.home() / ".cache/vibebuddy-fonts" + +FUSION_VERSION = "2026.09.25" +FUSION_URL = ("https://github.com/TakWolf/fusion-pixel-font/releases/download/" + f"{FUSION_VERSION}/fusion-pixel-font-12px-monospaced-bdf-v{FUSION_VERSION}.zip") +FUSION_SHA256 = "d75f5262f108757edb0f47ee8e3d2dfdfbecfd94558faf5ffb0c06dc5866fb3b" +FUSION_MEMBER = "fusion-pixel-12px-monospaced-zh_hans.bdf" +UNIHAN_URL = "https://www.unicode.org/Public/16.0.0/ucd/Unihan.zip" +UNIHAN_SHA256 = "b8f000df69de7828d21326a2ffea462b04bc7560022989f7cc704f10521ef3e0" +UNIHAN_MEMBER = "Unihan_OtherMappings.txt" + +LEVEL_ONE = 3500 +CELL = 12 +ASCENT = 10 +GLYPH_BYTES = CELL * CELL // 8 + +# Besides the level-one characters: what Chinese text needs around them. Ranges are inclusive; a codepoint +# the font has no full-width glyph for is skipped (the box draws a box for it). +EXTRA_RANGES = [ + (0x3000, 0x303F), # CJK symbols and punctuation: 、。〈〉《》「」『』【】〔〕〜 and the ideographic space + (0xFF01, 0xFF5E), # full-width ASCII forms: ，！？：；（）０-９ Ａ-Ｚ + (0xFFE0, 0xFFE6), # full-width ￠￡￢￣￤￥￦ +] +EXTRA_CHARACTERS = "‘’“”…—―‖‥※·→←↑↓" + + +def fetch(url, sha256, override): + if override: + data = Path(override).read_bytes() + else: + cached = CACHE / url.rsplit("/", 1)[1] + if not cached.exists(): + CACHE.mkdir(parents=True, exist_ok=True) + print(f"downloading {url}", file=sys.stderr) + with urllib.request.urlopen(url) as response: + cached.write_bytes(response.read()) + data = cached.read_bytes() + digest = hashlib.sha256(data).hexdigest() + if digest != sha256: + sys.exit(f"{override or url}: SHA-256 {digest}, expected {sha256}") + return zipfile.ZipFile(io.BytesIO(data)) + + +def level_one(unihan): + characters = [] + for line in unihan.read(UNIHAN_MEMBER).decode().splitlines(): + if not line.startswith("U+"): + continue + codepoint, field, value = line.split("\t") + if field == "kTGH" and int(value.split(":")[1]) <= LEVEL_ONE: + characters.append(int(codepoint[2:], 16)) + if len(characters) != LEVEL_ONE: + sys.exit(f"Unihan lists {len(characters)} level-one characters, expected {LEVEL_ONE}") + return characters + + +def parse_bdf(text): + """codepoint -> (advance, 12x12 cell as rows of 0/1, or None when the glyph reaches outside the cell).""" + lines = iter(text.splitlines()) + glyphs = {} + for line in lines: + if line.startswith("FONT_ASCENT ") and int(line.split()[1]) != ASCENT: + sys.exit(f"expected a font ascent of {ASCENT}: {line}") + if not line.startswith("STARTCHAR"): + continue + codepoint = advance = box = None + for line in lines: + key, _, value = line.partition(" ") + if key == "ENCODING": + codepoint = int(value) + elif key == "DWIDTH": + advance = int(value.split()[0]) + elif key == "BBX": + box = [int(number) for number in value.split()] + elif key == "BITMAP": + break + width, height, x_offset, y_offset = box + stride = (width + 7) // 8 * 8 + cell = [[0] * CELL for _ in range(CELL)] + for row in range(height): + bits = int(next(lines), 16) + for column in range(width): + if bits >> (stride - 1 - column) & 1: + # BDF rows run down from the box's top, which is y_offset + height above the baseline. + cell_row = ASCENT - (y_offset + height) + row + cell_column = x_offset + column + if cell is not None and 0 <= cell_row < CELL and 0 <= cell_column < CELL: + cell[cell_row][cell_column] = 1 + else: + cell = None + if next(lines) != "ENDCHAR": + sys.exit(f"U+{codepoint:04X}: malformed BDF") + glyphs[codepoint] = (advance, cell) + return glyphs + + +def pack(cell): + bits = 0 + for row in cell: + for pixel in row: + bits = bits << 1 | pixel + return bits.to_bytes(GLYPH_BYTES, "big") + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("--fusion", help="local copy of the Fusion Pixel Font BDF zip") + parser.add_argument("--unihan", help="local copy of Unihan.zip") + parser.add_argument("--output", type=Path, default=OUTPUT) + arguments = parser.parse_args() + + fusion = fetch(FUSION_URL, FUSION_SHA256, arguments.fusion) + glyphs = parse_bdf(fusion.read(FUSION_MEMBER).decode()) + hanzi = level_one(fetch(UNIHAN_URL, UNIHAN_SHA256, arguments.unihan)) + unusable = [codepoint for codepoint in hanzi + if codepoint not in glyphs or glyphs[codepoint][0] != CELL or glyphs[codepoint][1] is None] + if unusable: + sys.exit("no full-width 12x12 glyph for level-one characters: " + "".join(map(chr, unusable))) + + extras = [codepoint for first, last in EXTRA_RANGES for codepoint in range(first, last + 1)] + extras += [ord(character) for character in EXTRA_CHARACTERS] + # Only full-width glyphs inside the cell: every glyph in the table advances 12 px. The rest are left to + # the fallback box. + extras = [codepoint for codepoint in extras + if codepoint in glyphs and glyphs[codepoint][0] == CELL and glyphs[codepoint][1] is not None] + codepoints = sorted(set(hanzi) | set(extras)) + if codepoints[-1] > 0xFFFF: + sys.exit("the table's index holds 16-bit codepoints only") + + table = bytearray(struct.pack(" {arguments.output}") + if skipped: + print("not full-width in the font, left to the fallback box: " + "".join(map(chr, skipped))) + + +if __name__ == "__main__": + main() diff --git a/tools/mic-record.py b/tools/mic-record.py new file mode 100755 index 00000000..6e63a1dd --- /dev/null +++ b/tools/mic-record.py @@ -0,0 +1,153 @@ +#!/usr/bin/env python3 +"""Microphone bring-up check: send `device.mic.record` to a box, collect the recording, write a WAV and say +whether the microphone works. On this board the microphone is the left slot (the ES8311's ADC); the right +slot is the speaker's reference, silent while the box doesn't speak (docs/hardware.md, 2026-10-13). + +Needs exclusive use of the serial port, so quit the VibeBuddy app first. The port must be named: the box may +share a Mac with other ESP32-S3 devices, which enumerate the same way (LESSONS.md). + +Usage: + uv run --with pyserial python tools/mic-record.py --port /dev/cu.usbmodemXXXX [--ms 3000] [--gain-db 30] [--out mic.wav] + +Talk, clap or tap near the box while it records. The firmware sends the samples back after recording, +about 11 s per recorded second (the UART is always written at 115200 baud, whichever port you use). +See docs/firmware-bringup.md, "Microphone", for what the output means. +""" + +import argparse +import base64 +import json +import math +import sys +import time +import wave +import zlib + +import serial + +FULL_SCALE = 32768 +WINDOW_MS = 100 +# Quieter than this in every window is no signal, whatever the spread: a lone blip over digital silence +# (the right slot on 2026-10-13 went -71 then -120 dBFS) is not a microphone. +SIGNAL_DBFS = -70 + + +def dbfs(value): + return -120.0 if value <= 0 else 20 * math.log10(value / FULL_SCALE) + + +def slot_windows(samples, channels, slot, rate): + """One slot's samples with its DC offset removed, cut into 100 ms windows; the first window is dropped, + since the receiver starts mid-stream and the codec may still be settling.""" + values = [int.from_bytes(samples[index:index + 2], "little", signed=True) for index in range(slot * 2, len(samples), channels * 2)] + offset = sum(values) / max(len(values), 1) + size = rate * WINDOW_MS // 1000 + windows = [values[start:start + size] for start in range(size, len(values) - size + 1, size)] + return offset, [(math.sqrt(sum((value - offset) ** 2 for value in window) / len(window)), max(abs(value) for value in window)) for window in windows] + + +def verdict(windows): + """What one slot's windows (RMS without DC, raw peak) say about it.""" + if not windows: + return "TOO SHORT (record at least 300 ms)" + rms = [window[0] for window in windows] + peaks = [window[1] for window in windows] + if max(peaks) == 0: + return "SILENT (all zeros)" + if sum(peak >= FULL_SCALE - 1 for peak in peaks) > len(peaks) // 2: + return "SATURATED (pinned at full scale in most windows: gain too high or a floating pin)" + loudest = dbfs(max(rms)) + if loudest < SIGNAL_DBFS: + return f"NO SIGNAL (loudest window {loudest:.1f} dBFS)" + spread = loudest - dbfs(max(min(rms), 1)) + if spread < 6: + return f"FLAT (level barely moves, {spread:.1f} dB: noise floor only; make some noise and retry)" + return f"LIVE (loudest window {loudest:.1f} dBFS, {spread:.1f} dB above the quietest)" + + +def main(): + parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + parser.add_argument("--port", required=True, help="serial port of the box, e.g. /dev/cu.usbmodem8401") + parser.add_argument("--ms", type=int, default=3000, help="recording length, 100 to 5000 ms (default 3000)") + parser.add_argument("--gain-db", type=int, default=None, help="ES8311 microphone gain, 0 to 42 dB in 6 dB steps (firmware default 30)") + parser.add_argument("--out", default="mic.wav", help="WAV file to write (default mic.wav)") + args = parser.parse_args() + + fields = {"version": 1, "event": "device.mic.record", "ms": args.ms} + if args.gain_db is not None: + fields["gain_db"] = args.gain_db + line = json.dumps(fields, separators=(",", ":")).encode() + b"\n" + + port = serial.Serial(args.port, 115200, timeout=0.5) + port.reset_input_buffer() + # Short enough for the UART bridge in one piece (it corrupts writes over about 200 bytes). + port.write(line) + + rate = channels = bits = size = None + chunks = [] + levels = [] + end = None + deadline = time.monotonic() + 15 + while time.monotonic() < deadline: + text = port.readline().decode("utf-8", "replace").strip() + if not text.startswith("MIC "): + continue + words = text.split(" ") + kind = words[1] + if kind == "DATA": + if int(words[2]) != len(chunks): + print(f"missing data: got line {words[2]}, expected {len(chunks)}", file=sys.stderr) + return 1 + chunks.append(base64.b64decode(words[3])) + deadline = time.monotonic() + 5 + continue + print(text) + if kind in ("UNAVAILABLE", "BUSY", "ERROR"): + return 1 + if kind in ("RECORDING", "CAPTURED"): + deadline = time.monotonic() + 10 + elif kind == "LEVEL": + levels.append(((int(words[4]), int(words[5])), (int(words[7]), int(words[8])))) + elif kind == "BEGIN": + rate, channels, bits, size = (int(word) for word in words[2:6]) + # About 11 s per recorded second over the UART, plus margin. + deadline = time.monotonic() + 30 + size / 8000 + elif kind == "END": + end = (int(words[2]), int(words[3])) + break + port.close() + + if end is None: + print(f"no MIC END before the timeout ({len(chunks)} data lines received)", file=sys.stderr) + return 1 + samples = b"".join(chunks) + if len(samples) != end[0] or zlib.crc32(samples) != end[1]: + print(f"recording damaged on the way: {len(samples)} bytes, CRC {zlib.crc32(samples)}; box sent {end[0]} bytes, CRC {end[1]}", file=sys.stderr) + return 1 + + with wave.open(args.out, "wb") as output: + output.setnchannels(channels) + output.setsampwidth(bits // 8) + output.setframerate(rate) + output.writeframes(samples) + print(f"wrote {args.out}: {len(samples) // (channels * bits // 8) / rate:.2f} s, {rate} Hz, {channels} ch, {bits} bit") + + if levels: + print("\n time left rms/peak (dBFS) right rms/peak (dBFS)") + for index, (left, right) in enumerate(levels): + print(f"{index * 100:5d} ms {dbfs(left[0]):6.1f} / {dbfs(left[1]):6.1f} {dbfs(right[0]):6.1f} / {dbfs(right[1]):6.1f}") + # The verdict comes from the samples themselves, DC removed and the first window dropped; the box's + # LEVEL lines above include DC and the start-up window. + slots = [slot_windows(samples, channels, slot, rate) for slot in range(channels)] + print() + for name, (offset, windows) in zip(("left (microphone)", "right (speaker reference)"), slots): + print(f"{name}: {verdict(windows)}; DC offset {offset:.0f}") + live = verdict(slots[0][1]).startswith("LIVE") + if verdict(slots[1][1]).startswith("LIVE"): + print("note: the right slot moves too; fine if the box spoke while recording, otherwise the slots may be swapped") + print(f"\nmicrophone on this box: {'WORKS' if live else 'NOT SHOWN'} (listen to the left channel of {args.out} to be sure)") + return 0 if live else 3 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/package-licenses.py b/tools/package-licenses.py index 3fec7b5d..7d8cc392 100644 --- a/tools/package-licenses.py +++ b/tools/package-licenses.py @@ -59,6 +59,11 @@ def main(): (licenses / "THIRD-PARTY-FIRMWARE.txt").write_text( dependency_notices("firmware-rs/device/Cargo.toml", "xtensa-esp32s3-none-elf")) shutil.copyfile(ROOT / "packaging/BOOTLOADER-NOTICES.txt", licenses / "BOOTLOADER-NOTICES.txt") + # The firmware's Chinese glyphs come from Fusion Pixel Font (OFL), whose licenses travel with them. + fonts = ROOT / "firmware-rs/core/fonts" + (licenses / "FONTS.txt").write_text("\n\n".join( + f"{path.relative_to(fonts)}\n{'=' * len(str(path.relative_to(fonts)))}\n\n{path.read_text()}" + for path in sorted(fonts.rglob("*.txt")))) else: (licenses / "THIRD-PARTY-APP.txt").write_text(dependency_notices("Cargo.toml", app_target)) revision = run("git", "rev-parse", "HEAD")