docs(design): processors, and libraries of sounds and effects

docs/src/SUMMARY.md

@@ -94,6 +94,8 @@
 - [0007 Which samples we ship](design/0007-sample-licensing.md)
 - [0008 egui for native and web](design/0008-egui.md)
 - [0009 Releases through pull requests](design/0009-releases.md)
+- [0010 Processors: effects with several tracks in](design/0010-processors.md)
+- [0011 Libraries of sounds and effects](design/0011-libraries.md)
 
 # Contributing
 

docs/src/design/0010-processors.md

@@ -0,0 +1,110 @@
+# 0010 Processors: effects with several tracks in
+
+Status: proposed
+
+## Context
+
+Every effect in claudius works on the audio of one track, a bus, a return or the master. The only signal that crosses
+from one track to another is the sidechain duck, and that is built from the kick's notes, not its sound. Producers
+reach for more: a pad gated by the hi-hats, chords pumped by the real kick, a bass whose filter opens with the drums'
+envelope, strings that crossfade into a pad over a build, a synth vocoded by a voice, several tracks mixed with gains
+that move through the song.
+
+What is asked for is effects as functions: each takes up to some number of tracks, has parameters a person turns as
+knobs and that can follow time, and gives one track back, which then mixes like any other. They have to be available
+wherever the rest of claudius is: scripts, the studio and MCP.
+
+The constraints:
+
+- Rendering is offline, over whole-song buffers: a four-minute stereo track is about 96 MB at 48 kHz.
+- Scripts are Rhai, an interpreter: far too slow to call per sample, and its closures cannot be stored. A song has to
+  print back to a script that rebuilds it exactly ([printing](../language/printing.md)), and saved songs have to keep
+  opening in later releases ([0003](0003-blueprint.md)).
+- The engine already stores parameters that change over time (`Param`: a constant or (beat, value) points), but only
+  the arranger writes them. The compressor already takes a sidechain key, which nothing passes.
+
+## Decision
+
+**A processor is a named node of the blueprint** (`Blueprint.processors`). It has:
+
+- a `name`, which is the name of the track it outputs;
+- its `inputs`: tracks of the song, or other processors;
+- a graph of operations from a built-in set, over those inputs;
+- `knobs`: named parameters with a range and a default, which the graph reads;
+- the output's `mix`: level, pan, sends, bus and inserts, as any track has.
+
+**Scripts define one with a function whose arguments are its inputs.** The number of arguments is the number of inputs,
+up to eight:
+
+```js
+process("pump", ["chords", "kick"], |chords, kick| chords.duck(kick, knob("depth", 0.7, 0, 1)));
+process("swell", ["pad", "strings"], |pad, strings| pad.crossfade(strings, at(#{ intro: 0, drop: 1 })));
+process("talk", ["chords", "vox"], |chords, vox| chords.vocode(vox, #{ bands: 24 }).gain(-3));
+```
+
+The function runs once, when the script runs, on stand-ins for the tracks. What it does to them is recorded as a graph,
+the way a pattern function is printed by running it on a placeholder. The graph is saved; the function is not, so the
+song prints back as `process(...)` lines that rebuild it.
+
+An effect saved in a library ([0011](0011-libraries.md)) takes the function's place:
+`process("pump", ["chords", "kick"], "mine:pump")`.
+
+**Values** wherever an operation takes a number:
+
+- a number;
+- a knob, `knob(name, default, min, max)`, which the studio shows and MCP can set;
+- a curve over the song's beats, `curve([[0, 0.2], [64, 1]])`;
+- a value for each section, `at(#{ intro: 0.2, drop: 1 })`;
+- a pattern in mini-notation, one cycle a bar, `"0 0.5 1 0.5"`;
+- a function of the beat, `|beat| 0.5 + 0.5 * sin(beat / 8.0)`, sampled into a curve when the script runs;
+- another signal's envelope, `kick.envelope(5, 120)`.
+
+**The operations** start with:
+
+- `gain`, `pan`, `lpf`, `hpf`, `bpf`, `saturate`, `delay`, and `reverb`, with a recorded space or the generated one;
+- `compress(key, ...)`, `duck(key, depth)`, `gate(key, depth)`;
+- `mix(other, gain)`, `crossfade(other, position)`, `ringmod(other)`, `vocode(modulator, ...)`;
+- `envelope(attack, release)`;
+- `convolve(other, seconds)`, capped in length.
+
+`catalog` lists them, each with its arguments and their ranges.
+
+**Graphs stream**, block by block, in the runtime that [0011](0011-libraries.md) describes. Songs render offline
+first; the studio plays processors live later.
+
+**Rendering** splits the per-track pass in three:
+
+1. Every track's source and insert effects: its signal before the fader.
+2. The processors, in the order of their inputs. A cycle is refused when the script runs. Each processor reads the
+   pre-fader signals of its inputs.
+3. Level, duck, pan, sends and bus, for each track that still plays and for each processor's output.
+
+A track a processor processes no longer plays on its own, unless it is marked `keep`. Keys do: the inputs of `duck`,
+`gate`, `compress` and `envelope` that only steer.
+
+**The studio** gives each processor a strip in the mixer, with its knobs. A turned knob sends `set_knob`, done as a
+script call like the mixer's other edits. **MCP** builds processors through `run_script`; `catalog` lists the operations
+and the recorded spaces; `get_song` lists each processor with its inputs and knobs; and `command` takes `set_knob`.
+
+**Saved songs.** `processors` has a default, so songs without processors stay in blueprint format 1. A song with
+processors is written in format 2. An older claudius would otherwise drop the field silently and play the raw inputs;
+with format 2, it refuses the song with its "update claudius to open it" message.
+
+Rejected:
+
+- **Functions called per sample**: about 10^7 interpreter calls per track and second of audio.
+- **Only a fixed list of processor kinds** (duck, crossfade, vocoder): it cannot express the mixing the request is
+  about.
+- **A graph of tracks edited in the studio**: the script is where songs are built, and a graph editor would be a second
+  language.
+
+## Consequences
+
+- **Memory.** A processor's output is one more track buffer, and its inputs stay alive until the processors have run.
+  `MAX_TRACKS` counts processors.
+- **Convolution.** Convolving one track with another needs partitioned convolution, or a short cap on the kernel:
+  today's `convolve` sizes one FFT to the whole kernel.
+- **Slices and evolution.** `Project::slice`, which previews and evolution use, shifts processor curves as it does
+  filter curves. Mutation and evolution leave processors alone at first.
+- **The documentation grows.** The language pages get a page on processors; the mixing pages, the script reference and
+  the songs skill describe them; `catalog` documents each operation.

docs/src/design/0011-libraries.md

@@ -0,0 +1,167 @@
+# 0011 Libraries of sounds and effects, made, brought and shared
+
+Status: proposed
+
+## Context
+
+Today claudius plays its own synth presets and effects, the curated packs in `registry.json` ([0007](0007-sample-licensing.md)),
+Strudel sample maps added with `claudius-packs add-map`, the person's sample folders, and Apple Loops. People want more
+than that:
+
+- to bring their own sounds: recordings they made, or processed sounds they are proud of;
+- to make sounds and effects in claudius itself, and keep them;
+- to find what others made in a marketplace, free for now, and to publish their own there;
+- to keep some libraries for themselves only.
+
+A sound is either recorded audio, or a procedure that makes it. An effect is a procedure over up to N input tracks
+with one output ([0010](0010-processors.md)), which should be able to stream in real time.
+
+The constraints:
+
+- **Speed.** Scripts are Rhai, an interpreter, far too slow per sample, as 0010 explains.
+- **Strudel.** Strudel patterns run in claudius; using sounds has to stay Strudel's way.
+- **Downloads.** A downloaded library must not be able to do anything but make sound.
+- **Saved songs.** They have to keep opening.
+- **Publishing.** Licences matter more once anyone can publish.
+
+## Decision
+
+**One format, the library.** A library is a folder, or a `.zip` of one, with a `library.json` manifest: name, version,
+author, licence and description, its files, and its items. An item is one of:
+
+- **a recorded sound**: audio files and their mapping (root notes, velocity layers, round robins, loop points), as the
+  packs' instruments have today;
+- **a procedural sound**: a recorded graph of synthesis blocks, played per note with the note's pitch, velocity and
+  length, and with knobs;
+- **an effect**: a recorded graph over 1 to 8 inputs, with knobs, as in 0010.
+
+The curated packs are read as libraries too, so there is one loader and one resolver.
+
+**Definitions are data.** Sounds and effects are defined in the script language, by a function that runs once on
+stand-ins and is recorded as a graph, the way 0010 defines processors:
+
+```js
+sound("mine:glass", |note| osc("sine", note).fm(osc("sine", note * 3.5), knob("bite", 2, 0, 8))
+    .env(2, 900).lpf(knob("tone", 6000, 200, 16000)));
+effect("mine:pump", |music, kick| music.duck(kick, knob("depth", 0.7, 0, 1)));
+```
+
+A library therefore holds audio and graphs, not code. Installing one cannot run anything; it can only make sound.
+Graphs have size limits, as scripts do. The graph's blocks are native Rust, so a procedural sound costs what a built-in
+preset costs.
+
+**Graphs stream.** The graph runtime processes blocks of samples (128 by default), keeping each block's state, rather
+than whole-song buffers:
+
+- An offline render runs a graph block by block over the song.
+- The studio can run the same graph live: audition a sound, or turn an effect's knob and hear it at once, without a new
+  render.
+- Memory stays bounded by the block, not the song.
+
+Today's whole-buffer effects keep working. They move to blocks one at a time, each move held to the tests below.
+
+**Using sounds stays Strudel's.** A library sound plays with `s("mine:glass")`, `.bank("mine")` and `n()`.
+`samples("github:user/repo")` and `samples(url)` load a Strudel sample map as a library, which is what `add-map` does
+today, so Strudel code that brings its own samples runs in claudius. The definitions (`sound`, `effect`, `process`) are
+claudius' own; the docs mark them as extensions to Strudel.
+
+**Where libraries live:**
+
+- **Personal libraries** are folders in `~/Music/claudius/libraries/`. They are made with `claudius library new`, with
+  the studio's *Save as sound* or *Save as effect*, or by a script's `library("mine")`. They stay on the computer.
+- **Installed libraries** come from the marketplace, a URL or GitHub, into claudius' data folder. They are pinned to a
+  version and its SHA-256, like the packs.
+- **Songs** name sounds and effects as `library:item@version`, and keep a copy of the graphs they use. A song still
+  plays when a library changes or goes away; its audio files are fetched again when needed.
+
+**The marketplace** is a small service, free for now:
+
+- a Cloudflare Worker for the API, R2 for the files and D1 for the index;
+- files reached only through the API, so that they can move without a change to claudius. R2 charges $15 per TB a
+  month with free downloads; past about a terabyte, Backblaze B2 behind Cloudflare costs less than half that, its
+  downloads also free;
+- sign-in with an emailed link, with no password, for publishing and for private libraries. Searching and installing
+  public libraries need no account. The command line signs in with a short code confirmed in the browser;
+- each published version is immutable and named by its content hash.
+
+Each library has a visibility:
+
+- *public*: listed and searchable;
+- *unlisted*: anyone with the link;
+- *private*: the owner only. This is also how a person carries personal libraries to another computer.
+
+The command line has `claudius library search`, `install`, `publish` and `remove`. The studio has a library browser.
+MCP has the same, through `run_script` and `catalog`, plus commands for searching and installing.
+
+**Licences and safety:**
+
+- Publishing asks for a licence (CC0, CC BY, CC BY-SA, personal use only, or the author's own text) and for the
+  author's word that they have the right to publish.
+- The index shows the licence, and claudius' `credits.txt` covers library sounds as it covers packs.
+- Uploads are checked: audio formats claudius decodes, sizes and lengths within limits, and safe names (no path
+  traversal, no zip bombs).
+- Samples whose terms forbid passing them on are never uploaded or hosted: the curated packs with such terms (the
+  TR-909 pack, SampleRadar's) stay fetched from their publishers on each computer, as 0007 decided.
+- A reported library is hidden while it is looked at, and a takedown request is answered by removing it.
+
+**Tests come first.** Before the engine changes, regression nets go in at every level of the pyramid. Each change in
+the work that follows then shows what it did to the sound:
+
+1. **The sound of today's songs.** Short synth-only renders of each genre, with their metrics: loudness, true peak,
+   energy per band, and a hash of the samples on each platform. A change meant to be bit-exact must keep the hash. A
+   change meant to alter the sound updates the metrics on purpose, in review. Null tests compare two engines sample by
+   sample.
+2. **Every DSP block and effect.**
+   - Impulse and frequency responses, and envelope timing, measured against expected values.
+   - Stability: no NaN, infinity or denormals.
+   - The same output in blocks of any size as over the whole buffer.
+3. **Properties.** Random graphs give finite, bounded and deterministic output whatever the block size.
+4. **The language.**
+   - Definitions print and rebuild the same graph.
+   - A corpus of Strudel snippets has to parse and play.
+   - Errors say what to do.
+5. **The format.**
+   - Manifests across versions, with fixtures, as blueprints have.
+   - Name resolution and its precedence, and credits.
+   - Safe unpacking, and the manifest parser under fuzzing.
+6. **Integration.**
+   - The server and MCP against a fake marketplace.
+   - The command line end to end.
+   - The studio's library browser through `CLAUDIUS_UI_PLAN`.
+7. **The marketplace service.**
+   - The Worker's tests: sign-in, visibility, immutability, quotas.
+   - A contract test of the schema claudius and the Worker share.
+8. **Performance budgets.** Benchmarks with limits: a graph's throughput as a multiple of real time, render time per
+   genre, and peak memory, checked in CI with a margin.
+
+**In this order:**
+
+1. The tests above.
+2. The format, personal libraries, and bringing sounds (folders, `samples()`).
+3. The graph runtime, with effects and procedural sounds, rendering offline.
+4. The studio's *Save as* and the library browser.
+5. The marketplace.
+6. Live playback of effects and procedural sounds in the studio.
+
+Rejected:
+
+- **Native plugins (VST3, CLAP, AU) inside libraries.** Untrusted native code, and a build for each platform.
+- **Script functions called per sample.** Too slow, as in 0010.
+- **A marketplace made only of a GitHub index.** Publishing waits for a pull request, and there is no private
+  visibility.
+- **Sign-in with GitHub.** Musicians and DJs rarely have a GitHub account; everyone has an email address.
+- **Libraries in the claudius repository.** The redistribution and size problems of 0007.
+
+Kept for later: WebAssembly modules for DSP the graph's blocks cannot express. Sandboxed, portable and fast, they would
+be the one kind of code a library could carry.
+
+## Consequences
+
+- **A service to run.** The marketplace has an operator: storage costs past R2's free tier, a service that sends the
+  sign-in emails, moderation, takedowns, and backups of D1.
+- **Licences are the authors'.** The marketplace states each one, claudius credits it, and nobody reviews the audio
+  before it is published.
+- **Songs carry graphs.** A song that uses a library carries the graphs it needs, so the blueprint gets a format 2 when
+  it does ([0010](0010-processors.md)).
+- **Effects are rewritten for streaming.** Every effect moves to blocks; the tests in step 1 keep each move honest.
+- **Live playback** asks more of the studio's audio path than playing a file. That is why it is the last step.