fix(locallibrary): keep rekordbox's demo samples out of backups

apps/locallibrary/GLOSSARY.md

@@ -127,6 +127,21 @@ _Avoid_: stick, device, volume, USB
 A saved copy of a drive's library in the store.
 _Avoid_: backup, version, commit
 
+**Backup**:
+What a change to a drive keeps on that drive so `undo` can put it back: the databases and the files
+the change replaces or deletes.
+_Avoid_: snapshot
+
+**Demo sample**:
+One of the tracks rekordbox ships in its collection and puts back with every export. `clean` and
+`refresh` remove them.
+_Avoid_: bundled track, sample track
+
+**Unbacked file**:
+A file a change deleted and left out of its backup, because rekordbox's next export puts it back.
+Only demo samples' files are. Undoing the change brings the track back without the file.
+_Avoid_: skipped file, excluded file
+
 **History**:
 A play history list a player recorded.
 _Avoid_: session, play log

apps/locallibrary/changes/loc-176-demo-backups.md

@@ -0,0 +1,5 @@
+### Fixed
+
+- `clean` and `refresh` no longer copy rekordbox's demo samples into a backup, so those backups
+  stop piling up, and a drive too full for that copy can be cleaned. `undo` brings the tracks back
+  without their files and says which, until the next export.

apps/locallibrary/docs/adr/0003-demo-samples-stay-out-of-backups.md

@@ -0,0 +1,31 @@
+# Demo samples stay out of backups, so undoing a clean can't restore them
+
+Every change to a drive in place is backed up with every file it deletes, which is what makes
+`undo` whole. `clean` is the exception. rekordbox puts its demo samples back with every export and
+`refresh` removes them after each one, so backing them up kept another copy of the same audio on
+the drive each time, and a drive too full for that copy couldn't be cleaned at all. A clean's
+backup holds only the databases, and undoing it brings the tracks and their playlist entries back
+without their files. The next export brings the files back, or another `clean` removes the tracks.
+
+`delete_unbacked` is its own edit, so the journal carries it and `edits replay` behaves as `clean`
+does. Only plugins locallibrary ships can ask for it (`library:delete-unbacked`), and `edit.plan`
+refuses it, since it deletes what no backup can bring back and a capability means little if the
+sidecar goes around it. A plan that deletes anything else as well backs up everything it deletes.
+Undo says which tracks come back without audio instead of refusing, because the refusal's `--force`
+also overrides "the drive changed since", which is the check worth keeping.
+
+A journal holding the edit is version 5, so builds from before it refuse that drive's journal.
+
+## Considered options
+
+- A flag on `delete_tracks`: older builds could still read the journal, but `keep_files` and the
+  flag together mean nothing, and every place that deletes tracks would have to say it doesn't
+  want it.
+- Deciding per operation, in the plugin's plan or its manifest: the journal wouldn't carry it, so
+  `edits replay` after an export would pile up the same audio.
+- Refusing to undo a clean unless forced: a clean is undone so rarely that the refusal would mostly
+  stop people stepping back to a change before it.
+- Letting sidecar clients ask for it, as they're ours today: that stops being true once plugins or
+  third parties speak RPC.
+- Removing the tracks from the database again after undoing: undo would no longer put back exactly
+  what the backup holds.

apps/locallibrary/docs/commands.md

@@ -113,7 +113,7 @@ its drives.
 | `locallibrary drive space [<drive>]` / `drive tidy <drive>` / `drive recordings [<drive>]` | What's using a drive and when it fills; remove OS clutter; copy off set recordings, see [Space](drives.md#space) |
 | `locallibrary drive speed <drive> [--measure]` | How fast a drive writes, which plans estimate their time from, see [Speed](drives.md#speed) |
 | `locallibrary drive firmware <drive> [file.UPD] [--remove]` | Stage a player or mixer firmware update, see [Setting up and handing over](drives.md#setting-up-and-handing-over) |
-| `locallibrary drive backups [<drive>]` / `drive restore <library> \| <drive>` | The backups taken before every change to a library made in place; put any of them back (`--backup ID`; `--dry-run` shows what would change; refuses a backup whose audio has since been removed, unless `--force`). `--playlist P` brings back one playlist or folder from a snapshot or backup, see [Bringing back a playlist](editing.md#bringing-back-a-playlist) |
+| `locallibrary drive backups [<drive>]` / `drive restore <library> \| <drive>` | The backups taken before every change to a library made in place; put any of them back (`--backup ID`; `--dry-run` shows what would change; refuses a backup whose audio has since been removed, unless `--force`; a `clean`'s backup doesn't hold the demo samples, so those tracks come back without audio, and it says so). `--playlist P` brings back one playlist or folder from a snapshot or backup, see [Bringing back a playlist](editing.md#bringing-back-a-playlist) |
 | `locallibrary provision <target>` | Set up a new drive in one step, see [Setting up and handing over](drives.md#setting-up-and-handing-over) |
 | `locallibrary doctor [<library>] [--hash-audio] [--by-sound] [--audio] [--analyze] [--only SECTION,...]` | Health report: duplicates with a suggested keeper (`--by-sound`: also the same recording in another file or encoding), missing or cut-short files, orphaned files, tracks in no playlist, database drift, files over 4 GB, how players read the drive, what plugins' checks find (`--audio`: audio files cut short or that don't decode; `--analyze`: tempos and keys an analysis of the audio disagrees with, and tracks too loud, clipped or far from the library's level); `--fix-orphans` deletes orphans after a backup; `--fix-tempo` doubles or halves the beat grids of tempos that are half or double |
 | `locallibrary analysis bench [<library> \| <drive>] [--keys PATH] [--tempos DIR] [--audio DIR]` | Score rekordbox's stored keys and tempos, and the analysis `doctor --analyze` runs, against the GiantSteps datasets (`--keys`, `--tempos` or both), see [Checking the analysis](drives.md#checking-the-analysis) |
@@ -161,7 +161,7 @@ its drives.
 | `locallibrary merge <source> <target>` | Add the source's missing tracks and playlists to the target library (`--history`: and its history playlists) |
 | `locallibrary pack <library> <drive> --playlist P` | A drive for a set holding only some playlists, with `--transcode` as copies its players play, see [Drives for a set](editing.md#drives-for-a-set) |
 | `locallibrary convert <library> --to onelibrary\|device-library` | Build OneLibrary from `export.pdb` or the reverse; one the drive has gains what it lacks (`--replace` rebuilds it), see [Convert](editing.md#convert) |
-| `locallibrary clean <library>` | Remove rekordbox's bundled demo tracks (which pile up with every export) with their files, analysis and playlist entries: the demos plugin's, see [Clean](editing.md#clean) |
+| `locallibrary clean <library>` | Remove rekordbox's bundled demo tracks (which pile up with every export) with their files, analysis and playlist entries: the demos plugin's, see [Clean](editing.md#clean). `undo` brings back the tracks, not their files, which the next export does |
 | `locallibrary settings [<drive>] ...` | The settings profile, or a drive's settings; `settings export [FILE]` and `settings import FILE` take everything set here (the player's preferences and plugins' settings included) to another computer, see [Settings](settings.md) |
 
 **Plays and players**

apps/locallibrary/docs/diagrams/changing-drives.md

@@ -87,7 +87,7 @@ flowchart LR
     subgraph before["Before the swap"]
         direction TB
         lock["Lock the drive,<br/>check its databases"] --> build["Build and read back<br/>in memory"]
-        build --> backup["Back up what changes, but<br/>for repairs, firmware, recordings"]
+        build --> backup["Back up what changes, but for repairs,<br/>firmware, recordings, demo samples' files"]
         backup --> marker["Mark it as being written"]
         marker --> stage["Stage files and databases,<br/>flush, set times"]
     end
@@ -232,9 +232,9 @@ How `undo` picks a backup and puts it back.
 
 ```mermaid
 flowchart TD
-    pick["undo picks the last change's<br/>backup the drive still holds"] --> changed{"Drive changed since,<br/>or audio gone?"}
+    pick["undo picks the last change's<br/>backup the drive still holds"] --> changed{"Drive changed since, or audio<br/>gone that no backup left out?"}
     changed -->|"yes, no --force"| refused["Refused"]
-    changed -->|"no, or --force"| plan["Plan: files, playlists, history lists<br/>dropped, journal entries taken off"]
+    changed -->|"no, or --force"| plan["Plan: files, playlists, history lists<br/>dropped, journal entries taken off,<br/>tracks coming back without audio"]
     plan --> redo["Once confirmed, back up what it<br/>replaces: putting that back is the redo"]
     redo --> write["Put the backup back through<br/>the one write routine"]
     write --> journal["Put the backup's journal back"]

apps/locallibrary/docs/editing.md

@@ -447,14 +447,21 @@ rekordbox's collection ships with demo samples ("rekordbox - House 1", "Techno 1
 and they end up on drives, often once per export (`House1.wav`, `House1-1.wav`, `House1-2.wav`...).
 `clean` is the demos plugin's (see [Plugins](plugins.md)): it deletes every track on its list,
 from `export.pdb` and OneLibrary's `exportLibrary.db` (one only OneLibrary lists too), as an edit
-(`delete_tracks`) along with its playlist and history entries, audio, analysis files and artwork,
+(`delete_unbacked`) along with its playlist and history entries, audio, analysis files and artwork,
 plus artists, albums, genres, labels, keys and artwork rows nothing else uses. Files another track
 (in either database) still points at are kept, and a demo track sharing its audio file with
 another track stays, as deleting the file's rows would delete that track too. The list lives in
 [the plugin](../../../plugins/demos/src/lib.rs) (`DEMO_TRACKS`), whose check lists them in `check`
 and `doctor`. The next export brings them back unless they're removed from the rekordbox collection
 too; `refresh` removes them first.
 
+A clean's backup holds only the databases: every export brings the samples back, so keeping a copy
+after each `refresh` would only fill the drive, and a drive too full to copy them can still be
+cleaned. `locallibrary undo` can't restore those files. It brings the tracks and their playlist
+entries back without them, says which, and they stay that way until rekordbox's next export or
+another `clean`. `drive backups` counts the files a backup left out. `edits replay` leaves them
+out too, but a clean an earlier version recorded is replayed as it was made, files backed up.
+
 ## Convert
 
 ```bash

apps/locallibrary/docs/how-it-works.md

@@ -56,7 +56,7 @@ plugins/coverart/           the coverart plugin: front covers and track details
                             all, asked to), each album looked up on MusicBrainz and its cover fetched from
                             the Cover Art Archive, offered as candidates locallibrary plans the edits of
 plugins/demos/              the demos plugin: rekordbox's demo tracks on a drive, found by its check and
-                            removed by `clean` as one `delete_tracks` edit
+                            removed by `clean` as one `delete_unbacked` edit
 plugins/interop/            the interop plugin: exporters for Traktor, Serato, Engine DJ and rekordbox XML,
                             writing a drive's library with loclib-interop on this computer, and an importer
                             and a library source reading Traktor's collection.nml back
@@ -297,7 +297,9 @@ apps/locallibrary/docs/rpc/ the sidecar's contract: schema.json and locallibrary
   packs: `locallibrary undo` puts back the backup of the last change the drive still holds, and run
   again, the one before (`drive restore --backup ID` puts back any). Backups include every file a
   change overwrote or deleted (artwork images, analysis files, settings files, removed tracks'
-  files), with their modification times. Each notes what its change left, and putting it back is
+  files), with their modification times; a `clean`'s hold only the databases, as rekordbox's next
+  export puts the demo samples back, so undoing one brings those tracks back without their files
+  and says so. Each notes what its change left, and putting it back is
   refused, unless `--force`, when the drive changed since: a rekordbox export, a sync or push, or
   history players recorded would be lost too. Only the drive a change was made on is backed up:
   copying it onto a library's other drives isn't, and an undo is copied onto them like any change.

apps/locallibrary/docs/plugins.md

@@ -31,7 +31,9 @@ are:
   Traktor](other-software.md#from-traktor).
 - **demos**: `locallibrary clean LIBRARY` removes rekordbox's demo tracks and their files (see
   [Editing](editing.md#clean)), and its check lists them in `check` and `doctor`, with `clean` as
-  the fix. `refresh` runs it first.
+  the fix. `refresh` runs it first. Its `delete_unbacked` edit leaves the files out of the backup,
+  which only plugins locallibrary ship can ask for (`library:delete-unbacked`), and `edit.plan`
+  refuses.
 - **coverart**: finds front covers on MusicBrainz and the Cover Art Archive for `locallibrary art
   fetch`, and track details on MusicBrainz for `locallibrary track fetch` (see
   [Providers](#providers)).

apps/locallibrary/docs/rpc/locallibrary.d.ts

@@ -629,6 +629,16 @@ export type Edit =
     tracks: TrackSelector;
     keep_files?: boolean;
   }
+  /**
+   * Delete tracks as `delete_tracks` does, leaving their files out of the backup: for tracks
+   * rekordbox's next export puts back, whose copies would only fill the drive. Undoing brings
+   * the tracks back without their files, until that export. Only a plugin with the
+   * `library:delete-unbacked` capability can ask for it: `edit.plan` refuses it.
+   */
+  | {
+    op: "delete_unbacked";
+    tracks: TrackSelector;
+  }
   /**
    * Replace one copy of a recording with another: playlist and history entries (and My Tags)
    * move to `keep`, then `duplicate` is deleted unless `keep_duplicate`.
@@ -3588,12 +3598,20 @@ export type BackupInfo = {
    * medium copy rekordbox didn't leave): putting it back removes them.
    */
   added?: string[];
+  /**
+   * Files its change deleted that it doesn't hold, as rekordbox's next export puts them back
+   * (see [`crate::edit::Edit::DeleteUnbacked`]): putting it back leaves their tracks without
+   * them until then.
+   */
+  unbacked?: string[];
 };
 
 export type RestorePreview = {
   backup: BackupInfo;
   /** From the drive as it is to the drive restored. */
   diff: PlaylistDiff;
+  /** Tracks that come back without the audio their change left out of its backup. */
+  unbacked_audio: string[];
   /**
    * Tracks in the backup whose audio is neither on the drive nor in the backup: restoring
    * refuses them unless forced.
@@ -3656,6 +3674,11 @@ export type UndoPlan = {
   history_lists: string[];
   /** Recorded edits taken off the journal: `edits replay` won't make them again. */
   journal: JournalEntry[];
+  /**
+   * Tracks that come back without the audio their change left out of its backup, which
+   * rekordbox's next export puts back.
+   */
+  unbacked_audio: string[];
   /**
    * Tracks the backup lists whose audio is neither on the drive nor in the backup: putting it
    * back is refused unless forced.
@@ -3735,6 +3758,12 @@ export type UndoableBackup = {
    * medium copy rekordbox didn't leave): putting it back removes them.
    */
   added?: string[];
+  /**
+   * Files its change deleted that it doesn't hold, as rekordbox's next export puts them back
+   * (see [`crate::edit::Edit::DeleteUnbacked`]): putting it back leaves their tracks without
+   * them until then.
+   */
+  unbacked?: string[];
   /** Recorded edits putting it back takes off the journal. */
   journal: number;
   /** It's the one `undo` puts back. */
@@ -4347,6 +4376,8 @@ export type EditPlan = {
   history_removed: number;
   /** Files to delete (removed tracks' audio, analysis and artwork), as drive keys. */
   files: string[];
+  /** Those of `files` the backup leaves out (see [`Edit::DeleteUnbacked`]). */
+  unbacked?: string[];
   bytes: number;
   /** `export.pdb` and/or `exportLibrary.db`. */
   databases: string[];
@@ -4708,6 +4739,16 @@ export type EditOutput =
     tracks: TrackSelector;
     keep_files: boolean;
   }
+  /**
+   * Delete tracks as `delete_tracks` does, leaving their files out of the backup: for tracks
+   * rekordbox's next export puts back, whose copies would only fill the drive. Undoing brings
+   * the tracks back without their files, until that export. Only a plugin with the
+   * `library:delete-unbacked` capability can ask for it: `edit.plan` refuses it.
+   */
+  | {
+    op: "delete_unbacked";
+    tracks: TrackSelector;
+  }
   /**
    * Replace one copy of a recording with another: playlist and history entries (and My Tags)
    * move to `keep`, then `duplicate` is deleted unless `keep_duplicate`.