From 58cebe5ea73f19b7d49125710c10e635aed45073 Mon Sep 17 00:00:00 2001 From: KodeStar Date: Wed, 14 Oct 2026 18:16:49 +0100 Subject: [PATCH] Audiobookshelf metadata.json: where book details come from, provenance, scanner and data model Documents the server reading an Audiobookshelf metadata.json in a book's folder as a layer over the tags (server feat/abs-metadata-json): the user guide's Coming from Audiobookshelf section, the new provenance marker, the metadata.json skip pattern, and the developer pages' layers, columns (0046) and scan backfill. --- docs-developers/server/data-model.md | 46 +++++++++++++++---- docs-developers/server/scanner.md | 26 ++++++++++- docs-users/admin/books.md | 3 ++ docs-users/admin/libraries.md | 5 ++ .../organizing-your-library.md | 12 +++++ 5 files changed, 83 insertions(+), 9 deletions(-) diff --git a/docs-developers/server/data-model.md b/docs-developers/server/data-model.md index ed7ac2d..c38c322 100644 --- a/docs-developers/server/data-model.md +++ b/docs-developers/server/data-model.md @@ -247,7 +247,7 @@ covered in [Auth & security](auth-and-security.md#authorization-shares--scope). `scan_error_file`, `scan_error_detail` and `suspect_parts` in 0017; `split_parent` in 0022; `cover_art` and `cover_color` in 0023; `chapters_source`, `chapters_fit`, `scanned_chapters` and `chapters_hash` in 0035; `released` and `released_checked` -in 0037; `series_key` in 0041; `series_index_end` in 0045)* - one row per book, +in 0037; `series_key` in 0041; `series_index_end` in 0045; `abs_sig`, `abs_indexed_at` and `abs_file` in 0046)* - one row per book, `UNIQUE (library_id, rel_path)`. Columns: `is_folder` (folder book vs single-file book), identity metadata (`title`, `author`, `series`, `series_index`, `narrator`; `series_index_end` is the end of a main series @@ -255,11 +255,16 @@ position that is a range, an omnibus of books 1-3 at `series_index` 1 and `series_index_end` 3, `0` for none: the `series_index` override holds the text `"1-3"` and `refreshEffective` writes its start and end), `duration`, `asin`/`isbn` (optional external ids - present so enrichment/metadata services can attach data without reshaping the -schema), `published` (`YYYY[-MM[-DD]]`) and `description` (only an admin edit or -an accepted community match supplies these today; the scanner reads neither), +schema), `published` (`YYYY[-MM[-DD]]`) and `description` (an admin edit, an +accepted community match or an Audiobookshelf `metadata.json` supplies these; the +audio's tags never do), `abs_file` (what an Audiobookshelf `metadata.json` in a +folder book's folder says, as JSON field -> value normalized like an edit, `''` +for nothing) with `abs_sig` (the file's size and mtime, `''` for no file) and +`abs_indexed_at` (the `indexed_at` it was read for, `NULL` until a scan has +read it; see [the layers](#metadata-overrides-and-effective-values)), `more_series` (the effective list of the other series the book is in beyond -`series`, as JSON `[{"name","position"}]` plus `"position_end"` for a range, `'[]'` for none: only an edit or a -community match supplies it, an override like any other field; an admin edit +`series`, as JSON `[{"name","position"}]` plus `"position_end"` for a range, `'[]'` for none: only an edit, a +community match or an Audiobookshelf `metadata.json` supplies it, an override like any other field; an admin edit that sets or reverts `series` to one of these swaps the two, writing the list and `series_index` as overrides (a value the scan already gives is reverted to it instead; the console sends the values itself, so its swap writes them as @@ -925,8 +930,10 @@ only served while a book is indexed at its path). A `books` row always holds the **effective** metadata, built in layers: 1. what the scan found (`books.scanned`, `chapters.scanned_title`); -2. then any `book_enrichment` (`asin`/`isbn`, non-blank fields only); -3. then any `book_overrides` / `chapter_overrides`. +2. then what an Audiobookshelf `metadata.json` in a folder book's folder says + (`books.abs_file`, below); +3. then any `book_enrichment` (`asin`/`isbn`, non-blank fields only); +4. then any `book_overrides` / `chapter_overrides`. `bookLayers.resolve` (in `internal/catalog/overrides.go`) is the single statement of that precedence, and of each field's source. `refreshEffective` @@ -951,11 +958,33 @@ Per field, `resolve` yields the effective value, its source, the scanned value, whether it is locked, and who last edited it. Sources are not stored: a scanned value is `path` when it equals what `metadata.DeriveFromPath` yields for the book's path (or, in a path-first library, what `metadata.FromPathLayout` -yields), otherwise `tag`; an ASIN/ISBN from enrichment is `community`; +yields), otherwise `tag`; a value the ABS file gives is `abs`; an ASIN/ISBN from +enrichment is `community`; an override carries its own (`edited` or `community`); `""` means no value. The rule is the same for every row, so the rows migration 0016 backfilled `scanned` for (from their current values) need no special case. +**An Audiobookshelf `metadata.json`** (`metadata.ParseABSFile`, +`catalog/absfile.go`) is the record ABS keeps in a book's folder when its "Store +metadata with item" setting is on: its matches and its users' corrections, often +with the ASIN/ISBN the tags lack. ABS ranks it above the tags by default, and so +does `bookLayers.overABS`, applied in `scannedFields` over the scanned values +(after a path-first library's layout, before the position a title gives): every +field the file names replaces the scanned one. The parse is lenient (a wrongly +typed value costs that value; at most 4 MiB): `title` (not `subtitle`), +`authors` and `narrators` joined with ", ", `series` as `"Name #seq"` (the first +is the main series with its position, the rest `more_series`; a series without a +position keeps the scanned one only while it is the same series, by +`seriesKey`), `publishedDate` else `publishedYear` as `published`, the HTML +`description` as plain text, `asin`, `isbn`; genres, tags, publisher, language +and chapters are not read, and an empty value says nothing. `absFields` stores +each as an edit of its field would (`CleanOverride`, a refused value left out), +so an ASIN from the file makes the book matched for Health, `/meta`, the +community chapter check and a refresh run. Only a folder book's own folder +counts (`UpsertBook` writes `NULL` for a file book), and only a regular file the +library's ignore rules don't skip, so an ignore rule `metadata.json` turns it off +for a library. Reverting an edit lands on the file's value. + **A library's metadata source** (`libraries.metadata_source`) changes only the first layer. In a `'path'` library, `bookLayers.scannedFields` reads the book's path with `metadata.FromPathLayout` (the folder LAYOUT from @@ -1038,6 +1067,7 @@ The migration history so far: | 0043 | `cast_tokens` | `tokens.kind` `'cast'` (cast tokens for a Chromecast receiver) and `cast_grants` (each one's book and parent token), with `idx_cast_grants_parent` and the partial `idx_tokens_cast_expiry` | | 0044 | `household_members` | household members: `users.managed_by` (FK `ON DELETE SET NULL`), `users.member_kind` and `users.member_access` (each with a column CHECK), `users.can_add_members` (`DEFAULT 1`) and `users.display_name` (`DEFAULT ''`), with the partial `idx_users_managed_by`; `tokens.created_on_token_id` (FK `ON DELETE SET NULL`) with the partial `idx_tokens_created_on`. No data moves: every existing account reads as no member, allowed to add members | | 0045 | `series_ranges` | `books.series_index_end` and `book_more_series.position_end` (`REAL NOT NULL DEFAULT 0`), the end of a series position that is a range (an omnibus, `1-3`); `idx_book_more_series_key` is rebuilt to cover `position_end`. No data moves: every existing row holds a single position | +| 0046 | `abs_file` | `books.abs_sig` (`TEXT NOT NULL DEFAULT ''`), `books.abs_indexed_at` (`TEXT`, `NULL` on every existing row) and `books.abs_file` (`TEXT NOT NULL DEFAULT ''`): an Audiobookshelf `metadata.json` in a folder book's folder. The next scan reads the file of each folder book that has one, without re-indexing it (`catalog.SetABSFiles`); a book without one is left as it is | ## SQLite choices diff --git a/docs-developers/server/scanner.md b/docs-developers/server/scanner.md index 040e448..5ab38ca 100644 --- a/docs-developers/server/scanner.md +++ b/docs-developers/server/scanner.md @@ -251,6 +251,29 @@ library. A skipped folder book indexed before migration `0017` gets the same kind of tag-only check for [several books in one folder](#folders-that-may-hold-several-books) (`suspectFromTags`, written by `catalog.SetSuspectParts`). +A skipped folder book also has its **Audiobookshelf `metadata.json`** read again +when it is new, changed or gone: ABS rewrites the file without touching the +audio, so the book's own signature can't tell. Discovery records each folder's +file signature (`metadata.ABSSignature`, size + mtime) from the walk it already +makes (`audioDirs`' `absFiles`, no extra directory read), and a skipped book whose +signature differs from the stored one (`Signature.ABSSig`), or that has a file +not read for this index of the row (`ABSUnchecked`: `abs_indexed_at` is `NULL` +after migration 0046, or an older server re-indexed it), is read and written by +`catalog.SetABSFiles` in batches of 250 beside the path backfill. A book with no +file and none before is left unwritten, so the first scan after the upgrade +touches only books that have one. A book is re-resolved unless its reading +equals the one recorded for this very index (an older server's re-index resolved +the row without the file). A re-indexed book reads the file in `enrich` +(`Scanner.readABS`) only when the walk found one; an on-demand index, with no +walk, stats it first (`absSigAt`). The file is read only when it is a regular +file, never through a link (one swapped in after the walk included). A file that +can't be used (not JSON, too large, not a regular file) keeps its signature, so +it isn't read again until it changes; one that can't be read right now (gone +since the walk, a flaky mount) is left as it was by a skipped book and recorded +as none by a re-index, so the next scan reads it again. Both log a warning. What +the file means for a book's metadata is in +[Data model](data-model.md#metadata-overrides-and-effective-values). + ### Prune: what a removed book leaves behind When discovery saw the whole tree, `catalog.DeleteBooksNotIn` drops every book @@ -484,7 +507,8 @@ embedded data winning where it is trustworthy: The scanner does not record where each value came from. The admin console's per-field provenance works it out when the book is read (`bookLayers.resolve`): a scanned value equal to what `DeriveFromPath` yields -for the book's path is `path`, anything else `tag` - one rule for every row, old +for the book's path is `path`, a value an Audiobookshelf `metadata.json` gave is +`abs`, anything else `tag` - one rule for every row, old or new (see [Data model](data-model.md#metadata-overrides-and-effective-values)). A library whose `metadata_source` is `path` takes the folder layout over the tags (`metadata.FromPathLayout`: the top folder is the author and the folder diff --git a/docs-users/admin/books.md b/docs-users/admin/books.md index 7b66691..8a6dcb9 100644 --- a/docs-users/admin/books.md +++ b/docs-users/admin/books.md @@ -172,6 +172,9 @@ Each value has a marker saying where it came from: - **File tag** - read from the audio file's tags. - **Path** - worked out from the folder and file names. +- **Audiobookshelf** - read from the `metadata.json` file Audiobookshelf keeps + in the book's folder (see + [Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)). - **Edited** - your edit, locked against rescans. - **Community** - accepted from community metadata. diff --git a/docs-users/admin/libraries.md b/docs-users/admin/libraries.md index e92c5f2..02d058b 100644 --- a/docs-users/admin/libraries.md +++ b/docs-users/admin/libraries.md @@ -175,6 +175,7 @@ leave out. One pattern per line: | `Extras/` | every folder named `Extras`, wherever it is, with everything in it (the trailing `/` means folders only) | | `Podcasts/*` | everything inside the `Podcasts` folder at the top of the library (a `/` in the middle means "starting at the library folder") | | `/Extras` | only the `Extras` folder at the top of the library (a `/` at the start also means "starting at the library folder") | +| `metadata.json` | the `metadata.json` files Audiobookshelf leaves in book folders, so this library takes no details from them (see [Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)) | | `# publisher samples` | nothing: a line starting with `#` is a note to yourself | A pattern without a `/` (or with one only at the end) matches a file or @@ -214,6 +215,10 @@ folders don't name one for. A tag title that is exactly the folder's name, number and all (`13 Reasons Why`), is kept whole, so a number that is part of the title isn't read as a series number. +Either way, a book whose folder has an Audiobookshelf `metadata.json` takes the +values in that file over both (see +[Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)). + Changing this setting updates every book in the library at once, without a rescan, and your own edits (and values you accepted from community metadata) stay on top. On a book's page, the values taken from the folders are marked diff --git a/docs-users/getting-started/organizing-your-library.md b/docs-users/getting-started/organizing-your-library.md index e3056a8..14cdef7 100644 --- a/docs-users/getting-started/organizing-your-library.md +++ b/docs-users/getting-started/organizing-your-library.md @@ -74,11 +74,23 @@ For each book, AudioSilo starts with the names on disk, then lets the audio file - **From your folders**: the book's folder (or file) name becomes the title; a leading number like `03 - The Title` is read as its position in a series (but not a number that runs into the name, like `3rd Rock` or `20,000 Leagues Under the Sea`); the parent folder is read as the series and the folder above that as the author. A book with only one folder above it (`Author/Title/`) takes that folder as its author, with no series. So an `Author/Series/03 - Title/` or an `Author/Title/` layout works with no tags at all. - **From embedded tags**: a title (audiobooks usually carry it in the *album* tag; when the album holds the series name and the *title* tag carries the full "Series: Book" title, the more specific title tag wins), the author (album artist or artist tag), the narrator (composer or narrator tag), and the series where a tag exists. Junk tag titles like "Track 01" are ignored in favour of the folder name. +- **From Audiobookshelf's `metadata.json`**, when the book's folder has one: its title, authors, narrators, series, published year, description, ASIN and ISBN go over the tags. See [Coming from Audiobookshelf](#coming-from-audiobookshelf). - **From ffmpeg's ffprobe**, when available: durations and any chapter list embedded in the file, so a chaptered M4B shows its real chapters in the player. - **From the community database**, for chapters only: a book whose files have no chapters of their own (one long file, or a folder with one chapter per file) gets the community's chapter list once it is [matched](../admin/books.md#matching-with-community-metadata) and the list fits your copy. A book with only broad chapters can be switched to the community's more detailed ones. See [Community chapters](../admin/books.md#community-chapters). Audiobook tags are famously messy, which is why the folder names always provide a sane baseline - a well-named folder beats a badly-tagged file. If a library's folders are named well and its tags aren't, you can set it to take book details from the [folder names first](../admin/libraries.md#where-book-details-come-from) instead. Anything still wrong can be corrected in the admin console without touching the files: see [Books and metadata](../admin/books.md). +### Coming from Audiobookshelf + +If Audiobookshelf manages (or managed) the same folders, it may have left a `metadata.json` file in each book's folder. Audiobookshelf writes one when its **Store metadata with item** setting is on. That file is Audiobookshelf's record of the book, including any match you ran there and any corrections you made, so AudioSilo reads it too: + +- The title, authors, narrators, series (with their numbers), published year, description, **ASIN** and **ISBN** in the file go **over** what the audio tags say, as they do in Audiobookshelf. A value the file leaves empty doesn't clear the tag's. +- A book whose file has an ASIN or ISBN counts as [matched](../admin/books.md#matching-with-community-metadata) straight away, so community details and chapters can be looked up for it without matching it by hand. +- Your own edits in the admin console still win over the file, and the book page marks each value the file gave as **Audiobookshelf**. +- When the file changes (you edit the book in Audiobookshelf), the next scan picks the change up without re-reading the audio. If the file goes, its values go with it. + +Only a book's own folder counts: a loose audio file at the top of a library takes nothing from a `metadata.json` beside it. AudioSilo never writes to the file. To stop a library using these files, add `metadata.json` to its [skipped files](../admin/libraries.md#skipping-files-and-folders). + ### Covers Cover art is found in this order: