Audiobookshelf metadata.json as a metadata source

docs-developers/server/data-model.md

@@ -247,19 +247,24 @@ covered in [Auth & security](auth-and-security.md#authorization-shares--scope).
 `scan_error_file`, `scan_error_detail` and `suspect_parts` in 0017; `split_parent` in
 0022; `cover_art` and `cover_color` in 0023; `chapters_source`, `chapters_fit`,
 `scanned_chapters` and `chapters_hash` in 0035; `released` and `released_checked`
-in 0037; `series_key` in 0041; `series_index_end` in 0045)* - one row per book,
+in 0037; `series_key` in 0041; `series_index_end` in 0045; `abs_sig`, `abs_indexed_at` and `abs_file` in 0046)* - one row per book,
 `UNIQUE (library_id, rel_path)`. Columns: `is_folder` (folder book vs
 single-file book), identity metadata (`title`, `author`, `series`,
 `series_index`, `narrator`; `series_index_end` is the end of a main series
 position that is a range, an omnibus of books 1-3 at `series_index` 1 and
 `series_index_end` 3, `0` for none: the `series_index` override holds the text
 `"1-3"` and `refreshEffective` writes its start and end), `duration`, `asin`/`isbn` (optional external ids -
 present so enrichment/metadata services can attach data without reshaping the
-schema), `published` (`YYYY[-MM[-DD]]`) and `description` (only an admin edit or
-an accepted community match supplies these today; the scanner reads neither),
+schema), `published` (`YYYY[-MM[-DD]]`) and `description` (an admin edit, an
+accepted community match or an Audiobookshelf `metadata.json` supplies these; the
+audio's tags never do), `abs_file` (what an Audiobookshelf `metadata.json` in a
+folder book's folder says, as JSON field -> value normalized like an edit, `''`
+for nothing) with `abs_sig` (the file's size and mtime, `''` for no file) and
+`abs_indexed_at` (the `indexed_at` it was read for, `NULL` until a scan has
+read it; see [the layers](#metadata-overrides-and-effective-values)),
 `more_series` (the effective list of the other series the book is in beyond
-`series`, as JSON `[{"name","position"}]` plus `"position_end"` for a range, `'[]'` for none: only an edit or a
-community match supplies it, an override like any other field; an admin edit
+`series`, as JSON `[{"name","position"}]` plus `"position_end"` for a range, `'[]'` for none: only an edit, a
+community match or an Audiobookshelf `metadata.json` supplies it, an override like any other field; an admin edit
 that sets or reverts `series` to one of these swaps the two, writing the list
 and `series_index` as overrides (a value the scan already gives is reverted to
 it instead; the console sends the values itself, so its swap writes them as
@@ -925,8 +930,10 @@ only served while a book is indexed at its path).
 A `books` row always holds the **effective** metadata, built in layers:
 
 1. what the scan found (`books.scanned`, `chapters.scanned_title`);
-2. then any `book_enrichment` (`asin`/`isbn`, non-blank fields only);
-3. then any `book_overrides` / `chapter_overrides`.
+2. then what an Audiobookshelf `metadata.json` in a folder book's folder says
+   (`books.abs_file`, below);
+3. then any `book_enrichment` (`asin`/`isbn`, non-blank fields only);
+4. then any `book_overrides` / `chapter_overrides`.
 
 `bookLayers.resolve` (in `internal/catalog/overrides.go`) is the single
 statement of that precedence, and of each field's source. `refreshEffective`
@@ -951,11 +958,33 @@ Per field, `resolve` yields the effective value, its source, the scanned
 value, whether it is locked, and who last edited it. Sources are not stored: a
 scanned value is `path` when it equals what `metadata.DeriveFromPath` yields for
 the book's path (or, in a path-first library, what `metadata.FromPathLayout`
-yields), otherwise `tag`; an ASIN/ISBN from enrichment is `community`;
+yields), otherwise `tag`; a value the ABS file gives is `abs`; an ASIN/ISBN from
+enrichment is `community`;
 an override carries its own (`edited` or `community`); `""` means no value. The
 rule is the same for every row, so the rows migration 0016 backfilled `scanned`
 for (from their current values) need no special case.
 
+**An Audiobookshelf `metadata.json`** (`metadata.ParseABSFile`,
+`catalog/absfile.go`) is the record ABS keeps in a book's folder when its "Store
+metadata with item" setting is on: its matches and its users' corrections, often
+with the ASIN/ISBN the tags lack. ABS ranks it above the tags by default, and so
+does `bookLayers.overABS`, applied in `scannedFields` over the scanned values
+(after a path-first library's layout, before the position a title gives): every
+field the file names replaces the scanned one. The parse is lenient (a wrongly
+typed value costs that value; at most 4 MiB): `title` (not `subtitle`),
+`authors` and `narrators` joined with ", ", `series` as `"Name #seq"` (the first
+is the main series with its position, the rest `more_series`; a series without a
+position keeps the scanned one only while it is the same series, by
+`seriesKey`), `publishedDate` else `publishedYear` as `published`, the HTML
+`description` as plain text, `asin`, `isbn`; genres, tags, publisher, language
+and chapters are not read, and an empty value says nothing. `absFields` stores
+each as an edit of its field would (`CleanOverride`, a refused value left out),
+so an ASIN from the file makes the book matched for Health, `/meta`, the
+community chapter check and a refresh run. Only a folder book's own folder
+counts (`UpsertBook` writes `NULL` for a file book), and only a regular file the
+library's ignore rules don't skip, so an ignore rule `metadata.json` turns it off
+for a library. Reverting an edit lands on the file's value.
+
 **A library's metadata source** (`libraries.metadata_source`) changes only the
 first layer. In a `'path'` library, `bookLayers.scannedFields` reads the
 book's path with `metadata.FromPathLayout` (the folder LAYOUT from
@@ -1038,6 +1067,7 @@ The migration history so far:
 | 0043 | `cast_tokens` | `tokens.kind` `'cast'` (cast tokens for a Chromecast receiver) and `cast_grants` (each one's book and parent token), with `idx_cast_grants_parent` and the partial `idx_tokens_cast_expiry` |
 | 0044 | `household_members` | household members: `users.managed_by` (FK `ON DELETE SET NULL`), `users.member_kind` and `users.member_access` (each with a column CHECK), `users.can_add_members` (`DEFAULT 1`) and `users.display_name` (`DEFAULT ''`), with the partial `idx_users_managed_by`; `tokens.created_on_token_id` (FK `ON DELETE SET NULL`) with the partial `idx_tokens_created_on`. No data moves: every existing account reads as no member, allowed to add members |
 | 0045 | `series_ranges` | `books.series_index_end` and `book_more_series.position_end` (`REAL NOT NULL DEFAULT 0`), the end of a series position that is a range (an omnibus, `1-3`); `idx_book_more_series_key` is rebuilt to cover `position_end`. No data moves: every existing row holds a single position |
+| 0046 | `abs_file` | `books.abs_sig` (`TEXT NOT NULL DEFAULT ''`), `books.abs_indexed_at` (`TEXT`, `NULL` on every existing row) and `books.abs_file` (`TEXT NOT NULL DEFAULT ''`): an Audiobookshelf `metadata.json` in a folder book's folder. The next scan reads the file of each folder book that has one, without re-indexing it (`catalog.SetABSFiles`); a book without one is left as it is |
 
 ## SQLite choices
 

docs-developers/server/scanner.md

@@ -251,6 +251,29 @@ library. A skipped folder book indexed before migration `0017` gets the same
 kind of tag-only check for [several books in one folder](#folders-that-may-hold-several-books)
 (`suspectFromTags`, written by `catalog.SetSuspectParts`).
 
+A skipped folder book also has its **Audiobookshelf `metadata.json`** read again
+when it is new, changed or gone: ABS rewrites the file without touching the
+audio, so the book's own signature can't tell. Discovery records each folder's
+file signature (`metadata.ABSSignature`, size + mtime) from the walk it already
+makes (`audioDirs`' `absFiles`, no extra directory read), and a skipped book whose
+signature differs from the stored one (`Signature.ABSSig`), or that has a file
+not read for this index of the row (`ABSUnchecked`: `abs_indexed_at` is `NULL`
+after migration 0046, or an older server re-indexed it), is read and written by
+`catalog.SetABSFiles` in batches of 250 beside the path backfill. A book with no
+file and none before is left unwritten, so the first scan after the upgrade
+touches only books that have one. A book is re-resolved unless its reading
+equals the one recorded for this very index (an older server's re-index resolved
+the row without the file). A re-indexed book reads the file in `enrich`
+(`Scanner.readABS`) only when the walk found one; an on-demand index, with no
+walk, stats it first (`absSigAt`). The file is read only when it is a regular
+file, never through a link (one swapped in after the walk included). A file that
+can't be used (not JSON, too large, not a regular file) keeps its signature, so
+it isn't read again until it changes; one that can't be read right now (gone
+since the walk, a flaky mount) is left as it was by a skipped book and recorded
+as none by a re-index, so the next scan reads it again. Both log a warning. What
+the file means for a book's metadata is in
+[Data model](data-model.md#metadata-overrides-and-effective-values).
+
 ### Prune: what a removed book leaves behind
 
 When discovery saw the whole tree, `catalog.DeleteBooksNotIn` drops every book
@@ -484,7 +507,8 @@ embedded data winning where it is trustworthy:
 The scanner does not record where each value came from. The admin console's
 per-field provenance works it out when the book is read
 (`bookLayers.resolve`): a scanned value equal to what `DeriveFromPath` yields
-for the book's path is `path`, anything else `tag` - one rule for every row, old
+for the book's path is `path`, a value an Audiobookshelf `metadata.json` gave is
+`abs`, anything else `tag` - one rule for every row, old
 or new (see [Data model](data-model.md#metadata-overrides-and-effective-values)).
 A library whose `metadata_source` is `path` takes the folder layout over the
 tags (`metadata.FromPathLayout`: the top folder is the author and the folder

docs-users/admin/books.md

@@ -172,6 +172,9 @@ Each value has a marker saying where it came from:
 
 - **File tag** - read from the audio file's tags.
 - **Path** - worked out from the folder and file names.
+- **Audiobookshelf** - read from the `metadata.json` file Audiobookshelf keeps
+  in the book's folder (see
+  [Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)).
 - **Edited** - your edit, locked against rescans.
 - **Community** - accepted from community metadata.
 

docs-users/admin/libraries.md

@@ -175,6 +175,7 @@ leave out. One pattern per line:
 | `Extras/` | every folder named `Extras`, wherever it is, with everything in it (the trailing `/` means folders only) |
 | `Podcasts/*` | everything inside the `Podcasts` folder at the top of the library (a `/` in the middle means "starting at the library folder") |
 | `/Extras` | only the `Extras` folder at the top of the library (a `/` at the start also means "starting at the library folder") |
+| `metadata.json` | the `metadata.json` files Audiobookshelf leaves in book folders, so this library takes no details from them (see [Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)) |
 | `# publisher samples` | nothing: a line starting with `#` is a note to yourself |
 
 A pattern without a `/` (or with one only at the end) matches a file or
@@ -214,6 +215,10 @@ folders don't name one for. A tag title that is exactly the folder's name,
 number and all (`13 Reasons Why`), is kept whole, so a number that is part of
 the title isn't read as a series number.
 
+Either way, a book whose folder has an Audiobookshelf `metadata.json` takes the
+values in that file over both (see
+[Coming from Audiobookshelf](../getting-started/organizing-your-library.md#coming-from-audiobookshelf)).
+
 Changing this setting updates every book in the library at once, without a
 rescan, and your own edits (and values you accepted from community metadata)
 stay on top. On a book's page, the values taken from the folders are marked

docs-users/getting-started/organizing-your-library.md

@@ -74,11 +74,23 @@ For each book, AudioSilo starts with the names on disk, then lets the audio file
 
 - **From your folders**: the book's folder (or file) name becomes the title; a leading number like `03 - The Title` is read as its position in a series (but not a number that runs into the name, like `3rd Rock` or `20,000 Leagues Under the Sea`); the parent folder is read as the series and the folder above that as the author. A book with only one folder above it (`Author/Title/`) takes that folder as its author, with no series. So an `Author/Series/03 - Title/` or an `Author/Title/` layout works with no tags at all.
 - **From embedded tags**: a title (audiobooks usually carry it in the *album* tag; when the album holds the series name and the *title* tag carries the full "Series: Book" title, the more specific title tag wins), the author (album artist or artist tag), the narrator (composer or narrator tag), and the series where a tag exists. Junk tag titles like "Track 01" are ignored in favour of the folder name.
+- **From Audiobookshelf's `metadata.json`**, when the book's folder has one: its title, authors, narrators, series, published year, description, ASIN and ISBN go over the tags. See [Coming from Audiobookshelf](#coming-from-audiobookshelf).
 - **From ffmpeg's ffprobe**, when available: durations and any chapter list embedded in the file, so a chaptered M4B shows its real chapters in the player.
 - **From the community database**, for chapters only: a book whose files have no chapters of their own (one long file, or a folder with one chapter per file) gets the community's chapter list once it is [matched](../admin/books.md#matching-with-community-metadata) and the list fits your copy. A book with only broad chapters can be switched to the community's more detailed ones. See [Community chapters](../admin/books.md#community-chapters).
 
 Audiobook tags are famously messy, which is why the folder names always provide a sane baseline - a well-named folder beats a badly-tagged file. If a library's folders are named well and its tags aren't, you can set it to take book details from the [folder names first](../admin/libraries.md#where-book-details-come-from) instead. Anything still wrong can be corrected in the admin console without touching the files: see [Books and metadata](../admin/books.md).
 
+### Coming from Audiobookshelf
+
+If Audiobookshelf manages (or managed) the same folders, it may have left a `metadata.json` file in each book's folder. Audiobookshelf writes one when its **Store metadata with item** setting is on. That file is Audiobookshelf's record of the book, including any match you ran there and any corrections you made, so AudioSilo reads it too:
+
+- The title, authors, narrators, series (with their numbers), published year, description, **ASIN** and **ISBN** in the file go **over** what the audio tags say, as they do in Audiobookshelf. A value the file leaves empty doesn't clear the tag's.
+- A book whose file has an ASIN or ISBN counts as [matched](../admin/books.md#matching-with-community-metadata) straight away, so community details and chapters can be looked up for it without matching it by hand.
+- Your own edits in the admin console still win over the file, and the book page marks each value the file gave as **Audiobookshelf**.
+- When the file changes (you edit the book in Audiobookshelf), the next scan picks the change up without re-reading the audio. If the file goes, its values go with it.
+
+Only a book's own folder counts: a loose audio file at the top of a library takes nothing from a `metadata.json` beside it. AudioSilo never writes to the file. To stop a library using these files, add `metadata.json` to its [skipped files](../admin/libraries.md#skipping-files-and-folders).
+
 ### Covers
 
 Cover art is found in this order: