From 2f74de20bb9cc5c751d4f7bb881b2f95a3d60bc3 Mon Sep 17 00:00:00 2001 From: ROHAN PANDEY <95585299+rohan-pandeyy@users.noreply.github.com> Date: Wed, 29 Jul 2026 15:31:40 +0530 Subject: [PATCH 1/4] docs: document Smart Memories - New `backend/backend_python/memories.md`: schema, curation triggers, scoring signals, API, pipeline hooks and preferences. Added to the nav. - Rewrote `frontend/memories.md`, which described the removed implementation (DBSCAN location clustering, reverse geocoding, `/api/memories/*`). - `database.md`: the four memories tables, their keys, cascades and indexes. - `directory-structure.md`: the new modules, the rewritten memories router, and the semantic label tables, which are no longer dormant. - `image-processing.md`: capture-date extraction from EXIF sub-IFDs, Takeout sidecars and video container boxes, and the `date_source` field. - `semantic-search.md`: corrected the pipeline ordering and recorded memory curation as a consumer of embeddings and event labels. - `features.md`, `architecture.md`, `state-management.md`, `ui-components.md` updated to match. Admonition bodies are indented so MkDocs renders them inside the box. --- docs/backend/backend_python/database.md | 169 +++++ .../backend_python/directory-structure.md | 75 ++- .../backend_python/image-processing.md | 115 +++- docs/backend/backend_python/memories.md | 606 ++++++++++++++++++ .../backend/backend_python/semantic-search.md | 145 +++-- docs/frontend/memories.md | 475 ++++++-------- docs/frontend/state-management.md | 70 ++ docs/frontend/ui-components.md | 111 +++- docs/overview/architecture.md | 16 +- docs/overview/features.md | 118 ++-- mkdocs.yml | 1 + 11 files changed, 1443 insertions(+), 458 deletions(-) create mode 100644 docs/backend/backend_python/memories.md diff --git a/docs/backend/backend_python/database.md b/docs/backend/backend_python/database.md index 172856cd4..cd3a1e4bb 100644 --- a/docs/backend/backend_python/database.md +++ b/docs/backend/backend_python/database.md @@ -11,3 +11,172 @@ PictoPy uses several SQLite databases to manage various aspects of the applicati Alternatively, [click here to view the interactive DB schema diagram in a new tab](https://dbdiagram.io/d/PictoPy-6a593dd1c3a90dd98d55554d). + +!!! note "Diagram" + The embedded ER diagram above does not yet include the four Smart Memories tables (`memories`, `memory_images`, `memory_videos`, `memory_runs`). Use the [Memories tables](#memories-tables) section below for those. + +## Memories tables + +Defined and created by `db_create_memories_table()` in +`backend/app/database/memories.py`, which also holds every query helper the +Smart Memories feature uses. See [Memories](memories.md) for the feature +itself. + +The module connects through `_connect` imported from +`app/database/images.py`, which issues `PRAGMA foreign_keys = ON` on every +connection, so the `ON DELETE CASCADE` and `ON DELETE SET NULL` rules below +are actually enforced. + +### `memories` + +One row per curated memory. + +| Column | Type | Notes | +| ------------------ | -------- | -------------------------------------------------------------- | +| `memory_id` | TEXT | Primary key | +| `dedupe_key` | TEXT | NOT NULL, UNIQUE. Upserts conflict-resolve on this column | +| `event_type` | TEXT | NOT NULL, CHECK against `EVENT_TYPES` | +| `status` | TEXT | NOT NULL, DEFAULT `'pending'`, CHECK against `MEMORY_STATUSES` | +| `title` | TEXT | NOT NULL | +| `subtitle` | TEXT | Nullable | +| `place_label` | TEXT | Nullable | +| `center_lat` | REAL | Nullable | +| `center_lon` | REAL | Nullable | +| `surface_date` | DATE | NOT NULL. The date the memory is eligible to appear | +| `period_start` | DATETIME | Earliest capture time of the member images | +| `period_end` | DATETIME | Latest capture time of the member images | +| `cover_image_id` | TEXT | FK → `images(id)` ON DELETE SET NULL | +| `image_count` | INTEGER | NOT NULL, DEFAULT 0 | +| `video_count` | INTEGER | NOT NULL, DEFAULT 0 | +| `score` | REAL | NOT NULL, DEFAULT 0 | +| `signals` | TEXT | JSON blob, decoded on read | +| `params_signature` | TEXT | Identifies the scorer configuration that built the row | +| `error` | TEXT | Nullable | +| `notified_at` | DATETIME | Nullable | +| `viewed_at` | DATETIME | Nullable | +| `dismissed` | BOOLEAN | NOT NULL, DEFAULT 0 | +| `created_at` | DATETIME | DEFAULT `CURRENT_TIMESTAMP` | +| `updated_at` | DATETIME | DEFAULT `CURRENT_TIMESTAMP` | + +`db_upsert_memory()` writes every column except `memory_id`, `viewed_at`, +`notified_at` and `dismissed`, so re-curating a memory replaces its contents +without resetting what the user has already seen. + +### `memory_images` + +Join table between a memory and its curated photos. + +| Column | Type | Notes | +| ------------ | ------- | ------------------------------------------------------ | +| `memory_id` | TEXT | NOT NULL. FK → `memories(memory_id)` ON DELETE CASCADE | +| `image_id` | TEXT | NOT NULL. FK → `images(id)` ON DELETE CASCADE | +| `sort_order` | INTEGER | NOT NULL. Presentation order | +| `score` | REAL | Nullable. Per-image score within this memory | + +Primary key is the composite `(memory_id, image_id)`. Rows cascade away from +both parents: deleting a memory drops its members, and deleting a photo from +the library removes it from every memory holding it. + +### `memory_videos` + +The same shape for short video clips. + +| Column | Type | Notes | +| ------------ | ------- | ------------------------------------------------------ | +| `memory_id` | TEXT | NOT NULL. FK → `memories(memory_id)` ON DELETE CASCADE | +| `video_id` | TEXT | NOT NULL. FK → `videos(id)` ON DELETE CASCADE | +| `sort_order` | INTEGER | NOT NULL | +| `score` | REAL | Nullable | + +Primary key is `(memory_id, video_id)`. `sort_order` is a single sequence +shared with `memory_images`, so merging the two tables by `sort_order` +produces one chronological story. + +### `memory_runs` + +One row per curation run, keyed by date. A run that generates zero memories +is a legitimate outcome, so the `memories` table alone cannot answer "has +today's run already happened?". + +| Column | Type | Notes | +| ------------------ | -------- | ------------------------------------------ | +| `run_date` | DATE | Primary key | +| `status` | TEXT | NOT NULL, CHECK against `RUN_STATUSES` | +| `params_signature` | TEXT | Nullable | +| `generated_count` | INTEGER | NOT NULL, DEFAULT 0 | +| `error` | TEXT | Nullable | +| `started_at` | DATETIME | DEFAULT `CURRENT_TIMESTAMP` | +| `finished_at` | DATETIME | Set when the run reaches a terminal status | + +### Constrained vocabularies + +Three module-level constants in `app/database/memories.py` define the allowed +values, and the `CHECK` clauses in the DDL are generated from those same +tuples by the `_check_in()` helper, so constants and constraints cannot drift +apart. + +| Constant | Column | Values | +| ----------------- | --------------------- | ----------------------------------------------- | +| `EVENT_TYPES` | `memories.event_type` | `anniversary`, `import_event`, `semantic_event` | +| `MEMORY_STATUSES` | `memories.status` | `pending`, `complete`, `failed`, `empty` | +| `RUN_STATUSES` | `memory_runs.status` | `running`, `complete`, `failed` | + +`db_finish_memory_run()` additionally rejects any terminal status other than +`complete` or `failed`. + +### Indexes + +| Index | Table | Serves | +| ---------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------- | +| `ix_memories_surface_date` | `memories(surface_date DESC)` | Newest-first listing | +| `ix_memories_status_surface` | `memories(status, surface_date DESC)` | The filtered, paginated card list | +| `ix_memories_surfaceable` | `memories(surface_date DESC, score DESC)` | Partial index (see below) — the "is there anything to show?" lookup | +| `ix_memory_images_image_id` | `memory_images(image_id)` | Reverse lookup by photo; without it SQLite scans `memory_images` on every image delete | +| `ix_memory_videos_video_id` | `memory_videos(video_id)` | The same reverse lookup for clips | + +`ix_memories_surfaceable` is a partial index, covering only the rows a +surface check can possibly return: + +```sql +CREATE INDEX IF NOT EXISTS ix_memories_surfaceable +ON memories(surface_date DESC, score DESC) +WHERE viewed_at IS NULL AND dismissed = 0 AND status = 'complete' +``` + +### Table creation order + +`memory_images` references `images(id)` and `memory_videos` references +`videos(id)`, so both parent tables must exist first. In `main.py`'s +`lifespan()` the calls run in this order: + +1. `db_create_images_table()` +2. `db_create_videos_table()` +3. …other tables… +4. `db_create_memories_table()` + +`db_create_videos_table()` must run before `db_create_memories_table()`. The +test fixtures in `backend/tests/conftest.py` create the tables in the same +order. + +`memories.video_count` is also added by a guarded `ALTER TABLE`: the module +reads `PRAGMA table_info(memories)` and appends the column only when it is +missing, because `CREATE TABLE IF NOT EXISTS` will not add a column to a +table that a pre-existing database already has. + +### Memories capture dates + +`images.captured_at` (and `videos.captured_at`) is a distinct column from the +`date_created` value stored inside the `metadata` JSON blob. `date_created` +drives gallery display; `captured_at` is consumed only by memories, and every +memories query filters on `captured_at IS NOT NULL`. + +`metadata` also carries `date_source`, recording where the date came from: +`exif`, `sidecar`, `container`, `filesystem` or `unknown`. Only the first +three are treated as trusted capture times (`TRUSTED_DATE_SOURCES` in +`app/utils/extract_location_metadata.py`); a filesystem mtime is import day +for a copied library, so it never reaches `captured_at`, which is left `NULL` +and therefore invisible to memories. + +Both `images` and `videos` carry an index on `captured_at` +(`ix_images_captured_at`, `ix_videos_captured_at`), and `images` additionally +has `ix_images_favourite_captured_at` on `(isFavourite, captured_at)`. diff --git a/docs/backend/backend_python/directory-structure.md b/docs/backend/backend_python/directory-structure.md index b7b65aa41..32d545f15 100644 --- a/docs/backend/backend_python/directory-structure.md +++ b/docs/backend/backend_python/directory-structure.md @@ -33,17 +33,18 @@ This directory contains files related to database operations, including table cr These files are the places where most of the SQL queries are written. By default, on startup this directory is where the databases (`.db` files) is created. -| Name | Description | -| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `albums.py` | Handles operations related to photo albums, including creating, deleting, and managing albums and their contents. | -| `face_clusters.py` | Provides functions to create, insert, update, retrieve, and delete face cluster records along with related images. | -| `faces.py` | Manages face-related data, including storing and retrieving face embeddings for facial recognition. | -| `folders.py` | Handles operations to create, insert, update, retrieve, and delete folder records, while handling folder hierarchies and AI tagging status. | -| `image_embeddings.py` | Stores/retrieves SigLIP2 image embeddings as float32 BLOBs, filtered by `model_version`; see [Semantic Search](semantic-search.md). | -| `images.py` | Deals with image-related operations, such as storing image metadata, managing image IDs, and handling image classifications. | -| `metadata.py` | Manages the metadata and provides functions to create the table, retrieve stored metadata as a dictionary, and update the metadata with new values. | -| `semantic_labels.py` | Creates the (currently dormant) `semantic_labels`/`image_semantic_labels` tables reserved for a future curated-label browsing feature. | -| `yolo_mapping.py` | Creates and manages mappings for YOLO object detection classes. | +| Name | Description | +| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `albums.py` | Handles operations related to photo albums, including creating, deleting, and managing albums and their contents. | +| `face_clusters.py` | Provides functions to create, insert, update, retrieve, and delete face cluster records along with related images. | +| `faces.py` | Manages face-related data, including storing and retrieving face embeddings for facial recognition. | +| `folders.py` | Handles operations to create, insert, update, retrieve, and delete folder records, while handling folder hierarchies and AI tagging status. | +| `image_embeddings.py` | Stores/retrieves SigLIP2 image embeddings as float32 BLOBs, filtered by `model_version`; see [Semantic Search](semantic-search.md). | +| `images.py` | Deals with image-related operations, such as storing image metadata, managing image IDs, and handling image classifications. | +| `memories.py` | Creates the `memories`, `memory_images`, `memory_videos` and `memory_runs` tables and holds every memories query helper; see [Memories](memories.md). | +| `metadata.py` | Manages the metadata and provides functions to create the table, retrieve stored metadata as a dictionary, and update the metadata with new values. | +| `semantic_labels.py` | Creates and syncs the curated `semantic_labels` vocabulary (definitions plus cached label embeddings) and the `image_classes_display` view; its `event` labels drive the `semantic_event` memory trigger. | +| `yolo_mapping.py` | Creates and manages mappings for YOLO object detection classes. | ## models @@ -66,10 +67,11 @@ This directory contains API route definitions for different functionalities of t | Name | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `albums.py` | Handles API routes for album-related operations (create, delete, add/remove photos, view albums). | +| `dependencies.py` | Shared `get_state` FastAPI dependency, which hands routers the application state holding the process-pool executor. | | `face_clusters.py` | Rename clusters, list clusters, and fetch cluster images. | | `folders.py` | Add, sync, update AI tagging, delete, and list folders, managing folder hierarchy and image processing asynchronously. Triggers the SigLIP2 embedding pass last, after YOLO/face processing (see [Semantic Search](semantic-search.md)). | | `images.py` | Deals with image-related operations (adding, deleting, retrieving images and their metadata), plus `GET /images/semantic-search` (see [Semantic Search](semantic-search.md)). | -| `memories.py` | Provides endpoints to generate photo memories grouped by location and time, retrieve a timeline, and fetch on-this-day recollections. | +| `memories.py` | Router mounted at `/memories`: queue a curation run, report run status, fetch today's memory, list cards, fetch a full story, mark viewed/dismissed/notified, and delete a memory (see [Memories](memories.md)). | | `models.py` | Installs/uninstalls model tiers (including the `semantic` SigLIP2 bundle), reports install status, tracks SSE download progress, and exposes routes to get hardware recommendations. | | `shutdown.py` | Provides a single endpoint to gracefully terminate the PictoPy backend process on all platforms. | | `user_preferences.py` | Get and update user preferences stored in the metadata database. | @@ -78,33 +80,38 @@ This directory contains API route definitions for different functionalities of t This directory contains Pydantic models defining the structure and validation of data exchanged through the API endpoints. -| Name | Description | -| --------------------- | --------------------------------------------------------------- | -| `album.py` | For validating and structuring album-related API requests. | -| `face_clusters.py` | For requests and responses related to face cluster management. | -| `facetagging.py` | Face matching, clustering, related images, and error responses. | -| `folders.py` | Folder-related API requests, responses, and data structures | -| `images.py` | Image management requests and responses, including deletions. | -| `test.py` | Tests image detection requests, responses, and error handling. | -| `user_preferences.py` | User preferences API requests, responses, and error handling. | +| Name | Description | +| --------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `album.py` | For validating and structuring album-related API requests. | +| `face_clusters.py` | For requests and responses related to face cluster management. | +| `facetagging.py` | Face matching, clustering, related images, and error responses. | +| `folders.py` | Folder-related API requests, responses, and data structures | +| `images.py` | Image management requests and responses, including deletions. | +| `memories.py` | Memory cards, stories, and the generate/status/update/delete request and response models. | +| `test.py` | Tests image detection requests, responses, and error handling. | +| `user_preferences.py` | User preferences API requests, responses, and error handling, including the `memories` block and its scoring weights. | ## utils This directory contains utility functions and helper modules used across the application. -| Name | Description | -| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `API.py` | Sends POST request to restart sync microservice, logs success or failure | -| `face_clusters.py` | Clusters face embeddings, updates clusters, generates cluster images. | -| `FaceNet.py` | Preprocesses images, normalizes embeddings, computes similarity. | -| `folders.py` | Manages folder trees: add, delete, sync folders in database and filesystem. | -| `image_metadata.py` | Extracts image metadata including EXIF,size,format, and creation date safely | -| `images.py` | Processes images in folders: thumbnails, detects faces, classifies,updates DB | -| `memory_monitor.py` | Decorator logs memory usage and execution time of functions. | -| `microservice.py` | Starts sync microservice with virtual environment or bundled executable. | -| `ONNX.py` | Returns ONNX execution providers list based on GPU acceleration preference. | -| `SigLIP.py` | Preprocesses images for SigLIP2, tokenizes search queries (with a thread-safe tokenizer cache), and caches the text-tower session across `/semantic-search` requests. | -| `YOLO.py` | YOLO utilities for NMS, drawing, and model path from preferences. | +| Name | Description | +| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `API.py` | Sends POST request to restart sync microservice, logs success or failure | +| `face_clusters.py` | Clusters face embeddings, updates clusters, generates cluster images. | +| `FaceNet.py` | Preprocesses images, normalizes embeddings, computes similarity. | +| `folders.py` | Manages folder trees: add, delete, sync folders in database and filesystem. | +| `image_metadata.py` | Extracts image metadata including EXIF,size,format, and creation date safely | +| `images.py` | Processes images in folders: thumbnails, detects faces, classifies,updates DB | +| `memory_curator.py` | Runs the three memory triggers (anniversary, import event, semantic event) and the rescore entry points; see [Memories](memories.md). | +| `memory_monitor.py` | Profiling decorator logging a function's RAM usage and execution time (unrelated to the Memories feature). | +| `memory_scoring.py` | Scores candidate images and videos from stored signals, then dedupes and time-spreads the survivors. | +| `microservice.py` | Starts sync microservice with virtual environment or bundled executable. | +| `ONNX.py` | Returns ONNX execution providers list based on GPU acceleration preference. | +| `SigLIP.py` | Preprocesses images for SigLIP2, tokenizes search queries (with a thread-safe tokenizer cache), and caches the text-tower session across `/semantic-search` requests. | +| `takeout_sidecar.py` | Reads capture date and GPS from a Google Takeout sidecar JSON file next to an image or video. | +| `video_capture_date.py` | Reads capture timestamps straight out of MP4/MOV container boxes, without ffprobe. | +| `YOLO.py` | YOLO utilities for NMS, drawing, and model path from preferences. | ## scripts diff --git a/docs/backend/backend_python/image-processing.md b/docs/backend/backend_python/image-processing.md index 390c45157..095274143 100644 --- a/docs/backend/backend_python/image-processing.md +++ b/docs/backend/backend_python/image-processing.md @@ -76,6 +76,105 @@ For the full technical breakdown — architecture diagrams, database schema, model calibration details, and known limitations — see the dedicated [Semantic Search](semantic-search.md) page. +## Capture Dates and Location + +Indexing records two different dates per file, and they are not +interchangeable. + +| Where it lives | What it holds | Who reads it | +| ------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------- | +| `date_created`, inside the `metadata` JSON column | The best date available, always populated — it falls back to the file's modification time | Gallery display and sorting | +| `captured_at`, a column on `images` and `videos` | When the shutter actually fired; `NULL` when nothing trustworthy was found | The Memories feature, and nothing else | + +The same `metadata` JSON carries a `date_source` field saying where the date +came from: + +| `date_source` | Meaning | Trusted | +| ------------- | ------------------------------------------------ | ------- | +| `exif` | Read from the image's own EXIF tags | Yes | +| `sidecar` | Read from a Google Takeout sidecar JSON file | Yes | +| `container` | Read from an MP4/MOV container box | Yes | +| `filesystem` | The file's mtime, standing in for a capture date | No | +| `unknown` | The file could not be opened or stat'd at all | No | + +The trusted set is `TRUSTED_DATE_SOURCES` in +`app/utils/extract_location_metadata.py`. `MetadataExtractor.extract_datetime()` +returns `None` when `date_source` is anything outside it, and +`video_util_prepare_video_records()` applies the same test, so an untrusted +date is written to `captured_at` as `NULL` while `date_created` keeps it for +the UI. + +### EXIF + +`DateTimeOriginal` is not in IFD0 — it sits behind the EXIF sub-IFD pointer +`0x8769`, and Pillow's `getexif()` returns IFD0 only. `_extract_capture_datetime()` +searches both: for each of `DateTimeOriginal`, `DateTimeDigitized` and +`DateTime`, in that order, it reads the sub-IFD first and then IFD0. If nothing +parses, `date_created` falls back to the file mtime and `date_source` becomes +`filesystem`. + +GPS comes off the same EXIF object via `_extract_gps_coordinates()`, which +reads the GPSInfo IFD and converts degrees/minutes/seconds to signed decimal +degrees. + +### Google Takeout sidecars + +A Google Photos export strips EXIF from part of its own library and parks the +real values in a sibling JSON file. When EXIF yields no date or no coordinates, +`takeout_sidecar_read()` (`app/utils/takeout_sidecar.py`) looks for that file. + +| Sidecar spelling | How it is found | +| -------------------------------------- | -------------------------------------- | +| `.supplemental-metadata.json` | Probed by exact name | +| `.json` | Probed by exact name | +| `.supplemental-metadata(1).json` | Found by prefix in a directory listing | + +The exact spellings cost one `stat` each and are tried first; the directory is +only listed if neither produced a usable sidecar, and that listing is cached — +one entry, keyed on the directory path and its mtime — so a folder of N photos +does not cost N scans. + +Once a sidecar is open: + +- `photoTakenTime` is preferred over `creationTime`, which is the upload time. +- Coordinates come from `geoData`, falling back to `geoDataExif`. +- Takeout writes `0/0` rather than null for "no location", so an exact + (0.0, 0.0) pair is discarded. +- Album-level metadata files match no image and are skipped by key: a payload + containing `entries` or `albumData` is rejected. + +The datetime is returned in EXIF format (`%Y:%m:%d %H:%M:%S`) so callers parse +it exactly as they parse a real EXIF value. + +### Videos + +Video capture dates are parsed straight out of the ISO base media container +boxes by `app/utils/video_capture_date.py`. There is no ffmpeg or ffprobe +dependency — PictoPy ships through PyInstaller and cannot assume either exists +on the machine. OpenCV, which handles the poster frame, exposes no creation +date at all. + +`_resolve_capture_date()` in `app/utils/videos.py` takes the first source that +answers: + +| Order | Source | `date_source` | Notes | +| ----- | --------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------ | +| 1 | `com.apple.quicktime.creationdate` in `moov/meta` (keys + ilst) | `container` | Carries its own UTC offset, so it yields the wall-clock time of the shot | +| 2 | Google Takeout sidecar | `sidecar` | Same reader as photos | +| 3 | `moov/mvhd` creation time | `container` | UTC seconds since 1904-01-01, converted to local time; a re-encode rewrites it | +| 4 | File mtime | `filesystem` | Untrusted, so it never reaches `captured_at` | + +Values outside 1990-01-01 through two days from now are treated as noise — a +container can carry 0, or a clock that was never set — and a malformed keys +table is bounded at `MAX_METADATA_KEYS = 512` entries. + +The filesystem mtime is recorded in its own field, `metadata["file_modified"]`. +`video_util_source_is_unchanged()` compares a video's size and that stored mtime +to decide whether to re-index it; a row without `file_modified` is treated as +changed, so its container is read once and its date corrected. + +For what consumes `captured_at`, see the [Memories](memories.md) page. + ## How It All Fits Together When you add a new photo, we first look for objects and faces. If we find faces, we generate embeddings for them. These embeddings then get added to our face clusters. @@ -126,13 +225,13 @@ Here are some key parameters for the main models used in PictoPy's image process ### Semantic Search (SigLIP2) -| Parameter | Value | Description | -| -------------------------- | ---------------------------------- | ---------------------------------------------------------------------------- | -| Default checkpoint | `base` | Set via `SIGLIP2_ACTIVE_CHECKPOINT`; `large` and `so400m` also exist but ship placeholder registry entries only (see [Semantic Search](semantic-search.md#model-distribution-and-checkpoints)). | -| Input resolution (`base`) | 224 × 224 | Larger checkpoints use 384 × 384. | -| Embedding dimension | 768 | Same dimensionality for both the image and text towers. | -| `SIGLIP2_EMBED_BATCH_SIZE` | 8 | Images per batch during the background embedding pass. | -| `SIGLIP2_MATCH_THRESHOLD` | 0.01 | Minimum sigmoid score to count as a match. SigLIP2's absolute scores run low even for real matches — this is expected, not a bug. | -| Output | Sorted, scored image list | Scores are rounded to 4 decimal places server-side and never shown in the UI. | +| Parameter | Value | Description | +| -------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Default checkpoint | `base` | Set via `SIGLIP2_ACTIVE_CHECKPOINT`; `large` and `so400m` also exist but ship placeholder registry entries only (see [Semantic Search](semantic-search.md#model-distribution-and-checkpoints)). | +| Input resolution (`base`) | 224 × 224 | Larger checkpoints use 384 × 384. | +| Embedding dimension | 768 | Same dimensionality for both the image and text towers. | +| `SIGLIP2_EMBED_BATCH_SIZE` | 8 | Images per batch during the background embedding pass. | +| `SIGLIP2_MATCH_THRESHOLD` | 0.01 | Minimum sigmoid score to count as a match. SigLIP2's absolute scores run low even for real matches — this is expected, not a bug. | +| Output | Sorted, scored image list | Scores are rounded to 4 decimal places server-side and never shown in the UI. | Note: Some of these values are default parameters and can be adjusted when initializing the models or during runtime, depending on the specific use case or performance requirements. diff --git a/docs/backend/backend_python/memories.md b/docs/backend/backend_python/memories.md new file mode 100644 index 000000000..cab616e1b --- /dev/null +++ b/docs/backend/backend_python/memories.md @@ -0,0 +1,606 @@ +# Smart Memories + +This page is the backend reference for Smart Memories: persisted, scored +collections of photos and short clips that PictoPy surfaces proactively +instead of waiting to be browsed. For the story viewer, the grid and the +settings screen that consume this data, see +[Frontend Memories](../../frontend/memories.md). Scoring and titling both lean +on the SigLIP2 embeddings and the curated label vocabulary described in +[Semantic Search](semantic-search.md). + +## What a memory is + +A **memory** is a database row plus an ordered set of images and videos drawn +from the library. It is produced by a **curation run**, which writes rows; +the UI only ever reads them back. Nothing is computed at request time. + +Three triggers produce memories, and each run executes all three: + +| Trigger (`event_type`) | Candidate set | +| ---------------------- | ---------------------------------------------------------- | +| `anniversary` | Photos captured on today's calendar date in previous years | +| `import_event` | A burst of media that hangs together in time and place | +| `semantic_event` | Photos SigLIP2 recognizes as one occasion | + +The backend files involved: + +| Path | Role | +| ----------------------------------------- | ----------------------------------------------------------------- | +| `backend/app/database/memories.py` | Schema and every memories query helper | +| `backend/app/utils/memory_scoring.py` | Per-item signals, normalization, dedupe, cohesion, time spreading | +| `backend/app/utils/memory_curator.py` | The three triggers, run entry point, rescore entry points | +| `backend/app/routes/memories.py` | API router, mounted at `/memories` | +| `backend/app/schemas/memories.py` | Pydantic request/response models | +| `backend/app/schemas/user_preferences.py` | `MemoriesPreferences`, `MemoryScoringWeights` | + +## Database schema + +Four tables, all created by `db_create_memories_table()` +(`backend/app/database/memories.py`). + +```mermaid +erDiagram + memories ||--o{ memory_images : "curated stills" + memories ||--o{ memory_videos : "curated clips" + images ||--o{ memory_images : "" + images ||--o{ memories : "cover_image_id" + videos ||--o{ memory_videos : "" + + memories { + TEXT memory_id PK + TEXT dedupe_key UK "identity across re-curation" + TEXT event_type "CHECK against EVENT_TYPES" + TEXT status "CHECK against MEMORY_STATUSES" + TEXT title + TEXT subtitle + TEXT place_label + REAL center_lat + REAL center_lon + DATE surface_date "not shown before this date" + DATETIME period_start "min captured_at of members" + DATETIME period_end "max captured_at of members" + TEXT cover_image_id FK "ON DELETE SET NULL" + INTEGER image_count + INTEGER video_count + REAL score "memory-level rank" + TEXT signals "JSON, cover's signal breakdown" + TEXT params_signature + TEXT error + DATETIME notified_at + DATETIME viewed_at + BOOLEAN dismissed "default 0" + DATETIME created_at + DATETIME updated_at + } + memory_images { + TEXT memory_id PK, FK "ON DELETE CASCADE" + TEXT image_id PK, FK "ON DELETE CASCADE" + INTEGER sort_order + REAL score + } + memory_videos { + TEXT memory_id PK, FK "ON DELETE CASCADE" + TEXT video_id PK, FK "ON DELETE CASCADE" + INTEGER sort_order + REAL score + } + memory_runs { + DATE run_date PK + TEXT status "CHECK against RUN_STATUSES" + TEXT params_signature + INTEGER generated_count + TEXT error + DATETIME started_at + DATETIME finished_at + } +``` + +`memory_videos` is a second join table rather than a `media_type` column on +`memory_images`, because one id column cannot carry a foreign key to two +parents. **`sort_order` is a single sequence shared across both tables**, so a +story reads chronologically no matter how photos and clips interleave. +`interleave_by_time()` in `memory_scoring.py` numbers them. + +`memory_runs` records one row per curation run. Producing zero memories on a +given day is a legitimate outcome, so "has today already run?" cannot be +answered from the `memories` table alone. + +### Vocabularies + +Three module constants define the allowed values, and the `CHECK` constraints +are built from those same tuples by `_check_in()` so the two cannot drift: + +```python +EVENT_TYPES = ("anniversary", "import_event", "semantic_event") +MEMORY_STATUSES = ("pending", "complete", "failed", "empty") +RUN_STATUSES = ("running", "complete", "failed") +``` + +Curation writes `status = 'complete'`; `'pending'` is the column default and +`'empty'` is set by `db_prune_empty_memories()` when a memory's live image +count falls below `min_images`. The `Literal` types in +`backend/app/schemas/memories.py` mirror these three vocabularies. + +### `dedupe_key` + +`dedupe_key` is the stable identity of a memory across re-curation. +`db_upsert_memory()` resolves conflicts on it: the pre-existing `memory_id` +survives, and `viewed_at` / `notified_at` / `dismissed` are deliberately +excluded from the update column list, so rebuilding a memory never resets what +the user has already seen. + +| Trigger | Format | Example | +| ---------------- | ---------------------------------- | ------------------------------- | +| `anniversary` | `anniv:{MM-DD}:{year}` | `anniv:07-29:2021` | +| `import_event` | `import:{start_date}..{end_date}` | `import:2026-07-11..2026-07-13` | +| `semantic_event` | `semantic:{class_id}:{start_date}` | `semantic:1184:2026-02-14` | + +### Indexes and creation order + +Five indexes are created alongside the tables: + +| Index | On | +| ---------------------------- | --------------------------------------------------------------------------------------------- | +| `ix_memories_surface_date` | `memories(surface_date DESC)` | +| `ix_memories_status_surface` | `memories(status, surface_date DESC)` | +| `ix_memories_surfaceable` | Partial index over `(surface_date DESC, score DESC)` where unviewed, undismissed and complete | +| `ix_memory_images_image_id` | `memory_images(image_id)` | +| `ix_memory_videos_video_id` | `memory_videos(video_id)` | + +`ix_memories_surfaceable` backs the "is there anything to show?" lookup; the +two `image_id`/`video_id` indexes exist so deleting a photo does not scan the +join tables. + +`db_create_memories_table()` runs inside `lifespan()` in `backend/main.py`, +**after** `db_create_images_table()` and `db_create_videos_table()` — it +declares foreign keys into both. Databases created before `video_count` +existed get it through a guarded `ALTER`, since `CREATE TABLE IF NOT EXISTS` +is a no-op against an existing table. + +### Query helpers + +`backend/app/database/memories.py` is the only module that touches these +tables. The helpers worth knowing: + +| Helper | Purpose | +| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `db_upsert_memory` | Write a memory and both join tables in one transaction | +| `db_get_memory` / `db_get_memory_images` / `db_get_memory_videos` | Story payload, ordered by `sort_order` | +| `db_list_memories` | Paginated cards plus a total count | +| `db_get_surfaceable_memory` | The single best memory to surface now | +| `db_delete_stale_memories` | Drop memories whose members' capture dates moved outside `period_start`..`period_end` | +| `db_prune_empty_memories` | Mark memories that shrank below `min_images` as `'empty'` | +| `db_get_scoring_signals` / `db_get_video_scoring_signals` | Every raw signal for a candidate set, in one pass | +| `db_get_anniversary_candidates` / `db_get_recent_dated_images` / `db_get_images_in_period` | The three candidate pools | +| `db_get_event_labels` / `db_get_event_label_hits` / `db_get_top_memory_label` | Semantic label reads | +| `db_get_gps_histogram` / `db_get_gps_cell_centre` | Home-location detection | +| `db_start_memory_run` / `db_finish_memory_run` / `db_get_memory_run` / `db_reap_stale_memory_runs` | Run bookkeeping | +| `db_is_indexing_busy` | Whether any folder is mid-index or mid-tagging | + +`db_get_scoring_signals` uses correlated subqueries rather than joins across +`faces` / `album_images` / `image_classes`, and chunks its id list at +`SQLITE_ID_CHUNK`. + +## Curation + +### Entry point + +```python +memory_curator_run(reference_date=None, force=False, trigger="manual") -> int +``` + +Returns the number of memories written. **It never raises** — its callers are +import hooks and background tasks where a curation failure must not fail the +surrounding work. + +The run sequence: + +```mermaid +sequenceDiagram + participant Caller as API / folder hook + participant Run as memory_curator_run() + participant Runs as memory_runs + participant Triggers as anniversary → semantic_event → import_event + participant DB as memories tables + + Caller->>Run: reference_date, force, trigger + Run->>Run: memory_curator_get_preferences() + alt disabled and not force + Run->>Runs: release any claimed run as 'failed' + Run-->>Caller: 0 + end + Run->>Runs: db_start_memory_run(run_date, params_signature) + Run->>DB: db_delete_stale_memories() (guarded; failure is non-fatal) + Run->>Run: build _CurationContext + Note over Run: home location, active model_version,
recently-used ids, library cohesion baseline + loop each trigger, in order + Run->>Triggers: curate(context) + Note over Run,Triggers: each trigger is individually guarded —
one failure does not cost the others their output + Run->>Run: refresh recently_used so the next trigger
cannot reuse what this one just claimed + end + Run->>Runs: db_finish_memory_run('complete', generated) + Run-->>Caller: generated +``` + +`GENERATOR_VERSION = 2` is bumped when curation changes in a way that makes +existing memories stale. `memory_curator_params_signature()` hashes it +together with `min_images`, `max_images` and the weight set, and the resulting +16-character digest is stored on every memory as `params_signature`. + +`_CurationContext` holds what every trigger shares within one run: the +reference date, preferences and weights, the detected home location, the +active SigLIP2 `model_version`, the recently-used image ids, and +`cohesion_baseline` — the mean pairwise cosine of a `COHESION_SAMPLE_SIZE` +(500) sample of the library's embeddings. + +### The shared build path + +All three triggers differ only in how candidates are found. `_build_memory()` +does everything after that, and returns whether the set qualified: + +1. Optionally hard-exclude recently-used images (a memory's own photos, looked + up by `dedupe_key`, are never excluded from itself). +2. Bail if fewer than `min_images` candidates remain. +3. `db_get_scoring_signals()` → `score_candidates()`, highest first. When + recent use is a penalty rather than an exclusion, those ids are multiplied + by `RECENT_USE_PENALTY` (0.35). +4. Fetch embeddings, then `suppress_near_duplicates()`. +5. `trim_incoherent()`, for triggers that opt into it. +6. `spread_over_time()` down to `max_images`. +7. Re-title from the surviving set, if the trigger supplied a `rename` hook. +8. Cover = the highest-scoring **still**; clips are never covers. +9. Select videos (see below), then `interleave_by_time()` to build the shared + `sort_order`. +10. `db_upsert_memory()` with `status = 'complete'`. + +Every one of steps 2, 3, 4 and 6 re-checks `min_images` and abandons the +memory if it no longer clears the bar. + +### Trigger rules + +| | `anniversary` | `import_event` | `semantic_event` | +| ---------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------- | +| Candidate pool | `db_get_anniversary_candidates` over an `MM-DD` ±1 day window, years ≤ `reference.year - 1` | `db_get_recent_dated_images(5000)` | `db_get_event_label_hits` over active `event` labels | +| Grouping | By capture year | `segment_by_time_and_place`: split on a gap > 8 h or a jump > 40 km | `group_event_occurrences`: split each label's hits on a 36 h gap | +| Rejection | Years with fewer than `min_images` | Segments spanning more than 14 days, or under `min_images` | Fewer than 6 images, or cohesion below baseline + 0.15 | +| Ranking | Most photos first, then most recent year | Most recent segment first | Most images first | +| Cap | 2 | 3 | 3 | +| Recent use | Penalized | Excluded | Excluded | +| Outlier trimming | No | Yes | No | +| `surface_date` | The reference date | The run date | Upcoming anniversary within 3 days, else the run date | + +Notes on each: + +- **Anniversary** — the ±1 day window absorbs timezone and EXIF drift, since + an EXIF timestamp carries the camera's local time. The subtitle comes from + the photos' own timestamps rather than being rebuilt from the reference + date, because Feb 29 has no counterpart in a non-leap source year. Titles + are `"1 year ago today"` / `"N years ago today"`. Outlier trimming is off: + an anniversary spans years and is not supposed to look visually of a piece. +- **Import event** — segmentation is temporal, so two separate trips to the + same place stay two memories. Before naming, `trim_incoherent()` drops + photos that do not look like the rest of the segment. +- **Semantic event** — hits are selected by rank, not raw score (see + [Scoring](#scoring) and the label-rank discussion in + `db_get_event_label_hits`). Occurrences overlapping by ≥ 60% of the shorter + span merge into one, with the primary label being whichever contributed more + total confidence. Once an occurrence qualifies, `db_get_images_in_period()` + pulls in the unlabeled photos captured between the recognized ones. + +### Titling + +| Trigger | Title | Subtitle | +| ---------------- | ------------------------------------------------ | ------------------------------------- | +| `anniversary` | `"N years ago today"` | Month and year of the earliest member | +| `import_event` | Top label if one qualifies, else a generic title | The formatted date span | +| `semantic_event` | The occurrence's primary label | Month and year of the event start | + +`db_get_top_memory_label()` picks the label for an import event, and it runs +**after** trimming, on the final image set — a photo that was just dropped for +not belonging cannot vote on the name. It ranks by **summed percentile**, +which is count-dominant, and applies four gates: `EVENT_LABEL_TOP_N` (2) on +the per-image rank, `EVENT_LABEL_PERCENTILE` (0.50) on the within-label +percentile, `TITLE_MIN_IMAGES` (2), and `TITLE_MIN_SHARE` (0.15) of the +memory. The `event` and `scene` categories compete on equal terms; +`attribute` and `object` are never titles. + +When no label qualifies, a stand-in from `GENERIC_TITLES_ONE_DAY` or +`GENERIC_TITLES_MANY_DAYS` is chosen by `sha256(dedupe_key)`, so a rebuild +never renames a memory the user has already been shown, and the date moves to +the subtitle. `EVENT_DISPLAY_NAMES` overrides labels whose title-cased form +reads wrong (`valentines day` → `Valentine's Day`, `bbq` → `BBQ`). + +### Video selection + +Clips are punctuation between stills, and only run when +`MemoriesPreferences.include_videos` is on. + +```python +def video_quota(photo_count: int) -> int: + if photo_count <= 0: + return 0 + return min(MAX_VIDEOS_PER_MEMORY, max(1, round(photo_count / PHOTOS_PER_VIDEO))) +``` + +Candidates come from `db_get_video_candidates_in_period()`, bounded by the +span the selected photos already cover. A clip must have a **known** duration +in `0 < d <= MAX_VIDEO_SECONDS`; an unknown length is skipped rather than +gambled on. `select_videos_within_budget()` then takes the best-scoring clips +that fit `MAX_VIDEO_SECONDS_PER_MEMORY`, shortest winning a tie. The whole +video pass is wrapped in its own `try` — a story of photos is still a story. + +### Key constants + +All in `backend/app/utils/memory_curator.py`: + +| Constant | Value | Effect | +| -------------------------------------- | ------------ | ------------------------------------------------ | +| `GENERATOR_VERSION` | `2` | Feeds `params_signature` | +| `ANNIVERSARY_DAY_WINDOW` | `1` | ±1 day around today's `MM-DD` | +| `MAX_ANNIVERSARY_MEMORIES` | `2` | Anniversary memories per run | +| `IMPORT_GAP_HOURS` | `8.0` | Gap that ends an import segment | +| `IMPORT_JUMP_KM` | `40.0` | Distance jump that ends an import segment | +| `IMPORT_MAX_SPAN_DAYS` | `14` | Longest accepted import segment | +| `IMPORT_CANDIDATE_LIMIT` | `5000` | Import candidate pool cap | +| `MAX_IMPORT_MEMORIES` | `3` | Import memories per run | +| `EVENT_GAP_HOURS` | `36.0` | Gap that splits a label's hits into occurrences | +| `EVENT_MIN_IMAGES` | `6` | Minimum size of a semantic occurrence | +| `EVENT_LABEL_TOP_N` | `2` | Per-image rank a label must reach | +| `EVENT_LABEL_PERCENTILE` | `0.50` | Within-label percentile a hit must reach | +| `EVENT_COHESION_MARGIN` | `0.15` | Required margin over the library baseline | +| `COHESION_SAMPLE_SIZE` | `500` | Embeddings sampled to compute that baseline | +| `EVENT_MERGE_OVERLAP` | `0.60` | Overlap ratio that merges two occurrences | +| `EVENT_ANNIVERSARY_LOOKAHEAD_DAYS` | `3` | How far ahead a semantic anniversary may be held | +| `MAX_SEMANTIC_MEMORIES` | `3` | Semantic memories per run | +| `TITLE_MIN_SHARE` / `TITLE_MIN_IMAGES` | `0.15` / `2` | Coverage a label needs to name a memory | +| `PHOTOS_PER_VIDEO` | `9` | One clip earned per nine photos | +| `MAX_VIDEOS_PER_MEMORY` | `3` | Clip cap | +| `MAX_VIDEO_SECONDS` | `15.0` | Longest single clip | +| `MAX_VIDEO_SECONDS_PER_MEMORY` | `30.0` | Total clip budget | +| `RECENT_USE_WINDOW_DAYS` | `30` | Window that counts as "recently used" | +| `RECENT_USE_PENALTY` | `0.35` | Multiplier applied when recent use is a penalty | + +## Scoring + +`backend/app/utils/memory_scoring.py` scores one item at a time. Every signal +is normalized to 0–1, weighted, and summed. + +### The seven signals + +| Signal | Source | Default weight | +| --------------------- | --------------------------------------------------------------------------------- | -------------- | +| `favourite` | `images.isFavourite` | 0.22 | +| `known_people` | Distinct named `face_clusters` on the image, saturating at `MAX_NAMED_PEOPLE` (3) | 0.20 | +| `event_strength` | `MAX(image_classes.score)` where the label's `category = 'event'` | 0.18 | +| `face_presence` | Face count, saturating at `MAX_FACES` (4) | 0.12 | +| `semantic_confidence` | `MAX(image_classes.score)` for `class_id >= SEMANTIC_CLASS_ID_OFFSET` | 0.10 | +| `gps_novelty` | `1 - exp(-d / 50 km)`, `d` = haversine distance to the detected home | 0.10 | +| `in_album` | `EXISTS(album_images)` | 0.08 | + +Defaults live on `MemoryScoringWeights` +(`backend/app/schemas/user_preferences.py`) and are **normalized to sum to +1.0 on validation**, so a UI slider set never has to land on an exact total. +An all-zero set falls back to the defaults rather than dividing by zero. + +Home is the densest rounded GPS cell (`HOME_CELL_PRECISION = 1`, roughly +11 km) with at least `MIN_IMAGES_FOR_HOME` (20) geotagged photos, resolved to +the mean coordinate of that cell. Below the floor, `detect_home_location()` +returns `None`, which disables `gps_novelty` entirely. + +### Availability renormalization + +`compute_signals()` returns both the values and the set of signals that +actually have data behind them, and `composite_score()` divides by the weight +of what was available: + +```python +score = Σ(wᵢ · sᵢ) / Σ(wᵢ) for i in available +``` + +A missing sensor reading is therefore not a demerit. + +| Signal | Availability | +| -------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `favourite`, `known_people`, `face_presence`, `in_album` | Always (`ALWAYS_AVAILABLE`) — a landscape genuinely has no faces | +| `semantic_confidence`, `event_strength` | Only once the image has a `scored_signature` — an unscored image is not a boring image | +| `gps_novelty` | Only with both coordinates present **and** a detected home | + +### Videos + +`db_get_video_scoring_signals()` returns rows shaped like the image ones but +tagged `media_type = "video"`, which removes `UNAVAILABLE_FOR_VIDEO` — +`known_people`, `face_presence` and `in_album` — from the available set rather +than scoring them zero. It also sets `latitude`/`longitude` to `None`, so +`gps_novelty` is unavailable too. In practice a clip is ranked on +`favourite`, `event_strength` and `semantic_confidence`, read from +`video_classes` and from `video_frame_embeddings.scored_signature`. + +### Near-duplicate suppression + +`suppress_near_duplicates()` requires **both** conditions: + +| Condition | Constant | Value | +| -------------------- | -------------------- | --------------- | +| Cosine similarity ≥ | `DUP_COSINE` | `0.90` | +| Capture times within | `DUP_WINDOW_SECONDS` | `120.0` seconds | + +Candidates must arrive best-first; the survivor of a duplicate pair is +whichever was already ranked higher. An item missing an embedding or a +timestamp cannot satisfy the pair test and is kept. + +### Cohesion and trimming + +`mean_pairwise_cohesion()` is the mean cosine between every pair in a group, +excluding the diagonal. It is preferred over cosine-to-centroid because it +does not drift with group size. `cohesion_baseline()` runs it over a random +sample of the library, and every cohesion test in the codebase is expressed as +a margin over that baseline rather than as an absolute cosine. + +`trim_incoherent()` computes each item's leave-one-out cohesion to the rest, +then drops anything below: + +```python +threshold = median - max(COHESION_MAD_SCALE * MAD_TO_STD * mad, COHESION_MIN_MARGIN) +``` + +with `COHESION_MAD_SCALE = 2.0`, `MAD_TO_STD = 1.4826` and +`COHESION_MIN_MARGIN = 0.10`. Median and MAD rather than mean and standard +deviation, and the trim is abandoned entirely if it would remove more than +`COHESION_MAX_TRIM_RATIO` (0.4) of the group, or if fewer than three +embeddings are available. + +### Selection and ordering + +- `spread_over_time()` buckets the event's span into `target` slices and takes + the best candidate from each, backfilling by score where a bucket is empty. + This is what stops ten frames from one minute of a three-day trip filling + the whole story. +- `_chronological()` then orders the survivors for playback, breaking ties on + score. +- `interleave_by_time()` merges photos and clips into the one shared + `sort_order` sequence and returns the join-table rows for each. + +### Memory-level score + +`aggregate_memory_score()` produces the `memories.score` used to rank memories +against each other: + +```python +base = mean(top 5 image scores) +size_boost = 1.0 + 0.15 * (log1p(n) / log1p(30)) +diversity_boost = 1.0 + 0.10 * min(1.0, mean(class_count) / 10.0) +score = base * size_boost * diversity_boost +``` + +## API + +Router: `backend/app/routes/memories.py`, mounted at `/memories` in +`backend/main.py`. Every endpoint declares a `response_model` and returns the +project's `{success, message, data}` envelope. Full request and response +schemas are in the live [API Reference](api.md). + +| Method | Path | Purpose | +| -------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `POST` | `/memories/generate` | Queue a curation run. Body: `{force, reference_date}`. Returns once queued, not once finished. | +| `GET` | `/memories/status` | `run_date`, `run_status`, `run_started_at`, `indexing_busy`, `unviewed_count`, `latest_memory_id`, `memories_enabled`, `notifications_enabled` | +| `GET` | `/memories/today` | The one memory to surface now | +| `GET` | `/memories` | Paginated cards. Query: `limit`, `offset`, `event_type`, `include_viewed`, `include_dismissed` | +| `GET` | `/memories/{memory_id}` | Full story payload | +| `PATCH` | `/memories/{memory_id}` | `{viewed, dismissed, notified}` | +| `DELETE` | `/memories/{memory_id}` | Delete a memory; its photos remain | + +Behavior not obvious from the schemas: + +| Condition | Response | +| -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | +| Nothing qualifies for `/today` | `200` with `data.memory = null` — an empty library is a normal state for a polled endpoint, not a `404` | +| A run for that date is already `running` | `200`, `queued: false`, `status: "running"` | +| A run for that date is already `complete` and `force` is false | `200`, `queued: false`, `status: "complete"` | +| Memories are disabled and `force` is false | `200`, `queued: false` — no run is claimed | +| `reference_date` is not an ISO date | `400` | +| `PATCH` body sets none of `viewed`/`dismissed`/`notified` | `400` | + +`MemoryStory` carries `images` and `videos` as **two separate arrays**; the +viewer merges them on `sort_order`. Card and story rows both report a live +image count, recounted from `memory_images` rather than trusting the stored +`image_count`. + +### Run claiming + +`POST /generate` submits `memory_curator_run` to the app's shared +single-worker `ProcessPoolExecutor`, so curation serializes behind indexing +and semantic scoring instead of racing them. The route **claims the run** +(`db_start_memory_run`) before handing off, so a second caller arriving while +the executor is still starting sees `running` rather than queueing a duplicate +pass. Three paths release a claim nothing will finish: + +| Path | Guard | +| ---------------------------------------------------------- | ----------------------------------------------------------------- | +| `executor.submit` itself raises | `_release_run()` in the route | +| The worker process dies mid-curation | `_release_run_if_the_worker_died()`, via a `Future` done-callback | +| The curator declines the run because memories are disabled | `_release_claimed_run()` in `memory_curator.py` | + +A release records the run as `failed`, not `complete`, so the next attempt is +free to run. Anything still `running` past `STALE_RUN_MINUTES` (30) is reaped +by `db_reap_stale_memory_runs()`, which both `/generate` and `/status` call +before reading the run row. + +## Pipeline integration + +Curation is not on a timer inside the backend. It runs from the points where +the library has just changed. + +```mermaid +graph LR + FA["post_folder_add_sequence
(after indexing)"] --> CM["_curate_memories(trigger)"] + AT["post_AI_tagging_enabled_sequence
(after semantic_util_score_images)"] --> CM + SF["post_sync_folder_sequence
(after semantic_util_score_images)"] --> CM + CM --> Run["memory_curator_run()"] + API["POST /memories/generate"] --> Exec["shared ProcessPoolExecutor"] + Exec --> Run + Rename["PUT /face-clusters/{id}
(rename a person)"] --> Exec2["shared ProcessPoolExecutor"] + Exec2 --> Rescore["memory_curator_rescore_for_cluster()"] +``` + +`_curate_memories(trigger)` lives in `backend/app/routes/folders.py`. It +imports the curator late, to keep it out of the module import graph, and +swallows failures so a curation problem never fails an import. It never passes +`force`: a background import is not the user asking for memories. + +| Hook | Position | Trigger name | +| ---------------------------------- | ----------------------------------------------------------- | ------------- | +| `post_folder_add_sequence` | After indexing completes | `folder_add` | +| `post_AI_tagging_enabled_sequence` | After `semantic_util_score_images()`, before the video pass | `ai_tagging` | +| `post_sync_folder_sequence` | After `semantic_util_score_images()`, before the video pass | `sync_folder` | + +The two AI hooks run before the video pass because semantic labels are written +by that point and the video pass can run for minutes. At `folder_add` no AI +has run yet, so only the date-driven triggers can produce anything; semantic +events appear once tagging is enabled and curation runs again. + +`lifespan()` in `backend/main.py` creates the tables but **does not curate**. + +### Rescore on rename + +Renaming a face cluster changes the `known_people` signal for photos that are +already curated. `PUT /face-clusters/{cluster_id}` queues +`memory_curator_rescore_for_cluster` on the shared executor (failures +swallowed — renaming a person must succeed either way), which resolves the +affected memories through `db_get_memory_ids_for_cluster()` and calls +`memory_curator_rescore()`. + +The rescore is deliberately **in place**: `db_update_memory_scores()` rewrites +each image's score, the cover, and the memory's own score. Membership and +`sort_order` are untouched, so a memory the user may already have watched is +never reshuffled. + +## User preferences + +`MemoriesPreferences` sits under `user_preferences.memories` and is served by +`GET` / `PUT /user-preferences/`. `MemoriesPreferencesUpdate` and +`MemoryScoringWeightsUpdate` are the partial-update models — every field is +optional. + +| Field | Type | Default | Bounds | +| ------------------------ | ---------------------- | ------------------------ | ------------ | +| `enabled` | `bool` | `True` | — | +| `notifications_enabled` | `bool` | `True` | — | +| `story_music_enabled` | `bool` | `False` | — | +| `slide_duration_seconds` | `float` | `5.0` | 1.0–30.0 | +| `include_videos` | `bool` | `True` | — | +| `min_images` | `int` | `5` | 2–50 | +| `max_images` | `int` | `30` | 5–100 | +| `weights` | `MemoryScoringWeights` | The seven defaults above | Each 0.0–1.0 | + +A model validator rejects a set where `max_images < min_images`. A video slide +runs for its own length rather than `slide_duration_seconds`. + +The curator reads preferences through `memory_curator_get_preferences()`, +which is **read-only by design**: `db_update_metadata` rewrites the whole +metadata blob, so a write from the curator process would clobber a concurrent +settings save. Invalid stored preferences log a warning and fall back to +defaults rather than failing the run. + +`MemoryScoringWeightsUpdate` deliberately does **not** normalize — rescaling a +single slider to 1.0 would wipe out the others. Stored values stay raw, and +`resolve_weights()` normalizes them on read. diff --git a/docs/backend/backend_python/semantic-search.md b/docs/backend/backend_python/semantic-search.md index 8f6e54000..e0f5049e3 100644 --- a/docs/backend/backend_python/semantic-search.md +++ b/docs/backend/backend_python/semantic-search.md @@ -120,7 +120,7 @@ Notes on this flow, confirmed against the actual implementation (`backend/app/routes/images.py::semantic_search_images`, `frontend/src/pages/SearchResults/SearchResults.tsx`): - **The frontend never re-sorts.** `matched_pairs.sort(key=lambda x: x[1], reverse=True)` - is the *only* place sort order is established, inside the route handler. + is the _only_ place sort order is established, inside the route handler. Everything downstream (`matched_ids`, the `db_get_images_by_ids` call, the final response) just follows that order through — the response is built by iterating `db_get_images_by_ids`'s return value directly, in whatever order @@ -129,7 +129,7 @@ Notes on this flow, confirmed against the actual implementation be changed without also touching this endpoint. - **Empty tag results vs. a tag-search error are handled differently.** A successful tag search with zero results triggers the semantic fallback. - A tag-search *error* surfaces directly as an error — it does not fall back. + A tag-search _error_ surfaces directly as an error — it does not fall back. - **A 404 from `/semantic-search`** (text model or tokenizer file missing) is treated as "feature unavailable," not a generic error — the frontend detects it by HTTP status code (`error.response.status === 404`), not by @@ -182,7 +182,7 @@ called out explicitly: and re-enters `db_get_unembedded_images()` on the next pass. This deliberately does **not** follow the YOLO/face pipeline's mark-processed-regardless-of-outcome convention: that convention exists to - avoid re-running *expensive* inference on images that will never classify + avoid re-running _expensive_ inference on images that will never classify differently, but SigLIP2 preprocessing failure is a cheap check (PIL failing to open/decode), so the retry cost is low — and it means a file that becomes readable later (a transient lock, a restored backup) still @@ -194,7 +194,14 @@ called out explicitly: embedding and no record of the gap. - **Where it's wired in:** `post_AI_tagging_enabled_sequence()` and `post_sync_folder_sequence()` in `backend/app/routes/folders.py`, both - calling it *last*, after YOLO/face clustering. Gating is inherent in the + calling it after YOLO tagging and face clustering. It is _not_ the last + step in either sequence any more. The full tail of both is + `image_util_process_unembedded_images()` → `semantic_util_score_images()` + → `_curate_memories(trigger)` → the video pass + (`video_util_process_untagged_videos()`, + `video_util_process_unembedded_frames()`, `semantic_util_score_videos()`), + so memory curation runs against the semantic tags this pass and the + scoring sweep just wrote. Gating is inherent in the `AI_Tagging` join in the SQL query itself — non-AI-tagging folders never produce a single row from `db_get_unembedded_images()`, so no special-case code exists for "user has this feature off." @@ -293,8 +300,11 @@ results as **regular tags**: labels register as `mappings` rows at scoring pass writes plain `image_classes` rows with a `score`. Tag search, tag chips, and person-view tag lists pick them up with no consumer changes. The planned `image_semantic_labels` table was dropped in favor of this -reuse. Free-text search is unaffected — this layer is a cache/browse -feature, not a search gatekeeper. +reuse. Free-text search is unaffected — this layer is not a search +gatekeeper. `semantic_labels` and the scored `image_classes` rows it +produces are both live, populated tables: beyond tag search and chips, the +`semantic_event` memory trigger reads them (see +[Downstream consumers: memory curation](#downstream-consumers-memory-curation)). `semantic_labels` itself is a definition + cache table: @@ -342,6 +352,51 @@ Key behaviors (all in `backend/app/utils/semantic_labels.py` and `image_embeddings.scored_signature`) that check column presence before altering. +## Downstream consumers: memory curation + +Smart Memories reads both halves of what this feature produces — the stored +image embeddings and the curated event labels. Nothing is recomputed for +curation; a run only reads what the embedding pass and the scoring sweep +already wrote, and both reads filter on `model_version` exactly as search +does. The curation logic itself is documented in [Memories](memories.md); +what follows is only the SigLIP2 surface it depends on. + +**Stored embeddings** are loaded by two helpers in +`backend/app/database/image_embeddings.py`, called from +`backend/app/utils/memory_curator.py`: + +- `db_get_embeddings_for_image_ids(image_ids, model_version)` returns the + vectors for one explicit candidate set, keyed by image id. They feed + near-duplicate suppression (`suppress_near_duplicates` drops a photo only + when cosine ≥ `DUP_COSINE` = `0.90` **and** the two capture times are + within `DUP_WINDOW_SECONDS` = `120` s) and the cohesion gate that decides + whether a group of photos really looks like one occasion. +- `db_get_embedding_sample(model_version, limit)` reads an ordered slice of + the library's own embeddings (`COHESION_SAMPLE_SIZE` = 500) once per run. + SigLIP2 embeddings occupy a narrow cone, so cohesion is expressed as a + margin over that measured baseline (`EVENT_COHESION_MARGIN` = `0.15`) + rather than as an absolute cosine. + +**Event labels** drive the `semantic_event` trigger. `db_get_event_labels()` +(`backend/app/database/memories.py`) reads the `category = 'event'`, +`active = 1` rows straight out of `semantic_labels`, so a vocabulary edit +reaches curation without a code change. `db_get_event_label_hits()` then +selects the images where such a label is genuinely the subject **by rank, +not by raw score**, using two SQL window functions over the +`class_id >= 1000` rows in `image_classes`: + +| Window function | Partition | Question it answers | +| ------------------------------------------------------------ | --------- | ------------------------------------------------------------------------------------------- | +| `RANK() OVER (PARTITION BY image_id ORDER BY score DESC)` | per image | Is this label among the top few this image matched at all? | +| `PERCENT_RANK() OVER (PARTITION BY class_id ORDER BY score)` | per label | Is this image a strong example of the label, relative to every other image that matched it? | + +A row must clear both cuts — `image_rank <= EVENT_LABEL_TOP_N` (`2`) and +`label_rank >= EVENT_LABEL_PERCENTILE` (`0.50`). This is the same +absolute-scores-run-low property described in the +[score-range table](#scoring-metadata), handled in SQL: scores are not +comparable _across_ labels either, so an absolute cut would fire on the +handful of labels that ever score high and silence the rest of the ~395. + ## Model distribution and checkpoints SigLIP2 ships as three separate files per checkpoint — a vision-tower ONNX @@ -349,11 +404,11 @@ graph, a text-tower ONNX graph, and a tokenizer JSON — following the same `MODEL_REGISTRY` + GitHub Release + SHA-256 verification pattern already used for YOLO/FaceNet, not a new distribution mechanism. -| Checkpoint | Status | Vision size | Text size | Tier | -| --- | --- | --- | --- | --- | -| `base` | **Shipped** (`models-v1.0` release) | 354.5 MB | 1077.1 MB | `semantic` | -| `large` | Placeholder (`PLACEHOLDER_URL`/`PLACEHOLDER_SHA256`) | — | — | `medium` | -| `so400m` | Placeholder (`PLACEHOLDER_URL`/`PLACEHOLDER_SHA256`) | — | — | `manual` | +| Checkpoint | Status | Vision size | Text size | Tier | +| ---------- | ---------------------------------------------------- | ----------- | --------- | ---------- | +| `base` | **Shipped** (`models-v1.0` release) | 354.5 MB | 1077.1 MB | `semantic` | +| `large` | Placeholder (`PLACEHOLDER_URL`/`PLACEHOLDER_SHA256`) | — | — | `medium` | +| `so400m` | Placeholder (`PLACEHOLDER_URL`/`PLACEHOLDER_SHA256`) | — | — | `manual` | Only `base` has real registry entries (URL, SHA-256, size). `large` and `so400m` are deliberately kept **out of `TIER_MODELS`** with placeholder @@ -388,11 +443,11 @@ Each checkpoint has its own calibration constants in directly from the checkpoint (not tunable in the sense that changing them would require re-deriving from the model, not just picking a new number): -| Checkpoint | `logit_scale` | `logit_bias` | `model_version` | `input_resolution` | -| --- | --- | --- | --- | --- | -| `base` | 4.724453449249268 | -16.771724700927734 | `siglip2-base-patch16-224` | 224 | -| `large` | 4.6823530197143555 | -16.347614288330078 | `siglip2-large-patch16-384` | 384 | -| `so400m` | 4.699519157409668 | -15.932647705078125 | `siglip2-so400m-patch14-384` | 384 | +| Checkpoint | `logit_scale` | `logit_bias` | `model_version` | `input_resolution` | +| ---------- | ------------------ | ------------------- | ---------------------------- | ------------------ | +| `base` | 4.724453449249268 | -16.771724700927734 | `siglip2-base-patch16-224` | 224 | +| `large` | 4.6823530197143555 | -16.347614288330078 | `siglip2-large-patch16-384` | 384 | +| `so400m` | 4.699519157409668 | -15.932647705078125 | `siglip2-so400m-patch14-384` | 384 | Scoring formula (`backend/app/routes/images.py`): @@ -403,7 +458,7 @@ score = sigmoid(scaled_logits) = 1 / (1 + exp(-scaled_logits)) This is SigLIP2's own learned scale/bias (the sigmoid-loss calibration baked into the model), applied to a **raw cosine similarity** (both vectors -are unit-norm, so the dot product *is* the cosine similarity) — not the +are unit-norm, so the dot product _is_ the cosine similarity) — not the `pipeline()` API's opaque scoring, which was one of the first things ruled out early in this feature's design because it hides reusable embeddings and only exposes a top-1 label. @@ -412,11 +467,11 @@ only exposes a top-1 label. SigLIP2 community phenomenon (consistent with the sigmoid loss's negative bias initialization), not a bug. Empirically, on a real production library: -| Score range | Meaning | -| --- | --- | -| 0.6 – 0.9 | Strong match (descriptive phrases hit this range easily) | +| Score range | Meaning | +| ----------- | ---------------------------------------------------------------------------- | +| 0.6 – 0.9 | Strong match (descriptive phrases hit this range easily) | | 0.01 – 0.05 | Weak-but-real match (common for bare-noun queries on thumbnail-grade images) | -| < 0.005 | Noise | +| < 0.005 | Noise | `SIGLIP2_MATCH_THRESHOLD` defaults to `0.01` (moved down from an initially measured `0.02`, which was empirically cutting true positives at that low @@ -429,14 +484,14 @@ overridable via the project's existing `_get_env_str`/`_get_env_int`/`_get_env_f helpers (which log a warning and fall back to the default on an invalid or out-of-range value, rather than crashing): -| Setting | Default | Notes | -| --- | --- | --- | -| `SIGLIP2_ACTIVE_CHECKPOINT` | `"base"` | Falls back to `"base"` with a logged warning if set to anything not in `SIGLIP2_SCORING_METADATA`. | -| `SIGLIP2_QUERY_TEMPLATE` | `"This is a photo of {query}."` | Applied to every query before tokenizing — see [Preprocessing and calibration](#preprocessing-and-calibration-the-part-that-must-not-drift) below. | -| `SIGLIP2_EMBED_BATCH_SIZE` | `8` | Minimum enforced at `1`. Matches the batch size validated during the original PoC benchmarking. | -| `SIGLIP2_TEXT_MAX_LENGTH` | `64` | Fixed at export time (the ONNX text graph's sequence dimension is a **fixed** 64, not dynamic) — changing this constant without re-exporting the model produces a shape-mismatch error, not silently wrong numbers. | -| `SIGLIP2_TOKENIZER_PAD_ID` / `SIGLIP2_TOKENIZER_PAD_TOKEN` | `0` / `""` | Padding config passed to the `tokenizers` library. | -| `SIGLIP2_MATCH_THRESHOLD` | `0.01` | See the score-range table above. | +| Setting | Default | Notes | +| ---------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `SIGLIP2_ACTIVE_CHECKPOINT` | `"base"` | Falls back to `"base"` with a logged warning if set to anything not in `SIGLIP2_SCORING_METADATA`. | +| `SIGLIP2_QUERY_TEMPLATE` | `"This is a photo of {query}."` | Applied to every query before tokenizing — see [Preprocessing and calibration](#preprocessing-and-calibration-the-part-that-must-not-drift) below. | +| `SIGLIP2_EMBED_BATCH_SIZE` | `8` | Minimum enforced at `1`. Matches the batch size validated during the original PoC benchmarking. | +| `SIGLIP2_TEXT_MAX_LENGTH` | `64` | Fixed at export time (the ONNX text graph's sequence dimension is a **fixed** 64, not dynamic) — changing this constant without re-exporting the model produces a shape-mismatch error, not silently wrong numbers. | +| `SIGLIP2_TOKENIZER_PAD_ID` / `SIGLIP2_TOKENIZER_PAD_TOKEN` | `0` / `""` | Padding config passed to the `tokenizers` library. | +| `SIGLIP2_MATCH_THRESHOLD` | `0.01` | See the score-range table above. | ## Preprocessing and calibration (the part that must not drift) @@ -463,7 +518,7 @@ Measured specifics, from real debugging of a production quality regression: does not reproduce exactly. Shipping bit-exact HF parity would require bundling `transformers`, which was judged not worth it: production is internally consistent (images and queries share the same preprocessing - path), and the threshold/scale/bias are all calibrated against *that* + path), and the threshold/scale/bias are all calibrated against _that_ path, not against HF's. - **Any preprocessing change invalidates existing embeddings.** After the `cv2` → PIL switch, every previously stored embedding was stale and had to @@ -473,7 +528,7 @@ Measured specifics, from real debugging of a production quality regression: `strip()`'d and `lower()`'d before templating. Skipping this caused two reproducible bugs during development: (1) `"Beach"` scored very differently from `"beach"` because the SentencePiece tokenizer is - case-sensitive *and* a capitalized noun mid-template reads like a proper + case-sensitive _and_ a capitalized noun mid-template reads like a proper noun ("This is a photo of Beach." ≈ a place name); (2) un-templated raw queries land outside the calibration regime entirely, since every threshold/scale/bias number here was derived using the @@ -490,7 +545,7 @@ concurrency bug was found and fixed once it was consolidated. **Contract subclasses must follow:** `get_session()` must snapshot `self._session` and any tensor-name attributes into **local variables** -*before* releasing `_lock`, then return those locals — never re-read +_before_ releasing `_lock`, then return those locals — never re-read `self.*` after the lock is released. The bug this prevents: a concurrent `close()` can null those attributes between an in-lock check and an out-of-lock return, handing a caller a valid session object paired with a @@ -501,7 +556,7 @@ simultaneously. **Registration-leak bug (fixed):** `close()`'s cleanup used to be gated on `self._session is not None`. But `get_session()` can register a session (via `mark_model_session_active`, incrementing `session_registry`'s active -count) and *then* null `self._session` on a tensor-name validation failure, +count) and _then_ null `self._session` on a tensor-name validation failure, while `_session_registered` stays `True`. With the old gate, `close()` would see `self._session is None` and skip the entire cleanup block — including the `mark_model_session_inactive` call — leaking the registration forever @@ -538,14 +593,14 @@ verified end-to-end against the real ONNX models during development. [API Reference](api.md) (Swagger UI) for the full request/response schema. Summary of behavior not obvious from the schema alone: -| Condition | Response | -| --- | --- | -| Text model file missing | `404`, `message` mentions "text model not installed" | -| Tokenizer file missing | `404`, `message` mentions "tokenizer not installed" (checked independently of the text model) | +| Condition | Response | +| ----------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Text model file missing | `404`, `message` mentions "text model not installed" | +| Tokenizer file missing | `404`, `message` mentions "tokenizer not installed" (checked independently of the text model) | | Query is empty after `strip()` (e.g. whitespace-only) | `400` — note `min_length=1` on the FastAPI `Query` param only checks raw string length, so a whitespace-only string passes that check and is caught by this separate normalization step | -| No embeddings exist yet for the active checkpoint | `200`, empty result, friendly message ("No images have been embedded yet.") | -| Embeddings exist but none clear the threshold | `200`, empty result, message includes the threshold value used | -| Matches found | `200`, images sorted descending by score, each score rounded to 4 decimal places | +| No embeddings exist yet for the active checkpoint | `200`, empty result, friendly message ("No images have been embedded yet.") | +| Embeddings exist but none clear the threshold | `200`, empty result, message includes the threshold value used | +| Matches found | `200`, images sorted descending by score, each score rounded to 4 decimal places | ## Maintenance: `scripts/reset_embeddings.py` @@ -565,12 +620,12 @@ developer runs deliberately. ## Test coverage -| File | Covers | -| --- | --- | -| `tests/test_image_embeddings.py` | `image_embeddings` table CRUD: round-trip storage/retrieval, `model_version` filtering, upsert-overwrites-existing-row, FK cascade delete. Runs against a disposable per-test SQLite file (see note below), not the real database. | +| File | Covers | +| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `tests/test_image_embeddings.py` | `image_embeddings` table CRUD: round-trip storage/retrieval, `model_version` filtering, upsert-overwrites-existing-row, FK cascade delete. Runs against a disposable per-test SQLite file (see note below), not the real database. | | `tests/test_semantic_search_route.py` | The `/semantic-search` endpoint: 404s (text model / tokenizer missing, checked independently), 400 on a whitespace-only query, friendly empty-result responses, and — critically — descending sort order verified with two results that both clear the threshold (an earlier version of this test only had one matching result, which couldn't have detected a broken sort). | -| `tests/test_embedding_pipeline.py` | `image_util_process_unembedded_images`: skips cleanly with no vision model installed, batches per `SIGLIP2_EMBED_BATCH_SIZE`, excludes corrupt images from both the embeddings upsert and the embedded-marking (so they're retried on a later pass), always closes the vision session even if scoring raises mid-batch. | -| `tests/test_onnx_session_base.py` | `ONNXSessionBase.close()`: normal decrement, the registration-leak regression scenario, no-op-when-never-opened, idempotency. Fully mocks `onnxruntime.InferenceSession` and `os.path.exists` — does not depend on the real (multi-hundred-MB, not checked into git) ONNX files existing on disk. | +| `tests/test_embedding_pipeline.py` | `image_util_process_unembedded_images`: skips cleanly with no vision model installed, batches per `SIGLIP2_EMBED_BATCH_SIZE`, excludes corrupt images from both the embeddings upsert and the embedded-marking (so they're retried on a later pass), always closes the vision session even if scoring raises mid-batch. | +| `tests/test_onnx_session_base.py` | `ONNXSessionBase.close()`: normal decrement, the registration-leak regression scenario, no-op-when-never-opened, idempotency. Fully mocks `onnxruntime.InferenceSession` and `os.path.exists` — does not depend on the real (multi-hundred-MB, not checked into git) ONNX files existing on disk. | !!! warning "Local test runs and the real database" `DATABASE_PATH` only redirects to a throwaway SQLite file when the diff --git a/docs/frontend/memories.md b/docs/frontend/memories.md index 199a0bdc5..5983d751e 100644 --- a/docs/frontend/memories.md +++ b/docs/frontend/memories.md @@ -1,352 +1,237 @@ -# Memories Feature Documentation +# Memories -## Overview +Memories are curated collections of photos and short clips that the backend +scores and stores, and the frontend plays back as a full-screen, Instagram-style +story. The frontend never clusters or scores anything itself: it lists memories, +opens one, marks it viewed, and asks for a new curation run. -The Memories feature automatically organizes photos into meaningful collections based on location and date, providing a Google Photos-style experience for reliving past moments. +For how memories are produced — the schema, the three triggers and the scoring +signals — see [Memories (backend)](../backend/backend_python/memories.md). -## Features +## Routes and Pages -### 1. On This Day +| Route | Page component | Purpose | +| ------------------- | ----------------------------------- | ------------------------------------------- | +| `memories` | `pages/Memories/Memories.tsx` | Grid of memory cards, plus the story viewer | +| `memories/settings` | `pages/Memories/MemorySettings.tsx` | Memory preferences | -Shows photos from the same date in previous years with a prominent featured card. +Both are registered in `routes/AppRoutes.tsx`, with path names in +`constants/routes.ts` (`ROUTES.MEMORIES`, `ROUTES.MEMORIES_SETTINGS`). -**Display:** +## The Grid Page -- "On this day last year" for photos from exactly 1 year ago -- "[X] years ago" for photos from multiple years ago -- Featured hero image with gradient overlay -- Photo count and year badges +`Memories.tsx` requests up to 60 cards and renders them in a responsive grid +(2 columns, up to 5 at `xl`). -### 2. Memory Types +- **Header** – Title, plus an "N new to look back on" line driven by + `unviewed_count` from the status endpoint. +- **Refresh** – Requests a curation run. If the status snapshot reports + `indexing_busy`, the click opens an info dialog instead of firing the request, + because the backend declines to curate a half-indexed library. +- **Settings** – A gear button that navigates to the settings page. +- **Loading** – Ten pulsing placeholder tiles while the list query is in flight. +- **Empty state** – A dashed panel whose copy changes depending on whether a + run is currently in progress. +- **Errors** – A failed list query is surfaced through the shared info dialog + (`showInfoDialog`), not inline. -#### Location-Based Memories - -Photos grouped by GPS coordinates using DBSCAN clustering: - -- **Radius**: 5km (configurable) -- **Title Format**: "Trip to [City Name], [Year]" -- **Example**: "Trip to Jaipur, 2025" -- **Reverse Geocoding**: Maps coordinates to actual city names -- **Supported Cities**: 30+ major cities worldwide (Indian, European, American, Asian, etc.) - -#### Date-Based Memories - -Photos grouped by month for images without GPS: - -- **Grouping**: Monthly clusters -- **Title Format**: "[Month] [Year]" -- **Flexibility**: Works even without location data - -### 3. Memory Sections - -#### Recent Memories - -- **Timeframe**: Last 30 days -- **Use Case**: Recent trips and events -- **API**: `GET /api/memories/timeline?days=30` - -#### This Year - -- **Timeframe**: Last 365 days (current year) -- **Use Case**: Year-in-review -- **API**: `GET /api/memories/timeline?days=365` - -#### All Memories - -- **Timeframe**: All time -- **Use Case**: Complete memory collection -- **API**: `POST /api/memories/generate` - -### 4. Filtering - -**Filter Options:** - -- **All**: Shows all memories (default) -- **Location**: Only memories with GPS coordinates -- **Date**: Only memories without GPS (date-based) - -**Implementation:** - -```typescript -const applyFilter = (memories: Memory[]) => { - if (filter === "location") { - return memories.filter((m) => m.center_lat !== 0 || m.center_lon !== 0); - } - if (filter === "date") { - return memories.filter((m) => m.center_lat === 0 && m.center_lon === 0); - } - return memories; // 'all' -}; -``` - -### 5. Memory Viewer - -Full-screen modal for viewing memory photos: - -**Features:** - -- Image grid with hover effects -- Click to open MediaView -- Zoom and pan support -- Slideshow mode -- Keyboard navigation -- Info panel with metadata -- Thumbnail strip - -**Controls:** - -- **Zoom**: Mouse wheel or +/- keys -- **Navigation**: Arrow keys or buttons -- **Slideshow**: Play/Pause button or Space key -- **Info Panel**: Toggle with 'I' key -- **Close**: ESC key or X button - -## Components - -### MemoriesPage - -Main page component with sections: - -- Header with refresh button -- Filter buttons -- On This Day section -- Recent Memories grid -- This Year grid -- All Memories grid +The page also mirrors the user's slide interval preference into the Redux slice +that the viewer reads, and renders `MemoryStoryViewer` whenever +`activeMemoryId` is set. ### MemoryCard -Individual memory card display: - -- Thumbnail image -- Memory title (formatted based on type) -- Date range (relative format) -- Location (if available) -- Photo count badge -- Type badge (Location/Date) - -### FeaturedMemoryCard - -Large featured card for "On This Day": - -- Hero image with gradient overlay -- "On this day last year" text -- Photo count and year info -- Additional image previews - -### MemoryViewer - -Modal for viewing memory album: - -- Conditionally rendered to prevent event bubbling -- Grid layout of all photos -- MediaView integration for full-screen viewing -- Proper z-index layering - -## State Management - -Using Redux Toolkit with slices: - -```typescript -// Store structure -{ - memories: { - onThisDay: { - images: MemoryImage[], - meta: { today: string, years: number[] } - }, - recent: Memory[], - year: Memory[], - all: Memory[], - selectedMemory: Memory | null, - loading: { onThisDay, recent, year, all }, - error: { onThisDay, recent, year, all } - } -} -``` - -**Key Actions:** +`components/Memories/MemoryCard.tsx` renders one tile as a `