Skip to content
Merged

Fixes #4918

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion app/Console/Commands/RecomputeBuckets.php
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

namespace App\Console\Commands;

use App\Events\AlbumListingCacheFlushRequested;
use App\Events\PhotoListingCacheFlushRequested;
use App\Services\BucketKeyWriter;
use Illuminate\Console\Command;
use Illuminate\Support\Collection;
Expand All @@ -17,7 +19,8 @@
/**
* Rewrites `bucket_key` of every album, then of every `photo_album` row,
* in the granularity-free format (FR-092-15), chunked by
* {@see BucketKeyWriter} (FR-092-14).
* {@see BucketKeyWriter} (FR-092-14), flushing the album, then the photo
* listings once their keys are written.
*/
class RecomputeBuckets extends Command
{
Expand All @@ -37,8 +40,10 @@ public function handle(BucketKeyWriter $writer): int
$chunk_size = max(1, (int) $this->option('chunk'));

$this->recompute('albums', DB::table('albums')->count(), fn (\Closure $on_chunk): int => $writer->writeAlbumKeys(fn ($query) => $query, $chunk_size, $on_chunk));
AlbumListingCacheFlushRequested::dispatch();
$this->newLine();
$this->recompute('photo_album rows', DB::table('photo_album')->count(), fn (\Closure $on_chunk): int => $writer->writePhotoAlbumKeys(fn ($query) => $query, $chunk_size, $on_chunk));
PhotoListingCacheFlushRequested::dispatch();

return Command::SUCCESS;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
* FulfillPreCompute - Maintenance controller for backfilling album precomputed fields.
*
* This controller is equivalent to running the artisan command:
* php artisan lychee:recompute-album-fields
* php artisan lychee:recompute-album-stats
*
* It processes albums where ALL precomputed fields are null and dispatches
* jobs to compute:
Expand Down
39 changes: 39 additions & 0 deletions app/Http/Controllers/Admin/Maintenance/RecomputeBuckets.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<?php

/**
* SPDX-License-Identifier: MIT
* Copyright (c) 2017-2018 Tobias Reich
* Copyright (c) 2018-2026 LycheeOrg.
*/

namespace App\Http\Controllers\Admin\Maintenance;

use App\Http\Requests\Maintenance\RecomputeBucketsRequest;
use App\Jobs\RecomputeAllAlbumBucketsJob;
use App\Jobs\RecomputeAllPhotoBucketsJob;
use Illuminate\Routing\Controller;

/**
* Rewrites every album and `photo_album` bucket key.
*
* This controller is equivalent to running the artisan command:
* php artisan lychee:recompute-buckets
*
* The two jobs run on the `low` queue (inline on `sync`) and flush the
* album and photo listings once their keys are written.
*/
class RecomputeBuckets extends Controller
{
/**
* Queue the rewrite of every bucket key.
*
* @param RecomputeBucketsRequest $_request
*
* @return void
*/
public function do(RecomputeBucketsRequest $_request): void
{
RecomputeAllAlbumBucketsJob::dispatch();
RecomputeAllPhotoBucketsJob::dispatch();
}
}
25 changes: 25 additions & 0 deletions app/Http/Requests/Maintenance/RecomputeBucketsRequest.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<?php

/**
* SPDX-License-Identifier: MIT
* Copyright (c) 2017-2018 Tobias Reich
* Copyright (c) 2018-2026 LycheeOrg.
*/

namespace App\Http\Requests\Maintenance;

/**
* Bucket keys only feed the struct-of-arrays listings: the maintenance
* action is gated by the `features.struct-of-array` flag on top of the
* admin check.
*/
class RecomputeBucketsRequest extends MaintenanceRequest
{
/**
* {@inheritDoc}
*/
public function authorize(): bool
{
return config('features.struct-of-array') === true && parent::authorize();
}
}
18 changes: 14 additions & 4 deletions app/Services/BucketScheme.php
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
use App\Repositories\ConfigManager;
use Illuminate\Support\Facades\DB;
use function Safe\gmmktime;
use function Safe\preg_match;

/**
* Read-time bucket ids (ADR-092-01): one scheme per listing request,
Expand Down Expand Up @@ -57,6 +58,8 @@ final class BucketScheme

private const DATE_LENGTHS = [4, 7, 10, 13];
private const TITLE_DATE_LENGTH = 10;
/** Shape of a date id: `Y`, `Y-m`, `Y-m-d` or `Y-m-d-H`. */
private const DATE_ID_PATTERN = '/^\d{4}(?:-\d{2}){0,3}$/';

/**
* @param string $kind one of the private kind constants
Expand Down Expand Up @@ -284,7 +287,7 @@ private static function spansSeveralDays(array $bucket_ids): bool
{
$days = [];
foreach ($bucket_ids as $bucket_id) {
if ($bucket_id !== self::UNKNOWN && substr_count($bucket_id, '-') === 3) {
if (self::isDateId($bucket_id) && substr_count($bucket_id, '-') === 3) {
$days[substr($bucket_id, 0, 10)] = true;
}
}
Expand All @@ -294,12 +297,14 @@ private static function spansSeveralDays(array $bucket_ids): bool

/**
* Label of one id: date ids are formatted by their own precision (year,
* month, day, hour); other ids and `unknown` are returned as is. Hour ids
* use `$hour_format` when given (the multi-day format, FR-095-08).
* month, day, hour); other ids, `unknown` and ids of a date scheme that
* are not date-shaped (keys written for another scheme, FR-092-13) are
* returned as is. Hour ids use `$hour_format` when given (the multi-day
* format, FR-095-08).
*/
public function label(string $bucket_id, ?string $hour_format = null): string
{
if ($bucket_id === self::UNKNOWN || ($this->kind !== self::DATE && $this->kind !== self::CREATED)) {
if (($this->kind !== self::DATE && $this->kind !== self::CREATED) || !self::isDateId($bucket_id)) {
return $bucket_id;
}

Expand Down Expand Up @@ -364,6 +369,11 @@ private function whereSourceId($query, string $bucket_id): void
};
}

private static function isDateId(string $bucket_id): bool
{
return preg_match(self::DATE_ID_PATTERN, $bucket_id) === 1;
}

/**
* `[year, month, day, hour, precision]` of a date id (`Y`, `Y-m`,
* `Y-m-d` or `Y-m-d-H`); missing parts read as the start of the period.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Open questions for [Feature 092](spec.md). Log every high- and medium-impact que
| ~~Q-092-06~~ | 092 – Bucketing refactor | Medium | Tag, person and search album listings now return `unknown` as every per-row album `bucket_id`, which no FR describes (A record it in FR-092-05 · B keep a computed id) | Resolved (Option A; owner, 2026-10-14; spec FR-092-05) | 2026-10-14 | 2026-10-14 |
| ~~Q-092-07~~ | 092 – Bucketing refactor | Medium | FR-092-13 is silent on `sorting_photos_order`; no settings list flushes cached photo listings when it changes (A add it to the photo-listing flush · B leave out of scope) | Resolved (Option A; owner, 2026-10-14; spec FR-092-13, S-092-20) | 2026-10-14 | 2026-10-14 |
| ~~Q-092-08~~ | 092 – Bucketing refactor | Low | How does the title date read `YYYYMMDD`? (A optional dashes `-?` · B a separate eight-digit branch) | Resolved (Option B; owner, 2026-10-14; spec FR-092-02, S-092-21) | 2026-10-14 | 2026-10-14 |
| ~~Q-092-09~~ | 092 – Bucketing refactor | Medium | Album listings read the stored keys with the new scheme as soon as `title_bucket_mode` or `sorting_albums_col` is saved, but the keys are rewritten only when the queued `RecomputeAllAlbumBucketsJob` runs (A rewrite album keys in the settings request · B queued job, listings unbucketed while it is pending · C accept the window, stop date-formatting non-date ids) | Resolved (Option C; owner, 2026-10-23; spec FR-092-08, FR-092-13, S-092-23) | 2026-10-23 | 2026-10-23 |

## Question Details

Expand Down Expand Up @@ -156,3 +157,31 @@ Measured on SQLite 3.46 (scratchpad, synthetic data: 300,000 photos, one album w
- ❌ Harder to read, branch reset is PCRE-only so JS reads different groups.

**Resolution:** Option B (the owner first chose A, then switched to B). The owner also asked that every parser reject dates that are not on the calendar: `TimelineData::parseDateFromTitle()` no longer rolls `2023-13-01` over to January 2024 and `dateScrubber.ts` now checks the date, matching the `checkdate()` already in `BucketKeys` and `BucketKeyBackfill`. Recorded in FR-092-02 and S-092-21.

### ~~Q-092-09~~ – Album listings between a settings change and the key rewrite

**Context:** FR-092-13: saving `title_bucket_mode` or `sorting_albums_col` queues `RecomputeAllAlbumBucketsJob` on `low`. The read scheme (`BucketScheme::forAlbums()`) follows the new values on the next request, while `albums.bucket_key` still holds keys written for the old ones until a worker runs the job. Reproduced on a scratch instance (v8, struct-of-arrays, database queue, root sorted by title ascending, root timeline on, albums `2019-05 Paris`, `2020 Christmas`, `2021-07-14 Bastille`, `Alpha`, `alpha two`, `Bravo`, `Mountains`, `Zulu`):

| Step | Root buckets (ids = labels unless noted) |
|------|------------------------------------------|
| `date_prefix`, job done | `2019`, `2020`, `2021`, `unknown` |
| `alphabetical` saved, job pending | `2`, `unknown` (first character of the date keys, titles without a date stay `unknown`) |
| job done | `2`, `a`, `b`, `m`, `z` |
| `date_prefix` saved, job pending | ids `2019`, `2020`, `2021`, `alph`, `brav`, `moun`, `zulu`; labels `2019`, `2020`, `2021`, `2000`, `2000`, `2000`, `2000` |

The listing keeps the previous grouping (or shows `2000` labels: `BucketScheme::label()` formats any id of a date scheme as a date) until the worker reaches the job, which waits behind every job queued before it on `low`. The photo side has the same gap for `photo_title_bucket_mode` and `sorting_photos_col` (`RecomputeAllPhotoBucketsJob`). Album counts are small next to `photo_album` rows: on the owner's instance `lychee:recompute-buckets` logged 869 albums and 70,657 `photo_album` rows, the photo rows taking about 5 s.

- **Option A (recommended):** Run `RecomputeAllAlbumBucketsJob` inside the settings request (`dispatchSync`) instead of queueing it; the photo side keeps its queued job.
- ✅ No gap for album listings: the response that confirms the save comes after the keys are written and the listings flushed.
- ✅ One-line change in `SettingsController`; the job and its tests stay.
- ❌ The settings request pays one upsert per 1,000 albums.
- ❌ The photo side keeps the gap.
- **Option B:** Keep the queued jobs; record a pending recompute when one is dispatched and clear it when the job finishes; while it is pending, album (and photo) listings answer `bucketable: false`.
- ✅ Covers albums and photos, no request-time cost.
- ❌ Headers disappear until a worker reaches the job (indefinitely on a worker that does not listen on `low`).
- ❌ New shared state between the settings request, the job and the cached listings.
- **Option C:** Accept the gap: FR-092-13 states that listings follow the previous grouping until the recompute job has run, and `BucketScheme::label()` returns ids that are not date-shaped as they are.
- ✅ Smallest change.
- ❌ The reported symptom stays until the worker runs the job.

**Resolution:** Option C. FR-092-13 states that listings read the keys written for the previous values with the new scheme until the recompute job has run; FR-092-08 returns ids of a date scheme that are not date-shaped as they are. Scenario S-092-23.
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,15 @@ Prompts: "we also want to support YYYYMMDD", "why not make the - optional with ?
- _Steps:_ tests first; the four `timeline_photo_date_format_*` keys join `PHOTO_LISTING_COARSE_FLUSH_CONFIGS`, the three `timeline_album_date_format_*` keys join `ALBUM_LISTING_COARSE_FLUSH_CONFIGS`; the child `buckets`/`data`/`rights` cache entries of `AlbumChildrenController` carry `albumListingGlobalTag()`, like the root ones.
- _Commands:_ `php artisan test --filter=BucketSettingsV3Test`, `--filter=UpdateSettingsTest`, `make phpstan`, `vendor/bin/php-cs-fixer fix`.

17. **I16 – Ids written for another scheme (Q-092-09)**
- _Goal:_ FR-092-08, FR-092-13 (S-092-23): until a recompute job has run, a date scheme reads keys written for another scheme; ids that are not date-shaped are their own labels.
- _Steps:_ tests first in `BucketSchemeTest` and `AlbumRootV3Test`; `BucketScheme::label()` formats only date-shaped ids.
- _Commands:_ `php artisan test --filter=BucketSchemeTest`, `--filter=AlbumRootV3Test`, `make phpstan`, `vendor/bin/php-cs-fixer fix`.
18. **I17 – `lychee:recompute-buckets` flushes listings**
- _Goal:_ FR-092-15 (S-092-17): the command flushes the album and photo listing caches after rewriting their keys, like the two recompute jobs.
- _Steps:_ test first in `RecomputeBucketsCommandTest`; `RecomputeBuckets::handle()` dispatches `AlbumListingCacheFlushRequested` and `PhotoListingCacheFlushRequested`.
- _Commands:_ `php artisan test --filter=RecomputeBucketsCommandTest`, `make phpstan`, `vendor/bin/php-cs-fixer fix`.

## Scenario Tracking

| Scenario ID | Increment / Task reference | Notes |
Expand All @@ -186,12 +195,13 @@ Prompts: "we also want to support YYYYMMDD", "why not make the - optional with ?
| S-092-14 | I2 / T-092-02 | `BucketKeysTest` |
| S-092-15 | I7, I8b / T-092-11, T-092-14 | both directions |
| S-092-16 | I11 / T-092-19 | browser check |
| S-092-17 | I5 / T-092-08 | `RecomputeBucketsCommandTest` |
| S-092-17 | I5, I17 / T-092-08, T-092-27 | `RecomputeBucketsCommandTest` |
| S-092-18 | I10 / T-092-18 | `ManagedCachePhotoListingInvalidatorTest` |
| S-092-19 | I13 / T-092-22 | `AlbumKeysWriteTest` |
| S-092-20 | I13 / T-092-23 | `BucketSettingsV3Test` |
| S-092-21 | I14 / T-092-24 | `BucketKeysTest`, `TimelineDataTest` |
| S-092-22 | I15 / T-092-25 | `BucketSettingsV3Test` |
| S-092-23 | I16 / T-092-26 | `BucketSchemeTest`, `AlbumRootV3Test` |

## Analysis Gate

Expand Down
Loading
Loading