Repository navigation
feat(backup): copy cache files to a bucket, restore lost ones - #486
Open
MoonBoi9001 wants to merge 9 commits into
Conversation
The scores and freshness files live on 1 disk, so losing it refused selections until the next daily push and lost the freshness history. With IISA_BACKUP_URL set, IISA copies both files to that bucket when they change and restores a missing or unreadable one at startup.
…le-instead-of-overwriting' into mb9/back-up-scores-and-freshness-to-a-bucket # Conflicts: # src/iisa/iisa_http_endpoints.py
A reply from the bucket or the metadata server that ended early or came garbled raised an error the backup code didn't expect, which could stop the service starting. It now counts as a failure that may clear up, so startup retries for a while, then starts without the copy.
A copy in a layout this release can't read was restored, set aside, and then replaced in the bucket within the hour by the short history the service started again. Copying the freshness file now stops until the next restart in that case, as it does when the bucket can't be read.
Copies were named after the file on disk, so a scores file and a freshness file configured with the same file name in different folders would overwrite each other's copy. Each copy now has a fixed name of its own.
A restored file reached the disk, but the rename that put it in place could still be lost to a power cut. The folder is now flushed too, as the service's other writes already do.
The backups section said startup tries an unreachable bucket for about a minute, but it tries each of the 2 files that long, and a request already under way can add to it.
The 3 workloads all run as the namespace's default service account, so granting that one the bucket would let the score job and the sync-status fetcher write the copies too.
The pause that keeps a freshness copy the service couldn't read is held in memory, so the next restart copies the new history over it. The backups section now says so and that the bucket's older versions are then what keeps it.
MoonBoi9001
marked this pull request as ready for review
November 20, 2026 11:47
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
IISA_BACKUP_URLset to a Cloud Storage bucket, IISA copies its scores file after each push and its freshness file hourly, and restores a missing or unreadable one at startup. A lost disk then no longer refuses selections until the next scores push or wipes the freshness history.Important
Before setting
IISA_BACKUP_URL, create the bucket with object versioning and give the IISA deployment its own Kubernetes service account with Storage Object User on it.