Skip to content

feat(backup): copy cache files to a bucket, restore lost ones - #486

Open
MoonBoi9001 wants to merge 9 commits into
mb9/keep-unreadable-freshness-file-instead-of-overwritingfrom
mb9/back-up-scores-and-freshness-to-a-bucket
Open

MoonBoi9001 wants to merge 9 commits into
mb9/keep-unreadable-freshness-file-instead-of-overwritingfrom
mb9/back-up-scores-and-freshness-to-a-bucket

Conversation

@MoonBoi9001

@MoonBoi9001 MoonBoi9001 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

With IISA_BACKUP_URL set to a Cloud Storage bucket, IISA copies its scores file after each push and its freshness file hourly, and restores a missing or unreadable one at startup. A lost disk then no longer refuses selections until the next scores push or wipes the freshness history.

Important

Before setting IISA_BACKUP_URL, create the bucket with object versioning and give the IISA deployment its own Kubernetes service account with Storage Object User on it.

The scores and freshness files live on 1 disk, so losing it refused selections until the next
daily push and lost the freshness history. With IISA_BACKUP_URL set, IISA copies both files to
that bucket when they change and restores a missing or unreadable one at startup.
@MoonBoi9001
MoonBoi9001 added this pull request to stack #485 November 20, 2026 11:27
…le-instead-of-overwriting' into mb9/back-up-scores-and-freshness-to-a-bucket

# Conflicts:
#	src/iisa/iisa_http_endpoints.py
A reply from the bucket or the metadata server that ended early or came garbled raised an
error the backup code didn't expect, which could stop the service starting. It now counts as
a failure that may clear up, so startup retries for a while, then starts without the copy.
A copy in a layout this release can't read was restored, set aside, and then replaced in the
bucket within the hour by the short history the service started again. Copying the freshness
file now stops until the next restart in that case, as it does when the bucket can't be read.
Copies were named after the file on disk, so a scores file and a freshness file configured with
the same file name in different folders would overwrite each other's copy. Each copy now has a
fixed name of its own.
A restored file reached the disk, but the rename that put it in place could still be lost to a
power cut. The folder is now flushed too, as the service's other writes already do.
The backups section said startup tries an unreachable bucket for about a minute, but it tries
each of the 2 files that long, and a request already under way can add to it.
The 3 workloads all run as the namespace's default service account, so granting that one the
bucket would let the score job and the sync-status fetcher write the copies too.
The pause that keeps a freshness copy the service couldn't read is held in memory, so the next
restart copies the new history over it. The backups section now says so and that the bucket's
older versions are then what keeps it.
@MoonBoi9001
MoonBoi9001 marked this pull request as ready for review November 20, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant