Skip to content

fix(backups): distinguish flushed saves from verified write freezes - #907

Open
Baylem wants to merge 12 commits into
stack/870-backup-and-probesfrom
fix/870-backup-consistency
Open

Baylem wants to merge 12 commits into
stack/870-backup-and-probesfrom
fix/870-backup-consistency

Conversation

@Baylem

@Baylem Baylem commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Closes #870. Parent tracker #867. ARK: Survival Ascended and Project Zomboid now declare stopped-copy backup guarantees in the upstream module catalog. Module PR #59 · Merge 2a7bb6c8

Gameplane’s shared catalog delivery prerequisite is #926. The maintainer updated its gitlink to f9e55aa1 after the ARK: Survival Evolved 2.0.1 safeguard merged. This pin includes both this backup correction and the Palworld telemetry change. Installations using OCI catalogs need a release containing this revision. Catalog pin and CI, #926

The operator records pod-bound verified freeze, acknowledged flush with writes still enabled, and drained stopped capture separately. Historical unqualified hook successes remain restorable and report unknown consistency. The Minecraft Java checkpoint adapter, validation and agent protocol express flush-only without creating a resume obligation. CI exercises restored Minecraft world and scoreboard markers, a client join, another save, and SQLite integrity after each of two stopped Restic copies.

The durability (#899) and probe (#889) changes have merged. This review now stacks on #900 via stack/870-backup-and-probes for its existing one-request webhook fixture fix. The arm64 operator run against master exposed the BusyBox sink closing a connection that the real notification client reused. In a disposable reproduction, 150 of 300 requests failed on the original fixture. With the Connection: close change from #900, all 300 requests arrived exactly once. Commit 8cff7b87 retains the same 50 backup files with 473 additions and 61 deletions. Merge #900 before retargeting #907 to master. Failed operator job · Existing fixture fix · Webhook rerun. Real production recovery evidence on other engines depends on separately qualified engine adapters.

On 8cff7b87, the arm64 operator test passed, as did the backup and offline-recovery drills. The amd64 placement drill independently failed before applying the quota because its Helm release and its disposable tenant shared a namespace. #886 now renders this fixture using the control-plane release namespace and keeps both the admission binding and the quota scoped to the disposable tenant. The overlapping namespace still fails admission validation in Helm. #907 consumes this change at 30db243f; its scoped diff remains 50 files, 473 additions and 61 deletions. Passing arm64 webhook test · Placement failure · Shared fix · Full backup workflow on 30db243f.

Recovery and placement on signed head 30db243fd92687365561e88395962fccb27d2a2b

The workflow finished with 73 successful jobs and four conditional skips. The arm64 operator suite also passed on this head. Final CI report · Arm64 operator

The backup preparation and same-node copy drill passed in 113.69 seconds. Two independent stopped Restic snapshots restored a SQLite database with PRAGMA integrity_check returning ok and the player score still 17 (190.94 seconds). The multi-volume drill passed its suspended, idle, missing-claim and changed-path cases in 506.11 seconds. Same-node copy · SQLite and multi-volume restores

Minecraft Java 1.21.4 accepted the client after Restic restored the captured world marker and player score 17. The restored game accepted score 18 and flushed it to disk. The live drill took 385.73 seconds. Minecraft restore, rejoin and save

All three placement drills passed. Two workloads recovered after drain in 3 and 4 seconds with the same world markers and volume identities. The leaking tenant reached its 32 MiB limit with the Pod cgroup OOM counter increasing from 0 to 1 and the kill counter from 0 to 2. The neighboring Minecraft server recorded p95/p99 tick durations of 0.3/2.8 ms under load and 0.1/0.2 ms after the kill. The Restic backup and repository read on the surviving worker took 19.63 seconds. Worker placement and disruption

Recovery qualification on a0891c6b9fa749a900ca1d7d925e576a6046e022

Minecraft Java 1.21.4 accepted the real client before capture and after Restic restored the save. The world marker and player score changed from their captured values during the acknowledged freeze, and changed again after the copy. The restored server returned the original world marker and score 17, accepted score 18 and acknowledged another flushed save. A post-resume marker reached disk through native autosave. The live test completed in 532.71 seconds. Minecraft capture, restore and rejoin

Two independent stopped Restic snapshots each restored a SQLite database with PRAGMA integrity_check returning ok and the expected player score of 17 (190.68 seconds). The multi-volume recovery test passed its suspended, idle, missing-claim and changed-path cases (556.46 seconds). ARM64 operator recovery tests

@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Pre-merge checks: Passed 1 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Passed Passed checks: 1 passed

Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…rld data

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem changed the base branch from fix/871-durable-backup-resume to stack/870-backup-and-probes November 11, 2026 05:32
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

CI Report - 30db243

run 43162327533 - attempt 1

0 failed - 29 passed - 4 skipped - 127m49s elapsed

Test selection: partial - per-suite table

Coverage

Module Measured Gate Margin
web L94.03 S92.14 B86.51 F90.19 L92 S92 B82 F76 +0.14

Changed

Area Files +/-
operator/ 15 +178/-20
agent/ 6 +83/-9
test/ 2 +78/-6
charts/ 18 +48/-22
gp-module/ 2 +47/-0
i18n/ 3 +19/-3
docs/ 2 +14/-1
changelog/ 1 +5/-0
web/ 1 +1/-0

Slowest jobs

Job Duration
e2e telemetry / amd64 (kind) 23m52s
e2e telemetry / arm64 (kind) 22m57s
build e2e images 18m4s
e2e restricted-workloads / arm64 (kind) 15m31s
e2e restricted-workloads / amd64 (kind) 14m52s

e2e buckets run

api-agent, api-auth, api-mods, api-rbac, api-roles, backup-preparation, bot-fast, management-placement, monitoring, multicluster, network-isolation, operator, optional-components, postgres-availability, ratelimit, recovery, restricted-workloads, telemetry, upgrade, wake-hostport

@Baylem Baylem linked an issue Nov 11, 2026 that may be closed by this pull request
…larations

Signed-off-by: Baylem <zelathis@gmail.com>
…indow

Signed-off-by: Baylem <zelathis@gmail.com>
…l backups

Signed-off-by: Baylem <zelathis@gmail.com>
Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem marked this pull request as ready for review November 11, 2026 09:55
@Baylem
Baylem requested a review from ValgulNecron as a code owner November 11, 2026 09:55
@ValgulNecron

Copy link
Copy Markdown
Collaborator

Hey, could you please add changelog entry?

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem

Baylem commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Added the requested unreleased changelog entry.

CI passed on a0891c6b. The diff through 955171aa changes only the changelog entry and its index link. I canceled the repeat CI run because this update changes only Markdown.

…obe parents

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem
Baylem changed the base branch from stack/870-backup-and-probes to master November 12, 2026 02:03
@Baylem
Baylem changed the base branch from master to stack/870-backup-and-probes November 12, 2026 03:21
…cation

Signed-off-by: Baylem <zelathis@gmail.com>
@Baylem Baylem self-assigned this Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: agent in-pod sidecar area: e2e kind-based end-to-end suite area: operator controller-runtime operator and CRDs type: fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(backups): distinguish flushed saves from verified write freezes

2 participants