test(longhaul): add kill-operator-pod and kill-primary-pod chaos operations

.github/workflows/longhaul-smoke.yml

@@ -16,10 +16,10 @@
 #   The operator + sidecar are built from THIS PR's source (via the shared
 #   test-build-and-package.yml reusable workflow) so a code change that breaks
 #   the operator is caught. The documentdb + gateway (database) images are NOT
-#   rebuilt: test-build-and-package.yml probes the public registry for the
-#   pinned documentDbVersion and, when present, runs in "registry mode" and
-#   simply re-tags the public images into the build artifact. The long-haul
-#   driver image is the only extra image this workflow builds.
+#   built at all: the smoke exercises an upgrade between two published database
+#   releases, pulling both the 0.110.0 base and the under-test target straight
+#   from the public registry. The long-haul driver image is the only extra
+#   image this workflow builds.
 #
 # Environment bring-up is delegated to the shared composite action
 #   .github/actions/setup-test-environment
@@ -42,23 +42,37 @@
 #   longhaul-report ConfigMap's result field.
 #
 # CI budget
-#   Scale/upgrade disruption ops are disabled for the smoke run
-#   (LONGHAUL_MIN_INSTANCES == LONGHAUL_MAX_INSTANCES) so the gate is a fast,
-#   deterministic data-durability check (writers + verifier) that fits a
-#   GitHub-hosted runner and finishes in a few minutes.
+#   Scale ops run for real (MIN=2, MAX=3). The gate exercises every registered
+#   operation once and finishes within a GitHub-hosted runner's budget.
+#
+# Operations exercised (sequence mode)
+#   The smoke runs the operation scheduler in sequence mode
+#   (LONGHAUL_OPERATION_MODE=sequence) with an explicit ordered list
+#   (LONGHAUL_OPERATION_SEQUENCE). Each operation — scale-up, scale-down,
+#   upgrade-documentdb, kill-operator-pod, kill-primary-pod — runs exactly once,
+#   in order, gated by the same steady-state / precondition / recovery logic the
+#   long-haul run uses. This is the deterministic path that builds confidence the
+#   operations execute end-to-end; the gate asserts every sequenced operation
+#   reached PASSED and the run reached COMPLETE. The upgrade is a real
+#   cross-version rolling upgrade: the cluster starts on the most recent
+#   published release before the under-test version (0.110.0) and the
+#   upgrade-documentdb operation moves documentDBVersion forward to the
+#   under-test version (the chart's documentDbVersion), exercising the
+#   operator's rolling-update path with clean semver versions the image-rollback
+#   webhook accepts. MAX_DURATION is only the completion watchdog.
 #
 # Data-protection gate
 #   The backup verifier is exercised for real, not just compiled in. The kind
 #   cluster already has CSI VolumeSnapshot support (setup-test-environment runs
 #   deploy-csi-driver.sh: external-snapshotter + a default csi-hostpath
-#   VolumeSnapshotClass), so a single-instance cluster can complete snapshot
-#   backups — exactly as the e2e scheduled-backup test proves. The smoke run
-#   sets a per-minute backup schedule and a 30s verify interval (vs the 5m
-#   default, via LONGHAUL_BACKUP_VERIFY_INTERVAL) so the verifier's periodic
-#   loop fires many times within the window and reliably observes a
-#   scheduled+completed backup. It then asserts scheduled AND completed >= 1
-#   (with no retention leak or completion stall), so a broken backup path fails
-#   the PR rather than passing silently as a no-op.
+#   VolumeSnapshotClass), so the cluster can complete snapshot backups —
+#   exactly as the e2e scheduled-backup test proves. The smoke run sets a
+#   per-minute backup schedule and a 30s verify interval (vs the 5m default,
+#   via LONGHAUL_BACKUP_VERIFY_INTERVAL) so the verifier's periodic loop fires
+#   many times within the window and reliably observes a scheduled+completed
+#   backup. It then asserts scheduled AND completed >= 1 (with no retention leak
+#   or completion stall), so a broken backup path fails the PR rather than
+#   passing silently as a no-op.
 
 name: Long-Haul Smoke Gate
 
@@ -74,9 +88,9 @@ on:
   workflow_dispatch:
     inputs:
       max_duration:
-        description: "Bounded driver run length (Go duration). Keep >= 6m so at least one per-minute backup is scheduled and completed within the window."
+        description: "Completion watchdog duration (Go duration, e.g. 20m). Keep >= 6m so at least one per-minute backup is scheduled and completed within the window."
         required: false
-        default: "6m"
+        default: "20m"
 
 permissions:
   contents: read
@@ -102,8 +116,9 @@ env:
 
 jobs:
   # ---------------------------------------------------------------------------
-  # Build operator + sidecar from this PR's source; reuse public documentdb /
-  # gateway images (registry mode) so the database images are NOT rebuilt.
+  # Build operator + sidecar from this PR's source; the database (documentdb /
+  # gateway) images are NOT built or used from the artifact — the smoke pulls
+  # the released base and upgrade-target database images directly.
   # Produces the platform-images tar + helm chart artifacts the setup action
   # loads into kind.
   # ---------------------------------------------------------------------------
@@ -119,7 +134,7 @@ jobs:
     needs: build
     if: always() && needs.build.result == 'success'
     runs-on: ubuntu-22.04
-    timeout-minutes: 40
+    timeout-minutes: 50
     env:
       IMAGE_TAG: ${{ needs.build.outputs.image_tag }}
       EXT_IMAGE_TAG: ${{ needs.build.outputs.ext_image_tag }}
@@ -128,8 +143,19 @@ jobs:
       # Must match the cluster name the composite action derives:
       #   documentdb-<test-type>-<architecture>-<test-scenario-name>
       KIND_CLUSTER: documentdb-longhaul-amd64-smoke
-      # The pruner's first tick is at 5m; the default must run beyond it.
-      MAX_DURATION: ${{ github.event.inputs.max_duration || '6m' }}
+      # Completion watchdog; also kept beyond the 5m pruner tick and long enough
+      # for at least one per-minute backup to schedule and complete.
+      MAX_DURATION: ${{ github.event.inputs.max_duration || '20m' }}
+      # Two published database releases the smoke upgrades between. The cluster
+      # STARTS at DB_BASE_VERSION and the upgrade operation moves it forward to
+      # DB_TARGET_VERSION (the under-test release; keep in sync with the chart's
+      # documentDbVersion). Both are pulled from the public registry — the smoke
+      # builds no database image. 0.110.0 is the most recent published release
+      # before 0.113.0 (0.111.0 / 0.112.0 were never published).
+      DB_BASE_VERSION: "0.110.0"
+      DB_TARGET_VERSION: "0.113.0"
+      DOCUMENTDB_REPO: ghcr.io/documentdb/documentdb-kubernetes-operator/documentdb
+      GATEWAY_REPO: ghcr.io/documentdb/documentdb-kubernetes-operator/gateway
     steps:
       - name: Checkout
         uses: actions/checkout@v4
@@ -167,7 +193,7 @@ jobs:
           runner: "ubuntu-22.04"
           test-scenario-name: "smoke"
           node-count: "1"
-          instances-per-node: "1"
+          instances-per-node: "2"
           cert-manager-namespace: ${{ env.CERT_MANAGER_NS }}
           operator-namespace: ${{ env.OPERATOR_NS }}
           db-namespace: ${{ env.DB_NS }}
@@ -177,6 +203,12 @@ jobs:
           db-port: ${{ env.GATEWAY_PORT }}
           image-tag: ${{ env.IMAGE_TAG }}
           documentdb-image-tag: ${{ env.EXT_IMAGE_TAG }}
+          # Start the cluster at the released base version so the upgrade
+          # operation performs a real cross-version rolling upgrade. These
+          # overrides make the composite action pull the released images and
+          # deploy the CR with explicit spec.image fields pinned to the base.
+          documentdb-image: ${{ env.DOCUMENTDB_REPO }}:${{ env.DB_BASE_VERSION }}
+          gateway-image: ${{ env.GATEWAY_REPO }}:${{ env.DB_BASE_VERSION }}
           chart-version: ${{ env.CHART_VERSION }}
           use-external-images: "false"
           github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -198,6 +230,51 @@ jobs:
             --from-literal=uri="${URI}" \
             --dry-run=client -o yaml | kubectl apply -f -
 
+      - name: Prepare cross-version upgrade target (0.110.0 -> under-test)
+        run: |
+          set -euo pipefail
+          target_version="${DB_TARGET_VERSION}"
+          echo "Upgrade: ${DB_BASE_VERSION} -> ${target_version}"
+
+          # Pull the RELEASED base and target database images straight from the
+          # public registry and load them into kind under their canonical repo +
+          # semver tag, so documentDBVersion resolves to the real releases. The
+          # smoke builds no database image: it upgrades between two published
+          # releases (base -> target), both pulled here (the base is also
+          # deployed by the setup step via the spec.image overrides).
+          for component in documentdb gateway; do
+            img="ghcr.io/documentdb/documentdb-kubernetes-operator/${component}:${target_version}"
+            docker pull "${img}"
+            kind load docker-image "${img}" --name "${KIND_CLUSTER}"
+          done
+
+          # setup-test-environment deployed the CR with explicit spec.image
+          # fields pinned to the base version, which take precedence over
+          # documentDBVersion. Convert to the equivalent version-based reference
+          # (same base payload, image cleared) so the driver's upgrade operation
+          # can move documentDBVersion from the base to the target. Because the
+          # resolved version is unchanged (base == running) the image-rollback
+          # admission webhook accepts this patch.
+          version_patch=$(BASE_VERSION="${DB_BASE_VERSION}" jq -nc '{
+              spec: {
+                documentDBVersion: env.BASE_VERSION,
+                image: {
+                  documentDB: null,
+                  gateway: null
+                }
+              }
+            }')
+          kubectl patch documentdb "${DB_NAME}" -n "${DB_NS}" \
+            --type merge -p "${version_patch}"
+
+          # The upgrade operation reads this desired version from the ConfigMap
+          # and patches documentDBVersion to it; both base and target are clean
+          # semvers so the forward move (0.110.0 -> target) is permitted.
+          kubectl create configmap longhaul-versions \
+            -n "${DB_NS}" \
+            --from-literal="desired-documentdb-version=${target_version}" \
+            --dry-run=client -o yaml | kubectl apply -f -
+
       - name: Deploy long-haul driver (real manifests, bounded override)
         run: |
           # RBAC applies unmodified (namespace matches DB_NS).
@@ -215,11 +292,18 @@ jobs:
 
           # Bounded, deterministic smoke override — patch ONLY runtime knobs on
           # the shipped ConfigMap; the manifest structure is unchanged.
-          #  - MAX_DURATION: finite run
+          #  - MAX_DURATION: finite run (completion watchdog)
           #  - RESET_DATA: fresh collection each CI run
           #  - RETAIN_PER_WRITER: low enough to force a real prune at 5m
-          #  - MIN==MAX instances: disable disruptive scale ops (fast + stable)
+          #  - OPERATION_MODE=sequence + OPERATION_SEQUENCE: run every operation
+          #    (scale-up, scale-down, upgrade-documentdb, kill-operator-pod,
+          #    kill-primary-pod) exactly once, in a fixed order. This is the
+          #    deterministic path that builds confidence the operations execute
+          #    end-to-end; MAX_DURATION becomes the completion watchdog.
           #  - short cadences so the verifier gets several cycles in the window
+          #  - short steady-state gate with a bounded recovery budget. Sequence
+          #    mode paces ops purely by the steady-state/recovery gates, so
+          #    LONGHAUL_OP_COOLDOWN (a random-mode-only rate limiter) is omitted.
           #  - BACKUP_*: exercise the data-protection verifier for real — a
           #    per-minute schedule so at least one backup is scheduled and
           #    completed within the bounded window, plus a 30s verify interval
@@ -233,12 +317,13 @@ jobs:
               LONGHAUL_RESET_DATA: "true",
               LONGHAUL_NUM_WRITERS: "2",
               LONGHAUL_RETAIN_PER_WRITER: "100",
-              LONGHAUL_OP_COOLDOWN: "30s",
-              LONGHAUL_STEADY_STATE_WAIT: "10s",
-              LONGHAUL_RECOVERY_TIMEOUT: "2m",
+              LONGHAUL_OPERATION_MODE: "sequence",
+              LONGHAUL_OPERATION_SEQUENCE: "scale-up,scale-down,upgrade-documentdb,kill-operator-pod,kill-primary-pod",
+              LONGHAUL_STEADY_STATE_WAIT: "5s",
+              LONGHAUL_RECOVERY_TIMEOUT: "5m",
               LONGHAUL_REPORT_INTERVAL: "30s",
-              LONGHAUL_MIN_INSTANCES: "1",
-              LONGHAUL_MAX_INSTANCES: "1",
+              LONGHAUL_MIN_INSTANCES: "2",
+              LONGHAUL_MAX_INSTANCES: "3",
               LONGHAUL_BACKUP_ENABLED: "true",
               LONGHAUL_BACKUP_SCHEDULE: "*/1 * * * *",
               LONGHAUL_BACKUP_RETENTION_DAYS: "1",
@@ -255,7 +340,7 @@ jobs:
         id: wait
         run: |
           set -euo pipefail
-          deadline=$(( $(date +%s) + 900 ))  # 15 min hard cap
+          deadline=$(( $(date +%s) + 1800 ))  # 30 min hard cap
           exit_code=""
           while [[ $(date +%s) -lt ${deadline} ]]; do
             pod=$(kubectl get pods -n "${DB_NS}" \
@@ -301,8 +386,14 @@ jobs:
             -o jsonpath='{.data.result}' 2>/dev/null || echo "MISSING")
           report=$(kubectl get configmap longhaul-report -n "${DB_NS}" \
             -o jsonpath='{.data.latest-report}' 2>/dev/null || echo "MISSING")
+          operation_status=$(kubectl get configmap longhaul-report -n "${DB_NS}" \
+            -o json | jq -r '.data["operation-status"] // "MISSING"')
+          operation_results=$(kubectl get configmap longhaul-report -n "${DB_NS}" \
+            -o json | jq -r '.data["operation-results"] // "MISSING"')
           echo "Driver exit code : ${exit_code}"
           echo "Report result    : ${result}"
+          echo "Operation status : ${operation_status}"
+          echo "Operation results: ${operation_results}"
 
           if [[ "${exit_code}" != "0" ]]; then
             echo "::error::Driver exited non-zero (${exit_code})."
@@ -312,11 +403,24 @@ jobs:
             echo "::error::longhaul-report result is '${result}', expected PASS."
             exit 1
           fi
+          if [[ "${operation_status}" != "COMPLETE" ]]; then
+            echo "::error::operation-status is '${operation_status}', expected COMPLETE."
+            exit 1
+          fi
+          # Sequence mode: assert every operation in the configured sequence ran
+          # and passed. Order is fixed by LONGHAUL_OPERATION_SEQUENCE above.
+          if ! jq -e '
+            (map(.name)) == ["scale-up","scale-down","upgrade-documentdb","kill-operator-pod","kill-primary-pod"] and
+            all(.[]; .status == "PASSED")
+          ' <<<"${operation_results}" >/dev/null; then
+            echo "::error::operation-results did not show every sequenced operation PASSED in order."
+            exit 1
+          fi
           if ! grep -Eq 'pruner: pruned [1-9][0-9]* docs' <<<"${report}"; then
             echo "::error::Retention pruner did not report deleting any documents."
             exit 1
           fi
-          echo "✅ Long-haul smoke gate passed (exit 0, report PASS, retention pruned documents)."
+          echo "✅ Long-haul smoke gate passed (sequence COMPLETE, all operations PASSED, report PASS, retention pruned documents)."
 
       - name: Assert data-protection verifier ran
         run: |

docs/designs/long-haul-test-design.md

@@ -44,7 +44,7 @@ flowchart LR
 | Component | Role | Output |
 |---|---|---|
 | **Writer/Verifier** | Data-plane workload. Connects via `mongodb://` only — no k8s imports. Writers insert monotonic sequences with checksums under majority write concern; verifiers scan for gaps and bad checksums. | Counters (acked, failed, verify passes, gaps, checksum errors); errors to journal. |
-| **Operation Scheduler** | Control plane. Applies weighted-random ops (scale, kill, failover, backup, upgrade) with preconditions and cooldowns. | Operation start/end events to journal. |
+| **Operation Runner** | Control plane. Applies weighted-random ops for production long-haul runs, a deterministic named sequence for smoke/reproduction, or no ops when disabled. | Bounded per-operation results/aggregates plus operation events to journal. |
 | **Monitor** | Polls pod RSS/CPU and checks readiness of operator + DB pods. | Periodic samples + readiness events to journal. |
 | **Journal** | In-process append-only event log shared by all components. | Reproducible event stream for the report. |
 | **Report** | Aggregates the journal into a markdown summary at a configurable interval; raises alerts on threshold breaches. | Markdown report; alert lines. |
@@ -77,7 +77,11 @@ The test runs **continuously** — no cycles, no scheduled resets. Workload, met
 
 ## Operations
 
-The scheduler picks operations from these categories with weighted randomization:
+Production runs use weighted randomization. Deterministic smoke and reproduction
+runs can instead request a comma-separated sequence of stable operation names;
+each operation runs exactly once in order and the driver exits as soon as the
+sequence completes or fails. A disabled mode leaves the workload running without
+management operations.
 
 | Category | Examples |
 |---|---|
@@ -87,16 +91,33 @@ The scheduler picks operations from these categories with weighted randomization
 | **Chaos** | kill primary pod, drain node, kill operator pod |
 | **Data protection** | trigger backup, verify backup |
 
-**Sequencing invariants** (enforced by the scheduler — exact values live in code):
+**Operation invariants** (exact values live in code):
 
-- One disruptive op at a time. Overlapping disruptions are non-diagnosable.
-- Per-category cooldown between ops. Lets the cluster stabilize.
-- Steady-state gate — health check must pass before the next op fires.
+- One disruptive op at a time in every mode. Overlapping disruptions are
+  non-diagnosable.
+- Random mode applies the global cooldown between attempts.
+- The steady-state gate must pass before each operation. Sequence mode also
+  requires each named precondition to become true within the recovery timeout.
 
 **Backup is not isolated.** It runs concurrently with topology changes and chaos so that backup-vs-topology serialization bugs surface here rather than in production — that serialization is the backup feature's job, not the harness's.
 
 Each operation declares an **outage policy**: tolerated write failures during its disruption window and a max recovery time. Breaching the policy is recorded as a Tier-1 failure (see Failure Tiers).
 
+Operation state is intentionally bounded for multi-day runs. Random mode keeps
+only passed/failed counters per registered operation type; sequence mode keeps
+one mutable `PENDING`/`RUNNING`/`PASSED`/`FAILED` result per requested item.
+Execution errors, precondition timeouts, outage-policy violations, and an
+incomplete sequence at shutdown all produce a failing final verdict.
+
+**Sequence mode** (used by the PR smoke gate) runs the registered operations in
+an explicit, fixed order (`LONGHAUL_OPERATION_SEQUENCE`), executing each exactly
+once behind the same steady-state / precondition / recovery gates as random
+mode, rather than selecting operations by weight for the full duration. This
+gives the smoke gate a deterministic PASS/FAIL verdict — every sequenced
+operation must reach `PASSED` and the run must reach `COMPLETE`; `MAX_DURATION`
+becomes the completion watchdog, and a sequence that has not finished at shutdown
+is a failing `INCOMPLETE` verdict.
+
 ---
 
 ## Data Plane Workload