Skip to content

cache: honor backend phase timeouts in origin fetches - #4372

Closed
larry-dalmeida wants to merge 1 commit into
larry-almeida-zse-cache-correctness-prfrom
larry-almeida-zse-cache-timeouts-pr
Closed

larry-dalmeida wants to merge 1 commit into
larry-almeida-zse-cache-correctness-prfrom
larry-almeida-zse-cache-timeouts-pr

Conversation

@larry-dalmeida

@larry-dalmeida larry-dalmeida commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary of changes

Apply configured backend phase timeouts to cache-origin requests. These bound transport phases, not the entire request or response-body read.

How to Verify

Shared setup

mkdir skipper-review && cd skipper-review
git clone https://github.com/zalando/skipper.git skipper && cd skipper
for PR in 4369 4370 4371 4372 4384 4385 4386 4387; do
  git fetch origin pull/$PR/head:pr-$PR
  git worktree add ../skipper-pr-$PR pr-$PR
done
cd ..
git clone https://github.com/larry-dalmeida/skipper-cache-e2e-tests.git
cd skipper-cache-e2e-tests
make verify SKIPPER_CHECKOUT=../skipper-pr-4387

make verify starts or reuses Minikube, the mock origin and the stack; runs the smoke and coalescing checks; and validates supported dashboard queries. It prints a report path and exits non-zero on failure. For outage evidence, run make chaos SKIPPER_CHECKOUT=../skipper-pr-4387 NAME=origin-stale; it includes its own timed load. Requirements: Python 3.9+, Make, Bash, Git, curl, kubectl, Minikube, Docker and Go compatible with Skipper's go.mod. Free ports: 8080, 8081, 6379, 4318, 9091, 9999, 9911 (8089 for endless load). Dynamic URLs: .runtime/urls.txt; stop with make stop.

Validated on #4387 (d767f636): smoke: 6,398 requests, 0 failures; origin-stale: 155 failed revalidations across 77 routes. Smoke metric checks: 42 PASS, 2 fault-only NOT EXERCISED, 0 FAIL.

Focused tests

go -C ../skipper-pr-4372 test -race -count=1 . -run '^(TestCacheNetOptions_BackendTimeoutMapping|TestCacheBackendHeaderTimeout_PreservesStaleAndRecovers)$'

Expected: 2 focused tests pass. Delayed-header coverage verifies the 50 ms response-header timeout, unchanged last-good stale body, then MISS/HIT recovery. origin-stale injects immediate 503s, not delays; origin p95 does not measure Skipper's wait.

Checklist

Rollback Plan

Revert this PR to restore previous cache-origin timeout behavior.

References & Additional Context

Depends on #4371. The combined candidate passed a 60,000-request local exercise; the larger staircase campaign remained incomplete after a load-generator admission drop.

E2E Testing Results

E2E runs exercising origin fetches with the stack applied. These runs set no backend phase timeouts and injected no origin delay, so no timeout fired. Full results: larry-dalmeida/skipper-cache-e2e-tests/results

Test Scenarios:

Scenario Build Requests Failures Status
Smoke bda674b (top of stack) 6,306 0% ✓ Pass
Origin Stale bda674b (top of stack) 50,870 33.3% (503s while origin was failing) ✓ Pass
Skipper Kill 8a80d62e + local changes 38,969 1.0% (during restart) ✓ Pass

Validation Panels:

  • Origin Latency (p95) — origin fetch latency with this PR's fetch path (bda674b smoke): ~9.5–10 ms p95 for every status. The timeouts were never reached, so this is a baseline, not timeout behavior
    Origin Latency p95

Not shown by these runs: a backend phase timeout firing. Time to First Hit (skipper_cache_first_hit_seconds) is added by #4385 later in the stack, so it is not evidence for this PR.

Wire TCP dial, response-header, TLS handshake, and expect-continue backend options to the cache client transport. Preserve dial-only semantics instead of using http.Client.Timeout. Match proxy defaults for zero header/expect-continue settings. Reproduce the ignored header deadline and verify timeout SIE retention, unchanged timestamps, and healthy recovery using a real delayed origin.

Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>

Signed-off-by: Larry D Almeida <hello@larrydalmeida.com>
@larry-dalmeida
larry-dalmeida force-pushed the larry-almeida-zse-cache-timeouts-pr branch from 509147c to 7e48ec0 Compare October 26, 2026 14:37
@larry-dalmeida
larry-dalmeida force-pushed the larry-almeida-zse-cache-correctness-pr branch from ebcd7ff to e003f9a Compare October 26, 2026 14:37
@zalando-robot

Copy link
Copy Markdown

Docker image "registry-write.opensource.zalan.do/teapot/skipper:7e48ec0fb570d5e26b9ad551ec7a9d25dfef209d" is not based on an approved base image. Any production deployment relying on this image will be blocked.

To create a compliant Docker image of your application, you must reference an allowed Docker image as its base image in your Dockerfile. This base image must come from the Zalando Container Registry namespace library and use a recommended version as listed in the documentation.

@larry-dalmeida

Copy link
Copy Markdown
Collaborator Author

Superseded by #4394. Changes consolidated into five focused commits with published e2e evidence.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants