[larry-dalmeida]

Summary of changes

Adds a dispatch label (direct | pipeline) to skipper_cache_origin_duration_seconds, introduced in #4386.

In force mode with a static backend, cold-miss (coalesced) and revalidation origin fetches are dispatched directly and bypass Skipper's proxy pipeline. They are invisible to MeasureBackend* metrics and the access log, and cache.reval_backend_dispatch only counts revalidation. #4370 documents this gap; this PR makes the split observable per route and status class.

Label cardinality is bounded: dispatch has 2 values, so each existing series is at most doubled.

Docs (docs/operation/operation.md) and CONTEXT.md are updated.

Stacked PR: this targets larry-almeida-zse-cache-timeouts-pr, so the GitHub diff also contains the #4384/#4385/#4386 commits. Only the top commit (d767f636) is new; please review that commit. Merge after #4386.

How to Verify

Shared setup

mkdir skipper-review && cd skipper-review
git clone https://github.com/zalando/skipper.git skipper && cd skipper
for PR in 4369 4370 4371 4372 4384 4385 4386 4387; do
  git fetch origin pull/$PR/head:pr-$PR
  git worktree add ../skipper-pr-$PR pr-$PR
done
cd ..
git clone https://github.com/larry-dalmeida/skipper-cache-e2e-tests.git
cd skipper-cache-e2e-tests
make verify SKIPPER_CHECKOUT=../skipper-pr-4387

make verify starts or reuses Minikube, the mock origin and the stack; runs the smoke and coalescing checks; and validates supported dashboard queries. It prints a report path and exits non-zero on failure. For outage evidence, run make chaos SKIPPER_CHECKOUT=../skipper-pr-4387 NAME=origin-stale; it includes its own timed load. Requirements: Python 3.9+, Make, Bash, Git, curl, kubectl, Minikube, Docker and Go compatible with Skipper's go.mod. Free ports: 8080, 8081, 6379, 4318, 9091, 9999, 9911 (8089 for endless load). Dynamic URLs: .runtime/urls.txt; stop with make stop.

Validated on #4387 (d767f636): smoke: 6,398 requests, 0 failures; origin-stale: 155 failed revalidations across 77 routes. Smoke metric checks: 42 PASS, 2 fault-only NOT EXERCISED, 0 FAIL.

Focused tests

go -C ../skipper-pr-4387 test -race -count=1 ./filters/cache -run '^(TestCacheOriginStatusClass|TestCacheRecordCacheControlBoundedLabels|TestCacheRecordCacheControlOncePerResponse|TestCacheFilterOriginMetrics|TestCacheRouteRevalDropped|TestCacheResponseOriginCacheControl|TestCacheRouteRevalFailurePaths|TestCacheRouteRevalTerminalOutcomes|TestCacheFilter_OriginDurationDispatch)$'
go -C ../skipper-pr-4387 test -race -count=1 ./metrics -run '^(TestPrometheusCacheRouteReval|TestPrometheusCacheOriginDurationRouteStatus|TestPrometheusCacheOriginCacheControl|TestPrometheusCacheOriginDuration)$'
go -C ../skipper-pr-4387 test -race -count=1 ./metrics/metricstest -run '^TestCacheOriginMockMetrics$'

Expected: 14 focused tests pass. All 30 observed origin-duration series were dispatch="direct" (force mode/static backend); allowed values: direct, pipeline. The histogram covers cold misses and refreshes; cache.reval_backend_dispatch counts refreshes only.

Checklist

• Added regression tests and documentation.
• Confirm CI passes on the current head.

Rollback Plan

Revert this PR; the metric returns to the route/status_class label set from #4386.

References & Additional Context

• Gap documented in cache: centralize dispatch and preserve response bodies #4370.
• Depends on cache metrics: route revalidation, origin duration, Cache-Control #4386 (stack: cache: improve metrics accuracy and storage observability #4369 → cache: centralize dispatch and preserve response bodies #4370 → cache: preserve representations across refresh and failure #4371 → cache: honor backend phase timeouts in origin fetches #4372 → cache metrics: route revalidation, origin duration, Cache-Control #4386).
• Adding a label to an existing histogram breaks queries or recording rules that aggregate on exact label sets only once the metric is released. cache metrics: route revalidation, origin duration, Cache-Control #4386 is unmerged, so no released series change.

[larry-dalmeida]

Superseded by #4395, stacked above #4394. Changes consolidated with published metric, dashboard and fault-test evidence.