Summary
Resume intermittently fails at its first MintCert call even though the worker→actor assignment has already been committed to the authoritative store.
The failure is flaky: the authorization decision reads a replica-local workercache.Cache updated by an asynchronous watch, while ResumeActor writes the assignment directly to the store. A normal propagation delay can therefore make a valid assignment appear absent.
CI failures
Representative error:
failed to resume Actor again: rpc error: code = PermissionDenied desc = mint actor certificate:
rpc error: code = PermissionDenied desc = caller is not permitted to mint credentials for this actor
Recent examples across unrelated PRs:
A local serial Full/Full experiment also reproduced the same PermissionDenied twice in 50 runs, so E2E test parallelism is not required.
Representative timeline
From run 31628573958, using each RPC's logged elapsed time to derive its start time:
18:50:06.844197 ateapi begins ResumeActor
18:50:06.844788 selects the worker and commits UpdateWorker
18:50:06.847159 atelet Restore begins
18:50:06.852643 ateom RestoreWorkload begins
18:50:06.856462 MintCert begins
18:50:06.856480 ActorIdentity denies: worker has no actor assignment
18:50:06.858441 ResumeActor fails at CallAteletRestore
The authoritative write precedes the denied read by only 11.7ms. No event loss is required.
Root cause
ResumeActor commits the worker assignment to the store before asking atelet to restore. Restore immediately mints actor credentials because the workload must have its identity before startup.
MintCert, however, authorizes using the worker cache. The store write and cache watch delivery are not synchronized, and requests can reach either ateapi replica. If the selected replica has not applied the worker update yet—or has not observed the worker entry at all—it denies a valid mint. There is no retry on this path, so the transient cache state fails the entire resume.
Reading the store instead is not a small substitution. A worker's key there is (namespace, pool, pod), while a mint request carries only (namespace, pod, podUID), so the cache is the only thing that can supply the missing pool — which is what puts an asynchronously updated copy on the authorization path in the first place. The pool earns nothing as part of that key: Kubernetes already guarantees Pod names are unique within a namespace, and no query, prefix scan or bulk operation depends on it.
Deterministic reproduction
- Start a worker cache whose initial relist sees an unassigned worker.
- Prevent its worker watch from delivering subsequent updates.
- Write an assignment for that worker to the authoritative store.
- Confirm
store.GetWorker returns the assignment.
- Call
MintCert.
The mint is denied because authorization reads the stale cache, even though the assignment is committed.
Expected behavior
Once the assignment is committed, MintCert must authorize from authoritative worker state without waiting for asynchronous cache propagation. The authorization must continue to verify the caller node, Pod UID, actor UID, and the reciprocal actor↔worker placement.
Summary
Resume intermittently fails at its first
MintCertcall even though the worker→actor assignment has already been committed to the authoritative store.The failure is flaky: the authorization decision reads a replica-local
workercache.Cacheupdated by an asynchronous watch, whileResumeActorwrites the assignment directly to the store. A normal propagation delay can therefore make a valid assignment appear absent.CI failures
Representative error:
Recent examples across unrelated PRs:
e2e-test (cert)— failed, then a rerun of the same workflow succeedede2e-test (token)e2e-test (token)e2e-test (cert)e2e-test (cert)A local serial Full/Full experiment also reproduced the same
PermissionDeniedtwice in 50 runs, so E2E test parallelism is not required.Representative timeline
From run 31628573958, using each RPC's logged elapsed time to derive its start time:
The authoritative write precedes the denied read by only 11.7ms. No event loss is required.
Root cause
ResumeActorcommits the worker assignment to the store before asking atelet to restore. Restore immediately mints actor credentials because the workload must have its identity before startup.MintCert, however, authorizes using the worker cache. The store write and cache watch delivery are not synchronized, and requests can reach either ateapi replica. If the selected replica has not applied the worker update yet—or has not observed the worker entry at all—it denies a valid mint. There is no retry on this path, so the transient cache state fails the entire resume.Reading the store instead is not a small substitution. A worker's key there is
(namespace, pool, pod), while a mint request carries only(namespace, pod, podUID), so the cache is the only thing that can supply the missing pool — which is what puts an asynchronously updated copy on the authorization path in the first place. The pool earns nothing as part of that key: Kubernetes already guarantees Pod names are unique within a namespace, and no query, prefix scan or bulk operation depends on it.Deterministic reproduction
store.GetWorkerreturns the assignment.MintCert.The mint is denied because authorization reads the stale cache, even though the assignment is committed.
Expected behavior
Once the assignment is committed,
MintCertmust authorize from authoritative worker state without waiting for asynchronous cache propagation. The authorization must continue to verify the caller node, Pod UID, actor UID, and the reciprocal actor↔worker placement.