Expected Behavior
ate.actor.lifecycle.operation.duration carries the ate.workerpool.namespace + ate.workerpool.name pair on every suspend and pause that ran on a worker. A successful suspend of an actor on ate-workers/pool-a reports that pool, so pool-level latency and error rates stay comparable across the five operations.
Actual Behavior
The pair is inverted for suspend and pause: a successful operation reports no pool, and a failed one reports it.
Both workflows record the histogram from a deferred call that reads the actor variable:
|
defer func() { |
|
w.instruments.recordLifecycleOp(ctx, ateattr.OperationSuspend, start, err, |
|
lifecycleOpAttrs(actor, actorTemplate, "", wireSnapshotScope)...) |
|
}() |
The happy path reassigns that variable to the finalized record as its last step:
workflow_suspend.go:85-90 — actor = finalized from ensureSuspendedFinalized
workflow_pause.go:77-82 — actor = finalized from ensurePausedFinalized
FinalizeSuspended and FinalizePaused commit STATUS_SUSPENDED/STATUS_PAUSED and the cleared WorkerAssignment in one update. By the time the defer runs, actor.GetWorkerAssignment() is nil, so lifecycleOpAttrs omits both keys. A failure returns early with the pre-finalize record, which still names the worker, so the pool pair lands there instead.
The re-entrant fast-forward path has the same result for a different reason: a workflow that finds the actor already SUSPENDED/PAUSED returns a record that carries no assignment.
Of the five operations, resume is the only one where the pool is reliable. create has no assignment yet, and delete builds its labels by hand (actor.go:313-322) and never stamps a pool at all.
Net effect: any dashboard that breaks lifecycle latency down by pool sees suspend and pause only when they fail.
Steps to Reproduce the Problem
- Resume an actor so it holds a
WorkerAssignment in a named pool.
- Suspend it, and let the workflow reach
FinalizeSuspended.
- Read
ate.actor.lifecycle.operation.duration for ate.actor.operation.name=suspend: the sample has neither ate.workerpool.namespace nor ate.workerpool.name.
- Repeat with a suspend that fails after the assign step. That sample carries both keys.
Suggested Fix
Snapshot the labels before the record loses the pointers they read, the way crash.go already does for the crash counter:
|
// Snapshot crash attributes before pod and pool pointers are cleared below; |
|
// the counter itself is emitted only after the transition commits. |
|
crashAttrs := ateattr.ActorMetricAttributes(actor, sandboxClass, opName, reason) |
Concretely, in SuspendActor and PauseActor, build lifecycleOpAttrs(...) into a variable just before the finalize step and let the defer use it when set, falling back to the current computation for paths that end earlier.
Worth deciding at the same time whether delete should carry the pool, since it is released before the operation completes and hits the same ordering problem.
Specifications
- Version:
main @ 2b3a471
- Platform: any
Follow-up from review of #953 (comment #953 (comment)).
Expected Behavior
ate.actor.lifecycle.operation.durationcarries theate.workerpool.namespace+ate.workerpool.namepair on every suspend and pause that ran on a worker. A successful suspend of an actor onate-workers/pool-areports that pool, so pool-level latency and error rates stay comparable across the five operations.Actual Behavior
The pair is inverted for suspend and pause: a successful operation reports no pool, and a failed one reports it.
Both workflows record the histogram from a deferred call that reads the
actorvariable:substrate/cmd/ateapi/internal/controlapi/workflow_suspend.go
Lines 45 to 48 in 2b3a471
The happy path reassigns that variable to the finalized record as its last step:
workflow_suspend.go:85-90—actor = finalizedfromensureSuspendedFinalizedworkflow_pause.go:77-82—actor = finalizedfromensurePausedFinalizedFinalizeSuspendedandFinalizePausedcommitSTATUS_SUSPENDED/STATUS_PAUSEDand the clearedWorkerAssignmentin one update. By the time the defer runs,actor.GetWorkerAssignment()is nil, solifecycleOpAttrsomits both keys. A failure returns early with the pre-finalize record, which still names the worker, so the pool pair lands there instead.The re-entrant fast-forward path has the same result for a different reason: a workflow that finds the actor already
SUSPENDED/PAUSEDreturns a record that carries no assignment.Of the five operations,
resumeis the only one where the pool is reliable.createhas no assignment yet, anddeletebuilds its labels by hand (actor.go:313-322) and never stamps a pool at all.Net effect: any dashboard that breaks lifecycle latency down by pool sees suspend and pause only when they fail.
Steps to Reproduce the Problem
WorkerAssignmentin a named pool.FinalizeSuspended.ate.actor.lifecycle.operation.durationforate.actor.operation.name=suspend: the sample has neitherate.workerpool.namespacenorate.workerpool.name.Suggested Fix
Snapshot the labels before the record loses the pointers they read, the way
crash.goalready does for the crash counter:substrate/cmd/ateapi/internal/controlapi/crash.go
Lines 74 to 76 in 2b3a471
Concretely, in
SuspendActorandPauseActor, buildlifecycleOpAttrs(...)into a variable just before the finalize step and let the defer use it when set, falling back to the current computation for paths that end earlier.Worth deciding at the same time whether
deleteshould carry the pool, since it is released before the operation completes and hits the same ordering problem.Specifications
main@ 2b3a471Follow-up from review of #953 (comment #953 (comment)).