Skip to content

ateapi: suspend and pause stamp the worker pool only when they fail #957

Description

@JeffLuoo

Expected Behavior

ate.actor.lifecycle.operation.duration carries the ate.workerpool.namespace + ate.workerpool.name pair on every suspend and pause that ran on a worker. A successful suspend of an actor on ate-workers/pool-a reports that pool, so pool-level latency and error rates stay comparable across the five operations.

Actual Behavior

The pair is inverted for suspend and pause: a successful operation reports no pool, and a failed one reports it.

Both workflows record the histogram from a deferred call that reads the actor variable:

defer func() {
w.instruments.recordLifecycleOp(ctx, ateattr.OperationSuspend, start, err,
lifecycleOpAttrs(actor, actorTemplate, "", wireSnapshotScope)...)
}()

The happy path reassigns that variable to the finalized record as its last step:

  • workflow_suspend.go:85-90actor = finalized from ensureSuspendedFinalized
  • workflow_pause.go:77-82actor = finalized from ensurePausedFinalized

FinalizeSuspended and FinalizePaused commit STATUS_SUSPENDED/STATUS_PAUSED and the cleared WorkerAssignment in one update. By the time the defer runs, actor.GetWorkerAssignment() is nil, so lifecycleOpAttrs omits both keys. A failure returns early with the pre-finalize record, which still names the worker, so the pool pair lands there instead.

The re-entrant fast-forward path has the same result for a different reason: a workflow that finds the actor already SUSPENDED/PAUSED returns a record that carries no assignment.

Of the five operations, resume is the only one where the pool is reliable. create has no assignment yet, and delete builds its labels by hand (actor.go:313-322) and never stamps a pool at all.

Net effect: any dashboard that breaks lifecycle latency down by pool sees suspend and pause only when they fail.

Steps to Reproduce the Problem

  1. Resume an actor so it holds a WorkerAssignment in a named pool.
  2. Suspend it, and let the workflow reach FinalizeSuspended.
  3. Read ate.actor.lifecycle.operation.duration for ate.actor.operation.name=suspend: the sample has neither ate.workerpool.namespace nor ate.workerpool.name.
  4. Repeat with a suspend that fails after the assign step. That sample carries both keys.

Suggested Fix

Snapshot the labels before the record loses the pointers they read, the way crash.go already does for the crash counter:

// Snapshot crash attributes before pod and pool pointers are cleared below;
// the counter itself is emitted only after the transition commits.
crashAttrs := ateattr.ActorMetricAttributes(actor, sandboxClass, opName, reason)

Concretely, in SuspendActor and PauseActor, build lifecycleOpAttrs(...) into a variable just before the finalize step and let the defer use it when set, falling back to the current computation for paths that end earlier.

Worth deciding at the same time whether delete should carry the pool, since it is released before the operation completes and hits the same ordering problem.

Specifications

  • Version: main @ 2b3a471
  • Platform: any

Follow-up from review of #953 (comment #953 (comment)).

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions