From 6fe1611d1a15afd1d593cfcba1dbddea53b75faa Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:19:35 -0300 Subject: [PATCH 01/42] docs: start the unified start API design (0.9) --- CHANGELOG.md | 7 +++++++ docs/design/unified-start.md | 4 ++++ 2 files changed, 11 insertions(+) create mode 100644 docs/design/unified-start.md diff --git a/CHANGELOG.md b/CHANGELOG.md index a15604bc..03398efc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## Unreleased +- Design (0.9, in review): a unified start API, `docs/design/unified-start.md`. + OTP's `start`, `start_link` and `start_monitor`, each optionally taking one + `StartOptions` (placement, registered name, startup timeout, per-instance + bounds). Linked and monitored children inherit their parent's placement. + It replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and + `start_linked`. A new TLA+ model, `Start.tla`, has negative controls. + - Tests: coverage follow-ups. `restart_modes_distinguish_normal_shutdown_and_panic` now sends a real `Normal` exit; a per-PR `MCSupervisionTemporary` TLC configuration makes `TemporaryStartsOnce` non-vacuous, with a negative diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md new file mode 100644 index 00000000..475fc87b --- /dev/null +++ b/docs/design/unified-start.md @@ -0,0 +1,4 @@ +# Unified start API (0.9): design contract + +Status: draft, in progress. Design only; the implementation follows after two +independent reviews. From 0f7e3911ac6dc579ab5e1ffaf08cae06e691028a Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:23:45 -0300 Subject: [PATCH 02/42] verification: Start.tla model of the unified start API --- verification/tla/MCStart.cfg | 26 ++ verification/tla/MCStart.tla | 9 + verification/tla/MCStartDeep.cfg | 26 ++ verification/tla/MCStartDeep.tla | 5 + verification/tla/MCStartLive.cfg | 27 ++ verification/tla/MCStartLive.tla | 5 + verification/tla/Start.tla | 513 +++++++++++++++++++++++++++++++ 7 files changed, 611 insertions(+) create mode 100644 verification/tla/MCStart.cfg create mode 100644 verification/tla/MCStart.tla create mode 100644 verification/tla/MCStartDeep.cfg create mode 100644 verification/tla/MCStartDeep.tla create mode 100644 verification/tla/MCStartLive.cfg create mode 100644 verification/tla/MCStartLive.tla create mode 100644 verification/tla/Start.tla diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg new file mode 100644 index 00000000..40e2ca3b --- /dev/null +++ b/verification/tla/MCStart.cfg @@ -0,0 +1,26 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1", "c2"} + Kinds = {"start", "link", "monitor"} + Explicit = {"none", "U"} + ParentPlaces = {"G", "U"} + Cap = 2 + Named = TRUE + TimeoutEnabled = TRUE + ParentCanExit = TRUE + CloseEnabled = TRUE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameReleasedAtDeath +INVARIANT AbortInvisible +INVARIANT AckMeansStarted +INVARIANT FailureAfterDeath +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla new file mode 100644 index 00000000..8c1a56f0 --- /dev/null +++ b/verification/tla/MCStart.tla @@ -0,0 +1,9 @@ +---------------------------- MODULE MCStart ---------------------------- +EXTENDS Start +\* Per-PR safety instance of the unified start model (design +\* docs/design/unified-start.md): two children racing for one name, every +\* start variant, inherited or explicit unmanaged placement, a managed or +\* unmanaged parent that may exit, a closing group with room for two actors, +\* startup timeouts. Every Start safety control fails here on its named +\* invariant. MCStartDeep (weekly) adds every explicit placement. +============================================================================= diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg new file mode 100644 index 00000000..b9157269 --- /dev/null +++ b/verification/tla/MCStartDeep.cfg @@ -0,0 +1,26 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1", "c2"} + Kinds = {"start", "link", "monitor"} + Explicit = {"none", "G", "U", "same"} + ParentPlaces = {"G", "U"} + Cap = 2 + Named = TRUE + TimeoutEnabled = TRUE + ParentCanExit = TRUE + CloseEnabled = TRUE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameReleasedAtDeath +INVARIANT AbortInvisible +INVARIANT AckMeansStarted +INVARIANT FailureAfterDeath +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce diff --git a/verification/tla/MCStartDeep.tla b/verification/tla/MCStartDeep.tla new file mode 100644 index 00000000..7184bfe9 --- /dev/null +++ b/verification/tla/MCStartDeep.tla @@ -0,0 +1,5 @@ +---------------------------- MODULE MCStartDeep ---------------------------- +EXTENDS Start +\* Weekly safety instance: every explicit placement (a group, unmanaged, +\* the parent's via placement_of) for every start variant. +============================================================================= diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg new file mode 100644 index 00000000..b975341d --- /dev/null +++ b/verification/tla/MCStartLive.cfg @@ -0,0 +1,27 @@ +SPECIFICATION LiveSpec +CONSTANTS + Kids = {"c1", "c2"} + Kinds = {"link", "monitor"} + Explicit = {"none"} + ParentPlaces = {"G"} + Cap = 3 + Named = TRUE + TimeoutEnabled = TRUE + ParentCanExit = TRUE + CloseEnabled = FALSE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameReleasedAtDeath +INVARIANT AbortInvisible +INVARIANT AckMeansStarted +INVARIANT FailureAfterDeath +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce +PROPERTY ParentExitReachesChild diff --git a/verification/tla/MCStartLive.tla b/verification/tla/MCStartLive.tla new file mode 100644 index 00000000..d9c6de8a --- /dev/null +++ b/verification/tla/MCStartLive.tla @@ -0,0 +1,5 @@ +---------------------------- MODULE MCStartLive ---------------------------- +EXTENDS Start +\* Per-PR liveness instance: a parent exit, also one held during the +\* child's started(), reaches every linked child. +============================================================================= diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla new file mode 100644 index 00000000..253edab5 --- /dev/null +++ b/verification/tla/Start.tla @@ -0,0 +1,513 @@ +------------------------------- MODULE Start ------------------------------- +EXTENDS Naturals, FiniteSets +(***************************************************************************) +(* Unified start API (0.9; design: docs/design/unified-start.md). *) +(* *) +(* One parent actor P starts children with start, start_link or *) +(* start_monitor, each with optional StartOptions (explicit placement, *) +(* a registered name). A start has two stages: *) +(* stage 1 (in the caller, before the child runs any code): *) +(* Admit placement: explicit, else inherited (link/monitor), else *) +(* unmanaged; a group admits only while open and below Cap; *) +(* Link start_link/start_monitor: the edge is recorded while P is *) +(* running, else the start is refused (NoProc-like); *) +(* Register the name, atomically, or AlreadyStarted; *) +(* Dispatch the child: start returns here (immediate); *) +(* stage 2 (on the child): started() begins and returns, panics or *) +(* stops; signals received meanwhile are held until it returns. *) +(* start_link and start_monitor return after it (the ack): Ok when *) +(* started() returned with no exit pending, otherwise the error after *) +(* the child's death point D, consuming the link notice (trapping *) +(* parent) or the Down. A startup timeout unlinks and demonitors *) +(* first, then kills. *) +(* An aborted stage 1 leaves no trace: no edge, no name, no slot, no *) +(* hook, no notice. Children never trap exits; a parent exit is held *) +(* during started() and applied after it (Normal is ignored). *) +(* *) +(* Steps are atomic; interleavings between steps of stage 1 are what the *) +(* implementation must make invisible (parent death, group closure and a *) +(* competing registration may happen between any two of them). *) +(***************************************************************************) +CONSTANTS + Kids, \* child names + Kinds, \* start variants used: subset of {"start","link","monitor"} + Explicit, \* explicit placements: subset of {"none","G","U","same"} + \* ("same" = StartOptions::placement_of(&ctx)) + ParentPlaces, \* where P runs: subset of {"G","U"} + Cap, \* live actors group G admits (thread_group workers) + Named, \* children may request the one shared name + TimeoutEnabled, \* start_link/start_monitor may time out + ParentCanExit, \* P may exit (with Normal or Panic) + CloseEnabled, \* G may close (system shutdown) + Fault \* "None" or a negative control, see below + +(* Negative controls (each must produce a counterexample): *) +(* NoInherit 0.8: a linked/monitored child of a managed parent is *) +(* unmanaged -> InheritedPlacement *) +(* LinkAfterStarted the link is set after started() began *) +(* -> LinkBeforeStarted *) +(* DoubleRegister check-then-insert name registration -> NameUnique *) +(* LateUnregister the name outlives D -> NameReleasedAtDeath *) +(* AbortLeaksLink a refused start keeps its edge -> AbortInvisible *) +(* EarlyFailAck the error returns before the child's D *) +(* -> FailureAfterDeath *) +(* AckBeforeStarted Ok before started() returned -> AckMeansStarted *) +(* NoMonitorFlush a failed start_monitor also delivers Down *) +(* -> FailureFlushed *) +(* NoLinkFlush a failed start_link also delivers Exit to a trapping *) +(* parent -> FailureFlushed *) +(* KillBeforeUnlink the timeout kills while still linked *) +(* -> TimeoutContained *) +(* LinkCheckLate the link ignores the parent's state *) +(* -> NoEdgeToDeadParent *) +(* AdmitClosed a closed group admits -> AdmittedOnlyOpen *) +(* IgnoreCap a full group admits -> CapacityRespected *) +(* DropHeld a parent exit held during started() is dropped *) +(* (liveness) -> ParentExitReachesChild*) + +None == "none" +Abnormal == {"Panic", "Killed", "NoProc", "Shutdown"} +Reasons == {"Normal"} \cup Abnormal +Phases == {"idle", "begun", "admitted", "linked", "registered", "starting", + "ready", "exiting", "dead", "aborted"} +Results == {"none", "pending", "ok", "failed", "timeout", "closed", + "capacity", "already", "noproc"} + +VARIABLES + pst, \* P: "run", "exiting" (exit begun, before D) or "dead" (past D) + pplace, \* P's placement: "G" (managed) or "U" (unmanaged) + ptrap, \* P traps exits + pwhy, \* P's exit reason + pcause, \* the child whose death notice made P exit, or None + gopen, \* G admits + kind, \* [Kids -> start variant] + expl, \* [Kids -> explicit placement option] + wname, \* [Kids -> requests the name] + phase, \* [Kids -> Phases] + place, \* [Kids -> "G", "U" or None] + admOpen, \* [Kids -> G was open when the child was admitted to it] + edge, \* [Kids -> link edge with P] + mon, \* [Kids -> P's monitor on the child] + linkedAt, \* [Kids -> None, "before" or "after" started() began] + monAt, \* same for the monitor + began, \* [Kids -> started() began] + retd, \* [Kids -> started() returned, panicked or stopped] + killed, \* [Kids -> startup-timeout kill received] + pex, \* [Kids -> parent exit queued at the child, or None] + pexAbn, \* [Kids -> an abnormal parent exit was ever queued] + why, \* [Kids -> exit reason or None] + res, \* [Kids -> what the start call returned] + reg, \* children holding the name + exits, \* Exit messages P's exit_received sees: [c, r] + downs \* Down messages P sees: [c, r] + +vars == <> +pVars == <> +optVars == <> +startVars == <> +obsVars == <> + +Live(c) == phase[c] \in {"admitted", "linked", "registered", "starting", + "ready", "exiting"} +LiveInG == Cardinality({c \in Kids : place[c] = "G" /\ Live(c)}) + + (IF pplace = "G" /\ pst # "dead" THEN 1 ELSE 0) + +\* The contract: explicit placement, else the parent's (link/monitor), +\* else unmanaged. +Expected(c) == + CASE expl[c] = "same" -> pplace + [] expl[c] \in {"G", "U"} -> expl[c] + [] kind[c] \in {"link", "monitor"} -> pplace + [] OTHER -> "U" +Target(c) == + IF expl[c] = None /\ kind[c] \in {"link", "monitor"} /\ Fault = "NoInherit" + THEN "U" ELSE Expected(c) + +TypeOK == + /\ pst \in {"run", "exiting", "dead"} + /\ pplace \in {"G", "U"} + /\ ptrap \in BOOLEAN + /\ pwhy \in Reasons \cup {None} + /\ pcause \in Kids \cup {None} + /\ gopen \in BOOLEAN + /\ kind \in [Kids -> {"start", "link", "monitor", None}] + /\ expl \in [Kids -> {"G", "U", "same", None}] + /\ wname \in [Kids -> BOOLEAN] + /\ phase \in [Kids -> Phases] + /\ place \in [Kids -> {"G", "U", None}] + /\ admOpen \in [Kids -> BOOLEAN] + /\ edge \in [Kids -> BOOLEAN] + /\ mon \in [Kids -> BOOLEAN] + /\ linkedAt \in [Kids -> {None, "before", "after"}] + /\ monAt \in [Kids -> {None, "before", "after"}] + /\ began \in [Kids -> BOOLEAN] + /\ retd \in [Kids -> BOOLEAN] + /\ killed \in [Kids -> BOOLEAN] + /\ pex \in [Kids -> Reasons \cup {None}] + /\ pexAbn \in [Kids -> BOOLEAN] + /\ why \in [Kids -> Reasons \cup {None}] + /\ res \in [Kids -> Results] + /\ reg \subseteq Kids + /\ exits \subseteq [c : Kids, r : Reasons] + /\ downs \subseteq [c : Kids, r : Reasons] + +Init == + /\ pst = "run" + /\ pplace \in ParentPlaces + /\ ptrap \in BOOLEAN + /\ pwhy = None + /\ pcause = None + /\ gopen = TRUE + /\ kind = [c \in Kids |-> None] + /\ expl = [c \in Kids |-> None] + /\ wname = [c \in Kids |-> FALSE] + /\ phase = [c \in Kids |-> "idle"] + /\ place = [c \in Kids |-> None] + /\ admOpen = [c \in Kids |-> FALSE] + /\ edge = [c \in Kids |-> FALSE] + /\ mon = [c \in Kids |-> FALSE] + /\ linkedAt = [c \in Kids |-> None] + /\ monAt = [c \in Kids |-> None] + /\ began = [c \in Kids |-> FALSE] + /\ retd = [c \in Kids |-> FALSE] + /\ killed = [c \in Kids |-> FALSE] + /\ pex = [c \in Kids |-> None] + /\ pexAbn = [c \in Kids |-> FALSE] + /\ why = [c \in Kids |-> None] + /\ res = [c \in Kids |-> "none"] + /\ reg = {} + /\ exits = {} + /\ downs = {} + +----------------------------------------------------------------------------- +(* Stage 1, in the caller. *) + +\* The caller asks for a start: variant, explicit placement, name. +Begin(c) == + /\ phase[c] = "idle" + /\ \E k \in Kinds, e \in Explicit, n \in (IF Named THEN BOOLEAN ELSE {FALSE}): + /\ kind' = [kind EXCEPT ![c] = k] + /\ expl' = [expl EXCEPT ![c] = IF e = "none" THEN None ELSE e] + /\ wname' = [wname EXCEPT ![c] = n] + /\ phase' = [phase EXCEPT ![c] = "begun"] + /\ UNCHANGED <> + +\* A refused start: nothing remains (edge, monitor, name and slot released; +\* no hook ran, no notice is sent). +Abort(c, r) == + /\ phase' = [phase EXCEPT ![c] = "aborted"] + /\ res' = [res EXCEPT ![c] = r] + /\ place' = [place EXCEPT ![c] = None] + /\ edge' = IF Fault = "AbortLeaksLink" THEN edge ELSE [edge EXCEPT ![c] = FALSE] + /\ mon' = [mon EXCEPT ![c] = FALSE] + /\ reg' = reg \ {c} + +Admit(c) == + /\ phase[c] = "begun" + /\ LET t == Target(c) IN + IF t = "U" + THEN /\ phase' = [phase EXCEPT ![c] = "admitted"] + /\ place' = [place EXCEPT ![c] = "U"] + /\ UNCHANGED <> + ELSE IF (gopen \/ Fault = "AdmitClosed") /\ (LiveInG < Cap \/ Fault = "IgnoreCap") + THEN /\ phase' = [phase EXCEPT ![c] = "admitted"] + /\ place' = [place EXCEPT ![c] = "G"] + /\ admOpen' = [admOpen EXCEPT ![c] = gopen] + /\ UNCHANGED <> + ELSE /\ Abort(c, IF ~gopen THEN "closed" ELSE "capacity") + /\ UNCHANGED admOpen + /\ UNCHANGED <> + +\* start_link / start_monitor: the edge is recorded in one transaction with +\* the parent's state; an exiting or dead parent refuses the start. +Link(c) == + /\ phase[c] = "admitted" + /\ IF kind[c] = "start" \/ (kind[c] = "link" /\ Fault = "LinkAfterStarted") + THEN /\ phase' = [phase EXCEPT ![c] = "linked"] + /\ UNCHANGED <> + ELSE IF pst = "run" \/ Fault = "LinkCheckLate" + THEN /\ phase' = [phase EXCEPT ![c] = "linked"] + /\ edge' = [edge EXCEPT ![c] = kind[c] = "link"] + /\ mon' = [mon EXCEPT ![c] = kind[c] = "monitor"] + /\ linkedAt' = IF kind[c] = "link" THEN [linkedAt EXCEPT ![c] = "before"] ELSE linkedAt + /\ monAt' = IF kind[c] = "monitor" THEN [monAt EXCEPT ![c] = "before"] ELSE monAt + /\ UNCHANGED <> + ELSE /\ Abort(c, "noproc") + /\ UNCHANGED <> + /\ UNCHANGED <> + +\* The name is taken atomically, or the start is refused with AlreadyStarted. +Register(c) == + /\ phase[c] = "linked" + /\ IF ~wname[c] + THEN /\ phase' = [phase EXCEPT ![c] = "registered"] + /\ UNCHANGED <> + ELSE IF reg = {} \/ Fault = "DoubleRegister" + THEN /\ phase' = [phase EXCEPT ![c] = "registered"] + /\ reg' = reg \cup {c} + /\ UNCHANGED <> + ELSE Abort(c, "already") + /\ UNCHANGED <> + +\* The child is handed to its executor; start returns Ok here, start_link +\* and start_monitor wait for the ack. +Dispatch(c) == + /\ phase[c] = "registered" + /\ phase' = [phase EXCEPT ![c] = "starting"] + /\ res' = [res EXCEPT ![c] = IF kind[c] = "start" THEN "ok" ELSE "pending"] + /\ UNCHANGED <> + +----------------------------------------------------------------------------- +(* Stage 2, on the child. *) + +Exit(c, r) == + /\ phase' = [phase EXCEPT ![c] = "exiting"] + /\ why' = [why EXCEPT ![c] = r] + +StartedBegin(c) == + /\ phase[c] = "starting" + /\ ~began[c] + /\ ~killed[c] + /\ began' = [began EXCEPT ![c] = TRUE] + /\ UNCHANGED <> + +\* Only under LinkAfterStarted: the link is set once started() is running. +LateLink(c) == + /\ Fault = "LinkAfterStarted" + /\ kind[c] = "link" + /\ began[c] + /\ linkedAt[c] = None + /\ phase[c] \in {"starting", "ready"} + /\ pst = "run" + /\ edge' = [edge EXCEPT ![c] = TRUE] + /\ linkedAt' = [linkedAt EXCEPT ![c] = "after"] + /\ UNCHANGED <> + +\* A kill received before started() began: the child ends Killed. +KilledEarly(c) == + /\ phase[c] = "starting" + /\ ~began[c] + /\ killed[c] + /\ Exit(c, "Killed") + /\ UNCHANGED <> + +\* started() returns, panics or stops; held signals are applied then: a kill +\* wins, then an abnormal parent exit; a Normal parent exit is ignored. +StartedEnd(c) == + /\ phase[c] = "starting" + /\ began[c] + /\ ~retd[c] + /\ retd' = [retd EXCEPT ![c] = TRUE] + /\ \E o \in {"ok", "panic", "stop"}: + IF killed[c] THEN /\ Exit(c, "Killed") /\ UNCHANGED pex + ELSE IF o = "panic" THEN /\ Exit(c, "Panic") /\ UNCHANGED pex + ELSE IF o = "stop" THEN /\ Exit(c, "Normal") /\ UNCHANGED pex + ELSE IF pex[c] \in Abnormal /\ Fault # "DropHeld" + THEN /\ Exit(c, pex[c]) + /\ pex' = [pex EXCEPT ![c] = None] + ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] + /\ pex' = [pex EXCEPT ![c] = None] + /\ UNCHANGED why + /\ UNCHANGED <> + +\* A parent exit reaching a running child. +ApplyParentExit(c) == + /\ phase[c] = "ready" + /\ pex[c] # None + /\ pex' = [pex EXCEPT ![c] = None] + /\ IF pex[c] \in Abnormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* The child's death point D: the name is released, then the link notice +\* and the Down are taken. While the start call still waits (res pending), +\* the call consumes them (a trapping parent sees no Exit, no Down is +\* delivered); a non-trapping parent still exits on an abnormal notice. +ChildDie(c) == + /\ phase[c] = "exiting" + /\ phase' = [phase EXCEPT ![c] = "dead"] + /\ reg' = IF Fault = "LateUnregister" THEN reg ELSE reg \ {c} + /\ LET r == why[c] + pending == res[c] = "pending" + linkTaken == edge[c] /\ pst # "dead" + monTaken == mon[c] /\ pst # "dead" + kills == linkTaken /\ ~ptrap /\ r \in Abnormal /\ pst = "run" + IN /\ exits' = IF linkTaken /\ ptrap /\ (~pending \/ Fault = "NoLinkFlush") + THEN exits \cup {[c |-> c, r |-> r]} ELSE exits + /\ downs' = IF monTaken /\ (~pending \/ Fault = "NoMonitorFlush") + THEN downs \cup {[c |-> c, r |-> r]} ELSE downs + /\ pst' = IF kills THEN "exiting" ELSE pst + /\ pwhy' = IF kills THEN r ELSE pwhy + /\ pcause' = IF kills THEN c ELSE pcause + /\ edge' = [edge EXCEPT ![c] = FALSE] + /\ mon' = [mon EXCEPT ![c] = FALSE] + /\ UNCHANGED <> + +----------------------------------------------------------------------------- +(* The ack of start_link / start_monitor. *) + +AckOk(c) == + /\ kind[c] \in {"link", "monitor"} + /\ res[c] = "pending" + /\ ~killed[c] + /\ phase[c] = "ready" \/ (Fault = "AckBeforeStarted" /\ phase[c] = "starting") + /\ res' = [res EXCEPT ![c] = "ok"] + /\ UNCHANGED <> + +\* The error is returned after the child's D (OTP: after the 'DOWN'). +AckFail(c) == + /\ res[c] = "pending" + /\ ~killed[c] + /\ phase[c] = "dead" \/ (Fault = "EarlyFailAck" /\ phase[c] = "exiting") + /\ res' = [res EXCEPT ![c] = "failed"] + /\ UNCHANGED <> + +\* startup_timeout: unlink and demonitor (fences), then kill. +Timeout(c) == + /\ TimeoutEnabled + /\ res[c] = "pending" + /\ phase[c] = "starting" + /\ ~killed[c] + /\ killed' = [killed EXCEPT ![c] = TRUE] + /\ edge' = IF Fault = "KillBeforeUnlink" THEN edge ELSE [edge EXCEPT ![c] = FALSE] + /\ mon' = IF Fault = "KillBeforeUnlink" THEN mon ELSE [mon EXCEPT ![c] = FALSE] + /\ UNCHANGED <> + +\* Returns Timeout after the child's D or after the bounded wait for it. +AckTimeout(c) == + /\ res[c] = "pending" + /\ killed[c] + /\ res' = [res EXCEPT ![c] = "timeout"] + /\ UNCHANGED <> + +----------------------------------------------------------------------------- +(* The parent and the group. *) + +PExit == + /\ ParentCanExit + /\ pst = "run" + /\ \E r \in {"Normal", "Panic"}: + /\ pst' = "exiting" + /\ pwhy' = r + /\ UNCHANGED <> + +\* P's D: each linked child gets a parent exit (held if its started() has +\* not returned); P's monitors are cancelled. +PDie == + /\ pst = "exiting" + /\ pst' = "dead" + /\ pex' = [c \in Kids |-> IF edge[c] THEN pwhy ELSE pex[c]] + /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (edge[c] /\ pwhy \in Abnormal)] + /\ edge' = [c \in Kids |-> FALSE] + /\ mon' = [c \in Kids |-> FALSE] + /\ UNCHANGED <> + +GClose == + /\ CloseEnabled + /\ gopen + /\ gopen' = FALSE + /\ UNCHANGED <> + +ChildStep(c) == + \/ Begin(c) \/ Admit(c) \/ Link(c) \/ Register(c) \/ Dispatch(c) + \/ StartedBegin(c) \/ LateLink(c) \/ KilledEarly(c) \/ StartedEnd(c) + \/ ApplyParentExit(c) \/ ChildDie(c) + \/ AckOk(c) \/ AckFail(c) \/ Timeout(c) \/ AckTimeout(c) + +Next == (\E c \in Kids : ChildStep(c)) \/ PExit \/ PDie \/ GClose + +\* Liveness assumes the child's own steps and P's death complete. +Progress(c) == + Register(c) \/ Dispatch(c) \/ StartedBegin(c) \/ KilledEarly(c) + \/ StartedEnd(c) \/ ApplyParentExit(c) \/ ChildDie(c) +Fairness == /\ \A c \in Kids : WF_vars(Progress(c)) /\ WF_vars(Link(c)) /\ WF_vars(Admit(c)) + /\ WF_vars(PDie) + +Spec == Init /\ [][Next]_vars +LiveSpec == Spec /\ Fairness + +----------------------------------------------------------------------------- +(* Safety. *) + +\* Explicit placement wins; a linked or monitored child takes its parent's +\* placement; a plain start is unmanaged. +InheritedPlacement == \A c \in Kids : place[c] # None => place[c] = Expected(c) + +\* The link or monitor exists before started() begins. +LinkBeforeStarted == + \A c \in Kids : + /\ (began[c] /\ kind[c] = "link") => linkedAt[c] = "before" + /\ (began[c] /\ kind[c] = "monitor") => monAt[c] = "before" + +NameUnique == Cardinality(reg) <= 1 + +\* The name is free once the holder is past D (and before any notice, +\* Down or failure result is observable). +NameReleasedAtDeath == \A c \in reg : phase[c] \notin {"dead", "aborted"} + +\* A refused start leaves no trace. +AbortInvisible == + \A c \in Kids : phase[c] = "aborted" => + /\ ~edge[c] /\ ~mon[c] /\ c \notin reg /\ ~began[c] /\ place[c] = None + /\ \A e \in exits \cup downs : e.c # c + /\ pcause # c + +\* Ok from start_link/start_monitor means started() returned. +AckMeansStarted == + \A c \in Kids : (res[c] = "ok" /\ kind[c] \in {"link", "monitor"}) => retd[c] + +\* A startup failure is returned after the child's D. +FailureAfterDeath == \A c \in Kids : res[c] = "failed" => phase[c] = "dead" + +\* A failed or timed-out start_monitor delivers no Down; a failed or +\* timed-out start_link delivers no Exit to a trapping parent. +FailureFlushed == + \A c \in Kids : res[c] \in {"failed", "timeout"} => + /\ kind[c] = "monitor" => \A d \in downs : d.c # c + /\ (kind[c] = "link" /\ ptrap) => \A e \in exits : e.c # c + +\* A startup-timeout kill never reaches the parent. +TimeoutContained == + \A c \in Kids : killed[c] => + /\ pcause # c + /\ \A e \in exits \cup downs : e.c # c + +\* No link or monitor edge survives the parent's D. +NoEdgeToDeadParent == \A c \in Kids : (edge[c] \/ mon[c]) => pst # "dead" + +AdmittedOnlyOpen == \A c \in Kids : place[c] = "G" => admOpen[c] + +CapacityRespected == LiveInG <= Cap + +NoticeOnce == + \A c \in Kids : + /\ Cardinality({e \in exits : e.c = c}) <= 1 + /\ Cardinality({d \in downs : d.c = c}) <= 1 + +Safety == + /\ InheritedPlacement /\ LinkBeforeStarted /\ NameUnique + /\ NameReleasedAtDeath /\ AbortInvisible /\ AckMeansStarted + /\ FailureAfterDeath /\ FailureFlushed /\ TimeoutContained + /\ NoEdgeToDeadParent /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce + +(* Liveness: a child linked to a parent that dies abnormally, also during *) +(* the child's started(), dies. *) +ParentExitReachesChild == + \A c \in Kids : pexAbn[c] ~> phase[c] \in {"dead", "aborted"} +============================================================================= From cf66b03f62447254206155b488a4b43cb1f977ef Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:28:41 -0300 Subject: [PATCH 03/42] verification: Start.tla models fallible init, continuations and spawn --- verification/tla/MCStart.cfg | 2 + verification/tla/MCStartDeep.cfg | 4 +- verification/tla/MCStartLive.cfg | 2 + verification/tla/Start.tla | 115 ++++++++++++++++++++++--------- 4 files changed, 91 insertions(+), 32 deletions(-) diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 40e2ca3b..986028e8 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -8,6 +8,7 @@ CONSTANTS Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE + MaxMsgs = 1 CloseEnabled = TRUE Fault = "None" INVARIANT TypeOK @@ -24,3 +25,4 @@ INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce +INVARIANT ContinueFirst diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index b9157269..d3da818e 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -1,13 +1,14 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2"} - Kinds = {"start", "link", "monitor"} + Kinds = {"spawn", "start", "link", "monitor"} Explicit = {"none", "G", "U", "same"} ParentPlaces = {"G", "U"} Cap = 2 Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE + MaxMsgs = 1 CloseEnabled = TRUE Fault = "None" INVARIANT TypeOK @@ -24,3 +25,4 @@ INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce +INVARIANT ContinueFirst diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index b975341d..8afba44f 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -8,6 +8,7 @@ CONSTANTS Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE + MaxMsgs = 0 CloseEnabled = FALSE Fault = "None" INVARIANT TypeOK @@ -24,4 +25,5 @@ INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce +INVARIANT ContinueFirst PROPERTY ParentExitReachesChild diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 253edab5..40a44199 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -3,7 +3,7 @@ EXTENDS Naturals, FiniteSets (***************************************************************************) (* Unified start API (0.9; design: docs/design/unified-start.md). *) (* *) -(* One parent actor P starts children with start, start_link or *) +(* One parent actor P starts children with spawn, start, start_link or *) (* start_monitor, each with optional StartOptions (explicit placement, *) (* a registered name). A start has two stages: *) (* stage 1 (in the caller, before the child runs any code): *) @@ -12,11 +12,13 @@ EXTENDS Naturals, FiniteSets (* Link start_link/start_monitor: the edge is recorded while P is *) (* running, else the start is refused (NoProc-like); *) (* Register the name, atomically, or AlreadyStarted; *) -(* Dispatch the child: start returns here (immediate); *) -(* stage 2 (on the child): started() begins and returns, panics or *) -(* stops; signals received meanwhile are held until it returns. *) -(* start_link and start_monitor return after it (the ack): Ok when *) -(* started() returned with no exit pending, otherwise the error after *) +(* Dispatch the child: spawn returns here (no ack); *) +(* stage 2 (on the child): started() returns Init (ready, or ready with *) +(* a continuation), returns an error or ignore (exit Normal), or *) +(* panics or stops abnormally; signals received meanwhile are held *) +(* until it returns. A continuation runs before any mailbox message. *) +(* start, start_link and start_monitor return after started() (the *) +(* ack): Ok when it returned ready with no exit pending, else the error after *) (* the child's death point D, consuming the link notice (trapping *) (* parent) or the Down. A startup timeout unlinks and demonitors *) (* first, then kills. *) @@ -30,7 +32,7 @@ EXTENDS Naturals, FiniteSets (***************************************************************************) CONSTANTS Kids, \* child names - Kinds, \* start variants used: subset of {"start","link","monitor"} + Kinds, \* variants used: subset of {"spawn","start","link","monitor"} Explicit, \* explicit placements: subset of {"none","G","U","same"} \* ("same" = StartOptions::placement_of(&ctx)) ParentPlaces, \* where P runs: subset of {"G","U"} @@ -39,6 +41,7 @@ CONSTANTS TimeoutEnabled, \* start_link/start_monitor may time out ParentCanExit, \* P may exit (with Normal or Panic) CloseEnabled, \* G may close (system shutdown) + MaxMsgs, \* messages others may send to each child Fault \* "None" or a negative control, see below (* Negative controls (each must produce a counterexample): *) @@ -62,6 +65,8 @@ CONSTANTS (* -> NoEdgeToDeadParent *) (* AdmitClosed a closed group admits -> AdmittedOnlyOpen *) (* IgnoreCap a full group admits -> CapacityRespected *) +(* ContinueQueued the continuation is queued behind earlier messages *) +(* -> ContinueFirst *) (* DropHeld a parent exit held during started() is dropped *) (* (liveness) -> ParentExitReachesChild*) @@ -94,7 +99,10 @@ VARIABLES retd, \* [Kids -> started() returned, panicked or stopped] killed, \* [Kids -> startup-timeout kill received] pex, \* [Kids -> parent exit queued at the child, or None] - pexAbn, \* [Kids -> an abnormal parent exit was ever queued] + pexAbn, \* [Kids -> a non-Normal parent exit was ever queued] + cont, \* [Kids -> a continuation is pending] + msgs, \* [Kids -> messages sent to the child by others] + early, \* [Kids -> a message was handled before the continuation] why, \* [Kids -> exit reason or None] res, \* [Kids -> what the start call returned] reg, \* children holding the name @@ -103,10 +111,10 @@ VARIABLES vars == <> + pex, pexAbn, cont, msgs, early, why, res, reg, exits, downs>> pVars == <> optVars == <> -startVars == <> +startVars == <> obsVars == <> Live(c) == phase[c] \in {"admitted", "linked", "registered", "starting", @@ -132,7 +140,7 @@ TypeOK == /\ pwhy \in Reasons \cup {None} /\ pcause \in Kids \cup {None} /\ gopen \in BOOLEAN - /\ kind \in [Kids -> {"start", "link", "monitor", None}] + /\ kind \in [Kids -> {"spawn", "start", "link", "monitor", None}] /\ expl \in [Kids -> {"G", "U", "same", None}] /\ wname \in [Kids -> BOOLEAN] /\ phase \in [Kids -> Phases] @@ -147,6 +155,9 @@ TypeOK == /\ killed \in [Kids -> BOOLEAN] /\ pex \in [Kids -> Reasons \cup {None}] /\ pexAbn \in [Kids -> BOOLEAN] + /\ cont \in [Kids -> BOOLEAN] + /\ msgs \in [Kids -> 0..MaxMsgs] + /\ early \in [Kids -> BOOLEAN] /\ why \in [Kids -> Reasons \cup {None}] /\ res \in [Kids -> Results] /\ reg \subseteq Kids @@ -175,6 +186,9 @@ Init == /\ killed = [c \in Kids |-> FALSE] /\ pex = [c \in Kids |-> None] /\ pexAbn = [c \in Kids |-> FALSE] + /\ cont = [c \in Kids |-> FALSE] + /\ msgs = [c \in Kids |-> 0] + /\ early = [c \in Kids |-> FALSE] /\ why = [c \in Kids |-> None] /\ res = [c \in Kids |-> "none"] /\ reg = {} @@ -225,7 +239,7 @@ Admit(c) == \* the parent's state; an exiting or dead parent refuses the start. Link(c) == /\ phase[c] = "admitted" - /\ IF kind[c] = "start" \/ (kind[c] = "link" /\ Fault = "LinkAfterStarted") + /\ IF kind[c] \in {"spawn", "start"} \/ (kind[c] = "link" /\ Fault = "LinkAfterStarted") THEN /\ phase' = [phase EXCEPT ![c] = "linked"] /\ UNCHANGED <> ELSE IF pst = "run" \/ Fault = "LinkCheckLate" @@ -252,12 +266,12 @@ Register(c) == ELSE Abort(c, "already") /\ UNCHANGED <> -\* The child is handed to its executor; start returns Ok here, start_link -\* and start_monitor wait for the ack. +\* The child is handed to its executor; spawn returns Ok here, start, +\* start_link and start_monitor wait for the ack. Dispatch(c) == /\ phase[c] = "registered" /\ phase' = [phase EXCEPT ![c] = "starting"] - /\ res' = [res EXCEPT ![c] = IF kind[c] = "start" THEN "ok" ELSE "pending"] + /\ res' = [res EXCEPT ![c] = IF kind[c] = "spawn" THEN "ok" ELSE "pending"] /\ UNCHANGED <> @@ -274,7 +288,8 @@ StartedBegin(c) == /\ ~killed[c] /\ began' = [began EXCEPT ![c] = TRUE] /\ UNCHANGED <> + monAt, retd, killed, pex, pexAbn, why, cont, msgs, early, + res, obsVars>> \* Only under LinkAfterStarted: the link is set once started() is running. LateLink(c) == @@ -296,27 +311,58 @@ KilledEarly(c) == /\ killed[c] /\ Exit(c, "Killed") /\ UNCHANGED <> + began, retd, killed, pex, pexAbn, cont, msgs, early, res, obsVars>> -\* started() returns, panics or stops; held signals are applied then: a kill -\* wins, then an abnormal parent exit; a Normal parent exit is ignored. +\* started() returns Init::ready, Init::continue_with, an error or ignore +\* (the child exits Normal), or panics (or Init::stop with a non-Normal +\* reason). Held signals apply when it returns: a kill wins, then a +\* non-Normal parent exit; a Normal parent exit is ignored. StartedEnd(c) == /\ phase[c] = "starting" /\ began[c] /\ ~retd[c] /\ retd' = [retd EXCEPT ![c] = TRUE] - /\ \E o \in {"ok", "panic", "stop"}: - IF killed[c] THEN /\ Exit(c, "Killed") /\ UNCHANGED pex - ELSE IF o = "panic" THEN /\ Exit(c, "Panic") /\ UNCHANGED pex - ELSE IF o = "stop" THEN /\ Exit(c, "Normal") /\ UNCHANGED pex + /\ \E o \in {"ready", "continue", "panic", "error"}: + IF killed[c] THEN /\ Exit(c, "Killed") /\ UNCHANGED <> + ELSE IF o = "panic" THEN /\ Exit(c, "Panic") /\ UNCHANGED <> + ELSE IF o = "error" THEN /\ Exit(c, "Normal") /\ UNCHANGED <> ELSE IF pex[c] \in Abnormal /\ Fault # "DropHeld" THEN /\ Exit(c, pex[c]) /\ pex' = [pex EXCEPT ![c] = None] + /\ UNCHANGED cont ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] /\ pex' = [pex EXCEPT ![c] = None] + /\ cont' = [cont EXCEPT ![c] = o = "continue"] /\ UNCHANGED why /\ UNCHANGED <> + began, killed, pexAbn, msgs, early, res, obsVars>> + +\* Others (a peer that found the name, the caller after the ack) send. +Send(c) == + /\ phase[c] \in {"starting", "ready"} + /\ msgs[c] < MaxMsgs + /\ msgs' = [msgs EXCEPT ![c] = msgs[c] + 1] + /\ UNCHANGED <> + +\* The continuation runs first; a mailbox message only after it. +RunContinue(c) == + /\ phase[c] = "ready" + /\ cont[c] + /\ cont' = [cont EXCEPT ![c] = FALSE] + /\ UNCHANGED <> + +Handle(c) == + /\ phase[c] = "ready" + /\ msgs[c] > 0 + /\ ~cont[c] \/ Fault = "ContinueQueued" + /\ msgs' = [msgs EXCEPT ![c] = msgs[c] - 1] + /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] + /\ UNCHANGED <> \* A parent exit reaching a running child. ApplyParentExit(c) == @@ -325,7 +371,7 @@ ApplyParentExit(c) == /\ pex' = [pex EXCEPT ![c] = None] /\ IF pex[c] \in Abnormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> /\ UNCHANGED <> + began, retd, killed, pexAbn, cont, msgs, early, res, obsVars>> \* The child's death point D: the name is released, then the link notice \* and the Down are taken. While the start call still waits (res pending), @@ -353,10 +399,10 @@ ChildDie(c) == monAt, startVars, res>> ----------------------------------------------------------------------------- -(* The ack of start_link / start_monitor. *) +(* The ack of start, start_link and start_monitor. *) AckOk(c) == - /\ kind[c] \in {"link", "monitor"} + /\ kind[c] \in {"start", "link", "monitor"} /\ res[c] = "pending" /\ ~killed[c] /\ phase[c] = "ready" \/ (Fault = "AckBeforeStarted" /\ phase[c] = "starting") @@ -383,7 +429,8 @@ Timeout(c) == /\ edge' = IF Fault = "KillBeforeUnlink" THEN edge ELSE [edge EXCEPT ![c] = FALSE] /\ mon' = IF Fault = "KillBeforeUnlink" THEN mon ELSE [mon EXCEPT ![c] = FALSE] /\ UNCHANGED <> + began, retd, pex, pexAbn, why, cont, msgs, early, res, + obsVars>> \* Returns Timeout after the child's D or after the bounded wait for it. AckTimeout(c) == @@ -415,7 +462,8 @@ PDie == /\ edge' = [c \in Kids |-> FALSE] /\ mon' = [c \in Kids |-> FALSE] /\ UNCHANGED <> + admOpen, linkedAt, monAt, began, retd, killed, why, cont, + msgs, early, res, obsVars>> GClose == /\ CloseEnabled @@ -428,6 +476,7 @@ ChildStep(c) == \/ Begin(c) \/ Admit(c) \/ Link(c) \/ Register(c) \/ Dispatch(c) \/ StartedBegin(c) \/ LateLink(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ ApplyParentExit(c) \/ ChildDie(c) + \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ AckOk(c) \/ AckFail(c) \/ Timeout(c) \/ AckTimeout(c) Next == (\E c \in Kids : ChildStep(c)) \/ PExit \/ PDie \/ GClose @@ -468,9 +517,12 @@ AbortInvisible == /\ \A e \in exits \cup downs : e.c # c /\ pcause # c -\* Ok from start_link/start_monitor means started() returned. +\* Ok from start, start_link or start_monitor means started() returned. AckMeansStarted == - \A c \in Kids : (res[c] = "ok" /\ kind[c] \in {"link", "monitor"}) => retd[c] + \A c \in Kids : (res[c] = "ok" /\ kind[c] \in {"start", "link", "monitor"}) => retd[c] + +\* No mailbox message is handled before the continuation. +ContinueFirst == \A c \in Kids : ~early[c] \* A startup failure is returned after the child's D. FailureAfterDeath == \A c \in Kids : res[c] = "failed" => phase[c] = "dead" @@ -505,6 +557,7 @@ Safety == /\ NameReleasedAtDeath /\ AbortInvisible /\ AckMeansStarted /\ FailureAfterDeath /\ FailureFlushed /\ TimeoutContained /\ NoEdgeToDeadParent /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce + /\ ContinueFirst (* Liveness: a child linked to a parent that dies abnormally, also during *) (* the child's started(), dies. *) From 13dad2a268e4edb19d056403714dee447c902082 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:30:46 -0300 Subject: [PATCH 04/42] verification: split the continuation check into MCStartContinue --- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStart.tla | 5 +++-- verification/tla/MCStartContinue.cfg | 28 ++++++++++++++++++++++++++++ verification/tla/MCStartContinue.tla | 6 ++++++ 4 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 verification/tla/MCStartContinue.cfg create mode 100644 verification/tla/MCStartContinue.tla diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 986028e8..001b8427 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -8,7 +8,7 @@ CONSTANTS Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 1 + MaxMsgs = 0 CloseEnabled = TRUE Fault = "None" INVARIANT TypeOK diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index 8c1a56f0..c55788e7 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -4,6 +4,7 @@ EXTENDS Start \* docs/design/unified-start.md): two children racing for one name, every \* start variant, inherited or explicit unmanaged placement, a managed or \* unmanaged parent that may exit, a closing group with room for two actors, -\* startup timeouts. Every Start safety control fails here on its named -\* invariant. MCStartDeep (weekly) adds every explicit placement. +\* startup timeouts. Every Start safety control except ContinueQueued +\* (MCStartContinue) fails here on its named invariant. MCStartDeep (weekly) +\* adds every explicit placement, spawn, and messages from others. ============================================================================= diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg new file mode 100644 index 00000000..ba7a6df9 --- /dev/null +++ b/verification/tla/MCStartContinue.cfg @@ -0,0 +1,28 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1"} + Kinds = {"spawn", "start", "link", "monitor"} + Explicit = {"none"} + ParentPlaces = {"G", "U"} + Cap = 2 + Named = FALSE + TimeoutEnabled = TRUE + ParentCanExit = TRUE + MaxMsgs = 2 + CloseEnabled = TRUE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameReleasedAtDeath +INVARIANT AbortInvisible +INVARIANT AckMeansStarted +INVARIANT FailureAfterDeath +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce +INVARIANT ContinueFirst diff --git a/verification/tla/MCStartContinue.tla b/verification/tla/MCStartContinue.tla new file mode 100644 index 00000000..8aca97db --- /dev/null +++ b/verification/tla/MCStartContinue.tla @@ -0,0 +1,6 @@ +---------------------------- MODULE MCStartContinue ---------------------------- +EXTENDS Start +\* Per-PR instance for the continuation (Init::continue_with): one child of +\* every variant, two messages from others racing started(), the parent's +\* exit and a startup timeout. ContinueFirst and its control run here. +============================================================================= From 72aa302e25994a7ab798ed9dda262f641e730e1a Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:31:40 -0300 Subject: [PATCH 05/42] verification: Start negative controls in negative_checks.py --- verification/tla/negative_checks.py | 30 +++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index d439866b..7e0f07ef 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -241,6 +241,35 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCDeferredJobsEmits", "NoEmitCredit", "EmitCreditsBounded"), ] +# Unified start (docs/design/unified-start.md): each mutation of the start +# protocol against the invariant it breaks. +START_CONTROLS = [ + ("MCStart", "NoInherit", "InheritedPlacement"), + ("MCStart", "LinkAfterStarted", "LinkBeforeStarted"), + ("MCStart", "DoubleRegister", "NameUnique"), + ("MCStart", "LateUnregister", "NameReleasedAtDeath"), + ("MCStart", "AbortLeaksLink", "AbortInvisible"), + ("MCStart", "EarlyFailAck", "FailureAfterDeath"), + ("MCStart", "AckBeforeStarted", "AckMeansStarted"), + ("MCStart", "NoMonitorFlush", "FailureFlushed"), + ("MCStart", "NoLinkFlush", "FailureFlushed"), + ("MCStart", "KillBeforeUnlink", "TimeoutContained"), + ("MCStart", "LinkCheckLate", "NoEdgeToDeadParent"), + ("MCStart", "AdmitClosed", "AdmittedOnlyOpen"), + ("MCStart", "IgnoreCap", "CapacityRespected"), + ("MCStartContinue", "ContinueQueued", "ContinueFirst"), +] +START_TEMPORAL_CONTROLS = [ + ("DropHeld", "ParentExitReachesChild"), +] + + +def start_controls(tlc): + for module, fault, invariant in START_CONTROLS: + named_fault(module, fault, invariant, tlc) + for fault, prop in START_TEMPORAL_CONTROLS: + named_temporal("MCStartLive", fault, prop, tlc) + def main(): parser = argparse.ArgumentParser(description=__doc__) @@ -307,6 +336,7 @@ def fail_instead_of_retry(events): named_fault(module, fault, invariant, args.tlc) for fault, prop in LIFECYCLE_TEMPORAL_CONTROLS: named_temporal("MCLifecycleLive", fault, prop, args.tlc, LIFECYCLE_SMALL) + start_controls(args.tlc) if __name__ == "__main__": From 296c347d3092e53dcecc4a617f8649a877c7a6a3 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:39:44 -0300 Subject: [PATCH 06/42] verification: Start.tla models abandoned start calls --- .github/workflows/verification.yml | 5 ++- verification/tla/MCStart.cfg | 1 + verification/tla/MCStartContinue.cfg | 1 + verification/tla/MCStartDeep.cfg | 1 + verification/tla/MCStartLive.cfg | 1 + verification/tla/Start.tla | 61 ++++++++++++++++++++++------ verification/tla/negative_checks.py | 1 + 7 files changed, 57 insertions(+), 14 deletions(-) diff --git a/.github/workflows/verification.yml b/.github/workflows/verification.yml index f2d1512d..27e0d137 100644 --- a/.github/workflows/verification.yml +++ b/.github/workflows/verification.yml @@ -36,8 +36,10 @@ jobs: select: --only 'MCLifecycle*' - leg: deferred select: --only 'MCDeferred*' + - leg: start + select: --only 'MCStart*' - leg: other - select: --exclude 'MCSupervision,MCLifecycle*,MCDeferred*' + select: --exclude 'MCSupervision,MCLifecycle*,MCDeferred*,MCStart*' steps: - uses: actions/checkout@v4 - uses: DeterminateSystems/nix-installer-action@v16 @@ -112,6 +114,7 @@ jobs: - MCLifecycleWideDeep - MCLifecycleLiveDeep - MCLifecycleDeep + - MCStartDeep steps: - uses: actions/checkout@v4 - uses: DeterminateSystems/nix-installer-action@v16 diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 001b8427..ecfb70bf 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -26,3 +26,4 @@ INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce INVARIANT ContinueFirst +INVARIANT AbandonedNoticeDelivered diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg index ba7a6df9..9943ed24 100644 --- a/verification/tla/MCStartContinue.cfg +++ b/verification/tla/MCStartContinue.cfg @@ -26,3 +26,4 @@ INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce INVARIANT ContinueFirst +INVARIANT AbandonedNoticeDelivered diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index d3da818e..1dbacc4c 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -26,3 +26,4 @@ INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce INVARIANT ContinueFirst +INVARIANT AbandonedNoticeDelivered diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index 8afba44f..c53d4619 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -26,4 +26,5 @@ INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce INVARIANT ContinueFirst +INVARIANT AbandonedNoticeDelivered PROPERTY ParentExitReachesChild diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 40a44199..502dddca 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -18,7 +18,8 @@ EXTENDS Naturals, FiniteSets (* panics or stops abnormally; signals received meanwhile are held *) (* until it returns. A continuation runs before any mailbox message. *) (* start, start_link and start_monitor return after started() (the *) -(* ack): Ok when it returned ready with no exit pending, else the error after *) +(* ack): Ok when it returned ready with no exit pending, else the *) +(* error after *) (* the child's death point D, consuming the link notice (trapping *) (* parent) or the Down. A startup timeout unlinks and demonitors *) (* first, then kills. *) @@ -67,6 +68,8 @@ CONSTANTS (* IgnoreCap a full group admits -> CapacityRespected *) (* ContinueQueued the continuation is queued behind earlier messages *) (* -> ContinueFirst *) +(* AbandonDropsNotice a dropped start future loses the diverted notice *) +(* -> AbandonedNoticeDelivered *) (* DropHeld a parent exit held during started() is dropped *) (* (liveness) -> ParentExitReachesChild*) @@ -76,7 +79,7 @@ Reasons == {"Normal"} \cup Abnormal Phases == {"idle", "begun", "admitted", "linked", "registered", "starting", "ready", "exiting", "dead", "aborted"} Results == {"none", "pending", "ok", "failed", "timeout", "closed", - "capacity", "already", "noproc"} + "capacity", "already", "noproc", "abandoned"} VARIABLES pst, \* P: "run", "exiting" (exit begun, before D) or "dead" (past D) @@ -103,6 +106,8 @@ VARIABLES cont, \* [Kids -> a continuation is pending] msgs, \* [Kids -> messages sent to the child by others] early, \* [Kids -> a message was handled before the continuation] + div, \* [Kids -> notice diverted to the waiting start call, or None] + divEver, \* [Kids -> a notice was ever diverted] why, \* [Kids -> exit reason or None] res, \* [Kids -> what the start call returned] reg, \* children holding the name @@ -111,11 +116,11 @@ VARIABLES vars == <> + pex, pexAbn, cont, msgs, early, div, divEver, why, res, reg, exits, downs>> pVars == <> optVars == <> startVars == <> -obsVars == <> +obsVars == <> Live(c) == phase[c] \in {"admitted", "linked", "registered", "starting", "ready", "exiting"} @@ -163,6 +168,8 @@ TypeOK == /\ reg \subseteq Kids /\ exits \subseteq [c : Kids, r : Reasons] /\ downs \subseteq [c : Kids, r : Reasons] + /\ div \in [Kids -> Reasons \cup {None}] + /\ divEver \in [Kids -> BOOLEAN] Init == /\ pst = "run" @@ -194,6 +201,8 @@ Init == /\ reg = {} /\ exits = {} /\ downs = {} + /\ div = [c \in Kids |-> None] + /\ divEver = [c \in Kids |-> FALSE] ----------------------------------------------------------------------------- (* Stage 1, in the caller. *) @@ -233,7 +242,7 @@ Admit(c) == /\ UNCHANGED <> ELSE /\ Abort(c, IF ~gopen THEN "closed" ELSE "capacity") /\ UNCHANGED admOpen - /\ UNCHANGED <> + /\ UNCHANGED <> \* start_link / start_monitor: the edge is recorded in one transaction with \* the parent's state; an exiting or dead parent refuses the start. @@ -251,7 +260,7 @@ Link(c) == /\ UNCHANGED <> ELSE /\ Abort(c, "noproc") /\ UNCHANGED <> - /\ UNCHANGED <> + /\ UNCHANGED <> \* The name is taken atomically, or the start is refused with AlreadyStarted. Register(c) == @@ -264,7 +273,7 @@ Register(c) == /\ reg' = reg \cup {c} /\ UNCHANGED <> ELSE Abort(c, "already") - /\ UNCHANGED <> + /\ UNCHANGED <> \* The child is handed to its executor; spawn returns Ok here, start, \* start_link and start_monitor wait for the ack. @@ -375,8 +384,9 @@ ApplyParentExit(c) == \* The child's death point D: the name is released, then the link notice \* and the Down are taken. While the start call still waits (res pending), -\* the call consumes them (a trapping parent sees no Exit, no Down is -\* delivered); a non-trapping parent still exits on an abnormal notice. +\* they are diverted to it (a trapping parent's Exit, the Down) and the +\* error consumes them; a non-trapping parent still exits on a non-Normal +\* notice. ChildDie(c) == /\ phase[c] = "exiting" /\ phase' = [phase EXCEPT ![c] = "dead"] @@ -390,6 +400,10 @@ ChildDie(c) == THEN exits \cup {[c |-> c, r |-> r]} ELSE exits /\ downs' = IF monTaken /\ (~pending \/ Fault = "NoMonitorFlush") THEN downs \cup {[c |-> c, r |-> r]} ELSE downs + /\ div' = IF pending /\ ((linkTaken /\ ptrap) \/ monTaken) + THEN [div EXCEPT ![c] = r] ELSE div + /\ divEver' = IF pending /\ ((linkTaken /\ ptrap) \/ monTaken) + THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver /\ pst' = IF kills THEN "exiting" ELSE pst /\ pwhy' = IF kills THEN r ELSE pwhy /\ pcause' = IF kills THEN c ELSE pcause @@ -416,8 +430,9 @@ AckFail(c) == /\ ~killed[c] /\ phase[c] = "dead" \/ (Fault = "EarlyFailAck" /\ phase[c] = "exiting") /\ res' = [res EXCEPT ![c] = "failed"] + /\ div' = [div EXCEPT ![c] = None] /\ UNCHANGED <> + monAt, startVars, reg, exits, downs, divEver>> \* startup_timeout: unlink and demonitor (fences), then kill. Timeout(c) == @@ -437,8 +452,23 @@ AckTimeout(c) == /\ res[c] = "pending" /\ killed[c] /\ res' = [res EXCEPT ![c] = "timeout"] + /\ div' = [div EXCEPT ![c] = None] /\ UNCHANGED <> + monAt, startVars, reg, exits, downs, divEver>> + +\* The start future is dropped before it resolves: the child keeps running +\* and stays linked or monitored; a notice already diverted goes to P as if +\* it had never been diverted. +Abandon(c) == + /\ res[c] = "pending" + /\ res' = [res EXCEPT ![c] = "abandoned"] + /\ div' = [div EXCEPT ![c] = None] + /\ LET r == div[c] + give == r # None /\ Fault # "AbandonDropsNotice" + IN /\ exits' = IF give /\ kind[c] = "link" THEN exits \cup {[c |-> c, r |-> r]} ELSE exits + /\ downs' = IF give /\ kind[c] = "monitor" THEN downs \cup {[c |-> c, r |-> r]} ELSE downs + /\ UNCHANGED <> ----------------------------------------------------------------------------- (* The parent and the group. *) @@ -477,7 +507,7 @@ ChildStep(c) == \/ StartedBegin(c) \/ LateLink(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ ApplyParentExit(c) \/ ChildDie(c) \/ Send(c) \/ RunContinue(c) \/ Handle(c) - \/ AckOk(c) \/ AckFail(c) \/ Timeout(c) \/ AckTimeout(c) + \/ AckOk(c) \/ AckFail(c) \/ Timeout(c) \/ AckTimeout(c) \/ Abandon(c) Next == (\E c \in Kids : ChildStep(c)) \/ PExit \/ PDie \/ GClose @@ -521,6 +551,11 @@ AbortInvisible == AckMeansStarted == \A c \in Kids : (res[c] = "ok" /\ kind[c] \in {"start", "link", "monitor"}) => retd[c] +\* A notice diverted to a start call that was then dropped reaches P. +AbandonedNoticeDelivered == + \A c \in Kids : (res[c] = "abandoned" /\ divEver[c]) => + \E e \in exits \cup downs : e.c = c + \* No mailbox message is handled before the continuation. ContinueFirst == \A c \in Kids : ~early[c] @@ -557,7 +592,7 @@ Safety == /\ NameReleasedAtDeath /\ AbortInvisible /\ AckMeansStarted /\ FailureAfterDeath /\ FailureFlushed /\ TimeoutContained /\ NoEdgeToDeadParent /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce - /\ ContinueFirst + /\ ContinueFirst /\ AbandonedNoticeDelivered (* Liveness: a child linked to a parent that dies abnormally, also during *) (* the child's started(), dies. *) diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 7e0f07ef..c4052e29 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -257,6 +257,7 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStart", "LinkCheckLate", "NoEdgeToDeadParent"), ("MCStart", "AdmitClosed", "AdmittedOnlyOpen"), ("MCStart", "IgnoreCap", "CapacityRespected"), + ("MCStart", "AbandonDropsNotice", "AbandonedNoticeDelivered"), ("MCStartContinue", "ContinueQueued", "ContinueFirst"), ] START_TEMPORAL_CONTROLS = [ From ce136f512e9c4debbfa442444067e606898c31b4 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:42:18 -0300 Subject: [PATCH 07/42] verification: MCStart checks default placements only (explicit ones are in MCStartDeep) --- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStart.tla | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index ecfb70bf..88a675fb 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -2,7 +2,7 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2"} Kinds = {"start", "link", "monitor"} - Explicit = {"none", "U"} + Explicit = {"none"} ParentPlaces = {"G", "U"} Cap = 2 Named = TRUE diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index c55788e7..b3f5bac1 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -2,7 +2,7 @@ EXTENDS Start \* Per-PR safety instance of the unified start model (design \* docs/design/unified-start.md): two children racing for one name, every -\* start variant, inherited or explicit unmanaged placement, a managed or +\* start variant with its default (inherited or unmanaged) placement, a managed or \* unmanaged parent that may exit, a closing group with room for two actors, \* startup timeouts. Every Start safety control except ContinueQueued \* (MCStartContinue) fails here on its named invariant. MCStartDeep (weekly) From 92ffa045c0b9f678c05e27d80589b6fc50e00c81 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 13:51:01 -0300 Subject: [PATCH 08/42] docs: unified start design, client mapping, model README and roadmap rank 5h --- CHANGELOG.md | 15 +- ROADMAP.md | 3 + docs/design/README.md | 7 + docs/design/unified-start-client-mapping.md | 135 +++ docs/design/unified-start.md | 979 +++++++++++++++++++- verification/tla/README.md | 66 +- 6 files changed, 1196 insertions(+), 9 deletions(-) create mode 100644 docs/design/unified-start-client-mapping.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 03398efc..acbdf4b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,12 +2,15 @@ ## Unreleased -- Design (0.9, in review): a unified start API, `docs/design/unified-start.md`. - OTP's `start`, `start_link` and `start_monitor`, each optionally taking one - `StartOptions` (placement, registered name, startup timeout, per-instance - bounds). Linked and monitored children inherit their parent's placement. - It replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and - `start_linked`. A new TLA+ model, `Start.tla`, has negative controls. +- Design (0.9, in review): a unified start API (`docs/design/unified-start.md`, + rank 5h). OTP's `start`, `start_link` and `start_monitor` wait for + `started()`, and `spawn` starts without waiting. All four take one + `StartOptions` (placement, typed name, init timeout, label, stack size). + `started()` may fail or continue (`Init`). Linked and monitored children + inherit their parent's placement. It replaces `start_on`, + `start_with_backend`, `ExecutionGroup::start` and `start_linked`. A new + TLA+ model, `Start.tla`, runs in a per-PR CI leg `start` with 16 negative + controls, and a client mapping covers 51 sites. - Tests: coverage follow-ups. `restart_modes_distinguish_normal_shutdown_and_panic` now sends a real `Normal` exit; a per-PR `MCSupervisionTemporary` TLC diff --git a/ROADMAP.md b/ROADMAP.md index fbb3d1e4..0272cd89 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -40,6 +40,8 @@ handlers and the existing default behavior. New policies are explicit and opt-in release. Client ports are ready on the clients' `spawned-v0.8` branches. - **Next feature:** dynamic supervisor (rank 12a), design and TLA+ model in review (PR #245). +- **v0.9 design in review:** unified start API (rank 5h, PR #259, + [design](docs/design/unified-start.md)). - **Process:** every PR merges through GitHub's merge queue with 19 required checks, including the simulated suites (`sim`). Robustness features must add no cost on the default path (see [Non-goals](#non-goals)). @@ -578,6 +580,7 @@ startup: OTP aborts `start_link` on any child start error, while Spawned retries a Permanent or Transient child's failed factory, panic or startup timeout during the first start (semantics §11; found reviewing #233). Behaviour change: extend the `Supervision`/`Cohorts` TLA+ models and their Apalache inductive checks (`BudgetBounded`) to the new rule, with a negative control for counting per child, and update the Kani restart-window proofs and the guide. | | 5g | One exit reason when the runtime drops an actor (v0.8) | Tasks mode reports `Kill` when the runtime drops an actor; threads mode falls back to `Panic("unexpected framework panic")` for an actor that never finishes cleanup. Pick one documented reason for both modes and test it in both. | +| 5h | Unified start API (v0.9, design in review, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (16 negative controls) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()`; `spawn` does not. They share one `StartOptions` (placement, typed name, init timeout, label, stack size, per-instance bounds; slots reserved for ranks 17 and 18a). `started()` may fail (`Init`: ready, continue, error, stop, ignore), and `Init::continue_with` runs before any message (`handle_continue`). Linked and monitored children inherit the parent's group or backend, which fixes API-review footgun 5.11. Replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and `start_linked`. Client impact: 51 mechanical sites, no behaviour change. | | 6 | Test kit with `ActorRef::mock()` | **Phase A implemented** in [`spawned-test`](#testing-crate-spawned-test) (unreleased): closure-backed `Mock` actors that implement `Handler` for every message, so `ActorRef` satisfies any `#[protocol]` and replaces hand-stubbed protocol refs with no production change; recording `Probe` recipients (`expect`, `expect_no_message`); bounded harness waits; fault injection (`faults::exit`, `restart_child`, mock delay/discard/crash). Unmocked messages fail loudly. **Remaining:** mocking a field typed as a concrete `ActorRef` needs an `Envelope` hook that downcasts to the concrete message and yields message and reply sender (a core change, deferred); controllable time for timers; a generic idle check; core `ChildHandle::exit(reason)` would make injected exits exact (today `Panic` and `Kill` go through a linked helper, so a panic surfaces as `Panic("panic in handler: ...")`). **Client evidence:** hand-stubbed protocols in ethlambda tests (`RecordingChain` in `column_checks.rs`, also `envelope_checks.rs`, `verdict.rs`). **Research ([actor testing](docs/research/actor-testing.md)):** three gaps against Akka/Erlang kits. (1) A synchronous behavior test kit (Akka `BehaviorTestKit`): run one handler with a test `Context` and record its effects (sends, spawns, timers, stop) without a runtime; needs a core change. (2) Manual time (Akka `ManualTime`): document Tokio's `start_paused` for async actors; thread actors need the framework-owned clock, the same seam as the [simulation backend spike](#simulation-backend-spike) and `ctx.now()`, so build one clock, not two. (3) Concrete `ActorRef` mocking stays deferred (actix's `Mocker` is the precedent); until then users hold protocol refs or `Recipient`s. Also: the idle check should report total deadlock Go-`checkdead` style (every actor idle, a request outstanding, no timer pending), and a replay-equivalence check for event-sourced actors ([event sourcing](docs/research/event-sourcing.md)). | | 6b | Shared-state and abort lints in `#[actor]` | Zero runtime cost. The macro warns when an actor's fields hold shared mutable state (`Arc>`, `Arc>`, atomics behind `Arc`, `static mut`), with an opt-out attribute for deliberate cases, and a build script check warns when the profile uses `panic = "abort"`, under which a handler panic kills the process instead of one actor. Syntactic only (it cannot see sharing behind a type alias), but it catches the common pattern, including the one LLM-written code reaches for first. Erlang gap: memory isolation. | | 7 | Request deadlock detection | **Client evidence:** requests awaited inside handlers exist (ethrex peer table), no cycle found; keep as a cheap safety net. `ActorError::WouldDeadlock` naming the cycle instead of a timeout. `ActorRef::request` takes only `&self`, so set a thread-local `CURRENT_ACTOR` only while the actor's own handler future is polled (and around the synchronous handler in thread mode); detached helpers create no edges. Keep a small wait set per actor for `join!`. Measure cost before enabling outside debug builds. **Approach chosen ([deadlock and stall detection](docs/research/deadlock-and-stall-detection.md)):** eager detection. Add a wait-for edge when a handler sends a request, walk the per-actor wait sets for a cycle, fail the request that would close it with `WouldDeadlock` naming the cycle, remove the edge on reply or timeout; a request to self fails at once (Erlang's `calling_self`). Waits on raw channels, locks or `spawn_blocking` create no edges, so it is a safety net, not a proof. Edges are exported to rank 30. | diff --git a/docs/design/README.md b/docs/design/README.md index 0cdf4eea..e38aef29 100644 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -20,6 +20,13 @@ Active design contracts. Usage is documented in the [guide](../GUIDE.md). runtime-dropped actors as in OTP (ROADMAP ranks 5e, 5f, 5g), with the base actor lifecycle model `Lifecycle.tla`. +- **[unified-start.md](unified-start.md)** (rank 5h, 0.9, design in review, + not implemented): one start API as OTP's `start`, `start_link` and + `start_monitor` plus `spawn`, with `StartOptions`, placement inheritance, + fallible init, continuations and atomic names. Model: + `verification/tla/Start.tla`. Client rewrites: + [unified-start-client-mapping.md](unified-start-client-mapping.md). + Earlier designs (the v0.5 API redesign alternatives, the framework comparison and superseded roadmap plans) are archived in [`legacy/docs/design`](../../legacy/docs/design). diff --git a/docs/design/unified-start-client-mapping.md b/docs/design/unified-start-client-mapping.md new file mode 100644 index 00000000..5a1c360a --- /dev/null +++ b/docs/design/unified-start-client-mapping.md @@ -0,0 +1,135 @@ +# Unified start (0.9): client mapping + +Every start call in the production clients, mapped from the 0.8 API to the +[unified start design](unified-start.md). Sources: shallow clones on fede of +`lambdaclass/ethrex` `spawned-v0.8` at `c1163b7` and `lambdaclass/ethlambda` +`spawned-v0.8` at `85990aa`. + +Method: grep for `.start(`, `start_on(`, `start_with_backend(`, +`start_linked(`, `ExecutionGroup::start`, `ChildSpec::new`, `.ready()`, +`register_name`, `TestSystem` and `Mock` in files that use spawned. Each hit +was then checked by hand. The enclosing function, and whether it runs inside +an actor's handler, come from the source. + +## Summary + +| | ethrex | ethlambda | Total | +|---|---|---|---| +| `.start()` → `.spawn()` | 20 | 4 | 24 | +| `start_with_backend(b)` → `with_options(b).spawn()` | 7 | 0 | 7 | +| `start_on(g)` → `with_options(g).spawn()` | 4 | 2 | 6 | +| `Mock::..start()` → `Mock::..spawn()` | 6 | 8 | 14 | +| **Sites that change** | **37** | **14** | **51** | +| `start_linked`, `ExecutionGroup::start`, `ready()`, `register_name`, `TestSystem` | 0 | 0 | 0 | +| `ChildSpec::new` and supervisor `start()` (unchanged) | 1 (+2 test fixtures) | 2 | 3 | + +- **Behaviour.** The rewrite in the "0.9" column keeps the current behaviour + at every site. +- **Inheritance changes behaviour at no site.** Neither client uses + `start_linked`, and `spawn`/`start` never inherit. +- **Error types.** Six sites (marked E) sit in functions whose error type + comes from `start_on`'s `SystemError`. They return or convert `StartError` + instead. +- **Optional upgrades** (marked U): an async site may use + `.start().await?` instead of `.spawn()` to get init failures (today they + are only logged or invisible). This is a behaviour change: the caller waits + for `started()`. +- **Handler sites** (marked H) start a child from inside an actor's turn. + They stay unmanaged, which is the S6 footgun. `start_link` (or + `placement_of(ctx)`) would admit the child to the parent's group, so that + it joins ordered shutdown and counts against the group's capacity. That + is a client decision. + +## ethrex (37) + +| # | Site | Enclosing function | 0.8 | 0.9 | Notes | +|---|---|---|---|---|---| +| 1 | `crates/l2/sequencer/native_rollup/l1_watcher.rs:334` | `NativeL1Watcher::spawn` (sync) | `watcher.start()` | `watcher.spawn()` | | +| 2 | `crates/l2/sequencer/native_rollup/l1_advancer.rs:488` | `NativeL1Advancer::spawn` (sync) | `advancer.start()` | `advancer.spawn()` | | +| 3 | `crates/l2/sequencer/native_rollup/block_producer.rs:504` | `NativeBlockProducer::spawn` (sync) | `producer.start()` | `producer.spawn()` | | +| 4 | `crates/l2/sequencer/l1_watcher.rs:194` | `L1Watcher::spawn` (sync) | `Ok(state.start())` | `Ok(state.spawn())` | | +| 5 | `crates/l2/sequencer/state_updater.rs:110` | `StateUpdater::spawn` (async) | `state_updater.start()` | `state_updater.spawn()` | U | +| 6 | `crates/l2/sequencer/proof_coordinator.rs:110` | `ProofCoordinator::spawn` (async) | `state.start()` | `state.spawn()` | U | +| 7 | `crates/l2/sequencer/proof_coordinator.rs:385` | `ConnectionHandler::spawn` (async), called from `ProofCoordinator::handle_incoming_connection` | `connection_handler.start()` | `connection_handler.spawn()` | U, H | +| 8 | `crates/l2/sequencer/l1_committer.rs:253` | `L1Committer::spawn` (async) | `state.start_with_backend(Backend::Blocking)` | `state.with_options(Backend::Blocking).spawn()?` | U | +| 9 | `crates/l2/sequencer/block_producer.rs:350` | `BlockProducer::spawn` (async) | `block_producer.start_with_backend(Backend::Blocking)` | `block_producer.with_options(Backend::Blocking).spawn()?` | U | +| 10 | `crates/l2/sequencer/metrics.rs:69` | `MetricsGatherer::spawn` (async) | `Ok(metrics.start())` | `Ok(metrics.spawn())` | U | +| 11 | `crates/l2/sequencer/l1_proof_sender.rs:176` | `L1ProofSender::spawn` (async) | `state.start()` | `state.spawn()` | U | +| 12 | `crates/l2/based/block_fetcher.rs:337` | `BlockFetcher::spawn` (async) | `state.start()` | `state.spawn()` | U | +| 13 | `crates/networking/rpc/subscription_manager.rs:98` | `SubscriptionManager::spawn` (sync) | `..default().start()` | `..default().spawn()` | | +| 14 | `crates/networking/rpc/subscription_manager.rs:104` | `SubscriptionManager::spawn_on` (sync) | `..default().start_on(group)` | `..default().with_options(group).spawn()` | E | +| 15 | `crates/networking/p2p/tx_broadcaster.rs:206` | `TxBroadcaster::spawn_in` (sync) | `Ok(state.start_on(group)?)` | `Ok(state.with_options(group).spawn()?)` | E | +| 16 | `crates/networking/p2p/tx_broadcaster.rs:207` | same | `Ok(state.start())` | `Ok(state.spawn())` | | +| 17 | `crates/networking/p2p/rlpx/connection/server.rs:213` | `PeerConnection::spawn_as_receiver` (sync), called from the accept loop in `network.rs:288` | `connection.start()` | `connection.spawn()` | | +| 18 | `crates/networking/p2p/rlpx/connection/server.rs:228` | `PeerConnection::spawn_as_initiator` (sync), called from `RLPxInitiator` handlers (`initiator.rs:97`, `:113`) | `connection.start()` | `connection.spawn()` | H; the dynamic supervisor (#245) replaces this site | +| 19 | `crates/networking/p2p/rlpx/connection/server.rs:292` | `PeerConnection::unestablished` (sync) | `connection.start()` | `connection.spawn()` | | +| 20 | `crates/networking/p2p/rlpx/connection/server.rs:4024` | test helper `connection()` | `server.start()` | `server.spawn()` | test | +| 21 | `crates/networking/p2p/rlpx/initiator.rs:50` | `RLPxInitiator::spawn_on` (sync) | `..new(context).start_on(group)?` | `..new(context).with_options(group).spawn()?` | E | +| 22 | `crates/networking/p2p/rlpx/initiator.rs:66` | `RLPxInitiator::spawn_with_backend` (sync, returns `ActorRef`) | `state.start_with_backend(b)` | `state.with_options(b).spawn().expect("a backend alone cannot be refused")` | it fails only on options | +| 23 | `crates/networking/p2p/rlpx/initiator.rs:67` | same | `state.start()` | `state.spawn()` | | +| 24 | `crates/networking/p2p/peer_table.rs:616` | `PeerTableServer::spawn_with_filter` (sync) | `..start()` | `..spawn()` | | +| 25 | `crates/networking/p2p/peer_table.rs:635` | `PeerTableServer::spawn_on` (sync) | `.start_on(group)` | `.with_options(group).spawn()` | E | +| 26 | `crates/networking/p2p/discovery/server.rs:156` | `DiscoveryServer::spawn` (async) | `.start()` | `.spawn()` | U | +| 27 | `crates/prover/src/prover.rs:251` | `start_prover` (async) | `prover.start_with_backend(Backend::Blocking)` | `prover.with_options(Backend::Blocking).spawn()?` | U | +| 28 | `crates/prover/src/prover.rs:260` | same | same | same | U | +| 29 | `crates/prover/src/prover.rs:268` | same | same | same | U | +| 30 | `crates/prover/src/prover.rs:276` | same | same | same | U | +| 31 | `tooling/monitor/src/app.rs:103` | `EthrexMonitor::spawn` (async) | `monitor.start();` | `monitor.spawn();` | U; the 0.8 statement would still start the actor (eager stage 1), with a `#[must_use]` warning that clippy `-D warnings` rejects | +| 32 | `test/tests/p2p/peer_table_timeout_tests.rs:33` | test (async) | `Mock::new()..start()` | `Mock::new()..spawn()` | test | +| 33 | `test/tests/p2p/peer_table_timeout_tests.rs:47` | test | same | same | test | +| 34 | `test/tests/p2p/peer_table_timeout_tests.rs:59` | test | same | same | test | +| 35 | `test/tests/p2p/shutdown_order_tests.rs:141` | test | `Mock::new()..start()` | `Mock::new()..spawn()` | test | +| 36 | `test/tests/p2p/shutdown_order_tests.rs:169` | test | same | same | test | +| 37 | `test/tests/p2p/shutdown_order_tests.rs:196` | test | `Mock::new().start()` | `Mock::new().spawn()` | test | + +Unchanged: + +- `crates/networking/p2p/discovery/supervised.rs:145`: `ChildSpec::new("discovery", group, ..)`, + no `startup_timeout` and no bounds. The supervisor's `start()` is + unchanged. Two test fixtures (`:244`, `:271`) build an `ActorSystem` and a + group. +- `cmd/ethrex/initializers.rs:1043`: `ActorSystem::new`, `async_group`. + +E sites: #14 and #25 return `Result<_, SystemError>`, which becomes +`StartError`. #15 and #21 propagate with `?` into the caller's error type, +which needs `From`. + +## ethlambda (14) + +| # | Site | Enclosing function | 0.8 | 0.9 | Notes | +|---|---|---|---|---|---| +| 1 | `crates/net/p2p/src/lib.rs:1277` | `P2P::spawn` (sync) | `server.start_on(group)?` | `server.with_options(group).spawn()?` | E | +| 2 | `crates/blockchain/src/lib.rs:452` | `BlockChain::start_actor` (sync; the `chain` thread group) | `.start_on(group)?` | `.with_options(group).spawn()?` | E | +| 3 | `crates/blockchain/src/lib.rs:6393` | test | `beacon_server(..).start()` | `beacon_server(..).spawn()` | test | +| 4 | `crates/blockchain/src/lib.rs:6428` | test | same | same | test | +| 5 | `crates/blockchain/src/lib.rs:6450` | test | same | same | test | +| 6 | `crates/net/p2p/src/beacon/verdict.rs:1832` | test helper `run_checks` (async) | `..await.start()` | `..await.spawn()` | test | +| 7 | `crates/net/p2p/src/beacon/column_checks.rs:133` | test | `Mock::new().allow_sends().start()` | `Mock::new().allow_sends().spawn()` | test | +| 8 | `crates/net/p2p/src/beacon/column_checks.rs:164` | test | same | same | test | +| 9 | `crates/net/p2p/src/beacon/envelope_checks.rs:191` | test | same | same | test | +| 10 | `crates/net/p2p/src/beacon/verdict.rs:764` | test helper `recording_chain` (sync) | same | same | test | +| 11 | `crates/net/p2p/src/beacon/verdict.rs:771` | test helper `silent_chain` (sync) | `Mock::new().start()` | `Mock::new().spawn()` | test | +| 12 | `crates/net/p2p/src/req_resp/envelope_client.rs:615` | test | `Mock::new().start()` | `Mock::new().spawn()` | test | +| 13 | `crates/net/p2p/src/req_resp/envelope_client.rs:702` | test | `Mock::new().allow_sends().start()` | `Mock::new().allow_sends().spawn()` | test | +| 14 | `crates/net/p2p/src/gossipsub/handler.rs:1736` | test | same | same | test | + +Unchanged: + +- `bin/ethlambda/src/main.rs:981` and `bin/ethlambda/src/http_server.rs:216`: + `ChildSpec::new("http", ..)` with `shutdown_timeout` only. +- The supervisor `start().await` calls. +- `#[actor(max_native_jobs = ..)]` (`crates/net/p2p/src/lib.rs:1572`): an + `Actor` constant, unchanged. + +## Related work + +- **ethrex.** Site #18 is the per-connection actor that the dynamic + supervisor design (#245) moves under a `DynamicSupervisor`. With unified + start, its RLPx handshake becomes `Init::continue_with(Handshake)`, so no + peer-table message or `terminate_child` can run before it (gap report + 3.2.1). +- **ethlambda.** Site #2 is the chain actor on its own one-worker + `thread_group` (0.8 replaced `Backend::Thread` with it). Unified start + keeps that placement, and a `start_link` from the chain actor would + inherit the `chain` group: it would need a second worker, or `Capacity` is + returned. diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 475fc87b..a848ce11 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1,4 +1,979 @@ # Unified start API (0.9): design contract -Status: draft, in progress. Design only; the implementation follows after two -independent reviews. +Status: design in review, not implemented. Target release 0.9. The +implementation follows after two independent reviews. It is designed against +the 0.8 API as #240 ships it (exit signals, `try_link`, `NoProc`, the 0.8 +API-review cleanups: `send_request`, `#[non_exhaustive]` builders). + +- Model: [`verification/tla/Start.tla`](../../verification/tla/Start.tla) + (configurations `MCStart`, `MCStartContinue`, `MCStartLive`, `MCStartDeep`). +- Client mapping: [unified-start-client-mapping.md](unified-start-client-mapping.md). +- Inputs: the 0.8 API review (Start cluster, footgun 5.11, rank 19); the OTP, + Elixir and Gleam gap report (G1, G2, G7, G8, G14, G15, section 3.1). +- OTP sources: `proc_lib.erl`, `gen.erl` and `gen_server.erl` at OTP-27.1. A + line reference like `gen.erl:191` points into those files. + +## Summary + +| # | Decision | OTP | +|---|---|---| +| 1 | `start`, `start_link` and `start_monitor` return after the child's `started()`: `Ok` when it is ready, otherwise the error, after the child's death point | `gen_server:start/start_link/start_monitor` are synchronous on `init/1` | +| 2 | `spawn` starts without waiting (fire and forget); `ActorRef::ready()` waits later | `proc_lib:spawn` | +| 3 | `started()` may fail: it returns `()`, `Result<(), E>` or `Init` (ready, continue, error, ignore, stop) | `init/1` returns `{ok,S}`, `{ok,S,{continue,C}}`, `{error,R}`, `{stop,R}`, `ignore` | +| 4 | `Init::continue_with(msg)`: a turn that runs before any mailbox message | `handle_continue` | +| 5 | One `StartOptions`: placement, typed name, init timeout, label, stack size, per-instance bounds; reserved slots for 0.10 | `{local,Name}`, `{timeout,T}`, `spawn_opt`, `set_label` | +| 6 | Linked and monitored children inherit the parent's placement (its group or backend). Plain `start`/`spawn` never inherit; `placement_of(&ctx)` opts in | none (one node, one scheduler pool); see [Placement](#4-placement-and-inheritance) | +| 7 | Stage 1 runs eagerly in the caller: admission, link or monitor, name, dispatch. A refused start leaves no trace | `spawn_opt([link, monitor])` is atomic; `gen:init_it` registers the name before `init` | +| 8 | A waiting `start_link` consumes the failed child's `Exit` (trapping parent) and `start_monitor` its `Down` | `proc_lib:sync_start` + `flush_EXIT`; `gen:monitor_return` | +| 9 | The init timeout unlinks or demonitors first, then kills | `kill_flush_EXIT` | +| 10 | Removed: `start_on`, `start_with_backend`, `ExecutionGroup::start`, `start_linked`; `ChildSpec::{startup_timeout, max_outstanding, max_native_jobs}` move into `StartOptions` | | + +## 1. The problem + +The 0.8 start surface: + +| Call | Placement | Link | Waits for `started()` | Fallible | +|---|---|---|---|---| +| `a.start()` | unmanaged `Async` | no | no | no | +| `a.start_with_backend(b)` | unmanaged `b` | no | no | no | +| `a.start_on(&g)` | group `g` | no | no (`ready()` waits) | admission only | +| `g.start(a)` | group `g` (duplicate of `start_on`) | no | no | admission only | +| `a.start_linked(&ctx)` | **always unmanaged `Async`** | yes, before `started()` | no | no | +| `ChildSpec::new(id, &g, f)` | group `g` | supervisor parent | yes (`startup_timeout`) | yes | +| `TestSystem::start`, `Mock::start{,_on}` | test group / unmanaged | no | no | panics | + +The resulting problems: + +- **Footgun (API review 5.11).** A child that a managed actor starts with + `start_linked` is outside the parent's group. `spawn_helper` then returns + `UnmanagedActor`, ordered shutdown never sees the child, and group + teardown can abort it without `stopped()` (semantics S6). +- **No atomic monitor at start.** `ctx.monitor` after `start()` races the + child's death, and since 0.8 a child that already died reports `NoProc` + instead of its real reason. +- **Startup failure is invisible outside supervision.** `started()` cannot + fail, and `start()` returns an `ActorRef` even when `started()` will + panic. The client sweep found the ethrex L2 sequencer starting actors one + by one and only logging startup failures. +- **No continuation.** A self-send from `started()` queues behind anything + that arrived during `started()`. The dynamic supervisor design (#245) + moves ethrex's RLPx handshake into "the first message", and a peer message + or a `terminate_child` can overtake it. +- **Linking at start covers only one placement:** `start_linked` is + Async-only and `start_on` has no linked form. + +## 2. API + +### 2.1 tasks (`spawned_concurrency::tasks`) + +```rust +pub trait ActorStart: Actor { + /// proc_lib:spawn. Unmanaged Async, no wait, no parent. + fn spawn(self) -> ActorRef; + /// gen_server:start. Unmanaged Async; resolves after started(). + fn start(self) -> Starting>; + /// gen_server:start_link. The parent's placement; linked before started(). + fn start_link(self, parent: &Context

) -> Starting>; + /// gen_server:start_monitor. The parent's placement; monitored before started(). + fn start_monitor>(self, parent: &Context

) + -> Starting, MonitorRef)>; + /// The same four verbs with options (spawn_opt and the gen_server Opts list). + fn with_options(self, opts: impl Into>) -> Starter; +} + +#[must_use = "call spawn, start, start_link or start_monitor"] +pub struct Starter { /* actor, options */ } +impl Starter { + pub fn spawn(self) -> Result, StartError>; + pub fn start(self) -> Starting>; + pub fn start_link(self, parent: &Context

) -> Starting>; + pub fn start_monitor>(self, parent: &Context

) + -> Starting, MonitorRef)>; +} + +/// The child is already started (stage 1 ran in the call); awaiting it +/// waits for started(). 'static + Send: it owns what it needs. +#[must_use = "the actor runs; await this to learn whether it started"] +pub struct Starting { /* ... */ } +impl IntoFuture for Starting { + type Output = Result; +} +impl Starting { + /// The child's reference without waiting: Err only for a stage-1 refusal. + /// Dropping the Starting abandons the wait (section 5.5). + pub fn actor_ref(&self) -> Result, &StartError>; +} + +impl ActorRef { + /// The ack for any placement (0.8: managed only, returning SystemError). + pub async fn ready(&self) -> Result<(), StartError>; +} +``` + +### 2.2 threads (`spawned_concurrency::threads`) + +The four verbs and `with_options` have the same names and meanings. The +waiting forms block the calling thread instead of returning a future: + +```rust +fn spawn(self) -> ActorRef; +fn start(self) -> Result, StartError>; +fn start_link(self, parent: &Context

) -> Result, StartError>; +fn start_monitor>(self, parent: &Context

) + -> Result<(ActorRef, MonitorRef), StartError>; +fn with_options(self, opts: impl Into>) -> Starter; +// ActorRef::ready(&self) -> Result<(), StartError> (blocking) +``` + +The only difference between the modes is `.await` against blocking, as for +requests. threads mode has one placement (a thread per actor), so its +`StartOptions` has no `group`, `backend` or `placement_of`. These are +tasks-only, like `ActorSystem` today (API review rank 27). A threads-mode +`start_link` from a threads parent inherits "a thread", which is the only +placement there is. + +### 2.3 Fallible `started()` and `Init` + +```rust +// tasks +fn started(&mut self, ctx: &Context) + -> impl Future> + Send> + Send { async {} } +// threads +fn started(&mut self, ctx: &Context) -> impl Into> {} + +#[must_use] +pub struct Init { /* private */ } +impl Init { + pub fn ready() -> Self; // {ok, S} + pub fn continue_with(msg: M) -> Self // {ok, S, {continue, C}} + where A: Handler; + pub fn error(reason: impl Display) -> Self; // {error, R}: exit Normal + pub fn stop(reason: ExitReason) -> Self; // {stop, R}: exit R + pub fn ignore() -> Self; // ignore: exit Normal +} +impl From<()> for Init; // ready +impl From> for Init; // Err -> error +impl From, E>> for Init; // Err -> error +``` + +The signature was prototyped with rustc 1.99 (scratch crate). All of these +compile unchanged: + +- the existing `async fn started(&mut self, ctx: &Context) {}`; +- the default body; +- `-> Result<(), io::Error>` and `-> Result, String>`; +- the threads form. + +The `#[started]` macro hook passes the method's return type through. No +existing actor changes. Return-position `impl Trait` in traits is stable +since Rust 1.75, below the workspace's 1.93. + +| `started()` outcome | Child | `stopped()` | Waiting starter gets | Link notice / `Down` reason | +|---|---|---|---|---| +| `()`, `Ok(())`, `Init::ready()` | runs | later, as usual | `Ok` | none | +| `Init::continue_with(m)` | runs; `m` is handled first | later | `Ok` (before `m` runs, as OTP's `init_ack`) | none | +| `Err(e)`, `Init::error(e)` | exits `Normal` | no | `InitFailed(e)` | `Normal` | +| `Init::ignore()` | exits `Normal` | no | `Ignored` | `Normal` | +| `Init::stop(r)` | exits `r` | no | `InitExited(r)` | `r` | +| panic | exits `Panic("panic in started(): ..")` | no | `InitExited(Panic(..))` | `Panic` | +| ready, but `ctx.stop*` called in `started()` | exits that reason | yes (L1, unchanged) | `InitExited(r)` | `r` | +| ready, but a held signal ends it (kill, non-`Normal` parent exit, link notice) | exits per X3 | per X3 | `InitExited(r)` | `r` | +| still running at the init timeout | killed | no | `InitTimeout` | none (edge removed first) | + +The ack is `Ok` exactly when `started()` returned a ready `Init` and no exit +was pending when it returned. That is also the point where the managed +readiness barrier commits (L2). Typed init data, as Gleam's `returning` and +`proc_lib:init_ack(Ret)`, is not included; see [alternatives](#15-alternatives-considered). + +### 2.4 `StartOptions` + +```rust +#[derive(Clone)] +#[non_exhaustive] +pub struct StartOptions { /* private fields */ } +impl Default for StartOptions { .. } +impl StartOptions { + pub fn new() -> Self; + // placement (tasks only) + pub fn group(self, group: &ExecutionGroup) -> Self; + pub fn backend(self, backend: Backend) -> Self; + pub fn placement_of(self, ctx: &Context

) -> Self; + // naming and diagnostics + pub fn name(self, name: impl Into>) -> Self; + pub fn label(self, label: impl Into>) -> Self; + // start behaviour + pub fn init_timeout(self, timeout: Duration) -> Self; + // per-instance resources + pub fn stack_size(self, bytes: usize) -> Self; + pub fn max_outstanding(self, max: usize) -> Self; + pub fn max_native_jobs(self, max: usize) -> Self; +} +impl From<&ExecutionGroup> for StartOptions; // .group(g) +impl From for StartOptions; // .backend(b) +``` + +| Field | Default | Meaning | OTP | +|---|---|---|---| +| placement | `start_link`/`start_monitor`: the parent's; `start`/`spawn`: unmanaged `Async` | [section 4](#4-placement-and-inheritance). The last of `group`/`backend`/`placement_of` wins | none | +| `name` | none | registered for this instance at stage 1, released at its D ([section 6](#6-registered-names)) | `{local, Name}` | +| `label` | none | shown in tracing spans, panic and stall diagnostics, `ShutdownReport` actor details and introspection | `proc_lib:set_label/1` (OTP 27) | +| `init_timeout` | none (wait forever); `ChildSpec` keeps its 5 s default | [section 7](#7-init-timeout) | `{timeout, T}`, default `infinity` (`gen.erl:716`) | +| `stack_size` | runtime default | honoured where the actor owns a thread: threads mode, `Backend::Thread`. Elsewhere `InvalidOptions`; group stacks are rank 8c | none (BEAM stacks grow) | +| `max_outstanding`, `max_native_jobs` | `Actor::MAX_OUTSTANDING`, `MAX_NATIVE_JOBS` | per-instance overrides; were on `ChildSpec` only | none | + +**Reserved for 0.10.** These are planned as additive builder methods, with no +new start API: + +- `mailbox(Unbounded | Bounded(n))` (rank 17); +- `max_memory` (rank 18a, OTP `max_heap_size`); +- a message-priority opt-in (OTP 28 priority messages, gap G11). + +**Not copied from `spawn_opt`:** + +- `fullsweep_after`, `min_heap_size` and `message_queue_data`: there is no + per-actor heap. +- `priority`: a cooperative runtime cannot honour per-actor CPU priority. + Groups are the bulkhead. +- `hibernate_after` and `debug`. + +`Name` is a typed name: `Name::new("peer_table")`, plus +`From<&'static str>` and `From`. `name.whereis() -> Option>` +cannot mistype a lookup (G14). It still keys the global registry until rank 12 +scopes it. + +### 2.5 `StartError` + +```rust +#[derive(Debug, Clone, thiserror::Error)] +#[non_exhaustive] +pub enum StartError { + Closed, // group, system or the parent's scope no longer admits + Capacity, // thread_group full + AlreadyStarted { name: String, existing: Option }, + ParentExited, // start_link/start_monitor through a Context whose actor's exit began + InvalidOptions(&'static str), // e.g. stack_size on a pooled placement, backend in a ChildSpec + InitTimeout, + InitFailed(String), // started() returned an error + InitExited(ExitReason), // the child exited before it was ready + Ignored, // started() returned Init::ignore() + System(SystemError), // runtime construction and other placement failures +} +``` + +The `Init*` names follow Gleam's `actor.StartError`. `Ignored` is an error +variant because Rust has no third return shape for OTP's bare `ignore`. +Supervisors treat it as "nothing started" (section 9). +`SystemError::StartupFailed` is removed: `ready()` now returns +`StartError::InitExited`. + +## 3. Two stages + +**Stage 1 runs in the caller, synchronously, inside the call** (`spawn`, +`start*` or `Starter::*`, before any `.await`). Its steps, in order: + +1. **Admit** to the placement: explicit, else inherited, else unmanaged. A + group admits while it is open and has room. Otherwise the start returns + `Closed` or `Capacity`. +2. **Link or monitor** (`start_link`/`start_monitor`). The edge is recorded + in one transaction with the parent's lifecycle state. A parent whose exit + began, or which is past its death point, refuses with `ParentExited`. +3. **Register** the name, atomically, or refuse with `AlreadyStarted`. +4. **Dispatch** the child to its executor. + +**A refusal at any step leaves no trace:** + +- the admission slot, the edge and the name are released; +- the state is dropped without either hook running; +- no `Exit` or `Down` is ever delivered, and no actor id is published. + +This is L2's rule for a closed admission, extended to every refusal. The +implementation may reorder the internal locks, as long as the result is +indistinguishable from this order. The model checks interleavings of parent +death, group closure and a competing registration between every two steps +(`AbortInvisible`). + +**Stage 2 runs on the child.** `started()` runs with signals held (phase 0 of +the exit-signals design), and they are resolved when it returns. The waiting +forms resolve then, as in the table in 2.3. + +Because stage 1 is eager, `x.start();` without `.await` still starts the +actor. A forgotten `.await` therefore degrades to `spawn` (plus the +`#[must_use]` warning) and never to "the actor silently never started". +This matters for the migration: a 0.8 statement `x.start();` keeps working. + +## 4. Placement and inheritance + +**Rule.** The placement is the first that applies: + +1. an explicit placement (`group`, `backend`, `placement_of`); +2. for `start_link` and `start_monitor`, the parent's placement; +3. unmanaged `Backend::Async` (threads: a thread). + +The parent's placement is fixed when the parent was admitted: + +| Parent | Child of `start_link`/`start_monitor` | +|---|---| +| managed, async group `g` | admitted to `g` | +| managed, thread group `g` | admitted to `g` (any free slot; `Capacity` if none) | +| managed, pinned group `g`, shard `i` | `g`, shard `i` (as `Scope::group()` keeps a pinned selection) | +| supervised child on `g` | `g` | +| unmanaged, backend `b` | unmanaged, backend `b` | +| threads mode | a thread | + +**No fallback.** If the inherited group is closed (system shutdown began, S1) +or full, the start fails with `Closed` or `Capacity`. Silently escaping the +group is the 0.8 bug this design removes. + +**Ownership of an inherited child.** + +- **`start_link` child of a managed parent.** It is admitted under the + parent's scope, as nested supervisors are (admission parent), not as a new + system root. Its parent for E3 is the linking actor. System shutdown reaches + it through that parent: the parent's turn in the reverse admission order + (S2) ends when the parent and its `start_link` descendants have finished, + or at the grace deadline. The parent's exit reaches them as a parent exit + (E3). If the parent is supervised, its generation counts as finished (V3) + only when its `start_link` descendants have also finished. A descendant + still alive one cleanup interval (the row for the parent in the E3 deadline + table) after the parent's D is killed; one still alive a second interval + later makes the slot stuck. Without this, a trapping child would get the + system's `Shutdown` as a non-parent signal, which it can ignore. +- **`start_monitor` child of a managed parent.** It is a standalone system + root admitted after its starter, as `start_on` is today; its E3 parent is + the system. The reverse admission order stops it before its starter. +- **Unmanaged parents.** Inheritance copies the backend only. Unmanaged + children stay unowned (S6). + +**Why only `start_link` and `start_monitor`.** OTP has one node and one +scheduler pool, so every process is "in the parent's placement". The closest +Spawned analogue of the node is the parent's group, and a linked child +outside its parent's group is the footgun. Plain `start` and `spawn` take no +`Context`. Inheriting for them would need an ambient "current actor" +(task-local or thread-local). That would cost something on every turn, would +not follow work into `tokio::spawn` or helper tasks, and would silently move +the 20 `.start()` calls in ethrex's actor code into groups. The explicit form +is `with_options(StartOptions::new().placement_of(ctx))`. OTP's +`gen_server:start` and `start_monitor` also give the child no parent +(`gen.erl:137-142`, parent `self`). Only `start_link` makes the caller the +parent. + +## 5. `start_link` and `start_monitor` + +### 5.1 Atomicity + +- The edge (link or monitor) is recorded before the child's `started()` + begins, in every mode and placement (model `LinkBeforeStarted`). OTP sets + it atomically with the spawn: `spawn_opt(.., [link, monitor])`. +- The child never runs without its edge. The edge never exists for a child + that a stage-1 refusal aborted. +- No edge survives the parent's death point (`NoEdgeToDeadParent`). + +### 5.2 Parent state + +- **The parent is running** (the normal case: the call comes from its turn). + The edge is recorded, and the parent's turn waits for the ack (M1: no other + message of the parent runs meanwhile, as OTP's caller blocks in + `sync_start`). +- **The parent's exit began, or it is past D.** This is possible only through + a cloned `Context` used from a helper or job. The start returns + `ParentExited`, and no child exists. 0.8 instead spawned a doomed child + that received a `NoProc` parent exit (otp-exit-signals E3, "a parent + already past D"). That bullet is replaced. OTP has no counterpart (a dead + process cannot call); the closest rule is X2, where a link from an exiting + caller is inert. +- **The parent dies after the edge is recorded, during the child's stage 1 + or `started()`.** Its D sends the child a parent exit. The exit is held + until `started()` returns and then applied by E3: a non-`Normal` reason ends + the child, and the ack reports `InitExited(reason)`; `Normal` is ignored by + a non-trapping child. For `start_monitor` the monitor is cancelled at the + parent's D, and the child continues (it has no parent). Model: + `ParentExitReachesChild` (liveness), control `DropHeld`. + +### 5.3 Startup failure + +Every failure is returned after the child's D, as OTP's `sync_start` awaits +the `'DOWN'` (`proc_lib.erl:434-449`, `FailureAfterDeath`). By then the +child's name is free, so the start can be retried at once +(`NameReleasedAtDeath`), as `gen_server` unregisters before notifying +(`gen_server.erl:2203-2210`). + +- **`start_link`, trapping parent.** The child's `Exit` is consumed by the + waiting call and never reaches `exit_received`, as `flush_EXIT` + (`proc_lib.erl:606`). +- **`start_link`, non-trapping parent.** A non-`Normal` reason is applied as + a link notice. The call returns the error to the handler, and the parent + then exits with that reason after its current handler (X3; cooperative, + L3). In OTP the link kills the caller before `start_link` returns (the + `proc_lib:start_link` docs). A `Normal` exit (`Init::error`, `ignore`) is + ignored. +- **`start_monitor`.** The `Down` is consumed, as `gen:monitor_return` + (`gen.erl:172-182`); the call returns the error. Model: `FailureFlushed`. + +### 5.4 Ok + +`Ok` means `started()` returned a ready `Init` (`AckMeansStarted`). After +`Ok`, the edge behaves as any `ctx.link`/`ctx.monitor` edge (X1, X3, X5). + +### 5.5 Abandoning the wait + +Dropping the `Starting` (tasks) before it resolves abandons only the wait. The +child keeps running with its edge and its name. A notice already diverted to +the waiting call is handed to the parent as if it had never been diverted. The +handover is linearised with the child's D, so the notice is delivered exactly +once (`AbandonedNoticeDelivered`, control `AbandonDropsNotice`). An abandoned +`start` is `spawn`. `Starting` is `'static + Send`, so a handler may also +hand it to a job and await it outside its turn. The consumption then happens +wherever it is awaited. + +**Mechanism (no cost on send or dispatch).** While a waiting call exists, the +edge carries an "ack pending" mark. The child's death notice for a marked edge +goes to the call's slot instead of the parent's mailbox. The ack (`Ok` or +abandon) clears the mark under the edge's lock. Unmarked edges, which are +every edge after startup, take the 0.8 path unchanged. + +## 6. Registered names + +- **When.** At stage 1, before `started()`, so the child can be found while + it initialises (messages queue; a continuation still runs first). OTP + registers in `gen:init_it` before `init/1` (`gen.erl:194-201`). +- **Clash.** The start is refused with + `AlreadyStarted { name, existing }`. `started()` does not run, and nothing + is linked or delivered. `existing` is the holder's `ChildHandle` when an + actor holds the name, or `None` when a value from the untyped `register` + does. OTP returns `{error, {already_started, Pid}}` and its loser exits + `normal` (`gen.erl:198-200`). A `normal` exit reaches no non-trapping + caller, and the trapping caller's `'EXIT'` is consumed, so nothing is + observable either. +- **Uniqueness.** Two concurrent starts with one name: exactly one is + admitted (`NameUnique`, control `DoubleRegister`). +- **Release.** At the holder's D, before any link notice, `Down`, + `wait_exit`, ack or `ShutdownReport` entry can observe the death + (`NameReleasedAtDeath`, control `LateUnregister`). This applies to every + placement, unmanaged ones included. 0.8's `ctx.register_name` is + managed-only and released at cleanup. A refused start never held the name. +- **Supervised children.** Each generation registers the spec's name, and + the replacement registers only after the previous generation's D (V3). + `name.whereis()` therefore keeps addressing the current generation. This + is Gleam's `named_subject` behaviour, without a proxy. +- **Exception.** An init timeout whose child is stuck inside `started()` + keeps the name until the child's D (section 7). +- `ctx.register_name` remains for names chosen after start. The registry + keeps its global scope until rank 12. + +## 7. Init timeout + +`init_timeout(d)` bounds stage 2. It is measured from dispatch until +`started()` returns, as OTP's timer starts after the spawn. + +At expiry, if a starter is still waiting: + +1. the link or monitor edge is removed (`unlink`/`demonitor` fences); +2. a kill is sent (`kill_flush_EXIT` unlinks before `exit(Pid, kill)`, + `proc_lib.erl:610-615`); +3. the call returns `InitTimeout` after the child's D, or after a bounded + wait for it: the placement's cleanup deadline (5 s unmanaged, otherwise + the group's or spec's). + +Model: `TimeoutContained`, control `KillBeforeUnlink`. + +Rust cannot preempt `started()` (L3), so a child stuck in blocking code +reaches D only when `started()` returns. Until then it holds its name and its +admission slot, and it is reported as unfinished if the system shuts down. + +**Without a waiting starter** (`spawn`, or an abandoned `start*`), the +timeout is enforced by a framework timer armed at dispatch, only when the +option is set. Expiry kills the child, and `ready()` reports `InitTimeout`. +An edge of an abandoned `start_link` stays, so the `Killed` notice reaches +the parent like any linked death (trappable, X4). + +**Default.** None, as OTP's `infinity`. `ChildSpec` keeps its 5 s startup +default (V7, a Spawned bound). The gap report suggested 5 s "as +`gen_server:start`", but OTP's default is `infinity` (`gen.erl:716-722`). A +5 s default would also kill client actors whose `started()` does slow I/O +today (open question 3). + +## 8. Continuation (`handle_continue`) + +`Init::continue_with(msg)` runs `Handler::handle(msg)` as the actor's +first turn after `started()`, before any mailbox message, including messages +that arrived during `started()` (`ContinueFirst`, control `ContinueQueued`). + +- The ack and managed readiness commit before the continuation runs, as + OTP's `init_ack` precedes `handle_continue` (`gen_server.erl:2197-2201`). +- A stop or exit observed before the continuation begins discards it, as D4 + discards a queued message. No caller waits on it; its `PendingReply` is + detached. +- A panic in the continuation is an ordinary handler panic after readiness. + Under a supervisor it is a crash of a ready child, not a startup failure. +- **Cost.** One branch after `started()`; the dispatch loop is unchanged. +- **Not included:** continuing from ordinary handlers (OTP's + `{noreply, S, {continue, C}}`). That needs a check after every turn, which + is a cost on the dispatch path (open question 4). + +## 9. Supervisors + +The OTP child spec separates `start` (the MFA and its options) from +`restart`, `shutdown`, `type` and `significant`. `ChildSpec` mirrors that: + +```rust +ChildSpec::new(id, &group, |fctx| async { Ok(MyActor::new(..)) }) + .options(StartOptions::new() + .name("rpc") + .init_timeout(Duration::from_secs(10)) + .label("http") + .max_native_jobs(8)) + .restart(RestartMode::Transient) + .shutdown_timeout(Duration::from_secs(5)) +``` + +- **Placement.** The positional group stays: a supervised child is always + managed, and the group must belong to the supervisor's system + (`ForeignGroup`). A `group` in `options` replaces it. A `backend` is + `InvalidOptions` at `add()`. +- **Moved into `StartOptions`.** `startup_timeout` becomes `init_timeout` + (default 5 s when unset, V7 unchanged). `max_outstanding` and + `max_native_jobs` also move. `restart` and `shutdown_timeout` stay; there + is room for a future `significant(bool)` (G15), since the builder has + private fields. +- **Each generation is a `start_link` with the supervisor as parent.** The + same stage-1 and ack rules apply. `Init::error`, `stop` or a panic is a + failed start: the first start aborts the tree (V7), and a replacement is + retried (V1). **`Init::ignore`** follows OTP: the spec stays and no + generation runs, in a new `ChildState::Ignored`. A Temporary child that + ignores is removed from the snapshot, as OTP deletes it. `ignore` does not + fail the initial start and is not a restart. +- **Name clashes.** A clash at a restart is a failed restart attempt, which + is retried with backoff and charged to the budget (V5). OTP also fails the + restart on `already_started`. +- **Continuations.** A continuation of a supervised child runs after + readiness. +- **Dynamic supervisor (#245).** `DynamicChildSpec` takes the same + `StartOptions`. `start_child` returns `Ignored` as "nothing started" + (gap report 3.2.3). The RLPx handshake becomes a continuation, so no peer + message or `terminate_child` can overtake it (3.2.1). +- **Not derived.** `#[actor]` does not derive a child spec; see + [alternatives](#15-alternatives-considered). + +## 10. Test kit (`spawned-test`) + +| 0.8 | 0.9 | +|---|---| +| `Mock::start()` | `Mock::spawn()` (immediate; a mock's `started()` is trivial) | +| `Mock::start_on(&g)` | `Mock::spawn_with(opts)` (`impl Into>`, so `&g` works) | +| none | `Mock::on_started(\|ctx\| Init)` and `Mock::start_delay(d)` (virtual time under the simulator): drive every `Init` outcome and the init timeout from tests | +| `TestSystem::start(a)` | `TestSystem::spawn(a)` (immediate, panics on refusal) and `TestSystem::start(a).await` (waits, panics with the `StartError`) | +| `TestSystem::start_mock(m)` | `TestSystem::spawn_mock(m)` | +| none | `TestSystem::options::()`: `StartOptions` with the test group set | + +`start_link` and `start_monitor` are exercised from tests through `in_actor`. +Its closure gets the parent's `Context`, and because `Starting` is `'static` +it can be returned and awaited outside the turn. + +## 11. Erlang/OTP alignment + +Categories as in semantics section 11: R (Rust forces it), T (Rust types add +strictness), B (a bound Erlang lacks). + +| OTP | Spawned 0.9 | Verdict | +|---|---|---| +| `gen_server:start/3,4`: synchronous on `init/1` | `start()`, `with_options(o).start()` | aligned | +| `gen_server:start_link`: link set atomically, caller is the parent | `start_link(&ctx)` | aligned; the parent is passed explicitly (R: no `self()`) | +| `gen_server:start_monitor` (OTP 23) → `{ok, {Pid, Mon}}` | `start_monitor(&ctx)` → `(ActorRef, MonitorRef)` | aligned | +| `proc_lib:spawn/spawn_opt`: no init ack | `spawn()`, `with_options(o).spawn()` | aligned | +| `{local, Name}`; `{error, {already_started, Pid}}` | `name(Name)`; `AlreadyStarted { name, existing: Option }` | aligned; typed name (T); `existing` is `None` when a non-actor value holds the name (T: the registry stores typed values) | +| name unregistered before the starter is notified | released at D, before any notice | aligned | +| `{timeout, T}`, default `infinity` | `init_timeout`, default none | aligned; the wait for the killed child's death is bounded (R: no preemption) | +| `kill_flush_EXIT` on timeout | unlink/demonitor, then kill | aligned | +| init `{ok, S}` / `{ok, S, {continue, C}}` / `{error, R}` / `{stop, R}` / `ignore` | `Init::ready` / `continue_with` / `error` / `stop` / `ignore` | aligned; `ignore` is `Err(Ignored)` (T) | +| init crash → `{error, R}`, exit `R` | panic → `InitExited(Panic)` | aligned | +| `handle_continue` from `init` | `Init::continue_with` | aligned | +| `{noreply, S, {continue, C}}` from callbacks | not in 0.9 | gap: dispatch-path cost (open question 4) | +| failed `start_link`, trapping caller: `'EXIT'` consumed | consumed | aligned | +| failed `start_link`, non-trapping caller: killed by the link before returning | returns the error, then exits after the current handler | R: cooperative signals (L3) | +| failed `start_monitor`: `'DOWN'` consumed (`gen:monitor_return`) | consumed | aligned | +| start from a dead process | `ParentExited` | R: a cloned `Context` outlives its actor | +| `spawn_opt` `link`/`monitor` | the verbs | aligned | +| `spawn_opt` `max_heap_size` | reserved (rank 18a) | planned | +| `spawn_opt` `priority`, `fullsweep_after`, `min_heap_size`, `message_queue_data`; `hibernate_after` | none | not applicable (no per-process heap or priority scheduler) | +| `proc_lib:set_label` (OTP 27) | `label` at start | aligned (set once, at start) | +| one node, one scheduler pool: no placement | groups and backends; linked/monitored children inherit | R: no VM; groups are the scheduler analogue | +| `process_flag(trap_exit, true)` in `init` | `ctx.trap_exit(true)` in `started()` | aligned; no start option is needed (signals are held during `started()`, X3) | + +## 12. Migration from 0.8 + +| 0.8 | 0.9 | Behaviour | +|---|---|---| +| `a.start()` | `a.spawn()` | same | +| | or `a.start().await?` | waits for `started()`; reports init errors | +| `a.start_on(&g)?` | `a.with_options(&g).spawn()?` | same | +| | or `a.with_options(&g).start().await?` | waits | +| `a.start_with_backend(b)` | `a.with_options(b).spawn()?` | same (now a `Result`; it fails only on options) | +| `g.start(a)?` | `a.with_options(&g).spawn()?` | same | +| `a.start_linked(&ctx)` | `a.start_link(&ctx).await?` | **changes:** inherits the parent's placement, waits for `started()`, refuses an exiting parent | +| `ctx.monitor(&a.start().child_handle())` | `a.start_monitor(&ctx).await?` | atomic; no `NoProc` race | +| `ctx.register_name(n)` in `started()` | `with_options(StartOptions::new().name(n))` | atomic with start; released at D; all placements | +| `actor.ready().await` → `Result<(), SystemError>` (managed only) | `Result<(), StartError>`, every placement | error type | +| `SystemError::StartupFailed(r)` | `StartError::InitExited(r)` | | +| `ChildSpec::startup_timeout(d)` | `.options(StartOptions::new().init_timeout(d))` | same | +| `ChildSpec::{max_outstanding, max_native_jobs}` | `StartOptions::{max_outstanding, max_native_jobs}` | same; now also for unsupervised actors | +| `async fn started(..) {}` | unchanged | same; may now return `Result<(), E>` or `Init` | +| self-send from `started()` to finish setup | `Init::continue_with(msg)` | runs before any other message | +| `Mock::start()` / `Mock::start_on(&g)` | `Mock::spawn()` / `Mock::spawn_with(&g)` | same | +| `TestSystem::start(a)` / `start_mock(m)` | `TestSystem::spawn(a)` / `spawn_mock(m)` | same | + +In tasks mode a 0.8 statement `x.start();` still compiles and starts the +actor (with a `#[must_use]` warning, section 3). `let r = x.start();` used as +an `ActorRef` does not compile, which flags every site to migrate. The +removed functions are compile errors (trybuild cases in the test plan). + +## 13. Client mapping + +Every start call on the clients' `spawned-v0.8` branches is mapped in +[unified-start-client-mapping.md](unified-start-client-mapping.md): ethrex +`c1163b7`, ethlambda `85990aa`. + +- **Sites that change:** 51 (ethrex 37, ethlambda 14). All are mechanical, + and the behaviour-preserving rewrite keeps the current behaviour at every + one. +- **Inheritance changes behaviour at no site:** neither client calls + `start_linked`, and plain starts never inherit. +- **Two handler sites** start per-connection actors from inside an actor: + ethrex's RLPx initiator (via `PeerConnection::spawn_as_initiator`) and the + L2 proof coordinator. They stay unmanaged. Moving them to `start_link` or + `placement_of` is a client decision with a behaviour change: group + ownership, ordered shutdown and capacity. +- **Optional upgrade at 14 async sites** (8 in the L2 sequencer and the + based block fetcher, 4 in the prover, discovery, the monitor): + `start().await?` surfaces init failures, which today are only logged. +- **Six client functions return `SystemError` from `start_on`.** Their error + type becomes `StartError`, or they map it. +- **Supervisor specs** (3 `ChildSpec::new` sites): unchanged. They set no + `startup_timeout` or bounds. + +## 14. Semantics contract changes + +New or changed guarantees, with ids for `semantics.md`. They are written into +the contract by the implementation PR, as with otp-exit-signals. + +- **L1 (changed).** `stopped()` runs only after `started()` returned a ready + `Init`. A failed init (error, ignore, stop, panic) skips it. +- **L2 (changed).** Readiness exists for every placement. `ready()` and the + waiting starts resolve at the same barrier: `started()` returned ready with + no exit pending (and, for managed actors, admission still open). +- **L8 (new), Start.** Two stages as in section 3. `spawn` returns after + stage 1. `start`, `start_link` and `start_monitor` return after `started()`: + `Ok` only if ready, otherwise the error after the child's D. A stage-1 + refusal leaves no trace. +- **L9 (new), Init outcomes.** The table in 2.3. +- **L10 (new), Continuation first.** No mailbox message is dispatched before + the continuation. A stop before it discards it. +- **X5 (changed).** "`start_linked` registers the link before the child's + `started()`" becomes "`start_link` and `start_monitor` register their edge + before the child's `started()`, on every placement. A start through an + exiting or dead parent is refused (`ParentExited`)." +- **X3 (changed).** The parent row reads "`start_link` caller" for + "`start_linked` caller". +- **X8 (new), Start-time edges and failure.** On a failure while the starter + waits: a trapping parent's `Exit` and the `Down` are consumed, and a + non-trapping parent applies a non-`Normal` notice after its current handler. + An abandoned wait delivers the notice exactly once. The init timeout + removes the edge before the kill. +- **S6 (changed).** `spawn` and `start` are unmanaged unless given a + placement. `start_link` and `start_monitor` take the parent's. +- **S7 (new), Inheritance.** Section 4: the rule, no fallback, `Closed` and + `Capacity`. +- **S8 (new), Linked descendants are owned by the parent's scope.** + Shutdown order, V3 extension and kill escalation, as in section 4. +- **S9 (new), Start-time names.** Atomic, unique and released at D before any + observation, on every placement; `AlreadyStarted` on a clash. +- **T5 (new), Init timeout.** Section 7. +- **V1/V7 (changed).** `Init::ignore` leaves the spec without a generation + (`Ignored`) and is not a failure. A name clash at a restart is a failed + attempt. +- **Section 11 (changed).** New rows: + - start synchronous on init: aligned in 0.9; + - fallible init and continue: aligned; + - failed `start_link` with a non-trapping caller: R; + - continue from handlers: gap. + +**Non-guarantees** (added to section 10): + +- The init timeout cannot interrupt `started()`. The killed child holds its + name and slot until `started()` returns. +- Inheritance does not follow work outside the start call: `tokio::spawn`, + helpers and plain `start`/`spawn` from a handler stay unmanaged. +- The continuation exists only after `started()`. +- An abandoned start does not consume the child's failure notice. +- Names are process-global until rank 12. + +## 15. The model + +[`Start.tla`](../../verification/tla/Start.tla) models: + +- one parent (managed in group `G` or unmanaged, trapping or not, which may + exit `Normal` or `Panic`); +- children started with `spawn`, `start`, `start_link` or `start_monitor`, + with an explicit placement (none, `G`, unmanaged, `placement_of`) and + possibly one shared name; +- `G`, which may close and admits at most `Cap` live actors. + +Stage 1 is four separate steps, so parent death, group closure and a +competing registration interleave between them. `started()` is one step with +every `Init` outcome. Signals are held until it returns. Others may send +messages, which a continuation must precede. A waiting starter can time out +or abandon the wait. + +**Invariants:** + +| Invariant | Meaning | +|---|---| +| `InheritedPlacement` | explicit placement wins; link/monitor take the parent's; otherwise unmanaged | +| `LinkBeforeStarted` | the link or monitor exists before `started()` begins | +| `NameUnique` | at most one live holder of a name | +| `NameReleasedAtDeath` | no holder past D or aborted | +| `AbortInvisible` | a refused start: no edge, name, slot, hook or notice | +| `AckMeansStarted` | `Ok` from a waiting start means `started()` returned | +| `FailureAfterDeath` | the init error is returned after the child's D | +| `FailureFlushed` | a failed or timed-out `start_monitor` delivers no `Down`; `start_link` no `Exit` to a trapping parent | +| `TimeoutContained` | an init-timeout kill never reaches the parent | +| `NoEdgeToDeadParent` | no edge survives the parent's D | +| `AdmittedOnlyOpen`, `CapacityRespected` | group admission | +| `NoticeOnce` | at most one `Exit` and one `Down` per child | +| `ContinueFirst` | no message before the continuation | +| `AbandonedNoticeDelivered` | an abandoned wait hands a diverted notice to the parent | + +Liveness (`MCStartLive`, weak fairness on the child's steps and the parent's +D): `ParentExitReachesChild`. A child linked to a parent that dies with a +non-`Normal` reason, also during its `started()`, dies (or was refused). + +**Results** (TLC 2.19, fede, 4 workers, `nice -n 19` on a loaded host): + +| Configuration | Run | Distinct states | Depth | Time | +|---|---|---|---|---| +| `MCStart` | per PR, safety | 2,896,748 | 24 | 23 s | +| `MCStartContinue` | per PR, safety | 17,720 | 16 | 1 s | +| `MCStartLive` | per PR, safety + liveness | 424,172 | 23 | 20 s | +| `MCStartDeep` | weekly (every explicit placement, `spawn`, messages) | see below | | | + +MCStartDeep is reported in the PR description once its run completes. + +**Negative controls.** `negative_checks.py` runs `named_fault` with +`TypeOK` and only the named invariant, and `named_temporal` for the +liveness control. Each control below produced its counterexample (1-7 s +each, one at a time): + +| Fault | Reintroduces | Fails | +|---|---|---| +| `NoInherit` | 0.8: a linked/monitored child of a managed parent is unmanaged | `InheritedPlacement` | +| `LinkAfterStarted` | the link is set once `started()` runs | `LinkBeforeStarted` | +| `DoubleRegister` | check-then-insert registration | `NameUnique` | +| `LateUnregister` | the name outlives D | `NameReleasedAtDeath` | +| `AbortLeaksLink` | a refused start keeps its edge | `AbortInvisible` | +| `EarlyFailAck` | the error returns before the child's D | `FailureAfterDeath` | +| `AckBeforeStarted` | `Ok` before `started()` returned | `AckMeansStarted` | +| `NoMonitorFlush` | a failed `start_monitor` also delivers `Down` | `FailureFlushed` | +| `NoLinkFlush` | a failed `start_link` also delivers `Exit` | `FailureFlushed` | +| `KillBeforeUnlink` | the timeout kills while linked | `TimeoutContained` | +| `LinkCheckLate` | the link ignores the parent's state | `NoEdgeToDeadParent` | +| `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | +| `IgnoreCap` | a full group admits | `CapacityRespected` | +| `AbandonDropsNotice` | an abandoned wait loses the diverted notice | `AbandonedNoticeDelivered` | +| `ContinueQueued` (`MCStartContinue`) | the continuation is queued behind earlier messages | `ContinueFirst` | +| `DropHeld` (`MCStartLive`, liveness) | a parent exit held during `started()` is dropped | `ParentExitReachesChild` | + +**Not modelled:** + +- the child trapping exits (E3's trapping-child rules are in `Lifecycle.tla`); +- more than one parent; +- the system's ordered shutdown of linked descendants (S8). That extends + `Coordinator.tla` and `NestedSupervision.tla` in the implementation PR; +- time; +- the init-timeout timer without a waiting starter. + +## 16. Test plan + +The implementation is held to this plan. Every guarantee in section 14 has a +check at each layer that applies, and each check must fail when the +guarantee breaks. + +### 16.1 Per-guarantee matrix + +Columns: TLA+ invariant (and trace validation); integration tests (both +modes); simulator (`#[sim_test]`, PCT and bounded DFS); fuzzer oracle; +trybuild. + +| Guarantee | TLA+ / trace | Integration (`concurrency/tests/start_contract.rs`) | Sim (`start_sim.rs`) | Fuzz oracle | trybuild | +|---|---|---|---|---|---| +| L8 two stages, ack | `AckMeansStarted`, `FailureAfterDeath` | `{spawn,start,start_link,start_monitor}_returns_{immediately,after_started}`; `start_without_await_still_starts` | `ack_races_child_death` | ack `Ok` ⇒ hook log shows `started()` returned | `start_result_is_not_an_actor_ref` | +| L8 refusal invisible | `AbortInvisible` | `refused_start_runs_no_hook_and_leaves_no_edge_or_name` (closed, capacity, clash, parent exited) | `refusal_races_parent_death` | aborted ids never appear in `Exit`/`Down`/registry | | +| L9 Init outcomes | (model abstracts the outcomes to their exits) | one test per row of the 2.3 table, each mode and placement, for `start`, `start_link`, `start_monitor`, `spawn`+`ready()` | | `started()` outcomes generated; oracle checks the reported `StartError` against the outcome | `started_may_return_unit_result_or_init` (ui-pass); `continue_with_requires_handler` | +| L1 `stopped()` skipped on failed init | | `failed_init_skips_stopped` (every outcome) | | existing L5 oracle extended | | +| L2 readiness for every placement | `AckMeansStarted` | `ready_resolves_for_unmanaged_and_threads` | | | | +| L10 continuation first | `ContinueFirst` | `continuation_runs_before_messages_sent_during_started`; `stop_before_continuation_discards_it` | `continuation_races_peer_send` | hook-log order | | +| X5 edge before `started()` | `LinkBeforeStarted` | `start_link_child_panicking_in_started_notifies_parent` on every placement; same for `start_monitor` | `child_dies_in_started_racing_link` | link/monitor atomicity: a child failing in `started()` always produces the parent's notice or the consumed error | | +| X5 parent exited | `NoEdgeToDeadParent` | `start_link_through_dead_context_is_refused` | `parent_stops_during_child_started` | | | +| X8 failure flush | `FailureFlushed` | `failed_start_link_consumes_exit_of_trapping_parent`; `failed_start_link_kills_nontrapping_parent_after_handler`; `failed_start_monitor_consumes_down` | `failure_races_ack` | no `Exit`/`Down` for a start that returned an init error | | +| X8 abandoned wait | `AbandonedNoticeDelivered` | `abandoned_start_monitor_still_delivers_down`; `abandoned_start_link_delivers_exit` | `abandon_races_child_death` | | | +| T5 init timeout | `TimeoutContained` | `init_timeout_returns_after_death_and_spares_parent` (each kind); `spawn_init_timeout_kills_and_ready_reports_it`; `stuck_started_holds_name_until_return` | `timeout_races_started_return` | timeout options generated; parent never ends `Killed` from a timed-out child | | +| S7 inheritance | `InheritedPlacement` | `link_and_monitor_inherit_{async_group,thread_group,pinned_shard,backend_async,backend_blocking,backend_thread}`; `plain_start_never_inherits`; `placement_of_opts_in` | | placement oracle: the child's `execution_group()` name (or backend) matches the rule | `threads_options_have_no_group_or_backend` | +| S7 closed or full group | `AdmittedOnlyOpen`, `CapacityRespected` | `inherited_start_into_closing_group_fails_closed`; `inherited_start_into_full_thread_group_fails_capacity` | `group_shutdown_during_start` | existing S1 oracle | | +| S8 linked descendants owned | (Coordinator/Nested extension in the impl PR) | `system_shutdown_stops_linked_children_with_their_parent`; `supervised_parent_generation_waits_for_linked_children`; `unlinked_descendant_is_killed_after_one_interval` | `parent_shutdown_racing_child_start` | existing S1-S4 and V3 oracles see linked children | | +| S9 names | `NameUnique`, `NameReleasedAtDeath` | `name_clash_returns_already_started_with_holder`; `failed_named_start_can_be_retried_at_once`; `name_released_before_down_and_exit`; `typed_name_whereis` | `name_registration_race` | at most one live holder; no name of a dead actor after its `Down` | `name_type_must_match_actor` | +| V1/V7 ignore, clash at restart | (Supervision model: `Ignored` state in the impl PR) | `supervised_ignore_keeps_spec_without_generation`; `temporary_ignore_is_removed`; `restart_name_clash_is_a_failed_attempt` | | V1 oracle treats `Ignored` | | +| API removal | | | | | `start_on_removed`, `start_with_backend_removed`, `start_linked_removed`, `execution_group_start_removed`, `start_options_struct_literal` (private/`non_exhaustive`), `start_monitor_requires_down_handler` | + +**Placements.** Every integration test runs in: + +- tasks: `Backend::Async`, `Blocking`, `Thread`, `async_group`, + `thread_group` (and `pinned_group` on Linux); +- threads mode, where the API exists there (all except the placement rows). + +They use a per-placement macro, as in `semantics_gaps`. "Start from outside +any actor" is the plain `spawn`/`start` rows run from the test body. + +### 16.2 TLA+ and trace validation + +- **Model.** `Start.tla`, per PR: `MCStart`, `MCStartContinue`, + `MCStartLive` (the CI leg `start`). Weekly: `MCStartDeep`. The 16 negative + controls are in `negative_checks.py` (`START_CONTROLS`, + `START_TEMPORAL_CONTROLS`). +- **Trace schema `start-v1`.** The `lifecycle-trace` feature records these + events from the real code: + - `SBegin(c, kind, explicit, named)`, `SAdmit(c, place | refusal)`, + `SLink(c)`, `SMonitor(c)`, `SRegister(c | already)`, `SDispatch(c)`; + - `SStartedBegin(c)`, `SStartedEnd(c, outcome)`, `SAck(c, result)`; + - `STimeout(c)`, `SAbandon(c)`, `SDie(c, reason)`; + - `PDie(reason)`, `GClose`, `SSend(c)`, `SHandle(c)`, `SContinue(c)`. + + `TraceStart.tla` replays them as a behaviour of `Start.tla` (`trace_check.py` + gains the schema). A new `concurrency/tests/start_trace.rs` records every + integration scenario of 16.1 on Async, async groups and thread groups. +- **Trace negative controls** (`negative_checks.py`, `require_rejected`): + - move `SLink` after `SStartedBegin`; + - give two children the same `SRegister`; + - change an `SAdmit` place to unmanaged under a managed parent; + - deliver an `Exit` after a failed `SAck`. +- **Trace coverage.** `trace_coverage.py` adds a `StartCoverage` projection; + the baseline requires every `Start.tla` action to be reached by a recorded + trace. + +### 16.3 Simulator + +`concurrency/tests/start_sim.rs`, each case under PCT (depths 1-3) and +bounded DFS, with seeds and tapes: + +| Race | Check | +|---|---| +| parent stops during the child's `started()` | the child ends with the parent's reason, or `Normal` ignored | +| link or monitor racing the child's death in `started()` | exactly one of: consumed error, or delivered notice (abandoned) | +| two starts with one name | exactly one `Ok`, one `AlreadyStarted` | +| group shutdown during a start | either admitted and owned by shutdown, or `Closed`; never unowned | +| init timeout racing `started()` return | `Ok` xor `InitTimeout`; the parent is never hit | +| abandon racing the child's death | the notice is delivered exactly once | +| continuation racing a peer's send during `started()` | the continuation always runs first | + +### 16.4 Scenario fuzzer (`fuzz/`) + +- **Generator.** `SpawnHow` becomes + `Start { verb: Spawn | Start | Link | Monitor, placement: Default | Group(g) | Unmanaged | SameAsParent, name: Option, init_timeout: Option, abandon: bool }`. + Worker hooks gain `started()` outcomes: ready, continue, error, ignore, + stop with a reason, panic, sleep. +- **New oracles:** + - placement: the inheritance rule; + - link/monitor atomicity and failure flush; + - names unique and released before any death observation; + - ack `Ok` ⇒ `started()` returned; + - continuation first. +- **Extended oracles:** the existing M1/L1/L5 and X1/X3/X5 oracles learn + that a consumed notice is expected. + +### 16.5 Planted bugs + +Added to `verification/planted/patches/` (the #247 matrix). Each must be +caught by at least two layers. + +| Patch | Bug | Must be caught by | +|---|---|---| +| `NN-start-link-ignores-parent-group` | `start_link` admits the child unmanaged when the parent is managed (the 0.8 behaviour) | `link_and_monitor_inherit_*`, fuzz placement oracle, trace `SAdmit` check (`InheritedPlacement`) | +| `NN-start-link-after-dispatch` | the edge is recorded after the child is dispatched | `start_sim::child_dies_in_started_racing_link` (deterministic; the integration test `start_link_child_panicking_in_started_notifies_parent` catches it only on some schedules), trace `LinkBeforeStarted` | +| `NN-start-name-check-then-insert` | the name is checked under a read lock and inserted under a separate write lock | `start_sim::name_registration_race`, fuzz name oracle | +| `NN-start-monitor-down-lost-on-abandon` | a diverted `Down` is dropped when the `Starting` is dropped | `abandoned_start_monitor_still_delivers_down`, `start_sim::abandon_races_child_death` | +| `NN-start-name-released-at-publication` | the name is released at publication instead of D | `failed_named_start_can_be_retried_at_once`, `name_released_before_down_and_exit`, trace `NameReleasedAtDeath` | + +### 16.6 Coverage matrix + +Each new or changed guarantee in section 14 gets: + +- a "Checked by" line in `semantics.md` naming the checks above; +- a row in `docs/TESTING.md` "Semantics coverage" (modes, placements, + covered or partial). + +New rows: L8, L9, L10, X8, S7, S8, S9, T5. Changed rows: L1, L2, X3, X5, S6, +V1, V7. A guarantee is merged as covered only when each clause has a failing +check under its planted bug or model control. + +### 16.7 Client checks + +The client-compat gate (`.github/workflows/client-compat.yml`) runs on the +implementation PR, against client branches that carry the mapped rewrite +(`spawned-v0.9`, created from `spawned-v0.8`): + +- **ethrex:** check, clippy `-D warnings`, `ethrex-p2p` tests, the `p2p::` + and `rpc::` groups, and the lib tests. +- **ethlambda:** clippy and `cargo test --workspace --profile release-fast`. + +Clippy `-D warnings` also turns any leftover un-awaited `start()` into an +error. + +## 17. Alternatives considered + +| Alternative | Why not | +|---|---| +| Keep `start()` immediate and add `ready()` (no waiting starts) | Startup failure stays invisible by default (G1). The init timeout and failure flush have no starter to act for, so a timed-out linked child would kill a non-trapping parent with `Killed`, unlike OTP's `kill_flush_EXIT` | +| Waiting starts with a lazy stage 1 (nothing happens until awaited) | A forgotten `.await` would silently never start the actor (a warning only), across 51 migrated client sites. The eager stage 1 makes that mistake equal `spawn` | +| Ambient inheritance for plain `start`/`spawn` (task- or thread-local current actor) | A cost per turn; it does not follow `tokio::spawn` or helpers; it silently moves 20 ethrex call sites into groups; OTP's plain `start` has no parent either | +| Fall back to unmanaged when the inherited group is closed or full | It re-creates the footgun at the worst moment (shutdown) | +| One `start(opts)` taking options always (`start(StartOptions::default())`) | Noise at the common call; the `with_options` builder keeps the zero-option verbs short and gives one options path | +| Six or eight `_with` methods (`start_with`, `start_link_with`, ..) | Doubles the surface; `with_options(o).verb()` is one entry point | +| Typed init data (`started() -> Init`, Gleam's `returning`) | The starter needs the type `R` statically, which needs an associated type on `Actor`, and stable Rust has no associated-type defaults, so every manual `impl Actor` would break. The main use (handing back a sub-handle) is covered by the typed `ActorRef` and `Recipient`s derived from it. `Starting` is a struct, so a later `data()` accessor stays additive | +| Typed init error (`InitFailed(E)`) | Same associated-type problem; `Display` text matches supervisors' factory errors and Gleam's `InitFailed(String)` | +| `trap_exit` as a start option | No window to close: signals are held during `started()` and resolved with the flag read when it returns (X3). One way to set it stays | +| `#[actor]` derives a child spec (Elixir `child_spec/1`, Gleam `supervised()`) | A spec needs a factory for fresh state per generation, which the type alone cannot provide. With `ChildSpec::new(id, &g, f).options(o)` the derive would save nothing. `ChildSpec` keeps room for `significant` | +| `spawn_link` / `spawn_monitor` (no waiting) | OTP's `gen_server` has none; slow initialisation belongs in a continuation, so a waiting `start_link` returns fast | +| Continuation from every handler | A check after every turn is a cost on the dispatch path; post-init is the case clients need (#245) | +| `init_timeout` default 5 s | OTP's default is `infinity`; it would kill client actors whose `started()` does slow I/O; `ChildSpec` keeps its 5 s bound | +| Keep `start_on` as sugar for `with_options(g).spawn()` | Two spellings again (`ExecutionGroup::start` was the same mistake) | + +## 18. Gap-report input: accepted and rejected + +| Suggestion | Decision | +|---|---| +| Fallible init on every start path, `InitTimeout`/`InitFailed`/`InitExited`, non-waiting `spawn` | accepted (sections 2.3, 2.5, 3) | +| `Init` carrying data for the starter | rejected for 0.9 (section 17), room kept | +| `Init` continuation, `Init::ignore` | accepted (sections 8, 9) | +| `start_monitor`; link and monitor on every backend and group | accepted | +| Options: placement, label, typed name, init timeout, stack size; reserve mailbox, memory, priority | accepted; default timeout `infinity`, not 5 s (section 7) | +| `trap_exit` at start | rejected (no window; section 17) | +| `spawn_opt` GC and priority knobs | not copied (section 2.4) | +| `#[actor]` derives the child spec; room for `significant` | derive rejected (section 17); room kept | +| Builder shape `A::new(..).spawn_opts(opts).start()` | adopted as `with_options(opts).start()` | + +## 19. Open questions for the reviewers + +1. **Should a waiting `start_link` from a parent's turn block that turn?** + This is OTP, and the continuation keeps it short. A `started()` that + requests its parent deadlocks until the request timeout, as in OTP. +2. **S8 kill escalation for `start_link` descendants:** one cleanup interval + after the parent's D, then stuck. Is tying a supervised generation's end + to its linked descendants (V3) acceptable for ethrex-style per-connection + children? +3. **`init_timeout` default:** none (OTP) or 5 s (bound)? +4. **Continuation from ordinary handlers** for 0.9, at a one-branch dispatch + cost, or later? +5. **`AlreadyStarted.existing: Option`** against a typed + `Option>`. The typed form makes `StartError` generic. +6. **Moving `ChildSpec::{startup_timeout, max_*}` into `StartOptions`:** + neither client uses them, but the dynamic supervisor (#245) should adopt + the same shape in its review. +7. **`Starting::actor_ref()`** exposes the reference before readiness. Is it + needed beyond `spawn`? diff --git a/verification/tla/README.md b/verification/tla/README.md index 74b81417..2467152d 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -17,6 +17,7 @@ this fits with Loom, Shuttle, Miri, ThreadSanitizer and mutation testing. | `DeferredReplies.tla`, `MCDeferredReplies*.tla/.cfg` | Rank 11 design model (not implemented), part 1: reply slots, send-time deadlines with shortening, deferral credits shared by a token and its first linked job, the exiting phase with unwind-local drops, a stuck phase (cleanup deadline passed while `stopped()` runs: nothing closed early, death later or never), death, and downstream request cleanup. Thirteen named negative controls. | | `DeferredJobs.tla`, `MCDeferredJobs*.tla/.cfg` | Rank 11 design model, part 2: pipe jobs with separate execution, cancellation and notification states; keyed replacement including notified jobs; optional job deadlines; a FIFO mailbox with emit credits and the terminal message last; `ActorId` fencing. `MCDeferredJobsEmits` is the fast configuration with emits. Nine named negative controls. | | `Lifecycle.tla`, `MCLifecycle{,Live,Fence,Deep,LiveDeep,WideDeep}.tla/.cfg` | Base actor lifecycle as proposed for v0.8 ([design](../../docs/design/otp-exit-signals.md)): links, monitors, `trap_exit`, exit signals (`Killed`, ignored `Normal`, `NoProc`, parent exits), administrative stops, requests around death, `Exit`/`Down` ordering. Seventeen negative-control faults (17 safety checks, 2 liveness checks): the v0.7 exit-signal behaviours (direct stops ignoring `trap_exit`, kills ending `Kill`, unfenced `unlink`/`demonitor`, real reasons for late registrations, `stopped()` after kills and untrapped signals, late signals ignored during exit), the PR #182/#185 bugs, early exit errors, kill and panic precedence, runtime-drop reasons, early notification, a split death point and a parent exit reaching `exit_received`. Per PR: `MCLifecycle` (7.5M states, fewer reasons, where the safety controls run), `MCLifecycleFence`, `MCLifecycleLive`; weekly: `MCLifecycleWideDeep` (the former 43M-state per-PR instance), `MCLifecycleDeep`, `MCLifecycleLiveDeep`. | +| `Start.tla`, `MCStart{,Continue,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design ([design](../../docs/design/unified-start.md), not implemented): `spawn`, `start`, `start_link`, `start_monitor`, with placement inheritance, an atomic link or monitor before `started()`, atomic names, fallible init, the init timeout, a continuation run before any mailbox message, abandoned waits, and parent death and group closure between the steps of stage 1. Sixteen negative controls (15 safety, 1 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartContinue`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | | `TraceCohorts.tla`, `TraceNested.tla` | Policy and ownership projections of real cohort/nested Rust executions. | | `trace_check.py`, `negative_checks.py` | Replay recorded traces; deliberately wrong selections, premature resume/join and broken propagation must be rejected or produce counterexamples. | | `SupervisionInd.tla`, `CohortsInd.tla`, `NestedSupervisionInd.tla`, `CoordinatorInd.tla`, `MC*Ind.tla`, `apalache.sh` | Inductive invariants (`IndInv`) checked with Apalache; see [Inductive invariants](#inductive-invariants-apalache). | @@ -36,7 +37,7 @@ nix develop -c tlc MCSupervision -workers auto # ad hoc, from verification/tla **CI.** On every PR (`.github/workflows/verification.yml`) the per-PR configurations run as parallel `tla-models` legs (`supervision`, `lifecycle`, -`deferred`, and `other` for every configuration no leg names), next to +`deferred`, `start`, and `other` for every configuration no leg names), next to `tla-traces` (trace validation, negative controls, trace coverage). Weekly and on dispatch, `tla-deep` checks each `*Deep.cfg` on its own runner. The required check `tla` passes only if all of them passed. @@ -580,3 +581,66 @@ Measured 2026-11-10 (TLC 2.19, fede): | `MCDeferredJobsDeep` (3 jobs, 2 instances, 1 emit each; safety) | 5,876,658 | 12,792,039 | 23 | 25 s, 12 workers | All 22 named controls find their counterexample, each in a few seconds. + +## Unified start (0.9 design) + +`Start.tla` backs the [unified start design](../../docs/design/unified-start.md), +written before the implementation, as the deferred-replies models were. The +design's section 15 lists what it covers and what it leaves out. + +**What is modelled.** + +- One parent: managed in group `G` or unmanaged, trapping or not, and able + to exit. +- Children started with `spawn`, `start`, `start_link` or `start_monitor`, + with an explicit or default placement and possibly one shared name. +- Stage 1 as four steps (admit, link/monitor, register, dispatch). Parent + death, group closure and a competing registration can interleave between + any two of them. +- `started()` as one step with every `Init` outcome. Signals are held until + it returns. +- Messages from others that a continuation must precede. +- A waiting starter that acknowledges, fails, times out or abandons the wait. + +**Safety invariants:** + +- `InheritedPlacement`, `LinkBeforeStarted`; +- `NameUnique`, `NameReleasedAtDeath`; +- `AbortInvisible`, `AckMeansStarted`, `FailureAfterDeath`, `FailureFlushed`; +- `TimeoutContained`, `NoEdgeToDeadParent`; +- `AdmittedOnlyOpen`, `CapacityRespected`, `NoticeOnce`; +- `ContinueFirst`, `AbandonedNoticeDelivered`. + +Liveness: `ParentExitReachesChild` (`MCStartLive`). + +| Configuration | Run | Instance | +|---|---|---| +| `MCStart` | per PR | two children, every waiting variant, default placements, one name, timeouts, a closing group with room for two | +| `MCStartContinue` | per PR | one child of every variant, two messages from others, no name | +| `MCStartLive` | per PR | two linked or monitored children of a managed parent, liveness | +| `MCStartDeep` | weekly | as `MCStart`, plus every explicit placement, `spawn` and messages | + +**Negative controls** (`START_CONTROLS` and `START_TEMPORAL_CONTROLS` in +`negative_checks.py`; each checks `TypeOK` and only the named property): + +| Fault | Breaks | +|---|---| +| `NoInherit` (the 0.8 `start_linked` placement) | `InheritedPlacement` | +| `LinkAfterStarted` | `LinkBeforeStarted` | +| `DoubleRegister` | `NameUnique` | +| `LateUnregister` | `NameReleasedAtDeath` | +| `AbortLeaksLink` | `AbortInvisible` | +| `EarlyFailAck` | `FailureAfterDeath` | +| `AckBeforeStarted` | `AckMeansStarted` | +| `NoMonitorFlush`, `NoLinkFlush` | `FailureFlushed` | +| `KillBeforeUnlink` | `TimeoutContained` | +| `LinkCheckLate` | `NoEdgeToDeadParent` | +| `AdmitClosed` | `AdmittedOnlyOpen` | +| `IgnoreCap` | `CapacityRespected` | +| `AbandonDropsNotice` | `AbandonedNoticeDelivered` | +| `ContinueQueued` (`MCStartContinue`) | `ContinueFirst` | +| `DropHeld` (`MCStartLive`, liveness) | `ParentExitReachesChild` | + +The trace schema `start-v1` and `TraceStart.tla` (real start, link, monitor, +name and placement events replayed against this model) belong to the +implementation PR, as the verification rule requires. From 49b565bc2080c0f59801ba36c316522fe4f91ebb Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 14:21:06 -0300 Subject: [PATCH 09/42] verification: MCStartDeep without messages (MCStartContinue covers them) --- verification/tla/MCStartDeep.cfg | 2 +- verification/tla/MCStartDeep.tla | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index 1dbacc4c..d96fa70c 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -8,7 +8,7 @@ CONSTANTS Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 1 + MaxMsgs = 0 CloseEnabled = TRUE Fault = "None" INVARIANT TypeOK diff --git a/verification/tla/MCStartDeep.tla b/verification/tla/MCStartDeep.tla index 7184bfe9..3c629f54 100644 --- a/verification/tla/MCStartDeep.tla +++ b/verification/tla/MCStartDeep.tla @@ -1,5 +1,7 @@ ---------------------------- MODULE MCStartDeep ---------------------------- EXTENDS Start \* Weekly safety instance: every explicit placement (a group, unmanaged, -\* the parent's via placement_of) for every start variant. +\* the parent's via placement_of) for every start variant, spawn included. +\* Messages from others are checked in MCStartContinue (with them this +\* instance passed 74M distinct states unfinished). ============================================================================= From b915c37f478942bf69c5f2a6907027a5af414f80 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 14:27:51 -0300 Subject: [PATCH 10/42] docs: MCStartDeep result --- docs/design/unified-start.md | 5 +++-- verification/tla/README.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index a848ce11..20d4cbf2 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -753,9 +753,10 @@ non-`Normal` reason, also during its `started()`, dies (or was refused). | `MCStart` | per PR, safety | 2,896,748 | 24 | 23 s | | `MCStartContinue` | per PR, safety | 17,720 | 16 | 1 s | | `MCStartLive` | per PR, safety + liveness | 424,172 | 23 | 20 s | -| `MCStartDeep` | weekly (every explicit placement, `spawn`, messages) | see below | | | +| `MCStartDeep` | weekly (every explicit placement, every variant including `spawn`) | 56,947,732 | 24 | 6 min 23 s | -MCStartDeep is reported in the PR description once its run completes. +`MCStartDeep` with messages from others as well passed 74M distinct states +with its queue still growing, so messages stay in `MCStartContinue`. **Negative controls.** `negative_checks.py` runs `named_fault` with `TypeOK` and only the named invariant, and `named_temporal` for the diff --git a/verification/tla/README.md b/verification/tla/README.md index 2467152d..890d3768 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -618,7 +618,7 @@ Liveness: `ParentExitReachesChild` (`MCStartLive`). | `MCStart` | per PR | two children, every waiting variant, default placements, one name, timeouts, a closing group with room for two | | `MCStartContinue` | per PR | one child of every variant, two messages from others, no name | | `MCStartLive` | per PR | two linked or monitored children of a managed parent, liveness | -| `MCStartDeep` | weekly | as `MCStart`, plus every explicit placement, `spawn` and messages | +| `MCStartDeep` | weekly | as `MCStart`, plus every explicit placement and `spawn` (57M states, about 6 minutes on 4 workers) | **Negative controls** (`START_CONTROLS` and `START_TEMPORAL_CONTROLS` in `negative_checks.py`; each checks `TypeOK` and only the named property): From 8c88e1ca7a093257807b803ce77c9cc10377dacb Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 14:58:44 -0300 Subject: [PATCH 11/42] verification: Start.tla revision 2 (startup cell, atomic commit, D/publication, S8 scope) --- verification/tla/MCStart.cfg | 21 +- verification/tla/MCStart.tla | 12 +- verification/tla/MCStartContinue.cfg | 21 +- verification/tla/MCStartContinue.tla | 7 +- verification/tla/MCStartDeep.cfg | 21 +- verification/tla/MCStartDeep.tla | 6 +- verification/tla/MCStartLive.cfg | 28 +- verification/tla/MCStartLive.tla | 5 +- verification/tla/MCStartScope.cfg | 39 ++ verification/tla/MCStartScope.tla | 7 + verification/tla/Start.tla | 975 +++++++++++++++------------ 11 files changed, 652 insertions(+), 490 deletions(-) create mode 100644 verification/tla/MCStartScope.cfg create mode 100644 verification/tla/MCStartScope.tla diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 88a675fb..786e04af 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -4,26 +4,35 @@ CONSTANTS Kinds = {"start", "link", "monitor"} Explicit = {"none"} ParentPlaces = {"G", "U"} - Cap = 2 + Cap = 3 Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 0 CloseEnabled = TRUE + MaxMsgs = 0 + ChildCanTrap = FALSE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE Fault = "None" INVARIANT TypeOK INVARIANT InheritedPlacement INVARIANT LinkBeforeStarted INVARIANT NameUnique -INVARIANT NameReleasedAtDeath +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation INVARIANT AbortInvisible -INVARIANT AckMeansStarted -INVARIANT FailureAfterDeath +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication INVARIANT FailureFlushed INVARIANT TimeoutContained INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce -INVARIANT ContinueFirst INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index b3f5bac1..52badb3d 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -1,10 +1,8 @@ ---------------------------- MODULE MCStart ---------------------------- EXTENDS Start -\* Per-PR safety instance of the unified start model (design -\* docs/design/unified-start.md): two children racing for one name, every -\* start variant with its default (inherited or unmanaged) placement, a managed or -\* unmanaged parent that may exit, a closing group with room for two actors, -\* startup timeouts. Every Start safety control except ContinueQueued -\* (MCStartContinue) fails here on its named invariant. MCStartDeep (weekly) -\* adds every explicit placement, spawn, and messages from others. +\* Per-PR safety instance (design docs/design/unified-start.md, section 15): +\* two children racing for one name in a group with room for both and the +\* parent, every waiting variant with its default placement, a managed or +\* unmanaged parent that may exit, group closure, init timeouts, abandoned +\* waits, failure and retry. Most safety controls run here. ============================================================================= diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg index 9943ed24..0d2a2bb0 100644 --- a/verification/tla/MCStartContinue.cfg +++ b/verification/tla/MCStartContinue.cfg @@ -2,28 +2,37 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1"} Kinds = {"spawn", "start", "link", "monitor"} - Explicit = {"none"} + Explicit = {"none", "G", "U", "same"} ParentPlaces = {"G", "U"} Cap = 2 Named = FALSE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 2 CloseEnabled = TRUE + MaxMsgs = 2 + ChildCanTrap = TRUE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE Fault = "None" INVARIANT TypeOK INVARIANT InheritedPlacement INVARIANT LinkBeforeStarted INVARIANT NameUnique -INVARIANT NameReleasedAtDeath +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation INVARIANT AbortInvisible -INVARIANT AckMeansStarted -INVARIANT FailureAfterDeath +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication INVARIANT FailureFlushed INVARIANT TimeoutContained INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce -INVARIANT ContinueFirst INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone diff --git a/verification/tla/MCStartContinue.tla b/verification/tla/MCStartContinue.tla index 8aca97db..15bd4d4c 100644 --- a/verification/tla/MCStartContinue.tla +++ b/verification/tla/MCStartContinue.tla @@ -1,6 +1,7 @@ ---------------------------- MODULE MCStartContinue ---------------------------- EXTENDS Start -\* Per-PR instance for the continuation (Init::continue_with): one child of -\* every variant, two messages from others racing started(), the parent's -\* exit and a startup timeout. ContinueFirst and its control run here. +\* Per-PR instance for the continuation and trapping children: one child +\* of every variant with every explicit placement, two messages from others, +\* a child that may trap exits (a held parent exit is then queued behind the +\* continuation, E3). ContinueFirst and its control run here. ============================================================================= diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index d96fa70c..5b074282 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -4,26 +4,35 @@ CONSTANTS Kinds = {"spawn", "start", "link", "monitor"} Explicit = {"none", "G", "U", "same"} ParentPlaces = {"G", "U"} - Cap = 2 + Cap = 3 Named = TRUE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 0 CloseEnabled = TRUE + MaxMsgs = 0 + ChildCanTrap = TRUE + UnlinkEnabled = TRUE + ScopeEnabled = TRUE Fault = "None" INVARIANT TypeOK INVARIANT InheritedPlacement INVARIANT LinkBeforeStarted INVARIANT NameUnique -INVARIANT NameReleasedAtDeath +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation INVARIANT AbortInvisible -INVARIANT AckMeansStarted -INVARIANT FailureAfterDeath +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication INVARIANT FailureFlushed INVARIANT TimeoutContained INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce -INVARIANT ContinueFirst INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone diff --git a/verification/tla/MCStartDeep.tla b/verification/tla/MCStartDeep.tla index 3c629f54..bdb15cae 100644 --- a/verification/tla/MCStartDeep.tla +++ b/verification/tla/MCStartDeep.tla @@ -1,7 +1,5 @@ ---------------------------- MODULE MCStartDeep ---------------------------- EXTENDS Start -\* Weekly safety instance: every explicit placement (a group, unmanaged, -\* the parent's via placement_of) for every start variant, spawn included. -\* Messages from others are checked in MCStartContinue (with them this -\* instance passed 74M distinct states unfinished). +\* Weekly safety instance: two children of every variant with every +\* explicit placement, trapping, unlinking and the S8 scope. ============================================================================= diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index c53d4619..6bda12ee 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -1,30 +1,40 @@ SPECIFICATION LiveSpec CONSTANTS - Kids = {"c1", "c2"} - Kinds = {"link", "monitor"} + Kids = {"c1"} + Kinds = {"start", "link", "monitor"} Explicit = {"none"} ParentPlaces = {"G"} - Cap = 3 - Named = TRUE + Cap = 2 + Named = FALSE TimeoutEnabled = TRUE ParentCanExit = TRUE - MaxMsgs = 0 CloseEnabled = FALSE + MaxMsgs = 0 + ChildCanTrap = TRUE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE Fault = "None" INVARIANT TypeOK INVARIANT InheritedPlacement INVARIANT LinkBeforeStarted INVARIANT NameUnique -INVARIANT NameReleasedAtDeath +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation INVARIANT AbortInvisible -INVARIANT AckMeansStarted -INVARIANT FailureAfterDeath +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication INVARIANT FailureFlushed INVARIANT TimeoutContained INVARIANT NoEdgeToDeadParent INVARIANT AdmittedOnlyOpen INVARIANT CapacityRespected INVARIANT NoticeOnce -INVARIANT ContinueFirst INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone PROPERTY ParentExitReachesChild +PROPERTY StartAnswered diff --git a/verification/tla/MCStartLive.tla b/verification/tla/MCStartLive.tla index d9c6de8a..c3e5c0e3 100644 --- a/verification/tla/MCStartLive.tla +++ b/verification/tla/MCStartLive.tla @@ -1,5 +1,6 @@ ---------------------------- MODULE MCStartLive ---------------------------- EXTENDS Start -\* Per-PR liveness instance: a parent exit, also one held during the -\* child's started(), reaches every linked child. +\* Per-PR liveness instance: one child of every waiting variant of a +\* managed parent; parent exits (also held during started()) reach linked +\* children, and every waiting start is answered. ============================================================================= diff --git a/verification/tla/MCStartScope.cfg b/verification/tla/MCStartScope.cfg new file mode 100644 index 00000000..e6ecb804 --- /dev/null +++ b/verification/tla/MCStartScope.cfg @@ -0,0 +1,39 @@ +SPECIFICATION LiveSpec +CONSTANTS + Kids = {"c1", "c2"} + Kinds = {"link"} + Explicit = {"none", "U"} + ParentPlaces = {"G"} + Cap = 3 + Named = FALSE + TimeoutEnabled = FALSE + ParentCanExit = TRUE + CloseEnabled = FALSE + MaxMsgs = 0 + ChildCanTrap = TRUE + UnlinkEnabled = TRUE + ScopeEnabled = TRUE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation +INVARIANT AbortInvisible +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce +INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone +PROPERTY OwnedScopeFinishes diff --git a/verification/tla/MCStartScope.tla b/verification/tla/MCStartScope.tla new file mode 100644 index 00000000..943f60c1 --- /dev/null +++ b/verification/tla/MCStartScope.tla @@ -0,0 +1,7 @@ +---------------------------- MODULE MCStartScope ---------------------------- +EXTENDS Start +\* Per-PR instance for S8 (linked children owned by the parent's scope): +\* a managed parent, two linked children in its group or unmanaged, which +\* may trap or unlink; the scope deadline and the generation end. Safety and +\* OwnedScopeFinishes. +============================================================================= diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 502dddca..1eb3e2af 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -1,134 +1,93 @@ ------------------------------- MODULE Start ------------------------------- EXTENDS Naturals, FiniteSets (***************************************************************************) -(* Unified start API (0.9; design: docs/design/unified-start.md). *) +(* Unified start API, revision 2 (0.9; design docs/design/unified-start.md).*) (* *) -(* One parent actor P starts children with spawn, start, start_link or *) -(* start_monitor, each with optional StartOptions (explicit placement, *) -(* a registered name). A start has two stages: *) -(* stage 1 (in the caller, before the child runs any code): *) -(* Admit placement: explicit, else inherited (link/monitor), else *) -(* unmanaged; a group admits only while open and below Cap; *) -(* Link start_link/start_monitor: the edge is recorded while P is *) -(* running, else the start is refused (NoProc-like); *) -(* Register the name, atomically, or AlreadyStarted; *) -(* Dispatch the child: spawn returns here (no ack); *) -(* stage 2 (on the child): started() returns Init (ready, or ready with *) -(* a continuation), returns an error or ignore (exit Normal), or *) -(* panics or stops abnormally; signals received meanwhile are held *) -(* until it returns. A continuation runs before any mailbox message. *) -(* start, start_link and start_monitor return after started() (the *) -(* ack): Ok when it returned ready with no exit pending, else the *) -(* error after *) -(* the child's death point D, consuming the link notice (trapping *) -(* parent) or the Down. A startup timeout unlinks and demonitors *) -(* first, then kills. *) -(* An aborted stage 1 leaves no trace: no edge, no name, no slot, no *) -(* hook, no notice. Children never trap exits; a parent exit is held *) -(* during started() and applied after it (Normal is ignored). *) +(* One parent actor P (managed in group G or unmanaged; it may trap exits *) +(* and exit) starts children with spawn, start, start_link or *) +(* start_monitor, each with an explicit or default placement and possibly *) +(* the one shared name. *) (* *) -(* Steps are atomic; interleavings between steps of stage 1 are what the *) -(* implementation must make invisible (parent death, group closure and a *) -(* competing registration may happen between any two of them). *) +(* Stage 1 is one commit (design section 3): placement admission, the *) +(* parent's state, the link or monitor edge and the name are decided under *) +(* the commit locks together, so a refused start is never observable. A *) +(* dispatch failure happens before the commit (PrepareFail). *) +(* *) +(* Every waiting start owns a startup cell (design section 5): one *) +(* terminal commitment (ready, failed, ignored, timedout) made exactly *) +(* once, by the child at the end of started() or by the init timer, and a *) +(* waiter state (waiting, abandoned, consumed). Readiness clears the start *) +(* marks on the edges, so a later death is an ordinary death. A failed *) +(* start's notice is diverted to the waiter at publication and consumed by *) +(* the error, or handed to P if the wait was abandoned. *) +(* *) +(* Death is three steps, as in otp-exit-signals: D (Die: edges closed and *) +(* their notices taken), destruction, publication (Publish: name and slot *) +(* released, then notices, then the waiter). Destruction is folded into *) +(* the gap between Die and Publish; a waiter may give up at its bounded *) +(* cleanup deadline (pend = TRUE) before publication. *) +(* *) +(* A linked child placed in P's group is owned by P's scope (S8): P's *) +(* generation is done only when the owned children are published; the *) +(* scope deadline kills owned children that outlive P's exit (unlinked or *) +(* not). *) (***************************************************************************) CONSTANTS Kids, \* child names - Kinds, \* variants used: subset of {"spawn","start","link","monitor"} - Explicit, \* explicit placements: subset of {"none","G","U","same"} - \* ("same" = StartOptions::placement_of(&ctx)) - ParentPlaces, \* where P runs: subset of {"G","U"} - Cap, \* live actors group G admits (thread_group workers) + Kinds, \* subset of {"spawn","start","link","monitor"} + Explicit, \* subset of {"none","G","U","same"} ("same" = placement_of) + ParentPlaces, \* subset of {"G","U"} + Cap, \* live actors G admits (P included); held until publication Named, \* children may request the one shared name - TimeoutEnabled, \* start_link/start_monitor may time out - ParentCanExit, \* P may exit (with Normal or Panic) + TimeoutEnabled, \* the init timer may fire + ParentCanExit, \* P may exit (Normal or Panic) CloseEnabled, \* G may close (system shutdown) - MaxMsgs, \* messages others may send to each child - Fault \* "None" or a negative control, see below - -(* Negative controls (each must produce a counterexample): *) -(* NoInherit 0.8: a linked/monitored child of a managed parent is *) -(* unmanaged -> InheritedPlacement *) -(* LinkAfterStarted the link is set after started() began *) -(* -> LinkBeforeStarted *) -(* DoubleRegister check-then-insert name registration -> NameUnique *) -(* LateUnregister the name outlives D -> NameReleasedAtDeath *) -(* AbortLeaksLink a refused start keeps its edge -> AbortInvisible *) -(* EarlyFailAck the error returns before the child's D *) -(* -> FailureAfterDeath *) -(* AckBeforeStarted Ok before started() returned -> AckMeansStarted *) -(* NoMonitorFlush a failed start_monitor also delivers Down *) -(* -> FailureFlushed *) -(* NoLinkFlush a failed start_link also delivers Exit to a trapping *) -(* parent -> FailureFlushed *) -(* KillBeforeUnlink the timeout kills while still linked *) -(* -> TimeoutContained *) -(* LinkCheckLate the link ignores the parent's state *) -(* -> NoEdgeToDeadParent *) -(* AdmitClosed a closed group admits -> AdmittedOnlyOpen *) -(* IgnoreCap a full group admits -> CapacityRespected *) -(* ContinueQueued the continuation is queued behind earlier messages *) -(* -> ContinueFirst *) -(* AbandonDropsNotice a dropped start future loses the diverted notice *) -(* -> AbandonedNoticeDelivered *) -(* DropHeld a parent exit held during started() is dropped *) -(* (liveness) -> ParentExitReachesChild*) + MaxMsgs, \* messages others send to each child + ChildCanTrap, \* started() may set trap_exit + UnlinkEnabled, \* a ready linked child may unlink from P + ScopeEnabled, \* model P's generation end and its scope deadline (S8) + Fault \* "None" or a negative control (README, Unified start) None == "none" -Abnormal == {"Panic", "Killed", "NoProc", "Shutdown"} -Reasons == {"Normal"} \cup Abnormal -Phases == {"idle", "begun", "admitted", "linked", "registered", "starting", - "ready", "exiting", "dead", "aborted"} -Results == {"none", "pending", "ok", "failed", "timeout", "closed", - "capacity", "already", "noproc", "abandoned"} +NonNormal == {"Panic", "Killed", "Shutdown"} +Reasons == {"Normal"} \cup NonNormal +Phases == {"idle", "begun", "reg1", "starting", "ready", "exiting", "dead", + "published", "aborted"} +Commits == {None, "ready", "failed", "ignored", "timedout"} +Waits == {"none", "waiting", "abandoned", "consumed"} +Results == {"none", "pending", "ok", "failed", "ignored", "timeout", "closed", + "capacity", "already", "parentexited", "system"} +Outcomes == {"ready", "continue", "panic", "stop", "error", "ignore"} VARIABLES - pst, \* P: "run", "exiting" (exit begun, before D) or "dead" (past D) - pplace, \* P's placement: "G" (managed) or "U" (unmanaged) - ptrap, \* P traps exits - pwhy, \* P's exit reason - pcause, \* the child whose death notice made P exit, or None - gopen, \* G admits - kind, \* [Kids -> start variant] - expl, \* [Kids -> explicit placement option] - wname, \* [Kids -> requests the name] - phase, \* [Kids -> Phases] - place, \* [Kids -> "G", "U" or None] - admOpen, \* [Kids -> G was open when the child was admitted to it] - edge, \* [Kids -> link edge with P] - mon, \* [Kids -> P's monitor on the child] - linkedAt, \* [Kids -> None, "before" or "after" started() began] - monAt, \* same for the monitor - began, \* [Kids -> started() began] - retd, \* [Kids -> started() returned, panicked or stopped] - killed, \* [Kids -> startup-timeout kill received] - pex, \* [Kids -> parent exit queued at the child, or None] - pexAbn, \* [Kids -> a non-Normal parent exit was ever queued] - cont, \* [Kids -> a continuation is pending] - msgs, \* [Kids -> messages sent to the child by others] - early, \* [Kids -> a message was handled before the continuation] - div, \* [Kids -> notice diverted to the waiting start call, or None] - divEver, \* [Kids -> a notice was ever diverted] - why, \* [Kids -> exit reason or None] - res, \* [Kids -> what the start call returned] - reg, \* children holding the name - exits, \* Exit messages P's exit_received sees: [c, r] - downs \* Down messages P sees: [c, r] - -vars == <> -pVars == <> -optVars == <> -startVars == <> -obsVars == <> - -Live(c) == phase[c] \in {"admitted", "linked", "registered", "starting", - "ready", "exiting"} + pst, pplace, ptrap, pwhy, pcause, gopen, gdone, + kind, expl, wname, + phase, place, admOpen, owned, readyOpen, + edge, mon, linkEver, monEver, + began, killed, skill, pex, qpex, pexAbn, ctrap, + commit, why, wait, res, pend, + lnote, mnote, div, divEver, + cont, msgs, early, + reg, seen, + exitN, downN, exitR, downR + +pV == <> +oV == <> +sV == <> +eV == <> +xV == <> +cV == <> +nV == <> +mV == <> +rV == <> +dV == <> +vars == <> + +Waiting(k) == k \in {"start", "link", "monitor"} +Live(c) == phase[c] \in {"reg1", "starting", "ready", "exiting", "dead"} LiveInG == Cardinality({c \in Kids : place[c] = "G" /\ Live(c)}) + (IF pplace = "G" /\ pst # "dead" THEN 1 ELSE 0) -\* The contract: explicit placement, else the parent's (link/monitor), -\* else unmanaged. Expected(c) == CASE expl[c] = "same" -> pplace [] expl[c] \in {"G", "U"} -> expl[c] @@ -139,75 +98,79 @@ Target(c) == THEN "U" ELSE Expected(c) TypeOK == - /\ pst \in {"run", "exiting", "dead"} - /\ pplace \in {"G", "U"} - /\ ptrap \in BOOLEAN - /\ pwhy \in Reasons \cup {None} - /\ pcause \in Kids \cup {None} - /\ gopen \in BOOLEAN + /\ pst \in {"run", "exiting", "dead"} /\ pplace \in {"G", "U"} + /\ ptrap \in BOOLEAN /\ pwhy \in Reasons \cup {None} + /\ pcause \in Kids \cup {None} /\ gopen \in BOOLEAN /\ gdone \in BOOLEAN /\ kind \in [Kids -> {"spawn", "start", "link", "monitor", None}] /\ expl \in [Kids -> {"G", "U", "same", None}] /\ wname \in [Kids -> BOOLEAN] - /\ phase \in [Kids -> Phases] - /\ place \in [Kids -> {"G", "U", None}] - /\ admOpen \in [Kids -> BOOLEAN] - /\ edge \in [Kids -> BOOLEAN] - /\ mon \in [Kids -> BOOLEAN] - /\ linkedAt \in [Kids -> {None, "before", "after"}] - /\ monAt \in [Kids -> {None, "before", "after"}] - /\ began \in [Kids -> BOOLEAN] - /\ retd \in [Kids -> BOOLEAN] - /\ killed \in [Kids -> BOOLEAN] - /\ pex \in [Kids -> Reasons \cup {None}] - /\ pexAbn \in [Kids -> BOOLEAN] - /\ cont \in [Kids -> BOOLEAN] - /\ msgs \in [Kids -> 0..MaxMsgs] + /\ phase \in [Kids -> Phases] /\ place \in [Kids -> {"G", "U", None}] + /\ admOpen \in [Kids -> BOOLEAN] /\ owned \in [Kids -> BOOLEAN] + /\ readyOpen \in [Kids -> BOOLEAN] + /\ edge \in [Kids -> BOOLEAN] /\ mon \in [Kids -> BOOLEAN] + /\ linkEver \in [Kids -> BOOLEAN] /\ monEver \in [Kids -> BOOLEAN] + /\ began \in [Kids -> BOOLEAN] /\ killed \in [Kids -> BOOLEAN] + /\ skill \in [Kids -> BOOLEAN] + /\ pex \in [Kids -> Reasons \cup {None}] /\ qpex \in [Kids -> Reasons \cup {None}] + /\ pexAbn \in [Kids -> BOOLEAN] /\ ctrap \in [Kids -> BOOLEAN] + /\ commit \in [Kids -> Commits] /\ why \in [Kids -> Reasons \cup {None}] + /\ wait \in [Kids -> Waits] /\ res \in [Kids -> Results] + /\ pend \in [Kids -> BOOLEAN] + /\ lnote \in [Kids -> BOOLEAN] /\ mnote \in [Kids -> BOOLEAN] + /\ div \in [Kids -> Reasons \cup {None}] /\ divEver \in [Kids -> BOOLEAN] + /\ cont \in [Kids -> BOOLEAN] /\ msgs \in [Kids -> 0..MaxMsgs] /\ early \in [Kids -> BOOLEAN] - /\ why \in [Kids -> Reasons \cup {None}] - /\ res \in [Kids -> Results] - /\ reg \subseteq Kids - /\ exits \subseteq [c : Kids, r : Reasons] - /\ downs \subseteq [c : Kids, r : Reasons] - /\ div \in [Kids -> Reasons \cup {None}] - /\ divEver \in [Kids -> BOOLEAN] + /\ reg \subseteq Kids /\ seen \in [Kids -> BOOLEAN] + /\ exitN \in [Kids -> 0..3] /\ downN \in [Kids -> 0..3] + /\ exitR \in [Kids -> Reasons \cup {None}] /\ downR \in [Kids -> Reasons \cup {None}] Init == - /\ pst = "run" - /\ pplace \in ParentPlaces - /\ ptrap \in BOOLEAN - /\ pwhy = None - /\ pcause = None - /\ gopen = TRUE - /\ kind = [c \in Kids |-> None] - /\ expl = [c \in Kids |-> None] + /\ pst = "run" /\ pplace \in ParentPlaces /\ ptrap \in BOOLEAN + /\ pwhy = None /\ pcause = None /\ gopen = TRUE /\ gdone = FALSE + /\ kind = [c \in Kids |-> None] /\ expl = [c \in Kids |-> None] /\ wname = [c \in Kids |-> FALSE] - /\ phase = [c \in Kids |-> "idle"] - /\ place = [c \in Kids |-> None] - /\ admOpen = [c \in Kids |-> FALSE] - /\ edge = [c \in Kids |-> FALSE] - /\ mon = [c \in Kids |-> FALSE] - /\ linkedAt = [c \in Kids |-> None] - /\ monAt = [c \in Kids |-> None] - /\ began = [c \in Kids |-> FALSE] - /\ retd = [c \in Kids |-> FALSE] - /\ killed = [c \in Kids |-> FALSE] - /\ pex = [c \in Kids |-> None] - /\ pexAbn = [c \in Kids |-> FALSE] - /\ cont = [c \in Kids |-> FALSE] - /\ msgs = [c \in Kids |-> 0] + /\ phase = [c \in Kids |-> "idle"] /\ place = [c \in Kids |-> None] + /\ admOpen = [c \in Kids |-> FALSE] /\ owned = [c \in Kids |-> FALSE] + /\ readyOpen = [c \in Kids |-> FALSE] + /\ edge = [c \in Kids |-> FALSE] /\ mon = [c \in Kids |-> FALSE] + /\ linkEver = [c \in Kids |-> FALSE] /\ monEver = [c \in Kids |-> FALSE] + /\ began = [c \in Kids |-> FALSE] /\ killed = [c \in Kids |-> FALSE] + /\ skill = [c \in Kids |-> FALSE] + /\ pex = [c \in Kids |-> None] /\ qpex = [c \in Kids |-> None] + /\ pexAbn = [c \in Kids |-> FALSE] /\ ctrap = [c \in Kids |-> FALSE] + /\ commit = [c \in Kids |-> None] /\ why = [c \in Kids |-> None] + /\ wait = [c \in Kids |-> "none"] /\ res = [c \in Kids |-> "none"] + /\ pend = [c \in Kids |-> FALSE] + /\ lnote = [c \in Kids |-> FALSE] /\ mnote = [c \in Kids |-> FALSE] + /\ div = [c \in Kids |-> None] /\ divEver = [c \in Kids |-> FALSE] + /\ cont = [c \in Kids |-> FALSE] /\ msgs = [c \in Kids |-> 0] /\ early = [c \in Kids |-> FALSE] - /\ why = [c \in Kids |-> None] - /\ res = [c \in Kids |-> "none"] - /\ reg = {} - /\ exits = {} - /\ downs = {} - /\ div = [c \in Kids |-> None] - /\ divEver = [c \in Kids |-> FALSE] + /\ reg = {} /\ seen = [c \in Kids |-> FALSE] + /\ exitN = [c \in Kids |-> 0] /\ downN = [c \in Kids |-> 0] + /\ exitR = [c \in Kids |-> None] /\ downR = [c \in Kids |-> None] + +----------------------------------------------------------------------------- +(* Notices reaching P. *) + +\* A link notice: a trapping P gets an Exit; a non-trapping running P exits +\* on a non-Normal reason. +DeliverLink(c, r) == + IF ptrap + THEN /\ exitN' = [exitN EXCEPT ![c] = @ + 1] + /\ exitR' = [exitR EXCEPT ![c] = r] + /\ UNCHANGED <> + ELSE IF r # "Normal" /\ pst = "run" + THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c + /\ UNCHANGED <> + ELSE UNCHANGED <> + +DeliverDown(c, r) == + /\ downN' = [downN EXCEPT ![c] = @ + 1] + /\ downR' = [downR EXCEPT ![c] = r] ----------------------------------------------------------------------------- (* Stage 1, in the caller. *) -\* The caller asks for a start: variant, explicit placement, name. Begin(c) == /\ phase[c] = "idle" /\ \E k \in Kinds, e \in Explicit, n \in (IF Named THEN BOOLEAN ELSE {FALSE}): @@ -215,74 +178,90 @@ Begin(c) == /\ expl' = [expl EXCEPT ![c] = IF e = "none" THEN None ELSE e] /\ wname' = [wname EXCEPT ![c] = n] /\ phase' = [phase EXCEPT ![c] = "begun"] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* A refused start: nothing remains (edge, monitor, name and slot released; -\* no hook ran, no notice is sent). -Abort(c, r) == +\* Preparation (executor resources, state) fails before anything is visible. +PrepareFail(c) == + /\ phase[c] = "begun" /\ phase' = [phase EXCEPT ![c] = "aborted"] - /\ res' = [res EXCEPT ![c] = r] - /\ place' = [place EXCEPT ![c] = None] - /\ edge' = IF Fault = "AbortLeaksLink" THEN edge ELSE [edge EXCEPT ![c] = FALSE] - /\ mon' = [mon EXCEPT ![c] = FALSE] - /\ reg' = reg \ {c} - -Admit(c) == + /\ res' = [res EXCEPT ![c] = "system"] + /\ UNCHANGED <> + +\* Why the commit refuses, or None. Order of the checks does not matter: all +\* are read under the commit locks. +Refusal(c) == + LET t == Target(c) IN + CASE t = "G" /\ ~gopen /\ Fault # "AdmitClosed" -> "closed" + [] t = "G" /\ LiveInG >= Cap /\ Fault # "IgnoreCap" -> "capacity" + [] kind[c] \in {"link", "monitor"} /\ pst # "run" /\ Fault # "LinkCheckLate" + -> "parentexited" + [] wname[c] /\ reg # {} /\ Fault # "DoubleRegister" -> "already" + [] OTHER -> None + +\* The stage-1 commit: everything at once, or nothing. +Commit(c) == /\ phase[c] = "begun" - /\ LET t == Target(c) IN - IF t = "U" - THEN /\ phase' = [phase EXCEPT ![c] = "admitted"] - /\ place' = [place EXCEPT ![c] = "U"] - /\ UNCHANGED <> - ELSE IF (gopen \/ Fault = "AdmitClosed") /\ (LiveInG < Cap \/ Fault = "IgnoreCap") - THEN /\ phase' = [phase EXCEPT ![c] = "admitted"] - /\ place' = [place EXCEPT ![c] = "G"] - /\ admOpen' = [admOpen EXCEPT ![c] = gopen] - /\ UNCHANGED <> - ELSE /\ Abort(c, IF ~gopen THEN "closed" ELSE "capacity") - /\ UNCHANGED admOpen - /\ UNCHANGED <> - -\* start_link / start_monitor: the edge is recorded in one transaction with -\* the parent's state; an exiting or dead parent refuses the start. -Link(c) == - /\ phase[c] = "admitted" - /\ IF kind[c] \in {"spawn", "start"} \/ (kind[c] = "link" /\ Fault = "LinkAfterStarted") - THEN /\ phase' = [phase EXCEPT ![c] = "linked"] - /\ UNCHANGED <> - ELSE IF pst = "run" \/ Fault = "LinkCheckLate" - THEN /\ phase' = [phase EXCEPT ![c] = "linked"] - /\ edge' = [edge EXCEPT ![c] = kind[c] = "link"] - /\ mon' = [mon EXCEPT ![c] = kind[c] = "monitor"] - /\ linkedAt' = IF kind[c] = "link" THEN [linkedAt EXCEPT ![c] = "before"] ELSE linkedAt - /\ monAt' = IF kind[c] = "monitor" THEN [monAt EXCEPT ![c] = "before"] ELSE monAt - /\ UNCHANGED <> - ELSE /\ Abort(c, "noproc") - /\ UNCHANGED <> - /\ UNCHANGED <> - -\* The name is taken atomically, or the start is refused with AlreadyStarted. -Register(c) == - /\ phase[c] = "linked" - /\ IF ~wname[c] - THEN /\ phase' = [phase EXCEPT ![c] = "registered"] - /\ UNCHANGED <> - ELSE IF reg = {} \/ Fault = "DoubleRegister" - THEN /\ phase' = [phase EXCEPT ![c] = "registered"] - /\ reg' = reg \cup {c} - /\ UNCHANGED <> - ELSE Abort(c, "already") - /\ UNCHANGED <> - -\* The child is handed to its executor; spawn returns Ok here, start, -\* start_link and start_monitor wait for the ack. -Dispatch(c) == - /\ phase[c] = "registered" - /\ phase' = [phase EXCEPT ![c] = "starting"] - /\ res' = [res EXCEPT ![c] = IF kind[c] = "spawn" THEN "ok" ELSE "pending"] - /\ UNCHANGED <> + /\ Fault # "SplitCommit" \/ ~wname[c] + /\ LET t == Target(c) + r == Refusal(c) + lk == kind[c] = "link" /\ Fault # "LinkAfterStarted" + mn == kind[c] = "monitor" + IN IF r # None + THEN /\ phase' = [phase EXCEPT ![c] = "aborted"] + /\ res' = [res EXCEPT ![c] = r] + /\ UNCHANGED <> + ELSE /\ phase' = [phase EXCEPT ![c] = "starting"] + /\ place' = [place EXCEPT ![c] = t] + /\ admOpen' = [admOpen EXCEPT ![c] = gopen] + /\ owned' = [owned EXCEPT ![c] = kind[c] = "link" /\ pplace = "G" /\ t = "G"] + /\ edge' = [edge EXCEPT ![c] = lk] + /\ mon' = [mon EXCEPT ![c] = mn] + /\ linkEver' = [linkEver EXCEPT ![c] = lk] + /\ monEver' = [monEver EXCEPT ![c] = mn] + /\ reg' = IF wname[c] THEN reg \cup {c} ELSE reg + /\ wait' = [wait EXCEPT ![c] = IF Waiting(kind[c]) THEN "waiting" ELSE "none"] + /\ res' = [res EXCEPT ![c] = IF Waiting(kind[c]) THEN "pending" ELSE "ok"] + /\ UNCHANGED <> + +\* SplitCommit (negative control): the name is published before the rest of +\* the commit, which may still refuse. +RegisterFirst(c) == + /\ Fault = "SplitCommit" + /\ phase[c] = "begun" /\ wname[c] /\ reg = {} + /\ reg' = reg \cup {c} + /\ phase' = [phase EXCEPT ![c] = "reg1"] + /\ UNCHANGED <> +CommitRest(c) == + /\ phase[c] = "reg1" + /\ LET t == Target(c) + r == CASE t = "G" /\ (~gopen \/ LiveInG >= Cap) -> "capacity" + [] kind[c] \in {"link", "monitor"} /\ pst # "run" -> "parentexited" + [] OTHER -> None + IN IF r # None + THEN /\ phase' = [phase EXCEPT ![c] = "aborted"] + /\ res' = [res EXCEPT ![c] = r] + /\ reg' = reg \ {c} + /\ UNCHANGED <> + ELSE /\ phase' = [phase EXCEPT ![c] = "starting"] + /\ place' = [place EXCEPT ![c] = t] + /\ admOpen' = [admOpen EXCEPT ![c] = gopen] + /\ owned' = [owned EXCEPT ![c] = kind[c] = "link" /\ pplace = "G" /\ t = "G"] + /\ edge' = [edge EXCEPT ![c] = kind[c] = "link"] + /\ mon' = [mon EXCEPT ![c] = kind[c] = "monitor"] + /\ linkEver' = [linkEver EXCEPT ![c] = kind[c] = "link"] + /\ monEver' = [monEver EXCEPT ![c] = kind[c] = "monitor"] + /\ wait' = [wait EXCEPT ![c] = IF Waiting(kind[c]) THEN "waiting" ELSE "none"] + /\ res' = [res EXCEPT ![c] = IF Waiting(kind[c]) THEN "pending" ELSE "ok"] + /\ UNCHANGED reg + /\ UNCHANGED <> + +\* Anyone resolves the name (whereis). +Lookup == + /\ \E c \in reg : + /\ ~seen[c] + /\ seen' = [seen EXCEPT ![c] = TRUE] + /\ UNCHANGED <> ----------------------------------------------------------------------------- (* Stage 2, on the child. *) @@ -292,231 +271,307 @@ Exit(c, r) == /\ why' = [why EXCEPT ![c] = r] StartedBegin(c) == - /\ phase[c] = "starting" - /\ ~began[c] - /\ ~killed[c] + /\ phase[c] = "starting" /\ ~began[c] /\ ~killed[c] /\ ~skill[c] /\ began' = [began EXCEPT ![c] = TRUE] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* Only under LinkAfterStarted: the link is set once started() is running. +\* LinkAfterStarted (negative control): the link is set once started() runs. LateLink(c) == /\ Fault = "LinkAfterStarted" - /\ kind[c] = "link" - /\ began[c] - /\ linkedAt[c] = None - /\ phase[c] \in {"starting", "ready"} - /\ pst = "run" + /\ kind[c] = "link" /\ began[c] /\ ~linkEver[c] + /\ phase[c] \in {"starting", "ready"} /\ pst = "run" /\ edge' = [edge EXCEPT ![c] = TRUE] - /\ linkedAt' = [linkedAt EXCEPT ![c] = "after"] - /\ UNCHANGED <> + /\ linkEver' = [linkEver EXCEPT ![c] = TRUE] + /\ UNCHANGED <> -\* A kill received before started() began: the child ends Killed. +\* A kill (init timer or scope deadline) before started() began. KilledEarly(c) == - /\ phase[c] = "starting" - /\ ~began[c] - /\ killed[c] + /\ phase[c] = "starting" /\ ~began[c] /\ (killed[c] \/ skill[c]) /\ Exit(c, "Killed") - /\ UNCHANGED <> - -\* started() returns Init::ready, Init::continue_with, an error or ignore -\* (the child exits Normal), or panics (or Init::stop with a non-Normal -\* reason). Held signals apply when it returns: a kill wins, then a -\* non-Normal parent exit; a Normal parent exit is ignored. + /\ commit' = IF commit[c] = None THEN [commit EXCEPT ![c] = "failed"] ELSE commit + /\ UNCHANGED <> + +\* started() returns (or panics); held signals are resolved now, with the +\* trap flag it leaves (design section 5.3, precedence): +\* 1. the init timer committed timedout, or a kill: exit Killed; +\* 2. a non-Normal held parent exit and the child does not trap: exit +\* with it (a held Normal parent exit is dropped); +\* 3. a panic, Init::stop (non-Normal), Init::error / ignore (Normal); +\* 4. ready: commit, unless the group closed (exit Shutdown). A held parent +\* exit to a trapping child is queued behind the continuation (E3). StartedEnd(c) == - /\ phase[c] = "starting" - /\ began[c] - /\ ~retd[c] - /\ retd' = [retd EXCEPT ![c] = TRUE] - /\ \E o \in {"ready", "continue", "panic", "error"}: - IF killed[c] THEN /\ Exit(c, "Killed") /\ UNCHANGED <> - ELSE IF o = "panic" THEN /\ Exit(c, "Panic") /\ UNCHANGED <> - ELSE IF o = "error" THEN /\ Exit(c, "Normal") /\ UNCHANGED <> - ELSE IF pex[c] \in Abnormal /\ Fault # "DropHeld" - THEN /\ Exit(c, pex[c]) - /\ pex' = [pex EXCEPT ![c] = None] - /\ UNCHANGED cont - ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] - /\ pex' = [pex EXCEPT ![c] = None] - /\ cont' = [cont EXCEPT ![c] = o = "continue"] - /\ UNCHANGED why - /\ UNCHANGED <> + /\ phase[c] = "starting" /\ began[c] + /\ \E o \in Outcomes, t \in (IF ChildCanTrap THEN BOOLEAN ELSE {FALSE}): + LET sigOut == pex[c] \in NonNormal /\ ~t /\ Fault # "DropHeld" + failWith(r) == /\ Exit(c, r) + /\ commit' = IF commit[c] = None + THEN [commit EXCEPT ![c] = "failed"] ELSE commit + /\ UNCHANGED <> + IN /\ ctrap' = [ctrap EXCEPT ![c] = t] + /\ pex' = [pex EXCEPT ![c] = None] + /\ IF killed[c] \/ skill[c] THEN failWith("Killed") + ELSE IF sigOut THEN failWith(pex[c]) + ELSE IF o = "panic" THEN failWith("Panic") + ELSE IF o = "stop" THEN failWith("Shutdown") + ELSE IF o \in {"error", "ignore"} + THEN /\ Exit(c, "Normal") + /\ commit' = [commit EXCEPT ![c] = IF o = "ignore" THEN "ignored" ELSE "failed"] + /\ UNCHANGED <> + ELSE IF place[c] = "G" /\ ~gopen /\ Fault # "ReadyAfterClose" + THEN failWith("Shutdown") + ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] + /\ commit' = [commit EXCEPT ![c] = "ready"] + /\ readyOpen' = [readyOpen EXCEPT ![c] = gopen \/ place[c] # "G"] + /\ cont' = [cont EXCEPT ![c] = o = "continue"] + /\ qpex' = [qpex EXCEPT ![c] = IF t THEN pex[c] ELSE None] + /\ UNCHANGED why + /\ UNCHANGED <> \* Others (a peer that found the name, the caller after the ack) send. Send(c) == - /\ phase[c] \in {"starting", "ready"} - /\ msgs[c] < MaxMsgs + /\ phase[c] \in {"starting", "ready"} /\ msgs[c] < MaxMsgs /\ msgs' = [msgs EXCEPT ![c] = msgs[c] + 1] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* The continuation runs first; a mailbox message only after it. RunContinue(c) == - /\ phase[c] = "ready" - /\ cont[c] + /\ phase[c] = "ready" /\ cont[c] /\ cont' = [cont EXCEPT ![c] = FALSE] - /\ UNCHANGED <> + /\ UNCHANGED <> Handle(c) == - /\ phase[c] = "ready" - /\ msgs[c] > 0 + /\ phase[c] = "ready" /\ msgs[c] > 0 /\ ~cont[c] \/ Fault = "ContinueQueued" /\ msgs' = [msgs EXCEPT ![c] = msgs[c] - 1] /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] - /\ UNCHANGED <> + /\ UNCHANGED <> + +\* A queued parent exit to a trapping child ends it, Normal included (E3); +\* it is a mailbox entry, so it waits for the continuation. +HandleQueuedExit(c) == + /\ phase[c] = "ready" /\ qpex[c] # None + /\ ~cont[c] \/ Fault = "ContinueQueued" + /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] + /\ qpex' = [qpex EXCEPT ![c] = None] + /\ Exit(c, qpex[c]) + /\ UNCHANGED <> -\* A parent exit reaching a running child. +\* A parent exit reaching a ready, non-trapping child (Normal is ignored). ApplyParentExit(c) == - /\ phase[c] = "ready" - /\ pex[c] # None + /\ phase[c] = "ready" /\ pex[c] # None /\ pex' = [pex EXCEPT ![c] = None] - /\ IF pex[c] \in Abnormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> - /\ UNCHANGED <> - -\* The child's death point D: the name is released, then the link notice -\* and the Down are taken. While the start call still waits (res pending), -\* they are diverted to it (a trapping parent's Exit, the Down) and the -\* error consumes them; a non-trapping parent still exits on a non-Normal -\* notice. -ChildDie(c) == + /\ IF pex[c] \in NonNormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* After readiness the child may stop or crash on its own. +ReadyExit(c) == + /\ phase[c] = "ready" + /\ \E r \in {"Normal", "Panic"} : Exit(c, r) + /\ UNCHANGED <> + +\* A kill reaching a ready child (the scope deadline; under +\* TimerIgnoresCommit also the init timer). +KillReady(c) == + /\ phase[c] = "ready" /\ (killed[c] \/ skill[c]) + /\ Exit(c, "Killed") + /\ UNCHANGED <> + +\* A ready linked child unlinks from P (ctx.unlink); ownership stays (S8). +ChildUnlink(c) == + /\ UnlinkEnabled /\ phase[c] = "ready" /\ edge[c] + /\ edge' = [edge EXCEPT ![c] = FALSE] + /\ UNCHANGED <> + +\* D: the edges close and their notices are taken (delivered at publication). +Die(c) == /\ phase[c] = "exiting" /\ phase' = [phase EXCEPT ![c] = "dead"] - /\ reg' = IF Fault = "LateUnregister" THEN reg ELSE reg \ {c} - /\ LET r == why[c] - pending == res[c] = "pending" - linkTaken == edge[c] /\ pst # "dead" - monTaken == mon[c] /\ pst # "dead" - kills == linkTaken /\ ~ptrap /\ r \in Abnormal /\ pst = "run" - IN /\ exits' = IF linkTaken /\ ptrap /\ (~pending \/ Fault = "NoLinkFlush") - THEN exits \cup {[c |-> c, r |-> r]} ELSE exits - /\ downs' = IF monTaken /\ (~pending \/ Fault = "NoMonitorFlush") - THEN downs \cup {[c |-> c, r |-> r]} ELSE downs - /\ div' = IF pending /\ ((linkTaken /\ ptrap) \/ monTaken) - THEN [div EXCEPT ![c] = r] ELSE div - /\ divEver' = IF pending /\ ((linkTaken /\ ptrap) \/ monTaken) - THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver - /\ pst' = IF kills THEN "exiting" ELSE pst - /\ pwhy' = IF kills THEN r ELSE pwhy - /\ pcause' = IF kills THEN c ELSE pcause + /\ lnote' = [lnote EXCEPT ![c] = edge[c] /\ pst # "dead"] + /\ mnote' = [mnote EXCEPT ![c] = mon[c] /\ pst # "dead"] /\ edge' = [edge EXCEPT ![c] = FALSE] /\ mon' = [mon EXCEPT ![c] = FALSE] - /\ UNCHANGED <> + /\ reg' = IF Fault = "UnregisterAtDeath" THEN reg \ {c} ELSE reg + /\ UNCHANGED <> + +\* Publication (after destruction): the name and the slot are released, +\* then each taken notice is disposed of by the startup cell: a failed start +\* whose waiter still waits (or consumed the error) diverts or drops it, +\* except that a non-trapping P still exits on a non-Normal init failure +\* (OTP: the link kills the caller); otherwise it is an ordinary notice. +Publish(c) == + /\ phase[c] = "dead" + /\ phase' = [phase EXCEPT ![c] = "published"] + /\ reg' = IF Fault = "LateUnregister" THEN reg ELSE reg \ {c} + /\ LET r == why[c] + failedStart == (commit[c] \in {"failed", "ignored", "timedout"} + \/ (Fault = "DivertAfterReady" /\ commit[c] = "ready")) + /\ wait[c] \in {"waiting", "consumed"} + keepTimeout == commit[c] = "timedout" /\ Fault = "KillBeforeUnlink" + ordinary == ~failedStart \/ keepTimeout + ln == lnote[c] /\ pst # "dead" + mn == mnote[c] /\ pst # "dead" + divert == failedStart /\ wait[c] = "waiting" + /\ ((ln /\ ptrap) \/ mn) /\ ~keepTimeout + IN /\ div' = IF divert THEN [div EXCEPT ![c] = r] ELSE div + /\ divEver' = IF divert THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver + /\ IF ln /\ (ordinary \/ (ptrap /\ Fault = "NoLinkFlush") + \/ (ptrap /\ divert /\ Fault = "DoubleNotice")) + THEN /\ DeliverLink(c, r) /\ UNCHANGED <> + ELSE IF ln /\ ~ptrap /\ commit[c] = "failed" /\ r # "Normal" /\ pst = "run" + THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c + /\ UNCHANGED dV + ELSE IF mn /\ (ordinary \/ Fault = "NoMonitorFlush") + THEN /\ DeliverDown(c, r) /\ UNCHANGED <> + ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* LateUnregister (negative control): the name is released only after the +\* notices. +LateRelease(c) == + /\ Fault = "LateUnregister" /\ phase[c] = "published" /\ c \in reg + /\ reg' = reg \ {c} + /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* The ack of start, start_link and start_monitor. *) +(* The init timer and the waiter. *) -AckOk(c) == - /\ kind[c] \in {"start", "link", "monitor"} - /\ res[c] = "pending" - /\ ~killed[c] - /\ phase[c] = "ready" \/ (Fault = "AckBeforeStarted" /\ phase[c] = "starting") - /\ res' = [res EXCEPT ![c] = "ok"] - /\ UNCHANGED <> - -\* The error is returned after the child's D (OTP: after the 'DOWN'). -AckFail(c) == - /\ res[c] = "pending" - /\ ~killed[c] - /\ phase[c] = "dead" \/ (Fault = "EarlyFailAck" /\ phase[c] = "exiting") - /\ res' = [res EXCEPT ![c] = "failed"] - /\ div' = [div EXCEPT ![c] = None] - /\ UNCHANGED <> - -\* startup_timeout: unlink and demonitor (fences), then kill. +\* The framework-owned init timer: it commits timedout only if nothing was +\* committed, then kills. The start edges stay; their notice is disposed of +\* at publication by the waiter state (contained while waiting or consumed). Timeout(c) == /\ TimeoutEnabled - /\ res[c] = "pending" - /\ phase[c] = "starting" + /\ Waiting(kind[c]) \/ kind[c] = "spawn" /\ ~killed[c] + /\ \/ commit[c] = None /\ phase[c] = "starting" + \/ Fault = "TimerIgnoresCommit" /\ phase[c] \in {"starting", "ready"} + /\ commit' = IF commit[c] = None THEN [commit EXCEPT ![c] = "timedout"] ELSE commit /\ killed' = [killed EXCEPT ![c] = TRUE] - /\ edge' = IF Fault = "KillBeforeUnlink" THEN edge ELSE [edge EXCEPT ![c] = FALSE] - /\ mon' = IF Fault = "KillBeforeUnlink" THEN mon ELSE [mon EXCEPT ![c] = FALSE] - /\ UNCHANGED <> - -\* Returns Timeout after the child's D or after the bounded wait for it. -AckTimeout(c) == - /\ res[c] = "pending" - /\ killed[c] - /\ res' = [res EXCEPT ![c] = "timeout"] + /\ UNCHANGED <> + +AckOk(c) == + /\ wait[c] = "waiting" + /\ \/ commit[c] = "ready" + \/ Fault = "AckBeforeStarted" /\ commit[c] = None /\ phase[c] = "starting" + \/ Fault = "ErrorAsOk" /\ commit[c] = "ignored" + /\ res' = [res EXCEPT ![c] = "ok"] + /\ wait' = [wait EXCEPT ![c] = "consumed"] + /\ UNCHANGED <> + +\* A failed, ignored or timed-out start answers after publication, or at its +\* bounded cleanup deadline with the cleanup still pending (pend). The error +\* consumes a diverted notice. +AckFail(c) == + /\ wait[c] = "waiting" + /\ \/ commit[c] \in {"failed", "ignored", "timedout"} + \/ Fault = "DivertAfterReady" /\ commit[c] = "ready" /\ phase[c] = "published" + /\ \E late \in BOOLEAN : + /\ IF late THEN phase[c] \in {"exiting", "dead"} + \/ (phase[c] = "starting" /\ commit[c] = "timedout") + ELSE \/ phase[c] = "published" + \/ Fault = "EarlyFailAck" /\ phase[c] = "dead" + /\ pend' = [pend EXCEPT ![c] = late] + /\ ~(Fault = "NoAnswerIgnore" /\ commit[c] = "ignored") + /\ res' = [res EXCEPT ![c] = CASE commit[c] = "ignored" -> "ignored" + [] commit[c] = "timedout" -> "timeout" + [] OTHER -> "failed"] + /\ wait' = [wait EXCEPT ![c] = "consumed"] /\ div' = [div EXCEPT ![c] = None] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* The start future is dropped before it resolves: the child keeps running -\* and stays linked or monitored; a notice already diverted goes to P as if -\* it had never been diverted. +\* The Starting is dropped: the child keeps running; a diverted notice goes +\* to P as an ordinary one; later notices are ordinary. Abandon(c) == - /\ res[c] = "pending" - /\ res' = [res EXCEPT ![c] = "abandoned"] + /\ wait[c] = "waiting" + /\ wait' = [wait EXCEPT ![c] = "abandoned"] /\ div' = [div EXCEPT ![c] = None] /\ LET r == div[c] give == r # None /\ Fault # "AbandonDropsNotice" - IN /\ exits' = IF give /\ kind[c] = "link" THEN exits \cup {[c |-> c, r |-> r]} ELSE exits - /\ downs' = IF give /\ kind[c] = "monitor" THEN downs \cup {[c |-> c, r |-> r]} ELSE downs - /\ UNCHANGED <> + IN IF give /\ kind[c] = "link" THEN DeliverLink(c, r) /\ UNCHANGED <> + ELSE IF give /\ kind[c] = "monitor" + THEN DeliverDown(c, r) /\ UNCHANGED <> + ELSE UNCHANGED <> + /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* The parent and the group. *) +(* The parent, its scope and the group. *) PExit == - /\ ParentCanExit - /\ pst = "run" - /\ \E r \in {"Normal", "Panic"}: - /\ pst' = "exiting" - /\ pwhy' = r - /\ UNCHANGED <> - -\* P's D: each linked child gets a parent exit (held if its started() has -\* not returned); P's monitors are cancelled. + /\ ParentCanExit /\ pst = "run" + /\ \E r \in {"Normal", "Panic"} : pst' = "exiting" /\ pwhy' = r + /\ UNCHANGED <> + +\* P's D: each linked child gets a parent exit, held during started(), +\* queued behind the continuation for a trapping ready child, applied +\* otherwise; P's monitors are cancelled. PDie == /\ pst = "exiting" /\ pst' = "dead" - /\ pex' = [c \in Kids |-> IF edge[c] THEN pwhy ELSE pex[c]] - /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (edge[c] /\ pwhy \in Abnormal)] + /\ pex' = [c \in Kids |-> IF edge[c] /\ ~(phase[c] = "ready" /\ ctrap[c]) + THEN pwhy ELSE pex[c]] + /\ qpex' = [c \in Kids |-> IF edge[c] /\ phase[c] = "ready" /\ ctrap[c] + THEN pwhy ELSE qpex[c]] + /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (edge[c] /\ pwhy \in NonNormal)] /\ edge' = [c \in Kids |-> FALSE] /\ mon' = [c \in Kids |-> FALSE] - /\ UNCHANGED <> + /\ UNCHANGED <> + +\* S8: once P's exit began, the scope deadline kills every owned child +\* still alive (linked or not). +ScopeDeadline == + /\ ScopeEnabled /\ pplace = "G" /\ pst # "run" + /\ \E c \in Kids : owned[c] /\ phase[c] \in {"starting", "ready"} /\ ~skill[c] + /\ (edge[c] \/ Fault # "NoScopeKill") + /\ skill' = [c \in Kids |-> skill[c] \/ (owned[c] /\ phase[c] \in {"starting", "ready"} + /\ (edge[c] \/ Fault # "NoScopeKill"))] + /\ UNCHANGED <> + +\* P's generation is done: P past D and every owned child published. +GenDone == + /\ ScopeEnabled /\ pplace = "G" /\ pst = "dead" /\ ~gdone + /\ \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} + \/ Fault = "EarlyGenDone" + /\ gdone' = TRUE + /\ UNCHANGED <> GClose == - /\ CloseEnabled - /\ gopen + /\ CloseEnabled /\ gopen /\ gopen' = FALSE - /\ UNCHANGED <> + /\ UNCHANGED <> ChildStep(c) == - \/ Begin(c) \/ Admit(c) \/ Link(c) \/ Register(c) \/ Dispatch(c) + \/ Begin(c) \/ PrepareFail(c) \/ Commit(c) \/ RegisterFirst(c) \/ CommitRest(c) \/ StartedBegin(c) \/ LateLink(c) \/ KilledEarly(c) \/ StartedEnd(c) - \/ ApplyParentExit(c) \/ ChildDie(c) - \/ Send(c) \/ RunContinue(c) \/ Handle(c) - \/ AckOk(c) \/ AckFail(c) \/ Timeout(c) \/ AckTimeout(c) \/ Abandon(c) + \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ HandleQueuedExit(c) + \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ ChildUnlink(c) + \/ Die(c) \/ Publish(c) \/ LateRelease(c) + \/ Timeout(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) -Next == (\E c \in Kids : ChildStep(c)) \/ PExit \/ PDie \/ GClose +Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ PDie + \/ ScopeDeadline \/ GenDone \/ GClose -\* Liveness assumes the child's own steps and P's death complete. +\* Liveness assumes the child's own steps, publication, the waiter's answer, +\* P's death and the scope deadline complete (started() and destructors +\* return). Progress(c) == - Register(c) \/ Dispatch(c) \/ StartedBegin(c) \/ KilledEarly(c) - \/ StartedEnd(c) \/ ApplyParentExit(c) \/ ChildDie(c) -Fairness == /\ \A c \in Kids : WF_vars(Progress(c)) /\ WF_vars(Link(c)) /\ WF_vars(Admit(c)) - /\ WF_vars(PDie) + \/ StartedBegin(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ RunContinue(c) + \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ KillReady(c) + \/ Die(c) \/ Publish(c) \/ AckOk(c) +Fairness == + /\ \A c \in Kids : /\ WF_vars(Progress(c)) /\ WF_vars(Commit(c)) + /\ SF_vars(AckFail(c)) + /\ WF_vars(PDie) /\ WF_vars(ScopeDeadline) /\ WF_vars(GenDone) Spec == Init /\ [][Next]_vars LiveSpec == Spec /\ Fairness @@ -524,78 +579,104 @@ LiveSpec == Spec /\ Fairness ----------------------------------------------------------------------------- (* Safety. *) -\* Explicit placement wins; a linked or monitored child takes its parent's -\* placement; a plain start is unmanaged. InheritedPlacement == \A c \in Kids : place[c] # None => place[c] = Expected(c) -\* The link or monitor exists before started() begins. +\* The edge really exists before started() begins. LinkBeforeStarted == \A c \in Kids : - /\ (began[c] /\ kind[c] = "link") => linkedAt[c] = "before" - /\ (began[c] /\ kind[c] = "monitor") => monAt[c] = "before" + /\ (began[c] /\ kind[c] = "link") => linkEver[c] + /\ (began[c] /\ kind[c] = "monitor") => monEver[c] NameUnique == Cardinality(reg) <= 1 -\* The name is free once the holder is past D (and before any notice, -\* Down or failure result is observable). -NameReleasedAtDeath == \A c \in reg : phase[c] \notin {"dead", "aborted"} +\* The name is held from the commit until publication. +NameHeldUntilPublished == \A c \in reg : phase[c] \in {"reg1", "starting", "ready", "exiting", "dead"} -\* A refused start leaves no trace. +\* A new holder never overlaps an old holder's destruction. +NoOverlapOnReuse == + \A c \in reg : \A d \in Kids \ {c} : wname[d] => phase[d] # "dead" + +\* No notice or final error about c is observable while c holds the name. +NameFreeBeforeObservation == + \A c \in Kids : + (exitN[c] + downN[c] > 0 \/ (res[c] \in {"failed", "ignored", "timeout"} /\ ~pend[c])) + => c \notin reg + +\* A refused start is never observable. AbortInvisible == \A c \in Kids : phase[c] = "aborted" => - /\ ~edge[c] /\ ~mon[c] /\ c \notin reg /\ ~began[c] /\ place[c] = None - /\ \A e \in exits \cup downs : e.c # c - /\ pcause # c + /\ ~edge[c] /\ ~mon[c] /\ ~linkEver[c] /\ ~monEver[c] /\ c \notin reg + /\ ~seen[c] /\ ~began[c] /\ place[c] = None + /\ exitN[c] = 0 /\ downN[c] = 0 /\ pcause # c -\* Ok from start, start_link or start_monitor means started() returned. -AckMeansStarted == - \A c \in Kids : (res[c] = "ok" /\ kind[c] \in {"start", "link", "monitor"}) => retd[c] +\* The answer is the committed outcome. +ResultMatchesCommit == + \A c \in Kids : + /\ res[c] = "ok" /\ Waiting(kind[c]) => commit[c] = "ready" + /\ res[c] = "failed" => commit[c] = "failed" + /\ res[c] = "ignored" => commit[c] = "ignored" + /\ res[c] = "timeout" => commit[c] = "timedout" -\* A notice diverted to a start call that was then dropped reaches P. -AbandonedNoticeDelivered == - \A c \in Kids : (res[c] = "abandoned" /\ divEver[c]) => - \E e \in exits \cup downs : e.c = c +\* Readiness is final: a ready child's later death is never a start error. +ReadyIsFinal == \A c \in Kids : commit[c] = "ready" => res[c] \notin {"failed", "ignored", "timeout"} -\* No mailbox message is handled before the continuation. -ContinueFirst == \A c \in Kids : ~early[c] +\* The init timer never kills a child that committed ready. +TimeoutNeverKillsReady == \A c \in Kids : commit[c] = "ready" => ~killed[c] + +\* Managed readiness commits only while admission is open. +ReadyOnlyWhileOpen == \A c \in Kids : commit[c] = "ready" => readyOpen[c] -\* A startup failure is returned after the child's D. -FailureAfterDeath == \A c \in Kids : res[c] = "failed" => phase[c] = "dead" +\* Without a pending cleanup, a start error is answered after publication. +FailureAfterPublication == + \A c \in Kids : (res[c] \in {"failed", "ignored", "timeout"} /\ ~pend[c]) => phase[c] = "published" -\* A failed or timed-out start_monitor delivers no Down; a failed or -\* timed-out start_link delivers no Exit to a trapping parent. +\* A failed start_monitor delivers no Down; a failed start_link no Exit to a +\* trapping parent. FailureFlushed == - \A c \in Kids : res[c] \in {"failed", "timeout"} => - /\ kind[c] = "monitor" => \A d \in downs : d.c # c - /\ (kind[c] = "link" /\ ptrap) => \A e \in exits : e.c # c + \A c \in Kids : res[c] \in {"failed", "ignored", "timeout"} => + /\ kind[c] = "monitor" => downN[c] = 0 + /\ (kind[c] = "link" /\ ptrap) => exitN[c] = 0 -\* A startup-timeout kill never reaches the parent. +\* An init timeout reaches P only if the wait was abandoned. TimeoutContained == - \A c \in Kids : killed[c] => - /\ pcause # c - /\ \A e \in exits \cup downs : e.c # c + \A c \in Kids : (commit[c] = "timedout" /\ wait[c] # "abandoned") => + /\ pcause # c /\ exitN[c] = 0 /\ downN[c] = 0 -\* No link or monitor edge survives the parent's D. NoEdgeToDeadParent == \A c \in Kids : (edge[c] \/ mon[c]) => pst # "dead" AdmittedOnlyOpen == \A c \in Kids : place[c] = "G" => admOpen[c] CapacityRespected == LiveInG <= Cap -NoticeOnce == - \A c \in Kids : - /\ Cardinality({e \in exits : e.c = c}) <= 1 - /\ Cardinality({d \in downs : d.c = c}) <= 1 +NoticeOnce == \A c \in Kids : exitN[c] <= 1 /\ downN[c] <= 1 + +\* A notice diverted to an abandoned wait reaches P, with the child's reason. +AbandonedNoticeDelivered == + \A c \in Kids : (wait[c] = "abandoned" /\ divEver[c]) => + /\ kind[c] = "link" => exitN[c] >= 1 /\ exitR[c] = why[c] + /\ kind[c] = "monitor" => downN[c] >= 1 /\ downR[c] = why[c] + +ContinueFirst == \A c \in Kids : ~early[c] + +\* P's generation is done only after its owned children. +GenDoneMeansOwnedDone == + gdone => \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} Safety == /\ InheritedPlacement /\ LinkBeforeStarted /\ NameUnique - /\ NameReleasedAtDeath /\ AbortInvisible /\ AckMeansStarted - /\ FailureAfterDeath /\ FailureFlushed /\ TimeoutContained - /\ NoEdgeToDeadParent /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce - /\ ContinueFirst /\ AbandonedNoticeDelivered - -(* Liveness: a child linked to a parent that dies abnormally, also during *) -(* the child's started(), dies. *) + /\ NameHeldUntilPublished /\ NoOverlapOnReuse /\ NameFreeBeforeObservation + /\ AbortInvisible /\ ResultMatchesCommit /\ ReadyIsFinal + /\ TimeoutNeverKillsReady /\ ReadyOnlyWhileOpen /\ FailureAfterPublication + /\ FailureFlushed /\ TimeoutContained /\ NoEdgeToDeadParent + /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce + /\ AbandonedNoticeDelivered /\ ContinueFirst /\ GenDoneMeansOwnedDone + +(* Liveness. *) +\* A child linked to a parent that dies with a non-Normal reason ends. ParentExitReachesChild == - \A c \in Kids : pexAbn[c] ~> phase[c] \in {"dead", "aborted"} + \A c \in Kids : pexAbn[c] ~> phase[c] \in {"published", "aborted"} +\* Every waiting start is answered or abandoned. +StartAnswered == \A c \in Kids : wait[c] = "waiting" ~> wait[c] # "waiting" +\* A managed parent's generation finishes once it died. +OwnedScopeFinishes == (ScopeEnabled /\ pplace = "G" /\ pst = "dead") ~> gdone ============================================================================= From 383ceb1f4899569fa6798a646c78b33e2cade559 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:02:22 -0300 Subject: [PATCH 12/42] verification: smaller per-PR MCStart --- verification/tla/MCStart.cfg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 786e04af..a876c191 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -1,9 +1,9 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2"} - Kinds = {"start", "link", "monitor"} + Kinds = {"link", "monitor"} Explicit = {"none"} - ParentPlaces = {"G", "U"} + ParentPlaces = {"G"} Cap = 3 Named = TRUE TimeoutEnabled = TRUE From ed08307f9a588a0f4c4970850328df24f5ca2b8c Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:04:55 -0300 Subject: [PATCH 13/42] verification: MCStartTimeout for the init-timer races --- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStart.tla | 6 ++--- verification/tla/MCStartTimeout.cfg | 38 +++++++++++++++++++++++++++++ verification/tla/MCStartTimeout.tla | 6 +++++ 4 files changed, 48 insertions(+), 4 deletions(-) create mode 100644 verification/tla/MCStartTimeout.cfg create mode 100644 verification/tla/MCStartTimeout.tla diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index a876c191..99a657bc 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -6,7 +6,7 @@ CONSTANTS ParentPlaces = {"G"} Cap = 3 Named = TRUE - TimeoutEnabled = TRUE + TimeoutEnabled = FALSE ParentCanExit = TRUE CloseEnabled = TRUE MaxMsgs = 0 diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index 52badb3d..486597b9 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -2,7 +2,7 @@ EXTENDS Start \* Per-PR safety instance (design docs/design/unified-start.md, section 15): \* two children racing for one name in a group with room for both and the -\* parent, every waiting variant with its default placement, a managed or -\* unmanaged parent that may exit, group closure, init timeouts, abandoned -\* waits, failure and retry. Most safety controls run here. +\* parent, start_link and start_monitor with their default placement, a +\* managed parent that may exit, group closure, abandoned +\* waits, failure and retry (the init timer is in MCStartTimeout). Most safety controls run here. ============================================================================= diff --git a/verification/tla/MCStartTimeout.cfg b/verification/tla/MCStartTimeout.cfg new file mode 100644 index 00000000..cc0c7308 --- /dev/null +++ b/verification/tla/MCStartTimeout.cfg @@ -0,0 +1,38 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1"} + Kinds = {"start", "link", "monitor"} + Explicit = {"none"} + ParentPlaces = {"G", "U"} + Cap = 3 + Named = FALSE + TimeoutEnabled = TRUE + ParentCanExit = TRUE + CloseEnabled = TRUE + MaxMsgs = 0 + ChildCanTrap = FALSE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation +INVARIANT AbortInvisible +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce +INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone diff --git a/verification/tla/MCStartTimeout.tla b/verification/tla/MCStartTimeout.tla new file mode 100644 index 00000000..19b741f5 --- /dev/null +++ b/verification/tla/MCStartTimeout.tla @@ -0,0 +1,6 @@ +---------------------------- MODULE MCStartTimeout ---------------------------- +EXTENDS Start +\* Per-PR instance for the startup cell's races: one child of every waiting +\* variant of a managed or unmanaged parent; the init timer racing readiness, +\* failure, abandonment and publication; group closure. +============================================================================= From 1066ea140f98413f092c2da836f6e2f9e64ad02e Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:13:22 -0300 Subject: [PATCH 14/42] docs: unified start revision 2 (review resolutions); MCStartScope with one child --- docs/design/unified-start.md | 1656 ++++++++++++++++------------- verification/tla/MCStartScope.cfg | 2 +- verification/tla/MCStartScope.tla | 4 +- 3 files changed, 909 insertions(+), 753 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 20d4cbf2..e8c78894 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1,32 +1,43 @@ # Unified start API (0.9): design contract -Status: design in review, not implemented. Target release 0.9. The -implementation follows after two independent reviews. It is designed against -the 0.8 API as #240 ships it (exit signals, `try_link`, `NoProc`, the 0.8 -API-review cleanups: `send_request`, `#[non_exhaustive]` builders). - -- Model: [`verification/tla/Start.tla`](../../verification/tla/Start.tla) - (configurations `MCStart`, `MCStartContinue`, `MCStartLive`, `MCStartDeep`). -- Client mapping: [unified-start-client-mapping.md](unified-start-client-mapping.md). -- Inputs: the 0.8 API review (Start cluster, footgun 5.11, rank 19); the OTP, - Elixir and Gleam gap report (G1, G2, G7, G8, G14, G15, section 3.1). -- OTP sources: `proc_lib.erl`, `gen.erl` and `gen_server.erl` at OTP-27.1. A - line reference like `gen.erl:191` points into those files. +Status: design in review, revision 2, not implemented. Target release 0.9. +The implementation follows after two independent reviews. It is designed +against the 0.8 API as #240 ships it: exit signals, `try_link`, `NoProc`, and +the 0.8 API-review cleanups (`send_request`, `#[non_exhaustive]` builders). + +Revision 2 answers both round-one reviews (Codex and agy, both "approve after +the listed changes"). What changed and why is in +[Review resolutions](#20-review-resolutions). + +- **Model:** [`verification/tla/Start.tla`](../../verification/tla/Start.tla), + with configurations `MCStart`, `MCStartTimeout`, `MCStartContinue`, + `MCStartScope`, `MCStartLive` and `MCStartDeep`. +- **Client mapping:** [unified-start-client-mapping.md](unified-start-client-mapping.md). +- **Agreed shapes with other designs:** + - the dynamic supervisor (#245, revision 3) and introspection v1 (#256); + - see [section 18](#18-cross-design-agreement). +- **Inputs:** + - the 0.8 API review: the Start cluster, footgun 5.11 and rank 19; + - the OTP, Elixir and Gleam gap report: G1, G2, G7, G8, G14, G15 and + section 3.1. +- **OTP sources:** `proc_lib.erl`, `gen.erl` and `gen_server.erl` at OTP-27.1. + A line reference such as `gen.erl:191` points into those files. ## Summary | # | Decision | OTP | |---|---|---| -| 1 | `start`, `start_link` and `start_monitor` return after the child's `started()`: `Ok` when it is ready, otherwise the error, after the child's death point | `gen_server:start/start_link/start_monitor` are synchronous on `init/1` | -| 2 | `spawn` starts without waiting (fire and forget); `ActorRef::ready()` waits later | `proc_lib:spawn` | -| 3 | `started()` may fail: it returns `()`, `Result<(), E>` or `Init` (ready, continue, error, ignore, stop) | `init/1` returns `{ok,S}`, `{ok,S,{continue,C}}`, `{error,R}`, `{stop,R}`, `ignore` | -| 4 | `Init::continue_with(msg)`: a turn that runs before any mailbox message | `handle_continue` | -| 5 | One `StartOptions`: placement, typed name, init timeout, label, stack size, per-instance bounds; reserved slots for 0.10 | `{local,Name}`, `{timeout,T}`, `spawn_opt`, `set_label` | -| 6 | Linked and monitored children inherit the parent's placement (its group or backend). Plain `start`/`spawn` never inherit; `placement_of(&ctx)` opts in | none (one node, one scheduler pool); see [Placement](#4-placement-and-inheritance) | -| 7 | Stage 1 runs eagerly in the caller: admission, link or monitor, name, dispatch. A refused start leaves no trace | `spawn_opt([link, monitor])` is atomic; `gen:init_it` registers the name before `init` | -| 8 | A waiting `start_link` consumes the failed child's `Exit` (trapping parent) and `start_monitor` its `Down` | `proc_lib:sync_start` + `flush_EXIT`; `gen:monitor_return` | -| 9 | The init timeout unlinks or demonitors first, then kills | `kill_flush_EXIT` | -| 10 | Removed: `start_on`, `start_with_backend`, `ExecutionGroup::start`, `start_linked`; `ChildSpec::{startup_timeout, max_outstanding, max_native_jobs}` move into `StartOptions` | | +| 1 | `start`, `start_link` and `start_monitor` wait for the child's `started()`. They answer `Started` when it is ready, `Ignored` for `Init::ignore()`, or the error. An error is returned only after the child's publication: it is dead, its state is dropped and its name is free | `gen_server:start*` are synchronous on `init/1`; `sync_start` awaits the `'DOWN'` | +| 2 | `spawn` does not wait; `ActorRef::ready()` returns the same committed result later | `proc_lib:spawn` | +| 3 | `started()` may fail. It returns `()`, `Result<(), E>` or `Init`: ready, continue, error, stop or ignore | `init/1` returns `{ok,S}`, `{ok,S,{continue,C}}`, `{error,R}` (OTP 26 `init_fail/3`), `{stop,R}` or `ignore` | +| 4 | Each start has one **startup cell** with one terminal commitment: ready, failed, ignored or timed out. It is committed exactly once, by the child at the end of `started()` or by the init timer. Readiness is final and independent of the waiter | `init_ack` is sent before the loop starts; the timer and the ack race in the caller's `receive` | +| 5 | `Init::continue_with(msg)`: a turn that runs before any mailbox message. It is not a barrier against termination | `handle_continue` | +| 6 | One `StartOptions`, typed by the actor: placement, typed name, init timeout (default infinity), label, stack size, per-instance bounds, and reserved slots for 0.10 | `{local,Name}`, `{timeout,T}`, `spawn_opt`, `set_label` | +| 7 | Linked and monitored children inherit the parent's placement. A linked child in the parent's system is owned by the parent's scope (S8) | none (one node); see [section 4](#4-placement-and-inheritance) | +| 8 | Stage 1 is one commit under a fixed lock order: admission, the parent's state, the edge and the name, all or nothing. A refused start is never observable | `spawn_opt([link, monitor])` is atomic | +| 9 | While the starter waits, a failed start's `Exit` (trapping parent) or `Down` is consumed by the error. A non-trapping parent still exits on a non-`Normal` init failure. An init timeout never reaches the parent | `sync_start`, `flush_EXIT`, `kill_flush_EXIT`, `gen:monitor_return` | +| 10 | Supervised children keep `startup_timeout`, the deadline for the whole attempt (factory and `started()`); infinity is allowed. `ignore` keeps the spec, as OTP | `supervisor.erl` | +| 11 | Removed: `start_on`, `start_with_backend`, `ExecutionGroup::start`, `start_linked` | | ## 1. The problem @@ -44,23 +55,16 @@ The 0.8 start surface: The resulting problems: -- **Footgun (API review 5.11).** A child that a managed actor starts with - `start_linked` is outside the parent's group. `spawn_helper` then returns - `UnmanagedActor`, ordered shutdown never sees the child, and group - teardown can abort it without `stopped()` (semantics S6). +- **Footgun (API review 5.11).** A linked child of a managed actor is outside + the parent's group, ordered shutdown and helpers. - **No atomic monitor at start.** `ctx.monitor` after `start()` races the - child's death, and since 0.8 a child that already died reports `NoProc` - instead of its real reason. + child's death, and a child that already died reports `NoProc`. - **Startup failure is invisible outside supervision.** `started()` cannot - fail, and `start()` returns an `ActorRef` even when `started()` will - panic. The client sweep found the ethrex L2 sequencer starting actors one - by one and only logging startup failures. + fail. - **No continuation.** A self-send from `started()` queues behind anything - that arrived during `started()`. The dynamic supervisor design (#245) - moves ethrex's RLPx handshake into "the first message", and a peer message - or a `terminate_child` can overtake it. + that arrived during `started()`. - **Linking at start covers only one placement:** `start_linked` is - Async-only and `start_on` has no linked form. + Async-only, and `start_on` has no linked form. ## 2. API @@ -68,16 +72,11 @@ The resulting problems: ```rust pub trait ActorStart: Actor { - /// proc_lib:spawn. Unmanaged Async, no wait, no parent. - fn spawn(self) -> ActorRef; - /// gen_server:start. Unmanaged Async; resolves after started(). - fn start(self) -> Starting>; - /// gen_server:start_link. The parent's placement; linked before started(). + fn spawn(self) -> ActorRef; // proc_lib:spawn + fn start(self) -> Starting>; // gen_server:start fn start_link(self, parent: &Context

) -> Starting>; - /// gen_server:start_monitor. The parent's placement; monitored before started(). fn start_monitor>(self, parent: &Context

) -> Starting, MonitorRef)>; - /// The same four verbs with options (spawn_opt and the gen_server Opts list). fn with_options(self, opts: impl Into>) -> Starter; } @@ -91,46 +90,42 @@ impl Starter { -> Starting, MonitorRef)>; } -/// The child is already started (stage 1 ran in the call); awaiting it -/// waits for started(). 'static + Send: it owns what it needs. +/// The child already exists (stage 1 ran in the call). Awaiting it reads the +/// startup cell; dropping it abandons the wait (section 5.4). 'static + Send. #[must_use = "the actor runs; await this to learn whether it started"] pub struct Starting { /* ... */ } impl IntoFuture for Starting { - type Output = Result; + type Output = Result, StartError>; } impl Starting { - /// The child's reference without waiting: Err only for a stage-1 refusal. - /// Dropping the Starting abandons the wait (section 5.5). + /// The unready instance, without waiting or consuming the result (Err + /// only for a stage-1 refusal). For wiring during initialisation and for + /// keeping control of a stuck initializer. pub fn actor_ref(&self) -> Result, &StartError>; } +/// OTP {ok, Pid} | ignore. Shared with the dynamic supervisor (section 18). +#[must_use] +pub enum StartOutcome { Started(T), Ignored } +impl StartOutcome { + pub fn started(self) -> Option; + pub fn is_ignored(&self) -> bool; +} + impl ActorRef { - /// The ack for any placement (0.8: managed only, returning SystemError). - pub async fn ready(&self) -> Result<(), StartError>; + /// The committed startup result of this instance, for every placement. + pub async fn ready(&self) -> Result, StartError>; } ``` ### 2.2 threads (`spawned_concurrency::threads`) -The four verbs and `with_options` have the same names and meanings. The -waiting forms block the calling thread instead of returning a future: - -```rust -fn spawn(self) -> ActorRef; -fn start(self) -> Result, StartError>; -fn start_link(self, parent: &Context

) -> Result, StartError>; -fn start_monitor>(self, parent: &Context

) - -> Result<(ActorRef, MonitorRef), StartError>; -fn with_options(self, opts: impl Into>) -> Starter; -// ActorRef::ready(&self) -> Result<(), StartError> (blocking) -``` - -The only difference between the modes is `.await` against blocking, as for -requests. threads mode has one placement (a thread per actor), so its -`StartOptions` has no `group`, `backend` or `placement_of`. These are -tasks-only, like `ActorSystem` today (API review rank 27). A threads-mode -`start_link` from a threads parent inherits "a thread", which is the only -placement there is. +The verbs and `with_options` have the same names. The waiting forms block +and return `Result, StartError>`. `ready()` blocks. threads +mode has one placement (a thread), so its `StartOptions` has no `group`, +`backend` or `placement_of`; these are tasks-only, like `ActorSystem`. The +only difference between the modes is `.await` against blocking, as for +requests. ### 2.3 Fallible `started()` and `Init` @@ -144,102 +139,125 @@ fn started(&mut self, ctx: &Context) -> impl Into> {} #[must_use] pub struct Init { /* private */ } impl Init { - pub fn ready() -> Self; // {ok, S} - pub fn continue_with(msg: M) -> Self // {ok, S, {continue, C}} - where A: Handler; - pub fn error(reason: impl Display) -> Self; // {error, R}: exit Normal - pub fn stop(reason: ExitReason) -> Self; // {stop, R}: exit R - pub fn ignore() -> Self; // ignore: exit Normal + pub fn ready() -> Self; // {ok, S} + pub fn continue_with(msg: M) -> Self where A: Handler; // {ok, S, {continue, C}} + pub fn error(cause: impl Display) -> Self; // {error, R}: exit Normal (OTP 26 init_fail/3) + pub fn stop(reason: ExitReason) -> Self; // {stop, R}: exit R + pub fn ignore() -> Self; // ignore: exit Normal } -impl From<()> for Init; // ready -impl From> for Init; // Err -> error +impl From<()> for Init; // ready +impl From> for Init; // Err -> error impl From, E>> for Init; // Err -> error ``` -The signature was prototyped with rustc 1.99 (scratch crate). All of these -compile unchanged: +The signature was prototyped with rustc 1.99. These all compile, and existing +actors change nothing: + +- `async fn started(..) {}` and the default body; +- `-> Result<(), io::Error>`, `-> Result, String>`, and the + threads form. -- the existing `async fn started(&mut self, ctx: &Context) {}`; -- the default body; -- `-> Result<(), io::Error>` and `-> Result, String>`; -- the threads form. +The `#[started]` macro moves the method into `impl Actor` with its signature +intact. `Init::continue_with` is generic over `M` with `A: Handler`. It +needs no associated type: an associated-type default (`type Continue = ()`) +is still unstable on stable Rust. The continuation is boxed only when one is +returned. -The `#[started]` macro hook passes the method's return type through. No -existing actor changes. Return-position `impl Trait` in traits is stable -since Rust 1.75, below the workspace's 1.93. +`Init::stop(ExitReason::Normal)` ends the child `Normal` and answers +`InitExited { reason: Normal }`. Use `Init::error` for a diagnostic and +`Init::ignore` for a deliberate opt-out. `Init::stop` is for a reason that +links should see, as `{stop, R}`. -| `started()` outcome | Child | `stopped()` | Waiting starter gets | Link notice / `Down` reason | +**Resolution when `started()` returns.** Signals received during `started()` +are held (0.8 phase 0, otp-exit-signals E2/E3). They are resolved with the +trap flag that `started()` leaves, in this precedence. The first matching row +decides both the starter's result and the final exit reason. + +| # | Condition | Exit reason | `stopped()` | Startup cell | |---|---|---|---|---| -| `()`, `Ok(())`, `Init::ready()` | runs | later, as usual | `Ok` | none | -| `Init::continue_with(m)` | runs; `m` is handled first | later | `Ok` (before `m` runs, as OTP's `init_ack`) | none | -| `Err(e)`, `Init::error(e)` | exits `Normal` | no | `InitFailed(e)` | `Normal` | -| `Init::ignore()` | exits `Normal` | no | `Ignored` | `Normal` | -| `Init::stop(r)` | exits `r` | no | `InitExited(r)` | `r` | -| panic | exits `Panic("panic in started(): ..")` | no | `InitExited(Panic(..))` | `Panic` | -| ready, but `ctx.stop*` called in `started()` | exits that reason | yes (L1, unchanged) | `InitExited(r)` | `r` | -| ready, but a held signal ends it (kill, non-`Normal` parent exit, link notice) | exits per X3 | per X3 | `InitExited(r)` | `r` | -| still running at the init timeout | killed | no | `InitTimeout` | none (edge removed first) | - -The ack is `Ok` exactly when `started()` returned a ready `Init` and no exit -was pending when it returned. That is also the point where the managed -readiness barrier commits (L2). Typed init data, as Gleam's `returning` and -`proc_lib:init_ack(Ret)`, is not included; see [alternatives](#15-alternatives-considered). +| 1 | the init timer committed `timedout` (section 6) | `Killed` | no | `timedout` → `InitTimeout` | +| 2 | a kill arrived (`kill()`, `exit(Kill)`, the S8 scope deadline) | `Killed` | no | failed → `InitExited { Killed }` | +| 3 | the child does not trap, and a held non-`Normal` signal (parent exit, link notice, exit signal) arrived; the first in arrival order wins | that reason | no | failed → `InitExited { r }` | +| 4 | `started()` panicked | `Panic("panic in started(): ..")` | no | failed → `InitExited { Panic }` | +| 5 | `ctx.stop*(r)` was called in `started()` | `r` | yes (a self-exit, L1) | failed → `InitExited { r }` | +| 6 | `Init::stop(r)` | `r` | no | failed → `InitExited { r }` | +| 7 | `Err(e)` / `Init::error(e)` | `Normal` | no | failed → `InitFailed { cause: e }` | +| 8 | `Init::ignore()` | `Normal` | no | ignored → `Ok(Ignored)` | +| 9 | `Init::ready()` / `continue_with(m)`, but a managed scope's admission closed | `Shutdown` | no | failed → `InitExited { Shutdown }` | +| 10 | `Init::ready()` / `continue_with(m)` | runs | later | **ready** → `Ok(Started(..))` | + +- **Held signals in rows 9-10.** A held `Normal` signal to a non-trapping + child is dropped (E2). For a child that traps, every held non-kill signal is + queued as a mailbox entry in arrival order. That includes a parent exit, + even a `Normal` one (E3: it ends the child when handled). The child + commits ready, runs its continuation first, then handles the queued entries. + This is OTP's behaviour for an `init/1` that sets `trap_exit` and returns + `{continue, C}`. +- **A superseded diagnostic.** When a higher row wins over a returned + `Init::error(e)`, `e` is logged at debug level with the actor id. The + result is the winning row's. +- **Cooperative deviation.** OTP applies a signal when it arrives, so a + signal before `process_flag(trap_exit, true)` kills a non-trapping process + immediately. Spawned reads the flag when `started()` returns. This is + 0.8's documented Rust deviation (phase 0, R: no preemption). Section 11 + labels it as such. ### 2.4 `StartOptions` ```rust -#[derive(Clone)] #[non_exhaustive] -pub struct StartOptions { /* private fields */ } -impl Default for StartOptions { .. } +pub struct StartOptions { /* private; PhantomData A> */ } +impl Clone for StartOptions; // no A: Clone bound +impl Default for StartOptions; impl StartOptions { pub fn new() -> Self; - // placement (tasks only) - pub fn group(self, group: &ExecutionGroup) -> Self; - pub fn backend(self, backend: Backend) -> Self; - pub fn placement_of(self, ctx: &Context

) -> Self; - // naming and diagnostics + pub fn group(self, group: &ExecutionGroup) -> Self; // tasks + pub fn backend(self, backend: Backend) -> Self; // tasks + pub fn placement_of(self, ctx: &Context

) -> Self; // tasks pub fn name(self, name: impl Into>) -> Self; - pub fn label(self, label: impl Into>) -> Self; - // start behaviour - pub fn init_timeout(self, timeout: Duration) -> Self; - // per-instance resources + pub fn label(self, label: impl Into>) -> Self; // initial set_label + pub fn init_timeout(self, timeout: impl Into) -> Self; pub fn stack_size(self, bytes: usize) -> Self; pub fn max_outstanding(self, max: usize) -> Self; pub fn max_native_jobs(self, max: usize) -> Self; } -impl From<&ExecutionGroup> for StartOptions; // .group(g) -impl From for StartOptions; // .backend(b) +pub enum Timeout { After(Duration), Infinity } // From +impl From<&ExecutionGroup> for StartOptions; +impl From for StartOptions; ``` | Field | Default | Meaning | OTP | |---|---|---|---| -| placement | `start_link`/`start_monitor`: the parent's; `start`/`spawn`: unmanaged `Async` | [section 4](#4-placement-and-inheritance). The last of `group`/`backend`/`placement_of` wins | none | -| `name` | none | registered for this instance at stage 1, released at its D ([section 6](#6-registered-names)) | `{local, Name}` | -| `label` | none | shown in tracing spans, panic and stall diagnostics, `ShutdownReport` actor details and introspection | `proc_lib:set_label/1` (OTP 27) | -| `init_timeout` | none (wait forever); `ChildSpec` keeps its 5 s default | [section 7](#7-init-timeout) | `{timeout, T}`, default `infinity` (`gen.erl:716`) | -| `stack_size` | runtime default | honoured where the actor owns a thread: threads mode, `Backend::Thread`. Elsewhere `InvalidOptions`; group stacks are rank 8c | none (BEAM stacks grow) | -| `max_outstanding`, `max_native_jobs` | `Actor::MAX_OUTSTANDING`, `MAX_NATIVE_JOBS` | per-instance overrides; were on `ChildSpec` only | none | +| placement | `start_link`/`start_monitor`: the parent's; otherwise unmanaged `Async` | section 4; the last of `group`/`backend`/`placement_of` wins | none | +| `name` | none | committed with the start, released at publication (section 7) | `{local, Name}` | +| `label` | none | the initial value of introspection's `ctx.set_label` (#256): tracing, diagnostics, `ActorInfo.label` | `set_label/1` (OTP 27) | +| `init_timeout` | `Infinity` | section 6 | `{timeout, T}`, default `infinity` (`gen.erl:716-722`) | +| `stack_size` | runtime default | only where the actor owns a thread (threads mode, `Backend::Thread`) | none | +| `max_outstanding`, `max_native_jobs` | the `Actor` constants | per instance | none | + +**Validation.** It happens at stage 1, before anything is visible, and +returns `InvalidOptions`: -**Reserved for 0.10.** These are planned as additive builder methods, with no -new start API: +- a zero `init_timeout` (OTP's `{timeout, 0}` would always fail; use a real + bound); +- a zero `max_outstanding`, `max_native_jobs` or `stack_size`; +- a `stack_size` on a pooled placement; +- a `backend` in a `ChildSpec`. -- `mailbox(Unbounded | Bounded(n))` (rank 17); -- `max_memory` (rank 18a, OTP `max_heap_size`); -- a message-priority opt-in (OTP 28 priority messages, gap G11). +These rules are the same in both modes. -**Not copied from `spawn_opt`:** +**Typing.** `StartOptions` is typed by the actor, and `name` takes a +`Name`. Passing a `Name` to an `A` start is a compile error (trybuild +`name_type_must_match_actor`). `Name` and `StartOptions` are `Clone` +and `Send` without bounds on `A`. `Name` keys the global registry until +rank 12 scopes it. -- `fullsweep_after`, `min_heap_size` and `message_queue_data`: there is no - per-actor heap. -- `priority`: a cooperative runtime cannot honour per-actor CPU priority. - Groups are the bulkhead. -- `hibernate_after` and `debug`. +**Reserved for 0.10:** `mailbox(..)` (rank 17), `max_memory` (rank 18a), and +a message-priority opt-in (G11). -`Name` is a typed name: `Name::new("peer_table")`, plus -`From<&'static str>` and `From`. `name.whereis() -> Option>` -cannot mistype a lookup (G14). It still keys the global registry until rank 12 -scopes it. +**Not copied from `spawn_opt`:** `fullsweep_after`, `min_heap_size`, +`message_queue_data`, `priority`, `hibernate_after` and `debug`. None has a +counterpart in a cooperative runtime without per-actor heaps. ### 2.5 `StartError` @@ -247,549 +265,650 @@ scopes it. #[derive(Debug, Clone, thiserror::Error)] #[non_exhaustive] pub enum StartError { + // stage 1: nothing exists Closed, // group, system or the parent's scope no longer admits Capacity, // thread_group full AlreadyStarted { name: String, existing: Option }, - ParentExited, // start_link/start_monitor through a Context whose actor's exit began - InvalidOptions(&'static str), // e.g. stack_size on a pooled placement, backend in a ChildSpec - InitTimeout, - InitFailed(String), // started() returned an error - InitExited(ExitReason), // the child exited before it was ready - Ignored, // started() returned Init::ignore() - System(SystemError), // runtime construction and other placement failures + ParentExited, // start_link/start_monitor through a Context whose exit began + InvalidOptions(&'static str), + System(SystemError), // executor or runtime construction failed + // stage 2: the child existed and is gone (or still cleaning up: `pending`) + InitFailed { cause: String, pending: Option }, + InitExited { reason: ExitReason, pending: Option }, + InitTimeout { pending: Option }, } ``` -The `Init*` names follow Gleam's `actor.StartError`. `Ignored` is an error -variant because Rust has no third return shape for OTP's bare `ignore`. -Supervisors treat it as "nothing started" (section 9). -`SystemError::StartupFailed` is removed: `ready()` now returns -`StartError::InitExited`. +- **`pending`.** It is `None` when the child was published before the + answer: dead, state dropped, notices sent, name and slot free. It is + `Some(handle)` when the bounded wait for publication expired first + (section 6). The handle is instance-bound: `wait_exit` reaches the real + end, and the name and slot stay held until then. +- **`ignore`** is the `Ok(StartOutcome::Ignored)` outcome, not an error, as + OTP returns `ignore` beside `{ok, Pid}` (Codex 9). +- **Removed:** `SystemError::StartupFailed`. `ready()` returns these + variants. + +## 3. Stage 1: one commit + +Stage 1 runs inside the call, before any `.await`, in two parts. + +1. **Prepare (invisible).** Validate the options; build the actor's shared + state, mailbox, startup cell and edges in private memory; and acquire the + executor resources. For `Backend::Thread` and threads mode, spawn the + thread parked on a start latch. For an async placement, reserve the task + slot. A failure returns `System` or `InvalidOptions`, and the state is + dropped outside every framework lock. Nothing was visible. +2. **Commit (atomic).** Take, in this order: + 1. the target group's admission lock; + 2. the parent's exit-request lock, the one an exit begin takes; + 3. the parent's link or monitor table lock; + 4. the registry lock. + + Under all of them, check in any order: + - the group is open and below capacity (`Closed`, `Capacity`); + - the parent's exit has not begun (`ParentExited`); + - the name is free (`AlreadyStarted`). + + On success, apply everything before releasing: + - the admission slot; + - the parent's edge, created with the startup cell's mark; + - the name; + - the actor's entry in introspection's actor table. + + Release in reverse order, then open the start latch, which cannot fail. + On a refusal nothing was applied: release the locks, drop the prepared + state and stop the parked thread. + +**Guarantees.** A refused start is never observable: no `whereis` can return +it, no `Capacity` refusal is caused by it, and no edge, notice or hook +exists. Every admission becomes visible at one instant. This is the +`spawn_opt([link, monitor])` atomicity. Model: `AbortInvisible`, with a +`Lookup` observer; control `SplitCommit`, which registers the name before the +rest of the commit. + +**Lock-order obligations for the implementation.** The order above is global. +No 0.8 path takes a registry lock and then a link-table, exit-request or +group lock, and the implementation keeps it so. A Loom test covers a commit +racing the parent's exit begin and a concurrent commit of the same name. + +**Bare `spawn()`** is infallible in its signature. An executor construction +failure in the prepare step panics with the `SystemError`, as +`std::thread::spawn` does. `with_options(..).spawn()` returns it instead. + +**Forgotten `.await`.** Stage 1 is eager, so `x.start();` without `.await` +still starts the actor, and the dropped `Starting` abandons the wait +(section 5.4). In the migration a 0.8 statement `x.start();` therefore keeps +working. It also warns under `#[must_use]`, which the clients' clippy +`-D warnings` gate turns into an error. -## 3. Two stages +## 4. Placement and inheritance -**Stage 1 runs in the caller, synchronously, inside the call** (`spawn`, -`start*` or `Starter::*`, before any `.await`). Its steps, in order: +**Rule.** The placement is the first of: -1. **Admit** to the placement: explicit, else inherited, else unmanaged. A - group admits while it is open and has room. Otherwise the start returns - `Closed` or `Capacity`. -2. **Link or monitor** (`start_link`/`start_monitor`). The edge is recorded - in one transaction with the parent's lifecycle state. A parent whose exit - began, or which is past its death point, refuses with `ParentExited`. -3. **Register** the name, atomically, or refuse with `AlreadyStarted`. -4. **Dispatch** the child to its executor. +1. an explicit placement; +2. for `start_link` and `start_monitor`, the parent's placement; +3. unmanaged `Backend::Async` (threads mode: a thread). -**A refusal at any step leaves no trace:** +| Parent | Child of `start_link`/`start_monitor` | +|---|---| +| managed, async group `g` | `g` | +| managed, thread group `g` | `g` (any free slot; `Capacity` if none) | +| managed, pinned group `g`, shard `i` | `g`, shard `i` | +| supervised child on `g` | `g` | +| unmanaged, backend `b` | unmanaged, backend `b` | +| threads mode | a thread | -- the admission slot, the edge and the name are released; -- the state is dropped without either hook running; -- no `Exit` or `Down` is ever delivered, and no actor id is published. +**No fallback.** A closed or full inherited group fails the start (`Closed`, +`Capacity`). + +**Only `start_link` and `start_monitor` inherit.** Plain `start`/`spawn` take +no `Context`. Inheriting for them would need an ambient current actor, which +costs something on every turn. It would also not follow work into helpers, +and would silently move 20 ethrex call sites into groups. +`placement_of(ctx)` is the explicit form. OTP's `gen_server:start` and +`start_monitor` also give the child no parent (`gen.erl:137-142`). + +### 4.1 S8: linked children owned by the parent's scope + +Ownership is decided separately from inheritance, by this rule: + +- **Owned.** A `start_link` child is owned by its parent's scope when the + parent is managed and the child is admitted to a group of the parent's + `ActorSystem`. That covers an inherited group, an explicit group of the same + system, and a pinned shard. +- **Not owned.** A `start_monitor` child, a child placed unmanaged, and a + child in another system. It is linked only, as in OTP, and is owned by its + own placement: the other system's shutdown, or nobody (S6). + +**What ownership means.** + +- **System shutdown.** It reaches an owned child through its parent. The + parent's turn in the reverse admission order (S2) ends when the parent and + its owned descendants are published, or at the grace deadline. +- **Supervised parents.** The generation counts as finished (V3) only when + its owned descendants are published too. A replacement never overlaps + them. +- **Unlink.** Ownership persists after `unlink`. That is not OTP link + semantics; it is scope ownership, justified by Rust resource lifetimes. A + descendant may hold resources (sockets, files, a native slot) that the + parent's replacement or the system's shutdown report must see released, as + for 0.8's registered helpers. + +**Deadlines.** One interval, the parent's row in the E3 deadline table, +covers the whole scope: + +- `shutdown_timeout` from the parent's signal when the supervisor or system + stops it; +- `shutdown_timeout` from the exit begin when the parent exits on its own + (5 s unmanaged default). + +At its expiry the scope deadline sends a kill to every still-alive member, +the parent and each owned descendant, linked or not. Each member is killed +directly; nothing waits for the parent's D. One more interval later, the +scope is stuck, with the existing V3/E3 consequences. + +A parent that never reaches D does not delay the descendants' kill. System +grace expiry likewise kills every remaining actor of every scope directly +(S2, S3). A descendant still linked normally ends earlier, by its parent +exit at the parent's D (E3). + +Model: `MCStartScope`, with invariant `GenDoneMeansOwnedDone` (control +`EarlyGenDone`) and liveness `OwnedScopeFinishes` (control `NoScopeKill`: the +deadline spares unlinked descendants). + +## 5. The startup cell and `start_link`/`start_monitor` + +### 5.1 The cell + +Each start that has a waiter (`start`, `start_link`, `start_monitor`), and +each start with an init timer, gets one cell. Its two parts: + +```text +commitment: Pending -> Ready | Failed(cause, reason) | Ignored | TimedOut (exactly once) +waiter: Waiting -> Consumed | Abandoned (exactly once) +``` -This is L2's rule for a closed admission, extended to every refusal. The -implementation may reorder the internal locks, as long as the result is -indistinguishable from this order. The model checks interleavings of parent -death, group closure and a competing registration between every two steps -(`AbortInvisible`). +The cell has one lock. The edges created at the commit carry the cell's mark. -**Stage 2 runs on the child.** `started()` runs with signals held (phase 0 of -the exit-signals design), and they are resolved when it returns. The waiting -forms resolve then, as in the table in 2.3. +**Ready.** The child commits it at the end of `started()` under the cell +lock, when row 10 of 2.3 applies and (for managed actors) admission is still +open. The same critical section: -Because stage 1 is eager, `x.start();` without `.await` still starts the -actor. A forgotten `.await` therefore degrades to `spawn` (plus the -`#[must_use]` warning) and never to "the actor silently never started". -This matters for the migration: a 0.8 statement `x.start();` keeps working. +1. commits `Ready`; +2. disarms the init timer; +3. clears the start marks on the edges; +4. publishes the result to the waiter and to `ready()`. -## 4. Placement and inheritance +Only then does the child run its continuation or dispatch. Readiness is +final: a later death of the child is an ordinary death with an ordinary +`Exit`/`Down`, whether or not the waiter has polled yet (agy P0-1, Codex 1). +OTP sends `init_ack` before entering the loop (`gen_server.erl:2185-2201`). +Model: `ReadyIsFinal` (control `DivertAfterReady`), `ResultMatchesCommit`, +`ReadyOnlyWhileOpen` (control `ReadyAfterClose`). -**Rule.** The placement is the first that applies: +**Failed / Ignored.** The child commits it at the end of `started()` (rows +2-9), or at a kill before `started()` began. -1. an explicit placement (`group`, `backend`, `placement_of`); -2. for `start_link` and `start_monitor`, the parent's placement; -3. unmanaged `Backend::Async` (threads: a thread). +**TimedOut.** The framework-owned init timer commits it (section 6). -The parent's placement is fixed when the parent was admitted: +Exactly one of these wins the `Pending` state. In particular, the timer can +win only while the cell is `Pending`, and the child's readiness commit can +win only while the timer has not committed (`TimeoutNeverKillsReady`, control +`TimerIgnoresCommit`; agy P0-2, Codex 2). -| Parent | Child of `start_link`/`start_monitor` | -|---|---| -| managed, async group `g` | admitted to `g` | -| managed, thread group `g` | admitted to `g` (any free slot; `Capacity` if none) | -| managed, pinned group `g`, shard `i` | `g`, shard `i` (as `Scope::group()` keeps a pinned selection) | -| supervised child on `g` | `g` | -| unmanaged, backend `b` | unmanaged, backend `b` | -| threads mode | a thread | +### 5.2 Edges, death and publication + +The child's death has three steps, as in otp-exit-signals: + +1. **D.** The edges close and their notices are taken. +2. **Destruction.** Queued messages and the state `A` are dropped, outside + framework locks. +3. **Publication**, in this order: + 1. release the name; + 2. release the admission slot; + 3. dispose of each taken notice through the cell; + 4. publish `wait_exit`; + 5. answer the waiter. + +The waiter answers only from publication (agy P0-3, Codex 5), so a retry never +overlaps the old instance's `Drop`. This is OTP's completed-failure boundary, +since `sync_start` awaits the `'DOWN'`. Model: `FailureAfterPublication` +(control `EarlyFailAck`), `NoOverlapOnReuse` (control `UnregisterAtDeath`), +`NameFreeBeforeObservation` (control `LateUnregister`). -**No fallback.** If the inherited group is closed (system shutdown began, S1) -or full, the start fails with `Closed` or `Capacity`. Silently escaping the -group is the 0.8 bug this design removes. - -**Ownership of an inherited child.** - -- **`start_link` child of a managed parent.** It is admitted under the - parent's scope, as nested supervisors are (admission parent), not as a new - system root. Its parent for E3 is the linking actor. System shutdown reaches - it through that parent: the parent's turn in the reverse admission order - (S2) ends when the parent and its `start_link` descendants have finished, - or at the grace deadline. The parent's exit reaches them as a parent exit - (E3). If the parent is supervised, its generation counts as finished (V3) - only when its `start_link` descendants have also finished. A descendant - still alive one cleanup interval (the row for the parent in the E3 deadline - table) after the parent's D is killed; one still alive a second interval - later makes the slot stuck. Without this, a trapping child would get the - system's `Shutdown` as a non-parent signal, which it can ignore. -- **`start_monitor` child of a managed parent.** It is a standalone system - root admitted after its starter, as `start_on` is today; its E3 parent is - the system. The reverse admission order stops it before its starter. -- **Unmanaged parents.** Inheritance copies the backend only. Unmanaged - children stay unowned (S6). - -**Why only `start_link` and `start_monitor`.** OTP has one node and one -scheduler pool, so every process is "in the parent's placement". The closest -Spawned analogue of the node is the parent's group, and a linked child -outside its parent's group is the footgun. Plain `start` and `spawn` take no -`Context`. Inheriting for them would need an ambient "current actor" -(task-local or thread-local). That would cost something on every turn, would -not follow work into `tokio::spawn` or helper tasks, and would silently move -the 20 `.start()` calls in ethrex's actor code into groups. The explicit form -is `with_options(StartOptions::new().placement_of(ctx))`. OTP's -`gen_server:start` and `start_monitor` also give the child no parent -(`gen.erl:137-142`, parent `self`). Only `start_link` makes the caller the -parent. - -## 5. `start_link` and `start_monitor` - -### 5.1 Atomicity - -- The edge (link or monitor) is recorded before the child's `started()` - begins, in every mode and placement (model `LinkBeforeStarted`). OTP sets - it atomically with the spawn: `spawn_opt(.., [link, monitor])`. -- The child never runs without its edge. The edge never exists for a child - that a stage-1 refusal aborted. -- No edge survives the parent's death point (`NoEdgeToDeadParent`). - -### 5.2 Parent state - -- **The parent is running** (the normal case: the call comes from its turn). - The edge is recorded, and the parent's turn waits for the ack (M1: no other - message of the parent runs meanwhile, as OTP's caller blocks in - `sync_start`). -- **The parent's exit began, or it is past D.** This is possible only through - a cloned `Context` used from a helper or job. The start returns - `ParentExited`, and no child exists. 0.8 instead spawned a doomed child - that received a `NoProc` parent exit (otp-exit-signals E3, "a parent - already past D"). That bullet is replaced. OTP has no counterpart (a dead - process cannot call); the closest rule is X2, where a link from an exiting - caller is inert. -- **The parent dies after the edge is recorded, during the child's stage 1 - or `started()`.** Its D sends the child a parent exit. The exit is held - until `started()` returns and then applied by E3: a non-`Normal` reason ends - the child, and the ack reports `InitExited(reason)`; `Normal` is ignored by - a non-trapping child. For `start_monitor` the monitor is cancelled at the - parent's D, and the child continues (it has no parent). Model: - `ParentExitReachesChild` (liveness), control `DropHeld`. - -### 5.3 Startup failure - -Every failure is returned after the child's D, as OTP's `sync_start` awaits -the `'DOWN'` (`proc_lib.erl:434-449`, `FailureAfterDeath`). By then the -child's name is free, so the start can be retried at once -(`NameReleasedAtDeath`), as `gen_server` unregisters before notifying -(`gen_server.erl:2203-2210`). - -- **`start_link`, trapping parent.** The child's `Exit` is consumed by the - waiting call and never reaches `exit_received`, as `flush_EXIT` - (`proc_lib.erl:606`). -- **`start_link`, non-trapping parent.** A non-`Normal` reason is applied as - a link notice. The call returns the error to the handler, and the parent - then exits with that reason after its current handler (X3; cooperative, - L3). In OTP the link kills the caller before `start_link` returns (the - `proc_lib:start_link` docs). A `Normal` exit (`Init::error`, `ignore`) is - ignored. -- **`start_monitor`.** The `Down` is consumed, as `gen:monitor_return` - (`gen.erl:172-182`); the call returns the error. Model: `FailureFlushed`. - -### 5.4 Ok - -`Ok` means `started()` returned a ready `Init` (`AckMeansStarted`). After -`Ok`, the edge behaves as any `ctx.link`/`ctx.monitor` edge (X1, X3, X5). - -### 5.5 Abandoning the wait - -Dropping the `Starting` (tasks) before it resolves abandons only the wait. The -child keeps running with its edge and its name. A notice already diverted to -the waiting call is handed to the parent as if it had never been diverted. The -handover is linearised with the child's D, so the notice is delivered exactly -once (`AbandonedNoticeDelivered`, control `AbandonDropsNotice`). An abandoned -`start` is `spawn`. `Starting` is `'static + Send`, so a handler may also -hand it to a job and await it outside its turn. The consumption then happens -wherever it is awaited. - -**Mechanism (no cost on send or dispatch).** While a waiting call exists, the -edge carries an "ack pending" mark. The child's death notice for a marked edge -goes to the call's slot instead of the parent's mailbox. The ack (`Ok` or -abandon) clears the mark under the edge's lock. Unmarked edges, which are -every edge after startup, take the 0.8 path unchanged. - -## 6. Registered names - -- **When.** At stage 1, before `started()`, so the child can be found while - it initialises (messages queue; a continuation still runs first). OTP - registers in `gen:init_it` before `init/1` (`gen.erl:194-201`). -- **Clash.** The start is refused with - `AlreadyStarted { name, existing }`. `started()` does not run, and nothing - is linked or delivered. `existing` is the holder's `ChildHandle` when an - actor holds the name, or `None` when a value from the untyped `register` - does. OTP returns `{error, {already_started, Pid}}` and its loser exits - `normal` (`gen.erl:198-200`). A `normal` exit reaches no non-trapping - caller, and the trapping caller's `'EXIT'` is consumed, so nothing is - observable either. -- **Uniqueness.** Two concurrent starts with one name: exactly one is - admitted (`NameUnique`, control `DoubleRegister`). -- **Release.** At the holder's D, before any link notice, `Down`, - `wait_exit`, ack or `ShutdownReport` entry can observe the death - (`NameReleasedAtDeath`, control `LateUnregister`). This applies to every - placement, unmanaged ones included. 0.8's `ctx.register_name` is - managed-only and released at cleanup. A refused start never held the name. -- **Supervised children.** Each generation registers the spec's name, and - the replacement registers only after the previous generation's D (V3). - `name.whereis()` therefore keeps addressing the current generation. This - is Gleam's `named_subject` behaviour, without a proxy. -- **Exception.** An init timeout whose child is stuck inside `started()` - keeps the name until the child's D (section 7). -- `ctx.register_name` remains for names chosen after start. The registry - keeps its global scope until rank 12. - -## 7. Init timeout - -`init_timeout(d)` bounds stage 2. It is measured from dispatch until -`started()` returns, as OTP's timer starts after the spawn. - -At expiry, if a starter is still waiting: - -1. the link or monitor edge is removed (`unlink`/`demonitor` fences); -2. a kill is sent (`kill_flush_EXIT` unlinks before `exit(Pid, kill)`, - `proc_lib.erl:610-615`); -3. the call returns `InitTimeout` after the child's D, or after a bounded - wait for it: the placement's cleanup deadline (5 s unmanaged, otherwise - the group's or spec's). - -Model: `TimeoutContained`, control `KillBeforeUnlink`. - -Rust cannot preempt `started()` (L3), so a child stuck in blocking code -reaches D only when `started()` returns. Until then it holds its name and its -admission slot, and it is reported as unfinished if the system shuts down. - -**Without a waiting starter** (`spawn`, or an abandoned `start*`), the -timeout is enforced by a framework timer armed at dispatch, only when the -option is set. Expiry kills the child, and `ready()` reports `InitTimeout`. -An edge of an abandoned `start_link` stays, so the `Killed` notice reaches -the parent like any linked death (trappable, X4). - -**Default.** None, as OTP's `infinity`. `ChildSpec` keeps its 5 s startup -default (V7, a Spawned bound). The gap report suggested 5 s "as -`gen_server:start`", but OTP's default is `infinity` (`gen.erl:716-722`). A -5 s default would also kill client actors whose `started()` does slow I/O -today (open question 3). +**Disposition of a taken notice** under the cell lock at publication: + +| Cell | Waiter | Trapping parent (`Exit`) | Non-trapping parent | Monitor (`Down`) | +|---|---|---|---|---| +| `Ready` (any waiter) | any | delivered | applied if non-`Normal` (X3) | delivered | +| `Failed`/`Ignored` | `Waiting` | diverted to the cell, consumed by the error | **applied if non-`Normal`**: the parent exits after its current handler (OTP: the link kills the caller) | diverted, consumed | +| `Failed`/`Ignored` | `Consumed` (bounded wait expired) | dropped | applied if non-`Normal` | dropped | +| `TimedOut` | `Waiting` / `Consumed` | dropped | **not applied** | dropped | +| any non-`Ready` | `Abandoned` | delivered | applied if non-`Normal` | delivered | + +The timeout rows are OTP's `kill_flush_EXIT`, which unlinks before the kill. +Spawned keeps the edge and suppresses its notice instead, which is the same +observably, and the waiter decides at publication. Model: `FailureFlushed` +(controls `NoLinkFlush`, `NoMonitorFlush`), `TimeoutContained` (control +`KillBeforeUnlink`), `NoticeOnce` (control `DoubleNotice`). + +### 5.3 Parent state + +- **Running**, the normal case: the edge is committed, and the parent's turn + waits. This is OTP and needs no extra protection, but the deadlock is + OTP's too. If the child's `started()` sends a request to its parent, it + waits until its request timeout (5 s by default, R3), because the + parent's turn is blocked in `start_link`. OTP documents the same + `gen_server:call`-from-`init` deadlock. Slow initialisation belongs in a + continuation; a waiter that must not block moves the `Starting` to a job + explicitly. +- **Exit began or past D** (only through a cloned `Context`): the commit + refuses with `ParentExited`. The check is under the parent's exit-request + lock, so a start from `stopped()` is refused too (agy P1-9). Model: + `NoEdgeToDeadParent` (control `LinkCheckLate`). +- **Dies after the commit:** its D sends each linked child a parent exit, + which resolves under section 2.3. Model: `ParentExitReachesChild` + (liveness; control `DropHeld`). + +### 5.4 Abandoning the wait + +Dropping the `Starting` sets the waiter to `Abandoned` under the cell lock. An +abandoned wait never consumes anything: + +- **Already diverted.** A notice already diverted to the cell is handed to + the parent as an ordinary notice, exactly once (`AbandonedNoticeDelivered`, + which checks the kind and the reason; control `AbandonDropsNotice`). +- **Later.** A later notice is ordinary (table above). +- **Timer.** The init timer is unaffected (section 6). + +An abandoned `start` is `spawn`. + +### 5.5 Answering + +| Committed | Waiter answers | +|---|---| +| `Ready` | at once: `Ok(Started(..))` | +| `Ignored` | after publication: `Ok(Ignored)` | +| `Failed` | after publication: `InitFailed`/`InitExited { pending: None }` | +| `TimedOut` | after publication: `InitTimeout { pending: None }` | +| any failure, publication not reached by the bounded wait | at the bound: `pending: Some(handle)` | + +The bounded wait is the placement's cleanup deadline: 5 s unmanaged, or the +group's or spec's otherwise. `ready()` reads the same cell and answers the +same value, at any time and any number of times (agy P1-6). Model: +`StartAnswered` (liveness; control `NoAnswerIgnore`). + +## 6. Init timeout + +`init_timeout(d)` arms a framework-owned timer at the commit, only when the +option is set. The waiter does not own it, so it runs whether or not anyone +polls, and abandonment does not change it (Codex 2). + +At expiry, under the cell lock: + +- **Cell `Pending`:** commit `TimedOut`, then send a kill. The start edges + stay. Their notice is disposed of at publication by the waiter: suppressed + while waiting or consumed, ordinary if abandoned (5.2). +- **Already committed:** nothing happens (`TimeoutNeverKillsReady`). + +**Stuck policy (explicit, agy P1-5).** The kill is applied cooperatively, +like every kill (L3). It does not interrupt a running `started()`: + +- a `started()` that never returns keeps the actor, its name, its admission + slot and its state; +- the waiter answers `InitTimeout { pending: Some(handle) }` at the bounded + wait; +- the system reports the actor as unfinished; +- under a supervisor the slot follows V3's stuck rule. + +There is no thread abort and no dropping of the `started()` future at an +`.await`. Aborting a turn at an await is the 0.10 design introspection lists +("turn abort at an await"). It would cover `started()` and handlers alike, +with its own model, rather than adding a special case here. + +**Default.** `Infinity`, as OTP. `with_options(..)` accepts +`Timeout::Infinity` explicitly. + +## 7. Registered names + +- **When.** At the stage-1 commit, before `started()`, so the child can be + found while it initialises. Messages queue, and a continuation still runs + first. OTP registers in `gen:init_it` before `init/1` + (`gen.erl:194-201`). +- **Clash.** `AlreadyStarted { name, existing }`, decided in the commit, so + nothing is created. `existing` is the holder's `ChildHandle`, or `None` for + a non-actor value. OTP returns `{already_started, Pid}`, and its loser exits + `normal` unobservably. +- **Release.** At publication, after destruction and before any notice, + `wait_exit` or answer (5.2). The name is held from the commit until then. + A new holder therefore never overlaps the old one's `Drop`, which is + stricter than OTP's unregistration at exit; Rust has destructors. A child + stuck in `started()` or in `Drop` keeps its name until publication, and + the waiter is told so through `pending`. +- **Supervised children.** Each generation registers the name, and the + replacement waits for the old one's publication (V3). +- **Kept from 0.8.** `ctx.register_name` remains for names chosen after + start. The registry stays global until rank 12. ## 8. Continuation (`handle_continue`) -`Init::continue_with(msg)` runs `Handler::handle(msg)` as the actor's -first turn after `started()`, before any mailbox message, including messages -that arrived during `started()` (`ContinueFirst`, control `ContinueQueued`). - -- The ack and managed readiness commit before the continuation runs, as - OTP's `init_ack` precedes `handle_continue` (`gen_server.erl:2197-2201`). -- A stop or exit observed before the continuation begins discards it, as D4 - discards a queued message. No caller waits on it; its `PendingReply` is - detached. -- A panic in the continuation is an ordinary handler panic after readiness. - Under a supervisor it is a crash of a ready child, not a startup failure. +`Init::continue_with(msg)` runs `Handler::handle(msg)` as the first turn +after the readiness commit, before any mailbox entry. Mailbox entries include +messages that arrived during `started()` and signals queued for a trapping +child (`ContinueFirst`, control `ContinueQueued`). + +**What it does not guarantee** (Codex 8): that it runs at all, or runs to +completion, before termination. + +- Readiness is committed before it starts, so the waiter and the supervisor + already see the child ready. +- Anything that ends a ready child can end it before or during the + continuation: a kill, an untrapped exit signal, a `terminate_child`, or + supervisor shutdown (non-trapping: applied when the continuation's turn + ends; trapping: queued behind it). +- A stop observed before the continuation begins discards it, as D4 discards + queued messages. + +OTP gives the same guarantee: `handle_continue` precedes `receive`, but +fatal signals still terminate the process. Work that must finish before the +start succeeds belongs in `started()`. + +- **Under supervision.** A panic in the continuation is a crash of a ready + child (restart policy), not a start failure. - **Cost.** One branch after `started()`; the dispatch loop is unchanged. -- **Not included:** continuing from ordinary handlers (OTP's - `{noreply, S, {continue, C}}`). That needs a check after every turn, which - is a cost on the dispatch path (open question 4). +- **Not in 0.9:** continuing from ordinary handlers. That needs a branch per + turn; maintainer decision 4. ## 9. Supervisors -The OTP child spec separates `start` (the MFA and its options) from -`restart`, `shutdown`, `type` and `significant`. `ChildSpec` mirrors that: +The OTP child spec separates `start` (MFA and options) from `restart`, +`shutdown`, `type` and `significant`. `ChildSpec` mirrors it: ```rust -ChildSpec::new(id, &group, |fctx| async { Ok(MyActor::new(..)) }) - .options(StartOptions::new() - .name("rpc") - .init_timeout(Duration::from_secs(10)) - .label("http") - .max_native_jobs(8)) +ChildSpec::new(id, &group, |fctx| async { Ok(MyActor::new(..)) }) // or Ok(ChildStart::Ignore) + .options(StartOptions::new().name("rpc").label("http").max_native_jobs(8)) + .startup_timeout(Duration::from_secs(10)) // whole attempt; Timeout::Infinity allowed .restart(RestartMode::Transient) .shutdown_timeout(Duration::from_secs(5)) ``` +- **Factory result.** The factory returns + `Result>, E>` with + `ChildStart::Actor(a) | ChildStart::Ignore`. `From for ChildStart` + keeps existing factories compiling. This is the dynamic supervisor's + shape (section 18). A factory `Ignore` and an `Init::ignore()` are the same + outcome. +- **`startup_timeout` stays on `ChildSpec`** (agy P1-7, Codex 3). It bounds + the whole attempt: factory, stage 1 and `started()`. The default is 5 s + (V7), and `Timeout::Infinity` is allowed. The actor's init timer gets the + remaining budget at the commit, or the smaller of that and + `options.init_timeout`. Expiry during the factory cancels it (V7, unchanged); + expiry after the commit is the cell's `TimedOut`. The dynamic supervisor + bounds construction and `started()` together the same way. - **Placement.** The positional group stays: a supervised child is always - managed, and the group must belong to the supervisor's system - (`ForeignGroup`). A `group` in `options` replaces it. A `backend` is - `InvalidOptions` at `add()`. -- **Moved into `StartOptions`.** `startup_timeout` becomes `init_timeout` - (default 5 s when unset, V7 unchanged). `max_outstanding` and - `max_native_jobs` also move. `restart` and `shutdown_timeout` stay; there - is room for a future `significant(bool)` (G15), since the builder has - private fields. -- **Each generation is a `start_link` with the supervisor as parent.** The - same stage-1 and ack rules apply. `Init::error`, `stop` or a panic is a - failed start: the first start aborts the tree (V7), and a replacement is - retried (V1). **`Init::ignore`** follows OTP: the spec stays and no - generation runs, in a new `ChildState::Ignored`. A Temporary child that - ignores is removed from the snapshot, as OTP deletes it. `ignore` does not - fail the initial start and is not a restart. -- **Name clashes.** A clash at a restart is a failed restart attempt, which - is retried with backoff and charged to the budget (V5). OTP also fails the - restart on `already_started`. -- **Continuations.** A continuation of a supervised child runs after - readiness. -- **Dynamic supervisor (#245).** `DynamicChildSpec` takes the same - `StartOptions`. `start_child` returns `Ignored` as "nothing started" - (gap report 3.2.3). The RLPx handshake becomes a continuation, so no peer - message or `terminate_child` can overtake it (3.2.1). -- **Not derived.** `#[actor]` does not derive a child spec; see - [alternatives](#15-alternatives-considered). + managed, in the supervisor's system. A `group` in `options` replaces it; + a `backend` is `InvalidOptions` at `add()`. +- **Each generation is a `start_link` with the supervisor as parent**, with + the cell rules of section 5. +- **`ignore` (static supervisors, OTP semantics; agy P1-7, Codex 9).** + - **The spec stays**, with no generation: `ChildState::Ignored`, + `ChildInfo { actor: None, ready: false, state: Ignored }`. + - **Counts.** It counts in `ChildCounts.specs` and `workers`, not `active`, + as OTP's `count_children`. + - **Initial start.** An ignore completes the child's first start: the + initial-start barrier advances, the tree's `start()` succeeds, and V7 sees + no failure. The ignored instance was already published before the + supervisor observes it (the answer follows publication), so its cleanup + is complete. + - **OneForOne.** Nothing restarts an ignored child: it has no process that + could exit. This is OTP's `undefined` pid. + - **OneForAll / RestForOne.** A later round that covers the ignored child + starts it again with the rest of the cohort, as OTP's `start_children` + after `terminate_children` does. The round's readiness barrier treats + `Ignored` as done for that child. + - **Temporary.** An ignored Temporary child is removed, as OTP deletes it. + - **Budget.** An ignored replacement keeps the restart attempt already + stamped (V5) and adds nothing. + - **Waiting.** `SupervisedChild::wait_ready()` and `current()` return + `ChildUnavailable(ChildState::Ignored)` at once rather than waiting. + - **Model and traces.** `Supervision.tla` and `Cohorts.tla` gain the + `Ignored` state and the cohort rule, and traces gain an `Ignore(c, g)` + event between `Attempt` and the round's barrier, with `trace_check.py` + updated. This is in the implementation PR, with its own controls: + `IgnoreRetried` (OneForOne restarts an ignored child) and + `IgnoreSkippedInCohort` (a round leaves an ignored Permanent child + stopped). +- **Errors.** `SupervisorError::FailedToStartChild { child, cause: ChildStartError }`, + where: + + ```rust + #[non_exhaustive] + pub enum ChildStartError { + Factory(String), // error or panic in the factory + Start(StartError), // stage 1 or stage 2 (InitFailed, InitExited, InitTimeout) + StartupTimeout { pending: Option }, // the whole-attempt deadline + } + ``` + + The dynamic supervisor's `FailedToStart` carries the same type (section + 18). +- **Continuations.** A supervised child's continuation runs after + readiness, and the supervisor does not wait for it (section 8). +- **`#[actor]` does not derive a child spec** (section 17); `ChildSpec` + keeps room for `significant`. ## 10. Test kit (`spawned-test`) | 0.8 | 0.9 | |---|---| -| `Mock::start()` | `Mock::spawn()` (immediate; a mock's `started()` is trivial) | -| `Mock::start_on(&g)` | `Mock::spawn_with(opts)` (`impl Into>`, so `&g` works) | -| none | `Mock::on_started(\|ctx\| Init)` and `Mock::start_delay(d)` (virtual time under the simulator): drive every `Init` outcome and the init timeout from tests | -| `TestSystem::start(a)` | `TestSystem::spawn(a)` (immediate, panics on refusal) and `TestSystem::start(a).await` (waits, panics with the `StartError`) | +| `Mock::start()` | `Mock::spawn()` | +| `Mock::start_on(&g)` | `Mock::spawn_with(opts)` | +| none | `Mock::on_started(\|ctx\| Init)`, `Mock::start_delay(d)` (virtual time under the simulator), `Mock::drop_flag(Arc)` (set by the mock's `Drop`; checks publication order) | +| `TestSystem::start(a)` | `TestSystem::spawn(a)`; `TestSystem::start(a).await` (waits, panics with the error) | | `TestSystem::start_mock(m)` | `TestSystem::spawn_mock(m)` | -| none | `TestSystem::options::()`: `StartOptions` with the test group set | +| none | `TestSystem::options::()` | -`start_link` and `start_monitor` are exercised from tests through `in_actor`. -Its closure gets the parent's `Context`, and because `Starting` is `'static` -it can be returned and awaited outside the turn. +`start_link`/`start_monitor` are exercised through `in_actor`. `Starting` is +`'static`, so it can be returned from the turn and awaited outside it. ## 11. Erlang/OTP alignment -Categories as in semantics section 11: R (Rust forces it), T (Rust types add -strictness), B (a bound Erlang lacks). +Categories: R (Rust forces it), T (Rust types add strictness), B (a bound +Erlang lacks). | OTP | Spawned 0.9 | Verdict | |---|---|---| -| `gen_server:start/3,4`: synchronous on `init/1` | `start()`, `with_options(o).start()` | aligned | -| `gen_server:start_link`: link set atomically, caller is the parent | `start_link(&ctx)` | aligned; the parent is passed explicitly (R: no `self()`) | -| `gen_server:start_monitor` (OTP 23) → `{ok, {Pid, Mon}}` | `start_monitor(&ctx)` → `(ActorRef, MonitorRef)` | aligned | -| `proc_lib:spawn/spawn_opt`: no init ack | `spawn()`, `with_options(o).spawn()` | aligned | -| `{local, Name}`; `{error, {already_started, Pid}}` | `name(Name)`; `AlreadyStarted { name, existing: Option }` | aligned; typed name (T); `existing` is `None` when a non-actor value holds the name (T: the registry stores typed values) | -| name unregistered before the starter is notified | released at D, before any notice | aligned | -| `{timeout, T}`, default `infinity` | `init_timeout`, default none | aligned; the wait for the killed child's death is bounded (R: no preemption) | -| `kill_flush_EXIT` on timeout | unlink/demonitor, then kill | aligned | -| init `{ok, S}` / `{ok, S, {continue, C}}` / `{error, R}` / `{stop, R}` / `ignore` | `Init::ready` / `continue_with` / `error` / `stop` / `ignore` | aligned; `ignore` is `Err(Ignored)` (T) | -| init crash → `{error, R}`, exit `R` | panic → `InitExited(Panic)` | aligned | -| `handle_continue` from `init` | `Init::continue_with` | aligned | -| `{noreply, S, {continue, C}}` from callbacks | not in 0.9 | gap: dispatch-path cost (open question 4) | +| `gen_server:start/start_link/start_monitor`, synchronous on `init/1` | `start`, `start_link`, `start_monitor` | aligned; the parent is explicit (R: no `self()`) | +| `proc_lib:spawn/spawn_opt` | `spawn`, `with_options(o).spawn()` | aligned | +| `{ok, Pid}` / `ignore` / `{error, R}` | `Ok(Started)` / `Ok(Ignored)` / `Err` | aligned | +| `init_ack` before the loop; the ack is final | readiness commits before the continuation and is final | aligned | +| `{timeout, T}` default `infinity`; on expiry `kill_flush_EXIT`, then `{error, timeout}` after `'DOWN'` | `init_timeout` default `Infinity`; the notice is suppressed instead of unlinked; `InitTimeout` after publication | aligned observably; the wait for publication is bounded (R) | +| a kill ends `init` at once | applied when `started()` returns (L3) | R: no preemption; abort at an await is the 0.10 design | +| a signal before `process_flag(trap_exit, true)` acts at once | held, resolved with the flag at return | R: 0.8 phase 0 (cooperative) | +| `init` returns `{stop, R}` / crashes / `{error, R}` (OTP 26 `init_fail/3`) / `ignore` | `Init::stop` / panic / `Init::error` / `Init::ignore` | aligned | +| failure returned after the process is gone (`await_DOWN`) | after publication: state dropped, name free | aligned; stricter (destructors, R) | +| `{local, Name}`, `{already_started, Pid}`, name freed before the starter is notified | `name`, `AlreadyStarted { existing }`, released at publication before notices | aligned; typed (T) | | failed `start_link`, trapping caller: `'EXIT'` consumed | consumed | aligned | -| failed `start_link`, non-trapping caller: killed by the link before returning | returns the error, then exits after the current handler | R: cooperative signals (L3) | +| failed `start_link`, non-trapping caller: killed by the link | the error is returned, then the caller exits after its handler | R: cooperative | | failed `start_monitor`: `'DOWN'` consumed (`gen:monitor_return`) | consumed | aligned | -| start from a dead process | `ParentExited` | R: a cloned `Context` outlives its actor | -| `spawn_opt` `link`/`monitor` | the verbs | aligned | -| `spawn_opt` `max_heap_size` | reserved (rank 18a) | planned | -| `spawn_opt` `priority`, `fullsweep_after`, `min_heap_size`, `message_queue_data`; `hibernate_after` | none | not applicable (no per-process heap or priority scheduler) | -| `proc_lib:set_label` (OTP 27) | `label` at start | aligned (set once, at start) | -| one node, one scheduler pool: no placement | groups and backends; linked/monitored children inherit | R: no VM; groups are the scheduler analogue | -| `process_flag(trap_exit, true)` in `init` | `ctx.trap_exit(true)` in `started()` | aligned; no start option is needed (signals are held during `started()`, X3) | +| `handle_continue` from `init` | `Init::continue_with` | aligned | +| `{noreply, S, {continue, C}}` | not in 0.9 | gap (dispatch cost) | +| `gen_server:call` to the parent from `init` deadlocks until timeout | same (request timeout) | aligned; documented | +| supervisor child `ignore`: spec kept, `undefined` pid, restarted in a later `one_for_all`/`rest_for_one` round, deleted if temporary | same | aligned | +| `set_label` (OTP 27) | `label` option, `ctx.set_label` | aligned | +| one node, no placement | inheritance for link/monitor; S8 ownership | R: no VM; ownership is a Rust resource-lifetime addition | ## 12. Migration from 0.8 | 0.8 | 0.9 | Behaviour | |---|---|---| | `a.start()` | `a.spawn()` | same | -| | or `a.start().await?` | waits for `started()`; reports init errors | +| | or `a.start().await?` | waits; reports init errors | | `a.start_on(&g)?` | `a.with_options(&g).spawn()?` | same | -| | or `a.with_options(&g).start().await?` | waits | -| `a.start_with_backend(b)` | `a.with_options(b).spawn()?` | same (now a `Result`; it fails only on options) | +| `a.start_with_backend(b)` | `a.with_options(b).spawn()?` | same; a `Result` (options or executor failure) | | `g.start(a)?` | `a.with_options(&g).spawn()?` | same | -| `a.start_linked(&ctx)` | `a.start_link(&ctx).await?` | **changes:** inherits the parent's placement, waits for `started()`, refuses an exiting parent | -| `ctx.monitor(&a.start().child_handle())` | `a.start_monitor(&ctx).await?` | atomic; no `NoProc` race | -| `ctx.register_name(n)` in `started()` | `with_options(StartOptions::new().name(n))` | atomic with start; released at D; all placements | -| `actor.ready().await` → `Result<(), SystemError>` (managed only) | `Result<(), StartError>`, every placement | error type | -| `SystemError::StartupFailed(r)` | `StartError::InitExited(r)` | | -| `ChildSpec::startup_timeout(d)` | `.options(StartOptions::new().init_timeout(d))` | same | -| `ChildSpec::{max_outstanding, max_native_jobs}` | `StartOptions::{max_outstanding, max_native_jobs}` | same; now also for unsupervised actors | -| `async fn started(..) {}` | unchanged | same; may now return `Result<(), E>` or `Init` | -| self-send from `started()` to finish setup | `Init::continue_with(msg)` | runs before any other message | -| `Mock::start()` / `Mock::start_on(&g)` | `Mock::spawn()` / `Mock::spawn_with(&g)` | same | -| `TestSystem::start(a)` / `start_mock(m)` | `TestSystem::spawn(a)` / `spawn_mock(m)` | same | - -In tasks mode a 0.8 statement `x.start();` still compiles and starts the -actor (with a `#[must_use]` warning, section 3). `let r = x.start();` used as -an `ActorRef` does not compile, which flags every site to migrate. The -removed functions are compile errors (trybuild cases in the test plan). +| `a.start_linked(&ctx)` | `a.start_link(&ctx).await?` | **changes:** inherits placement, waits, refuses an exiting parent | +| `ctx.monitor(&a.start().child_handle())` | `a.start_monitor(&ctx).await?` | atomic | +| `ctx.register_name(n)` in `started()` | `StartOptions::name(n)` | atomic; released at publication | +| `ready().await -> Result<(), SystemError>` (managed only) | `-> Result, StartError>`, every placement | type | +| `SystemError::StartupFailed(r)` | `StartError::InitExited { reason: r, .. }` | | +| `ChildSpec::startup_timeout(d)` | unchanged (whole attempt) | same | +| `ChildSpec::{max_outstanding, max_native_jobs}` | `.options(StartOptions::new().max_*(..))` | same | +| factory `Ok(actor)` | unchanged (`From for ChildStart`); `Ok(ChildStart::Ignore)` is new | same | +| `async fn started(..) {}` | unchanged | same | +| `Mock::start()` / `start_on` | `Mock::spawn()` / `spawn_with` | same | +| `TestSystem::start` / `start_mock` | `TestSystem::spawn` / `spawn_mock` | same | ## 13. Client mapping -Every start call on the clients' `spawned-v0.8` branches is mapped in -[unified-start-client-mapping.md](unified-start-client-mapping.md): ethrex -`c1163b7`, ethlambda `85990aa`. +See [unified-start-client-mapping.md](unified-start-client-mapping.md): +ethrex `c1163b7`, ethlambda `85990aa`. - **Sites that change:** 51 (ethrex 37, ethlambda 14). All are mechanical, - and the behaviour-preserving rewrite keeps the current behaviour at every - one. -- **Inheritance changes behaviour at no site:** neither client calls - `start_linked`, and plain starts never inherit. -- **Two handler sites** start per-connection actors from inside an actor: - ethrex's RLPx initiator (via `PeerConnection::spawn_as_initiator`) and the - L2 proof coordinator. They stay unmanaged. Moving them to `start_link` or - `placement_of` is a client decision with a behaviour change: group - ownership, ordered shutdown and capacity. -- **Optional upgrade at 14 async sites** (8 in the L2 sequencer and the - based block fetcher, 4 in the prover, discovery, the monitor): - `start().await?` surfaces init failures, which today are only logged. -- **Six client functions return `SystemError` from `start_on`.** Their error - type becomes `StartError`, or they map it. -- **Supervisor specs** (3 `ChildSpec::new` sites): unchanged. They set no - `startup_timeout` or bounds. + and none changes behaviour. +- **Inheritance changes behaviour at no site.** +- **Optional upgrades:** 14 async sites may use `start().await?`. +- **Error types:** six client functions change their error type from + `SystemError` to `StartError`. +- **Supervisor specs:** the 3 `ChildSpec::new` sites are unchanged + (`startup_timeout` stays). + +Revision 2 corrects the mapping on two points: +- the RLPx handshake continuation precedes peer messages, but a + `terminate_child` can still end the connection before it completes; +- bare `spawn()` panics on executor construction failure. ## 14. Semantics contract changes -New or changed guarantees, with ids for `semantics.md`. They are written into -the contract by the implementation PR, as with otp-exit-signals. - -- **L1 (changed).** `stopped()` runs only after `started()` returned a ready - `Init`. A failed init (error, ignore, stop, panic) skips it. -- **L2 (changed).** Readiness exists for every placement. `ready()` and the - waiting starts resolve at the same barrier: `started()` returned ready with - no exit pending (and, for managed actors, admission still open). -- **L8 (new), Start.** Two stages as in section 3. `spawn` returns after - stage 1. `start`, `start_link` and `start_monitor` return after `started()`: - `Ok` only if ready, otherwise the error after the child's D. A stage-1 - refusal leaves no trace. -- **L9 (new), Init outcomes.** The table in 2.3. -- **L10 (new), Continuation first.** No mailbox message is dispatched before - the continuation. A stop before it discards it. -- **X5 (changed).** "`start_linked` registers the link before the child's - `started()`" becomes "`start_link` and `start_monitor` register their edge - before the child's `started()`, on every placement. A start through an - exiting or dead parent is refused (`ParentExited`)." -- **X3 (changed).** The parent row reads "`start_link` caller" for - "`start_linked` caller". -- **X8 (new), Start-time edges and failure.** On a failure while the starter - waits: a trapping parent's `Exit` and the `Down` are consumed, and a - non-trapping parent applies a non-`Normal` notice after its current handler. - An abandoned wait delivers the notice exactly once. The init timeout - removes the edge before the kill. -- **S6 (changed).** `spawn` and `start` are unmanaged unless given a - placement. `start_link` and `start_monitor` take the parent's. -- **S7 (new), Inheritance.** Section 4: the rule, no fallback, `Closed` and - `Capacity`. -- **S8 (new), Linked descendants are owned by the parent's scope.** - Shutdown order, V3 extension and kill escalation, as in section 4. -- **S9 (new), Start-time names.** Atomic, unique and released at D before any - observation, on every placement; `AlreadyStarted` on a clash. -- **T5 (new), Init timeout.** Section 7. -- **V1/V7 (changed).** `Init::ignore` leaves the spec without a generation - (`Ignored`) and is not a failure. A name clash at a restart is a failed - attempt. -- **Section 11 (changed).** New rows: - - start synchronous on init: aligned in 0.9; - - fallible init and continue: aligned; - - failed `start_link` with a non-trapping caller: R; - - continue from handlers: gap. - -**Non-guarantees** (added to section 10): - -- The init timeout cannot interrupt `started()`. The killed child holds its - name and slot until `started()` returns. -- Inheritance does not follow work outside the start call: `tokio::spawn`, - helpers and plain `start`/`spawn` from a handler stay unmanaged. -- The continuation exists only after `started()`. -- An abandoned start does not consume the child's failure notice. +The implementation PR writes these into `semantics.md`, with the "Checked by" +lines from section 16. + +- **L1 (changed).** `stopped()` runs after a ready commit, or after a + `ctx.stop*` called in `started()`; never after a failed or ignored init. +- **L2 (changed).** Readiness exists for every placement. It is the startup + cell's `Ready`: committed by the child at the end of `started()`, while + admission is open, before any continuation or dispatch. It is final. + `ready()` and the waiting starts read the same committed value. +- **L8 (new), Start.** Stage 1 is one commit (section 3), and a refusal is + never observable. The waiting starts answer per 5.5. Errors come after + publication or carry `pending`. +- **L9 (new), Init resolution.** The precedence table in 2.3, including + trapping children. +- **L10 (new), Continuation.** It precedes every mailbox entry. It is not a + barrier against termination. +- **X5 (changed).** `start_link`/`start_monitor` commit their edge before the + child's `started()`, on every placement. A start through an exiting or + dead parent is refused. +- **X3 (changed).** The parent row reads "`start_link` caller". +- **X8 (new), Startup notices.** The disposition table in 5.2. The init + timeout never reaches the parent unless the wait was abandoned. An + abandoned wait delivers a diverted notice exactly once. +- **S6 (changed).** `spawn`/`start` are unmanaged unless given a placement. +- **S7 (new), Inheritance.** Section 4, with no fallback. +- **S8 (new), Scope ownership.** Section 4.1. +- **S9 (new), Start-time names.** Committed atomically, unique, held until + publication, released before any notice. +- **T5 (new), Init timeout.** A framework-owned timer that loses to a ready + commit, with the stuck policy. +- **V1/V5/V7 (changed).** `ignore` keeps the spec, as OTP. `startup_timeout` + bounds the whole attempt. A name clash at a restart is a failed attempt. +- **Section 11 (changed).** New rows as in section 11 above. + +**Non-guarantees.** + +- Neither the init timeout nor a kill interrupts `started()`. +- A continuation may not run at all. +- Inheritance does not follow helpers or plain starts. +- An abandoned wait consumes nothing. - Names are process-global until rank 12. +- The commit's global lock order is a cost at start time, never on send or + dispatch. ## 15. The model -[`Start.tla`](../../verification/tla/Start.tla) models: - -- one parent (managed in group `G` or unmanaged, trapping or not, which may - exit `Normal` or `Panic`); -- children started with `spawn`, `start`, `start_link` or `start_monitor`, - with an explicit placement (none, `G`, unmanaged, `placement_of`) and - possibly one shared name; -- `G`, which may close and admits at most `Cap` live actors. - -Stage 1 is four separate steps, so parent death, group closure and a -competing registration interleave between them. `started()` is one step with -every `Init` outcome. Signals are held until it returns. Others may send -messages, which a continuation must precede. A waiting starter can time out -or abandon the wait. - -**Invariants:** +[`Start.tla`](../../verification/tla/Start.tla), revision 2: + +- **Parent.** One parent P: managed in group `G` or unmanaged, trapping or + not, able to exit. +- **Children.** Started with `spawn`, `start`, `start_link` or + `start_monitor`, with a default or explicit placement and possibly the + shared name. +- **Stage 1.** Prepare failure, then the atomic `Commit`; a `Lookup` + observer resolves the name at any time. +- **End of `started()`.** One step with every `Init` outcome, a trap flag, + held parent exits and kills, resolved by the 2.3 precedence. +- **The startup cell.** The commitment and the waiter state, with `AckOk`, + `AckFail` (after publication, or at the bounded wait with `pend`), + `Abandon`, and the init `Timeout`. +- **Death.** `Die` (D) and `Publish`, with destruction between them; group + capacity is released at publication. +- **After readiness.** Ready children may exit, be killed or unlink; a + continuation and messages from others; a trapping child's queued parent + exit. +- **S8.** Owned children, the scope deadline, and the generation end. +- **Notices.** Counted per child (`exitN`, `downN`) with their reason, so a + duplicate delivery is visible. + +**Safety invariants:** | Invariant | Meaning | |---|---| -| `InheritedPlacement` | explicit placement wins; link/monitor take the parent's; otherwise unmanaged | -| `LinkBeforeStarted` | the link or monitor exists before `started()` begins | -| `NameUnique` | at most one live holder of a name | -| `NameReleasedAtDeath` | no holder past D or aborted | -| `AbortInvisible` | a refused start: no edge, name, slot, hook or notice | -| `AckMeansStarted` | `Ok` from a waiting start means `started()` returned | -| `FailureAfterDeath` | the init error is returned after the child's D | -| `FailureFlushed` | a failed or timed-out `start_monitor` delivers no `Down`; `start_link` no `Exit` to a trapping parent | -| `TimeoutContained` | an init-timeout kill never reaches the parent | -| `NoEdgeToDeadParent` | no edge survives the parent's D | -| `AdmittedOnlyOpen`, `CapacityRespected` | group admission | -| `NoticeOnce` | at most one `Exit` and one `Down` per child | -| `ContinueFirst` | no message before the continuation | -| `AbandonedNoticeDelivered` | an abandoned wait hands a diverted notice to the parent | - -Liveness (`MCStartLive`, weak fairness on the child's steps and the parent's -D): `ParentExitReachesChild`. A child linked to a parent that dies with a -non-`Normal` reason, also during its `started()`, dies (or was refused). - -**Results** (TLC 2.19, fede, 4 workers, `nice -n 19` on a loaded host): - -| Configuration | Run | Distinct states | Depth | Time | -|---|---|---|---|---| -| `MCStart` | per PR, safety | 2,896,748 | 24 | 23 s | -| `MCStartContinue` | per PR, safety | 17,720 | 16 | 1 s | -| `MCStartLive` | per PR, safety + liveness | 424,172 | 23 | 20 s | -| `MCStartDeep` | weekly (every explicit placement, every variant including `spawn`) | 56,947,732 | 24 | 6 min 23 s | +| `InheritedPlacement` | explicit wins; link/monitor take the parent's; otherwise unmanaged | +| `LinkBeforeStarted` | the edge was really created (at the commit) before `started()` began | +| `NameUnique` | at most one holder | +| `NameHeldUntilPublished` | a holder is between its commit and its publication | +| `NoOverlapOnReuse` | no holder while another named child is between D and publication | +| `NameFreeBeforeObservation` | no notice or final error while the child still holds the name | +| `AbortInvisible` | a refused start: no edge, name, lookup, slot, hook or notice | +| `ResultMatchesCommit` | the answer is the committed outcome | +| `ReadyIsFinal` | a ready commit is never answered as a start error | +| `TimeoutNeverKillsReady` | the init timer never kills a ready child | +| `ReadyOnlyWhileOpen` | managed readiness commits only while admission is open | +| `FailureAfterPublication` | an error without `pending` comes after publication | +| `FailureFlushed` | a failed start delivers no `Down`, and no `Exit` to a trapping parent | +| `TimeoutContained` | an init timeout reaches P only after an abandoned wait | +| `NoEdgeToDeadParent` | no edge outlives P's D | +| `AdmittedOnlyOpen`, `CapacityRespected` | admission; slots held until publication | +| `NoticeOnce` | at most one `Exit` and one `Down` per child (counters) | +| `AbandonedNoticeDelivered` | an abandoned wait hands the diverted notice to P with its kind and reason | +| `ContinueFirst` | no mailbox entry before the continuation | +| `GenDoneMeansOwnedDone` | P's generation ends only after its owned children | + +**Liveness** (weak fairness on the child's own steps, publication, P's D and +the scope deadline; strong fairness on the answer): + +| Property | Meaning | +|---|---| +| `ParentExitReachesChild` | a non-`Normal` parent exit ends a linked child, also when held during `started()` | +| `StartAnswered` | every waiting start is answered or abandoned | +| `OwnedScopeFinishes` | a managed parent's generation ends after it dies | -`MCStartDeep` with messages from others as well passed 74M distinct states -with its queue still growing, so messages stay in `MCStartContinue`. +**Configurations and results** (TLC 2.19, fede, 4 workers, `nice -n 19`, +run one at a time): -**Negative controls.** `negative_checks.py` runs `named_fault` with -`TypeOK` and only the named invariant, and `named_temporal` for the -liveness control. Each control below produced its counterexample (1-7 s -each, one at a time): +| Configuration | Run | Instance | Distinct states | Depth | Time | +|---|---|---|---|---|---| +| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3 (two children and P at once), closure, abandonment, retry | RESULT_MCStart | +| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | RESULT_MCStartTimeout | +| `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | RESULT_MCStartContinue | +| `MCStartScope` | per PR | managed parent, two linked children (owned or not), trapping, unlink, scope deadline; liveness `OwnedScopeFinishes` | RESULT_MCStartScope | +| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | RESULT_MCStartLive | +| `MCStartDeep` | weekly | two named children, every variant and explicit placement, trapping, unlink, scope, timer | RESULT_MCStartDeep | -| Fault | Reintroduces | Fails | -|---|---|---| -| `NoInherit` | 0.8: a linked/monitored child of a managed parent is unmanaged | `InheritedPlacement` | -| `LinkAfterStarted` | the link is set once `started()` runs | `LinkBeforeStarted` | -| `DoubleRegister` | check-then-insert registration | `NameUnique` | -| `LateUnregister` | the name outlives D | `NameReleasedAtDeath` | -| `AbortLeaksLink` | a refused start keeps its edge | `AbortInvisible` | -| `EarlyFailAck` | the error returns before the child's D | `FailureAfterDeath` | -| `AckBeforeStarted` | `Ok` before `started()` returned | `AckMeansStarted` | -| `NoMonitorFlush` | a failed `start_monitor` also delivers `Down` | `FailureFlushed` | -| `NoLinkFlush` | a failed `start_link` also delivers `Exit` | `FailureFlushed` | -| `KillBeforeUnlink` | the timeout kills while linked | `TimeoutContained` | -| `LinkCheckLate` | the link ignores the parent's state | `NoEdgeToDeadParent` | -| `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | -| `IgnoreCap` | a full group admits | `CapacityRespected` | -| `AbandonDropsNotice` | an abandoned wait loses the diverted notice | `AbandonedNoticeDelivered` | -| `ContinueQueued` (`MCStartContinue`) | the continuation is queued behind earlier messages | `ContinueFirst` | -| `DropHeld` (`MCStartLive`, liveness) | a parent exit held during `started()` is dropped | `ParentExitReachesChild` | +**Negative controls** (`negative_checks.py`, each with `TypeOK` and only the +named property, run one at a time): + +CONTROLS_TABLE **Not modelled:** -- the child trapping exits (E3's trapping-child rules are in `Lifecycle.tla`); +- the lock implementation, which is covered by the Loom test in section 16; - more than one parent; -- the system's ordered shutdown of linked descendants (S8). That extends - `Coordinator.tla` and `NestedSupervision.tla` in the implementation PR; -- time; -- the init-timeout timer without a waiting starter. +- time (the bounded wait is a nondeterministic choice); +- destruction as a separate step (it lies between `Die` and `Publish`); +- the supervisor's `ignore` and `startup_timeout` rules, which change + `Supervision.tla` and `Cohorts.tla` in the implementation PR (section 9). ## 16. Test plan @@ -799,182 +918,219 @@ guarantee breaks. ### 16.1 Per-guarantee matrix -Columns: TLA+ invariant (and trace validation); integration tests (both -modes); simulator (`#[sim_test]`, PCT and bounded DFS); fuzzer oracle; -trybuild. - -| Guarantee | TLA+ / trace | Integration (`concurrency/tests/start_contract.rs`) | Sim (`start_sim.rs`) | Fuzz oracle | trybuild | +| Guarantee | TLA+ / trace | Integration (`concurrency/tests/start_contract.rs`) | Sim (`start_sim.rs`, PCT + DFS) | Fuzz oracle | trybuild | |---|---|---|---|---|---| -| L8 two stages, ack | `AckMeansStarted`, `FailureAfterDeath` | `{spawn,start,start_link,start_monitor}_returns_{immediately,after_started}`; `start_without_await_still_starts` | `ack_races_child_death` | ack `Ok` ⇒ hook log shows `started()` returned | `start_result_is_not_an_actor_ref` | -| L8 refusal invisible | `AbortInvisible` | `refused_start_runs_no_hook_and_leaves_no_edge_or_name` (closed, capacity, clash, parent exited) | `refusal_races_parent_death` | aborted ids never appear in `Exit`/`Down`/registry | | -| L9 Init outcomes | (model abstracts the outcomes to their exits) | one test per row of the 2.3 table, each mode and placement, for `start`, `start_link`, `start_monitor`, `spawn`+`ready()` | | `started()` outcomes generated; oracle checks the reported `StartError` against the outcome | `started_may_return_unit_result_or_init` (ui-pass); `continue_with_requires_handler` | -| L1 `stopped()` skipped on failed init | | `failed_init_skips_stopped` (every outcome) | | existing L5 oracle extended | | -| L2 readiness for every placement | `AckMeansStarted` | `ready_resolves_for_unmanaged_and_threads` | | | | -| L10 continuation first | `ContinueFirst` | `continuation_runs_before_messages_sent_during_started`; `stop_before_continuation_discards_it` | `continuation_races_peer_send` | hook-log order | | -| X5 edge before `started()` | `LinkBeforeStarted` | `start_link_child_panicking_in_started_notifies_parent` on every placement; same for `start_monitor` | `child_dies_in_started_racing_link` | link/monitor atomicity: a child failing in `started()` always produces the parent's notice or the consumed error | | -| X5 parent exited | `NoEdgeToDeadParent` | `start_link_through_dead_context_is_refused` | `parent_stops_during_child_started` | | | -| X8 failure flush | `FailureFlushed` | `failed_start_link_consumes_exit_of_trapping_parent`; `failed_start_link_kills_nontrapping_parent_after_handler`; `failed_start_monitor_consumes_down` | `failure_races_ack` | no `Exit`/`Down` for a start that returned an init error | | -| X8 abandoned wait | `AbandonedNoticeDelivered` | `abandoned_start_monitor_still_delivers_down`; `abandoned_start_link_delivers_exit` | `abandon_races_child_death` | | | -| T5 init timeout | `TimeoutContained` | `init_timeout_returns_after_death_and_spares_parent` (each kind); `spawn_init_timeout_kills_and_ready_reports_it`; `stuck_started_holds_name_until_return` | `timeout_races_started_return` | timeout options generated; parent never ends `Killed` from a timed-out child | | -| S7 inheritance | `InheritedPlacement` | `link_and_monitor_inherit_{async_group,thread_group,pinned_shard,backend_async,backend_blocking,backend_thread}`; `plain_start_never_inherits`; `placement_of_opts_in` | | placement oracle: the child's `execution_group()` name (or backend) matches the rule | `threads_options_have_no_group_or_backend` | -| S7 closed or full group | `AdmittedOnlyOpen`, `CapacityRespected` | `inherited_start_into_closing_group_fails_closed`; `inherited_start_into_full_thread_group_fails_capacity` | `group_shutdown_during_start` | existing S1 oracle | | -| S8 linked descendants owned | (Coordinator/Nested extension in the impl PR) | `system_shutdown_stops_linked_children_with_their_parent`; `supervised_parent_generation_waits_for_linked_children`; `unlinked_descendant_is_killed_after_one_interval` | `parent_shutdown_racing_child_start` | existing S1-S4 and V3 oracles see linked children | | -| S9 names | `NameUnique`, `NameReleasedAtDeath` | `name_clash_returns_already_started_with_holder`; `failed_named_start_can_be_retried_at_once`; `name_released_before_down_and_exit`; `typed_name_whereis` | `name_registration_race` | at most one live holder; no name of a dead actor after its `Down` | `name_type_must_match_actor` | -| V1/V7 ignore, clash at restart | (Supervision model: `Ignored` state in the impl PR) | `supervised_ignore_keeps_spec_without_generation`; `temporary_ignore_is_removed`; `restart_name_clash_is_a_failed_attempt` | | V1 oracle treats `Ignored` | | -| API removal | | | | | `start_on_removed`, `start_with_backend_removed`, `start_linked_removed`, `execution_group_start_removed`, `start_options_struct_literal` (private/`non_exhaustive`), `start_monitor_requires_down_handler` | +| L8 commit, refusal invisible | `AbortInvisible` | `refused_start_runs_no_hook_and_leaves_no_edge_or_name` (closed, capacity, clash, parent exited, prepare failure); `refused_named_start_is_never_found_by_whereis` | `commit_races_whereis_and_competing_admission` | aborted ids never in `Exit`/`Down`/registry/`Capacity` causes | `start_result_is_not_an_actor_ref` | +| L8/L2 readiness final (P0-1) | `ReadyIsFinal`, `ResultMatchesCommit` | `ready_then_immediate_crash_with_unpolled_waiter_returns_started_and_delivers_down` (link and monitor); `continuation_panic_before_waiter_resumes_is_a_crash_not_a_start_error` | `ready_commit_races_post_ready_death` | ack `Started` ⇒ hook log shows ready; post-ready deaths deliver notices | | +| L2 readiness while open | `ReadyOnlyWhileOpen` | `group_closure_immediately_before_ready_commit_fails_the_start` | `group_shutdown_during_start` | existing S1 oracle | | +| L9 resolution | (abstracted outcomes) | one test per row of 2.3, each mode and placement, for every waiting verb and `spawn` + `ready()`; `error_with_held_abnormal_parent_exit_answers_init_exited` | `held_signals_resolve_by_precedence` | generated outcomes; the result and the exit reason match the table | `started_may_return_unit_result_or_init` (ui-pass); `continue_with_requires_handler` | +| L9 trapping child (agy P1-4) | `ContinueFirst`, `ParentExitReachesChild` | `trapping_child_with_normal_parent_exit_in_started_commits_ready_runs_continuation_then_ends`; the same for an abnormal parent exit | `trapping_child_parent_exit_during_started` | | | +| L10 continuation | `ContinueFirst` | `continuation_runs_before_messages_sent_during_started`; `stop_before_continuation_discards_it`; `terminate_child_can_end_a_child_before_its_continuation` | `continuation_races_peer_send` | hook-log order | | +| X5 edge before `started()` | `LinkBeforeStarted` | `start_link_child_panicking_in_started_notifies_parent` (every placement), the same for `start_monitor` | `child_dies_in_started_racing_commit` | link/monitor atomicity | | +| X5 parent exited | `NoEdgeToDeadParent` | `start_link_through_dead_context_is_refused`; `start_link_from_stopped_is_refused` | `parent_stops_during_child_started` | | | +| X8 disposition | `FailureFlushed`, `NoticeOnce`, `TimeoutContained` | `failed_start_link_consumes_exit_of_trapping_parent`; `failed_start_link_kills_nontrapping_parent_after_handler`; `failed_start_monitor_consumes_down`; `init_timeout_never_reaches_the_parent` | `failure_races_ack` | no notice for a start that returned an error; at most one per child | | +| X8 abandonment | `AbandonedNoticeDelivered` | `abandoned_start_monitor_still_delivers_down`; `abandoned_start_link_delivers_exit`; `abandoned_start_with_timeout_delivers_killed` | `abandon_races_publication` | | | +| L8 failure after publication (P0-3) | `FailureAfterPublication`, `NoOverlapOnReuse` | `state_drop_finishes_before_failure_answer_and_name_reuse` (`Mock::drop_flag`); `hung_drop_answers_with_pending_handle_at_the_bound` | `retry_never_overlaps_old_drop` | no two live states of one name | | +| T5 timer vs readiness (P0-2) | `TimeoutNeverKillsReady`, `ResultMatchesCommit` | `init_timeout_racing_ready_agrees_with_ready()` (failpoint at the commit); `spawn_init_timeout_kills_and_ready_reports_it`; `stuck_started_keeps_name_and_reports_pending` | `timeout_races_ready_commit` | timeout options generated; a ready child never ends `Killed` by its timer | | +| `ready()` parity (agy P1-6) | `ResultMatchesCommit` | `ready_returns_the_same_result_as_the_waiter` for `Started`, `Ignored`, `InitFailed`, `InitExited`, `InitTimeout`, via `spawn` and `Starting::actor_ref()` | | | | +| S7 inheritance | `InheritedPlacement` | `link_and_monitor_inherit_{async_group,thread_group,pinned_shard,backend_async,backend_blocking,backend_thread}`; `plain_start_never_inherits`; `placement_of_opts_in` | | placement oracle | `threads_options_have_no_group_or_backend` | +| S7 closed/full | `AdmittedOnlyOpen`, `CapacityRespected` | `inherited_start_into_closing_group_fails_closed`; `..._full_thread_group_fails_capacity`; `capacity_released_only_at_publication` | `group_shutdown_during_start` | | | +| S8 ownership | `GenDoneMeansOwnedDone`, `OwnedScopeFinishes` | `system_shutdown_stops_owned_children_with_their_parent`; `supervised_generation_waits_for_owned_children`; `unlinked_owned_child_is_killed_at_the_scope_deadline`; `stuck_parent_does_not_delay_descendant_kill`; `foreign_system_and_unmanaged_children_are_not_owned` | `parent_shutdown_racing_child_start` | S1-S4/V3 oracles see owned children | | +| S9 names | `NameUnique`, `NameHeldUntilPublished`, `NameFreeBeforeObservation` | `name_clash_returns_already_started_with_holder` (the holder kept alive across both calls); `failed_named_start_can_be_retried_at_once`; `name_released_before_down_and_exit`; `typed_name_whereis`; `unregister_then_reregister_keeps_ownership`; `cross_type_name_clash_reports_none_holder` | `name_registration_race` | one live holder | `name_type_must_match_actor` | +| V1/V5/V7 supervisors | `Supervision`/`Cohorts` `Ignored` (impl PR) | `factory_ignore_and_init_ignore_keep_the_spec`; `ignored_child_restarted_in_one_for_all_round`; `ignored_child_restarted_in_rest_for_one_round`; `temporary_ignore_is_removed`; `wait_ready_on_ignored_returns_unavailable`; `startup_timeout_bounds_a_hung_factory`; `startup_timeout_infinity`; `restart_name_clash_is_a_failed_attempt` (traced, `trace_check.py`) | | V1 oracle treats `Ignored` | | +| Options | | `zero_timeout_and_zero_bounds_are_invalid_options` (both modes); `stack_size_on_pooled_placement_is_invalid`; `spawn_panics_on_executor_failure` (failpoint) | | | `start_options_struct_literal`; `start_options_clone_without_actor_clone` (ui-pass) | +| API removal | | | | | `start_on_removed`, `start_with_backend_removed`, `start_linked_removed`, `execution_group_start_removed`, `start_monitor_requires_down_handler` | +| No cost on send/dispatch | | | | | benchmark gate: the default send, request and dispatch benchmarks within noise of 0.8 (`benchmarks/`, the gate introspection #256 uses); `future_layout` sizes unchanged for the run loop | +| Lock order | | Loom: `commit_races_parent_exit_begin`, `two_commits_one_name`, `ready_commit_races_timer` (the cell's CAS) | | | | **Placements.** Every integration test runs in: -- tasks: `Backend::Async`, `Blocking`, `Thread`, `async_group`, - `thread_group` (and `pinned_group` on Linux); -- threads mode, where the API exists there (all except the placement rows). - -They use a per-placement macro, as in `semantics_gaps`. "Start from outside -any actor" is the plain `spawn`/`start` rows run from the test body. - -### 16.2 TLA+ and trace validation - -- **Model.** `Start.tla`, per PR: `MCStart`, `MCStartContinue`, - `MCStartLive` (the CI leg `start`). Weekly: `MCStartDeep`. The 16 negative - controls are in `negative_checks.py` (`START_CONTROLS`, - `START_TEMPORAL_CONTROLS`). -- **Trace schema `start-v1`.** The `lifecycle-trace` feature records these - events from the real code: - - `SBegin(c, kind, explicit, named)`, `SAdmit(c, place | refusal)`, - `SLink(c)`, `SMonitor(c)`, `SRegister(c | already)`, `SDispatch(c)`; - - `SStartedBegin(c)`, `SStartedEnd(c, outcome)`, `SAck(c, result)`; - - `STimeout(c)`, `SAbandon(c)`, `SDie(c, reason)`; - - `PDie(reason)`, `GClose`, `SSend(c)`, `SHandle(c)`, `SContinue(c)`. - - `TraceStart.tla` replays them as a behaviour of `Start.tla` (`trace_check.py` - gains the schema). A new `concurrency/tests/start_trace.rs` records every - integration scenario of 16.1 on Async, async groups and thread groups. -- **Trace negative controls** (`negative_checks.py`, `require_rejected`): - - move `SLink` after `SStartedBegin`; - - give two children the same `SRegister`; - - change an `SAdmit` place to unmanaged under a managed parent; - - deliver an `Exit` after a failed `SAck`. -- **Trace coverage.** `trace_coverage.py` adds a `StartCoverage` projection; - the baseline requires every `Start.tla` action to be reached by a recorded - trace. +- tasks: `Async`, `Blocking`, `Thread`, `async_group`, `thread_group`, and + `pinned_group` on Linux; +- threads mode, except the placement rows. -### 16.3 Simulator +They use a per-placement macro, as `semantics_gaps` does. -`concurrency/tests/start_sim.rs`, each case under PCT (depths 1-3) and -bounded DFS, with seeds and tapes: +### 16.2 Trace validation (Codex 11, agy P1-9) -| Race | Check | -|---|---| -| parent stops during the child's `started()` | the child ends with the parent's reason, or `Normal` ignored | -| link or monitor racing the child's death in `started()` | exactly one of: consumed error, or delivered notice (abandoned) | -| two starts with one name | exactly one `Ok`, one `AlreadyStarted` | -| group shutdown during a start | either admitted and owned by shutdown, or `Closed`; never unowned | -| init timeout racing `started()` return | `Ok` xor `InitTimeout`; the parent is never hit | -| abandon racing the child's death | the notice is delivered exactly once | -| continuation racing a peer's send during `started()` | the continuation always runs first | +`lifecycle-trace` records the events below. A new +`concurrency/tests/start_trace.rs` records every scenario in 16.1 on `Async`, +`async_group` and `thread_group`. `TraceStart.tla` replays each trace as a +behaviour of `Start.tla`, and `trace_check.py` gains schema `start-v1`. + +Every event names the code point at which it is recorded, which is its +linearization point. + +| Event | Fields | Model action | Linearization point (code) | +|---|---|---|---| +| `SConfig` | parent placement, parent trap flag, Cap | `Init` | trace start | +| `SBegin` | c, kind, explicit, named | `Begin` | entry of the start call | +| `SPrepareFail` | c, error | `PrepareFail` | the prepare step's error return | +| `SCommit` | c, place, edge kind, named, owned | `Commit` | inside the commit section, all locks held | +| `SRefuse` | c, reason | `Commit` (refusal) | inside the commit section | +| `SLookup` | c | `Lookup` | `whereis` under the registry read lock (test probes only) | +| `SStartedBegin` | c | `StartedBegin` | before the first poll of `started()` | +| `SResolve` | c, outcome, trap flag, held signals, committed cell | `StartedEnd` / `KilledEarly` | under the cell lock at the end of `started()` | +| `STimer` | c, won | `Timeout` | under the cell lock in the timer | +| `SContinue` / `SHandle` | c | `RunContinue` / `Handle` | the start of the turn | +| `SExit` | c, reason | `Exit` (via `ReadyExit`, `KillReady`, `ApplyParentExit`, `HandleQueuedExit`) | exit begin (0.8 phase 2) | +| `SUnlink` | c | `ChildUnlink` | the unlink fence | +| `SDie` | c, taken link note, taken monitor note | `Die` | D (0.8 phase 4) | +| `SPublish` | c, name released, slot released, disposition per note (delivered, diverted, applied, dropped) | `Publish` | publication (0.8 phase 6), under the cell lock | +| `SAnswer` | c, result, pending | `AckOk` / `AckFail` | under the cell lock when the waiter takes the result | +| `SAbandon` | c, handed-on notice | `Abandon` | `Drop` of `Starting`, under the cell lock | +| `PExitBegin` | reason | `PExit` | the parent's exit begin | +| `PDie` | per child: queued or held parent exit | `PDie` | the parent's D | +| `PNotice` | c, kind, reason | (part of `Publish`/`Abandon`) | enqueue into the parent's mailbox, or the signal applied to it | +| `GClose`, `GScopeKill`, `GGenDone` | | `GClose`, `ScopeDeadline`, `GenDone` | group close; the scope deadline; the generation-end commit | + +- **Complete consumption.** Replay must consume every event, and it may not + invent observable events: every notice the parent sees has its own + `PNotice` and must match the disposition recorded in `SPublish`. +- **Fault-only actions** (`RegisterFirst`, `CommitRest`, `LateLink`, + `LateRelease`) are excluded from positive coverage. `trace_coverage.py` + requires every other action to be reached. +- **Trace negative controls** (`require_rejected`). Each mutates a recorded + trace, which the checker must reject: + - move `SCommit` after `SStartedBegin`; + - give two children overlapping names; + - change a managed child's `SCommit` place to unmanaged; + - add a `PNotice` after an `SAnswer` error for a trapping parent; + - move `SAnswer` (error) before `SPublish`; + - add an `STimer won` after an `SResolve` that committed ready. + +### 16.3 Simulator + +`concurrency/tests/start_sim.rs`, every race in 16.1's Sim column, each +under PCT (depths 1-3) and bounded DFS, with seeds and tapes. Each comes +with a mutation that makes it fail: the planted bugs below, or a failpoint +that reorders the commit. -### 16.4 Scenario fuzzer (`fuzz/`) +### 16.4 Scenario fuzzer - **Generator.** `SpawnHow` becomes `Start { verb: Spawn | Start | Link | Monitor, placement: Default | Group(g) | Unmanaged | SameAsParent, name: Option, init_timeout: Option, abandon: bool }`. - Worker hooks gain `started()` outcomes: ready, continue, error, ignore, - stop with a reason, panic, sleep. + Worker hooks gain `started()` outcomes (ready, continue, error, ignore, + stop, panic, sleep, set `trap_exit`). Supervisor factories gain `Ignore`. - **New oracles:** - - placement: the inheritance rule; - - link/monitor atomicity and failure flush; - - names unique and released before any death observation; - - ack `Ok` ⇒ `started()` returned; - - continuation first. -- **Extended oracles:** the existing M1/L1/L5 and X1/X3/X5 oracles learn - that a consumed notice is expected. + - placement inheritance; + - commit invisibility; + - readiness final; + - notice disposition, with the non-trapping-parent rule kept; + - names unique and released before observation; + - continuation first; + - S8 generation end. +- **Qualification.** Timeout containment applies only to unabandoned waits. -### 16.5 Planted bugs +### 16.5 Planted bugs (`verification/planted/patches/`, #247) -Added to `verification/planted/patches/` (the #247 matrix). Each must be -caught by at least two layers. +Each is caught by at least two layers: -| Patch | Bug | Must be caught by | +| Patch | Bug | Caught by | |---|---|---| -| `NN-start-link-ignores-parent-group` | `start_link` admits the child unmanaged when the parent is managed (the 0.8 behaviour) | `link_and_monitor_inherit_*`, fuzz placement oracle, trace `SAdmit` check (`InheritedPlacement`) | -| `NN-start-link-after-dispatch` | the edge is recorded after the child is dispatched | `start_sim::child_dies_in_started_racing_link` (deterministic; the integration test `start_link_child_panicking_in_started_notifies_parent` catches it only on some schedules), trace `LinkBeforeStarted` | -| `NN-start-name-check-then-insert` | the name is checked under a read lock and inserted under a separate write lock | `start_sim::name_registration_race`, fuzz name oracle | -| `NN-start-monitor-down-lost-on-abandon` | a diverted `Down` is dropped when the `Starting` is dropped | `abandoned_start_monitor_still_delivers_down`, `start_sim::abandon_races_child_death` | -| `NN-start-name-released-at-publication` | the name is released at publication instead of D | `failed_named_start_can_be_retried_at_once`, `name_released_before_down_and_exit`, trace `NameReleasedAtDeath` | +| `start-link-ignores-parent-group` | 0.8 placement for `start_link` | `link_and_monitor_inherit_*`, fuzz placement oracle, trace `InheritedPlacement` | +| `start-link-after-dispatch` | edge created after the latch opens | `start_sim::child_dies_in_started_racing_commit`, trace `LinkBeforeStarted` | +| `start-name-check-then-insert` | name checked under a read lock, inserted under a separate write lock | `start_sim::name_registration_race`, fuzz name oracle, Loom `two_commits_one_name` | +| `start-monitor-down-lost-on-abandon` | a diverted `Down` dropped with the `Starting` | `abandoned_start_monitor_still_delivers_down`, `start_sim::abandon_races_publication` | +| `start-name-released-at-death` | name released at D (before destruction) | `state_drop_finishes_before_failure_answer_and_name_reuse`, trace `NoOverlapOnReuse` | +| `start-mark-not-cleared-at-ready` | readiness leaves the start marks | `ready_then_immediate_crash_with_unpolled_waiter_*`, `start_sim::ready_commit_races_post_ready_death` | +| `start-timer-without-cas` | the timer kills without checking the cell | `init_timeout_racing_ready_agrees_with_ready`, Loom `ready_commit_races_timer` | ### 16.6 Coverage matrix -Each new or changed guarantee in section 14 gets: - -- a "Checked by" line in `semantics.md` naming the checks above; -- a row in `docs/TESTING.md` "Semantics coverage" (modes, placements, - covered or partial). - -New rows: L8, L9, L10, X8, S7, S8, S9, T5. Changed rows: L1, L2, X3, X5, S6, -V1, V7. A guarantee is merged as covered only when each clause has a failing -check under its planted bug or model control. +- **`semantics.md`:** each guarantee in section 14 gets a "Checked by" line + naming the checks above. +- **`docs/TESTING.md` "Semantics coverage":** a row per guarantee. + - New rows: L8, L9, L10, X8, S7, S8, S9, T5. + - Changed rows: L1, L2, X3, X5, S6, V1, V5, V7. +- **Merge rule.** A guarantee is merged as covered only when each clause + fails under its planted bug or model control. ### 16.7 Client checks -The client-compat gate (`.github/workflows/client-compat.yml`) runs on the -implementation PR, against client branches that carry the mapped rewrite -(`spawned-v0.9`, created from `spawned-v0.8`): +The client-compat gate runs on the implementation PR, against `spawned-v0.9` +client branches that carry the mapped rewrite: -- **ethrex:** check, clippy `-D warnings`, `ethrex-p2p` tests, the `p2p::` - and `rpc::` groups, and the lib tests. -- **ethlambda:** clippy and `cargo test --workspace --profile release-fast`. - -Clippy `-D warnings` also turns any leftover un-awaited `start()` into an -error. +- ethrex: check, clippy `-D warnings`, `ethrex-p2p`, `p2p::`/`rpc::`, and the + lib tests; +- ethlambda: clippy and `cargo test --workspace --profile release-fast`. ## 17. Alternatives considered | Alternative | Why not | |---|---| -| Keep `start()` immediate and add `ready()` (no waiting starts) | Startup failure stays invisible by default (G1). The init timeout and failure flush have no starter to act for, so a timed-out linked child would kill a non-trapping parent with `Killed`, unlike OTP's `kill_flush_EXIT` | -| Waiting starts with a lazy stage 1 (nothing happens until awaited) | A forgotten `.await` would silently never start the actor (a warning only), across 51 migrated client sites. The eager stage 1 makes that mistake equal `spawn` | -| Ambient inheritance for plain `start`/`spawn` (task- or thread-local current actor) | A cost per turn; it does not follow `tokio::spawn` or helpers; it silently moves 20 ethrex call sites into groups; OTP's plain `start` has no parent either | -| Fall back to unmanaged when the inherited group is closed or full | It re-creates the footgun at the worst moment (shutdown) | -| One `start(opts)` taking options always (`start(StartOptions::default())`) | Noise at the common call; the `with_options` builder keeps the zero-option verbs short and gives one options path | -| Six or eight `_with` methods (`start_with`, `start_link_with`, ..) | Doubles the surface; `with_options(o).verb()` is one entry point | -| Typed init data (`started() -> Init`, Gleam's `returning`) | The starter needs the type `R` statically, which needs an associated type on `Actor`, and stable Rust has no associated-type defaults, so every manual `impl Actor` would break. The main use (handing back a sub-handle) is covered by the typed `ActorRef` and `Recipient`s derived from it. `Starting` is a struct, so a later `data()` accessor stays additive | -| Typed init error (`InitFailed(E)`) | Same associated-type problem; `Display` text matches supervisors' factory errors and Gleam's `InitFailed(String)` | -| `trap_exit` as a start option | No window to close: signals are held during `started()` and resolved with the flag read when it returns (X3). One way to set it stays | -| `#[actor]` derives a child spec (Elixir `child_spec/1`, Gleam `supervised()`) | A spec needs a factory for fresh state per generation, which the type alone cannot provide. With `ChildSpec::new(id, &g, f).options(o)` the derive would save nothing. `ChildSpec` keeps room for `significant` | -| `spawn_link` / `spawn_monitor` (no waiting) | OTP's `gen_server` has none; slow initialisation belongs in a continuation, so a waiting `start_link` returns fast | -| Continuation from every handler | A check after every turn is a cost on the dispatch path; post-init is the case clients need (#245) | -| `init_timeout` default 5 s | OTP's default is `infinity`; it would kill client actors whose `started()` does slow I/O; `ChildSpec` keeps its 5 s bound | -| Keep `start_on` as sugar for `with_options(g).spawn()` | Two spellings again (`ExecutionGroup::start` was the same mistake) | - -## 18. Gap-report input: accepted and rejected - -| Suggestion | Decision | -|---|---| -| Fallible init on every start path, `InitTimeout`/`InitFailed`/`InitExited`, non-waiting `spawn` | accepted (sections 2.3, 2.5, 3) | -| `Init` carrying data for the starter | rejected for 0.9 (section 17), room kept | -| `Init` continuation, `Init::ignore` | accepted (sections 8, 9) | -| `start_monitor`; link and monitor on every backend and group | accepted | -| Options: placement, label, typed name, init timeout, stack size; reserve mailbox, memory, priority | accepted; default timeout `infinity`, not 5 s (section 7) | -| `trap_exit` at start | rejected (no window; section 17) | -| `spawn_opt` GC and priority knobs | not copied (section 2.4) | -| `#[actor]` derives the child spec; room for `significant` | derive rejected (section 17); room kept | -| Builder shape `A::new(..).spawn_opts(opts).start()` | adopted as `with_options(opts).start()` | - -## 19. Open questions for the reviewers - -1. **Should a waiting `start_link` from a parent's turn block that turn?** - This is OTP, and the continuation keeps it short. A `started()` that - requests its parent deadlocks until the request timeout, as in OTP. -2. **S8 kill escalation for `start_link` descendants:** one cleanup interval - after the parent's D, then stuck. Is tying a supervised generation's end - to its linked descendants (V3) acceptable for ethrex-style per-connection - children? -3. **`init_timeout` default:** none (OTP) or 5 s (bound)? -4. **Continuation from ordinary handlers** for 0.9, at a one-branch dispatch - cost, or later? -5. **`AlreadyStarted.existing: Option`** against a typed - `Option>`. The typed form makes `StartError` generic. -6. **Moving `ChildSpec::{startup_timeout, max_*}` into `StartOptions`:** - neither client uses them, but the dynamic supervisor (#245) should adopt - the same shape in its review. -7. **`Starting::actor_ref()`** exposes the reference before readiness. Is it - needed beyond `spawn`? +| Keep `start()` immediate with only `ready()` | Startup failure stays invisible (G1); there is no starter for the flush and timeout rules | +| Lazy stage 1 (nothing until awaited) | A forgotten `.await` would silently never start the actor | +| Multi-step stage 1 with rollback (revision 1) | A name or capacity reservation is observable before a later refusal (Codex 4) | +| Readiness committed by the waiter's ack (revision 1) | A death after readiness but before the poll became a start error, or swallowed its notice (P0-1) | +| Unlink at the timeout (revision 1) | It races abandonment and readiness. One cell with the notice disposed of at publication is equivalent and has one owner (P0-2) | +| Answer at D (revision 1) | A retry could overlap the old `Drop` (P0-3) | +| Abort `started()` at an `.await` on a kill or timeout (agy P1-5) | Not in 0.9. Kills are cooperative everywhere (L3); aborting a turn at an await is introspection's 0.10 design and should cover handlers and `started()` together, with its own model. Until then the stuck policy (section 6) applies | +| Typed start data (`A::StartData`, Gleam's `returning`, #245's G4) | OTP's `gen_server` start returns only `{ok, Pid}`; the data channel is `proc_lib`'s, which `gen_server` does not expose. Under the match-Erlang rule, deferred; #245 drops `Started::data` (section 18). The associated type would also be required in every `impl Actor` (no stable defaults) | +| `Err(Ignored)` (revision 1) | Not Rust-forced; OTP returns `ignore` beside `{ok, Pid}`; the dynamic supervisor's `StartOutcome` already has the shape (Codex 9) | +| Inheritance for plain starts (ambient current actor) | A cost per turn, misses helpers, moves 20 client sites | +| Continuation from every handler | A branch per turn | +| `#[actor]` derives a child spec | A spec needs a factory; the derive saves nothing | +| A separate `init_timeout` on `ChildSpec` replacing `startup_timeout` (revision 1) | It dropped the factory deadline (agy P1-7, Codex 3) | +| `trap_exit` as a start option | Held signals already resolve with the flag `started()` leaves | + +## 18. Cross-design agreement + +These shapes are shared with the dynamic supervisor (#245, revision 3) and +introspection v1 (#256, approved). Where #245 must change, the change is +listed for its next revision. Revision 2 of this design does not edit that +branch, which its own author maintains. + +| Item | Agreed shape | Unified start | #245 dynamic supervisor | #256 introspection | +|---|---|---|---|---| +| Start result | `StartOutcome { Started(T), Ignored }` | `T = ActorRef` or `(ActorRef, MonitorRef)` | `StartOutcome>` (rename from `StartOutcome`) | n/a | +| `Started` | `{ actor, child }` | n/a | **drop `data: A::StartData`** (deferred, section 17) | n/a | +| Factory result | `ChildStart { Actor(A), Ignore }`, `From` | static `ChildSpec` | as proposed | n/a | +| `ignore` | static: spec kept, `ChildState::Ignored`, restarted in later cohort rounds, Temporary removed; dynamic: nothing stored (first start) or removed (restart), no `ChildState` variant needed | section 9 | as proposed | `ChildState::Ignored` appears only for static children; counted in `specs` and `workers`, not `active` | +| Start errors | `StartError`; supervisors wrap it in `ChildStartError { Factory, Start, StartupTimeout { pending } }` | section 2.5, 9 | `FailedToStart { cause: ChildStartError, cleanup_stuck }` replaces `{ cause: String, exit }` | crash reports carry the `Display` | +| Whole-attempt deadline | `startup_timeout` covers the factory and `started()`; `Timeout::Infinity` allowed | section 9 | as proposed (default 5 s) | n/a | +| Options | `StartOptions` | section 2.4 | `DynamicChildSpec::options(StartOptions)`; the template's group stays positional | `label` is the initial `set_label` value | +| Continuation | `Init::continue_with(msg)` with `Handler`; precedes messages, not a termination barrier | section 8 | the ethrex sketch's `#[continue_start] async fn handshake` becomes `Init::continue_with(Handshake)` plus `Handler`; a `terminate_child` may end the child before or during it | n/a | +| Lifecycle | `started()` running is `Lifecycle::Starting`; the ready commit moves to `Idle` | section 5.1 | n/a | as specified | +| One spawn path | every verb funnels through stage 1's commit, which inserts the actor-table entry | section 3 | n/a | as specified ("one spawn path registers every actor") | +| `ChildInfo`, `ChildCounts`, `ChildKey` | introspection's types | n/a | as proposed | owner | + +## 19. Decisions on the round-one open questions + +| # | Question | Decision | Source | +|---|---|---|---| +| 1 | A waiting `start_link` blocks the parent's turn | **Yes**, as OTP, with the deadlock documented (5.3) | maintainer | +| 2 | Tie a generation's end to its linked descendants | **Yes, as scope ownership** with explicit placement and deadline rules (4.1). agy did not answer this question, so Codex's answer is followed | Codex | +| 3 | `init_timeout` default | **Infinity** (OTP). Supervised children keep the 5 s whole-attempt `startup_timeout` | maintainer | +| 4 | Continuation from ordinary handlers | **Not in 0.9** (dispatch cost) | maintainer | +| 5 | `AlreadyStarted.existing` type | **`Option`**: a global name may belong to another type or a non-actor. Codex's answer; agy's answered a different question (name scope), which rank 12 owns | Codex | +| 6 | Dynamic supervisor adopts `StartOptions` | **Yes** (section 18) | maintainer | +| 7 | Keep `Starting::actor_ref()` | **Yes**: documented as an unready instance; it neither consumes nor abandons the result | Codex | + +## 20. Review resolutions + +### Round 1 (revision 2) + +Each finding was treated as a claim and checked against the code, the design +and the model. + +| Finding | Claim checked | Resolution | +|---|---|---| +| agy P0-1 = Codex 1: readiness depends on the waiter | **True** in revision 1: `res` pending kept the edge mark, and `ChildDie` → `AckFail` was reachable after `StartedEnd` committed ready. Codex's `GClose → StartedEnd → AckOk` also passed | **Accepted.** A child-side ready commit clears the marks before any continuation or dispatch (5.1). Invariants `ReadyIsFinal`, `ResultMatchesCommit`, `ReadyOnlyWhileOpen`; controls `DivertAfterReady`, `ReadyAfterClose`; tests in 16.1 | +| agy P0-2 = Codex 2: timeout, abandonment and failure arbitration | **True**: revision 1's timer acted only while a waiter existed, and the waiter path removed edges | **Accepted.** One cell with a CAS commit and a separate waiter state; a framework-owned timer; the notice disposed of at publication (5.1, 5.2, 6). `TimeoutNeverKillsReady`, control `TimerIgnoresCommit`; the bounded wait is explicit (`pend`, `pending`) | +| agy P0-3 = Codex 5: D vs publication | **True**: revision 1 released the name and answered at D, before destruction | **Accepted.** The answer, the name release and the slot release come at publication (5.2, 7). `FailureAfterPublication`, `NoOverlapOnReuse`, `NameFreeBeforeObservation`; controls `EarlyFailAck`, `UnregisterAtDeath`, `LateUnregister`. A hung `Drop` answers with `pending` at the bound | +| agy P1-7 = Codex 9: `ignore` and cross-design types | **True**: cohort behaviour was undefined; `Err(Ignored)` diverged from #245 | **Accepted.** OTP `ignore` semantics for static supervisors (9), `Ok(StartOutcome::Ignored)` everywhere, and the shared types (18). The `Supervision`/`Cohorts` model and trace updates are in the implementation PR, with named controls | +| agy P1-7 = Codex 3: factory deadline | **True** | **Accepted.** `startup_timeout` stays, covering the whole attempt, with `Timeout::Infinity` (9) | +| agy P1-8/P1-9 = Codex 10/11: weak invariants, trace validation | **Partly true.** Revision 1's `LiveInG` already excluded dead children (agy quoted a different text). But `Cap = 2` with a managed parent never held two children at once (true), capacity was released at D rather than publication (true), and the other weaknesses were true | **Accepted.** `Cap = 3`; slots are held until publication; real edge-creation history; notice counters with reasons; committed-outcome history; liveness `StartAnswered`; controls `DoubleNotice`, `ErrorAsOk`, `NoAnswerIgnore`. The lock protocol is in 3 and 5.2, and the event-to-action mapping with linearization points is in 16.2. Fault-only actions are excluded from coverage | +| Codex 4: refusal needs a commit protocol | **True** | **Accepted.** Prepare and an atomic commit under a global lock order (3); `Lookup` observer; control `SplitCommit`; Loom tests | +| Codex 6: combined outcomes and held signals | **True** | **Accepted.** The precedence table in 2.3, both the result and the exit reason; the superseded diagnostic is logged; the trap flag at return is labelled the cooperative R deviation (11) | +| Codex 7: S8 placement and deadlines | **True** | **Accepted.** Ownership is decided apart from placement, retained after unlink (justified), with one scope interval and direct kills (4.1). Modelled: `MCStartScope`, `GenDoneMeansOwnedDone`, `OwnedScopeFinishes`, controls `EarlyGenDone`, `NoScopeKill` | +| Codex 8: a continuation is not a termination barrier | **True** | **Accepted.** The guarantee is restated (8), and the client mapping and the #245 coordination row corrected (18) | +| agy P1-4: trapping child plus parent exit during `started()` | **True**: revision 1 contradicted E3 and 0.8's `resolve_held_exits` | **Accepted.** A trapping child commits ready, runs its continuation, then handles the queued parent exit (`Normal` included) (2.3). Modelled with `ChildCanTrap` | +| agy P1-5: a held kill cannot cancel a hung `started()` | **True as a description**; the proposed future-drop is rejected for 0.9 | **Partly accepted.** An explicit stuck policy with no thread abort (6). The abort-at-await mechanism belongs to the 0.10 design (17) | +| agy P1-6: `ready()` contradictions | **True** | **Accepted.** `ready()` reads the cell (5.5); parity tests | +| agy P1-10: untyped `StartOptions` | **False for this design**: revision 1 already had `StartOptions` and `name(impl Into>)`. agy's text quotes a `named` method that never existed | **No change**, except Clone without bounds is now documented and the trybuild cases are listed | +| Codex 12 (P2) | | Zero and invalid options (2.4); `spawn()` dispatch failure (3); Clone without bounds; a no-cost benchmark gate; name ownership and cross-type tests; the two-name test holds the winner (16.1) | +| agy P2-11 | `NoticeOnce` had no control and used sets: true | Counters, control `DoubleNotice`; `MCStartContinue` runs every explicit placement | +| agy P2-12 | | `init_fail/3` cited; `Init::stop(Normal)` documented (2.3) | +| agy P2-13 | | All six tests added (16.1) | diff --git a/verification/tla/MCStartScope.cfg b/verification/tla/MCStartScope.cfg index e6ecb804..c3293990 100644 --- a/verification/tla/MCStartScope.cfg +++ b/verification/tla/MCStartScope.cfg @@ -1,6 +1,6 @@ SPECIFICATION LiveSpec CONSTANTS - Kids = {"c1", "c2"} + Kids = {"c1"} Kinds = {"link"} Explicit = {"none", "U"} ParentPlaces = {"G"} diff --git a/verification/tla/MCStartScope.tla b/verification/tla/MCStartScope.tla index 943f60c1..a11054ac 100644 --- a/verification/tla/MCStartScope.tla +++ b/verification/tla/MCStartScope.tla @@ -1,7 +1,7 @@ ---------------------------- MODULE MCStartScope ---------------------------- EXTENDS Start \* Per-PR instance for S8 (linked children owned by the parent's scope): -\* a managed parent, two linked children in its group or unmanaged, which -\* may trap or unlink; the scope deadline and the generation end. Safety and +\* a managed parent and one linked child, in its group (owned) or unmanaged +\* (not owned), which may trap or unlink; the scope deadline and the generation end. Safety and \* OwnedScopeFinishes. ============================================================================= From 96ddf44db9f2bc8150cff03527f901bc8efa663a Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:13:49 -0300 Subject: [PATCH 15/42] verification: revision-2 Start controls --- verification/tla/negative_checks.py | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index c4052e29..8d5e4d5d 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -247,29 +247,38 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStart", "NoInherit", "InheritedPlacement"), ("MCStart", "LinkAfterStarted", "LinkBeforeStarted"), ("MCStart", "DoubleRegister", "NameUnique"), - ("MCStart", "LateUnregister", "NameReleasedAtDeath"), - ("MCStart", "AbortLeaksLink", "AbortInvisible"), - ("MCStart", "EarlyFailAck", "FailureAfterDeath"), - ("MCStart", "AckBeforeStarted", "AckMeansStarted"), + ("MCStart", "LateUnregister", "NameFreeBeforeObservation"), + ("MCStart", "UnregisterAtDeath", "NoOverlapOnReuse"), + ("MCStart", "SplitCommit", "AbortInvisible"), + ("MCStart", "EarlyFailAck", "FailureAfterPublication"), + ("MCStart", "AckBeforeStarted", "ResultMatchesCommit"), + ("MCStart", "ErrorAsOk", "ResultMatchesCommit"), + ("MCStart", "DivertAfterReady", "ReadyIsFinal"), + ("MCStart", "ReadyAfterClose", "ReadyOnlyWhileOpen"), ("MCStart", "NoMonitorFlush", "FailureFlushed"), ("MCStart", "NoLinkFlush", "FailureFlushed"), - ("MCStart", "KillBeforeUnlink", "TimeoutContained"), ("MCStart", "LinkCheckLate", "NoEdgeToDeadParent"), ("MCStart", "AdmitClosed", "AdmittedOnlyOpen"), ("MCStart", "IgnoreCap", "CapacityRespected"), ("MCStart", "AbandonDropsNotice", "AbandonedNoticeDelivered"), + ("MCStart", "DoubleNotice", "NoticeOnce"), + ("MCStartTimeout", "TimerIgnoresCommit", "TimeoutNeverKillsReady"), + ("MCStartTimeout", "KillBeforeUnlink", "TimeoutContained"), ("MCStartContinue", "ContinueQueued", "ContinueFirst"), + ("MCStartScope", "EarlyGenDone", "GenDoneMeansOwnedDone"), ] START_TEMPORAL_CONTROLS = [ - ("DropHeld", "ParentExitReachesChild"), + ("MCStartLive", "DropHeld", "ParentExitReachesChild"), + ("MCStartLive", "NoAnswerIgnore", "StartAnswered"), + ("MCStartScope", "NoScopeKill", "OwnedScopeFinishes"), ] def start_controls(tlc): for module, fault, invariant in START_CONTROLS: named_fault(module, fault, invariant, tlc) - for fault, prop in START_TEMPORAL_CONTROLS: - named_temporal("MCStartLive", fault, prop, tlc) + for module, fault, prop in START_TEMPORAL_CONTROLS: + named_temporal(module, fault, prop, tlc) def main(): From 32d5a9900be581eb78884a5fa67392606c1e60ff Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:18:14 -0300 Subject: [PATCH 16/42] verification: MCStartCap (capacity binds; release at publication; retry) --- verification/tla/MCStartCap.cfg | 38 +++++++++++++++++++++++++++++ verification/tla/MCStartCap.tla | 8 ++++++ verification/tla/negative_checks.py | 4 +-- 3 files changed, 48 insertions(+), 2 deletions(-) create mode 100644 verification/tla/MCStartCap.cfg create mode 100644 verification/tla/MCStartCap.tla diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg new file mode 100644 index 00000000..0e4383ae --- /dev/null +++ b/verification/tla/MCStartCap.cfg @@ -0,0 +1,38 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1", "c2", "c3"} + Kinds = {"link"} + Explicit = {"none"} + ParentPlaces = {"G"} + Cap = 3 + Named = FALSE + TimeoutEnabled = FALSE + ParentCanExit = TRUE + CloseEnabled = TRUE + MaxMsgs = 0 + ChildCanTrap = FALSE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE + Fault = "None" +INVARIANT TypeOK +INVARIANT InheritedPlacement +INVARIANT LinkBeforeStarted +INVARIANT NameUnique +INVARIANT NameHeldUntilPublished +INVARIANT NoOverlapOnReuse +INVARIANT NameFreeBeforeObservation +INVARIANT AbortInvisible +INVARIANT ResultMatchesCommit +INVARIANT ReadyIsFinal +INVARIANT TimeoutNeverKillsReady +INVARIANT ReadyOnlyWhileOpen +INVARIANT FailureAfterPublication +INVARIANT FailureFlushed +INVARIANT TimeoutContained +INVARIANT NoEdgeToDeadParent +INVARIANT AdmittedOnlyOpen +INVARIANT CapacityRespected +INVARIANT NoticeOnce +INVARIANT AbandonedNoticeDelivered +INVARIANT ContinueFirst +INVARIANT GenDoneMeansOwnedDone diff --git a/verification/tla/MCStartCap.tla b/verification/tla/MCStartCap.tla new file mode 100644 index 00000000..83a2ec0a --- /dev/null +++ b/verification/tla/MCStartCap.tla @@ -0,0 +1,8 @@ +---------------------------- MODULE MCStartCap ---------------------------- +EXTENDS Start +\* Per-PR instance for group capacity: a managed parent and three linked +\* children in a group of three, so a start is refused while the slots are +\* held (until publication), a retry after a failure gets the slot, and two +\* children run in the group beside the parent. AdmitClosed and IgnoreCap +\* run here. +============================================================================= diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 8d5e4d5d..9bebbe24 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -258,8 +258,8 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStart", "NoMonitorFlush", "FailureFlushed"), ("MCStart", "NoLinkFlush", "FailureFlushed"), ("MCStart", "LinkCheckLate", "NoEdgeToDeadParent"), - ("MCStart", "AdmitClosed", "AdmittedOnlyOpen"), - ("MCStart", "IgnoreCap", "CapacityRespected"), + ("MCStartCap", "AdmitClosed", "AdmittedOnlyOpen"), + ("MCStartCap", "IgnoreCap", "CapacityRespected"), ("MCStart", "AbandonDropsNotice", "AbandonedNoticeDelivered"), ("MCStart", "DoubleNotice", "NoticeOnce"), ("MCStartTimeout", "TimerIgnoresCommit", "TimeoutNeverKillsReady"), From 5173fe552975f49eacb89aef12bdefdf41f1e41d Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:22:01 -0300 Subject: [PATCH 17/42] docs: client mapping corrections (continuation guarantee, spawn failure) --- docs/design/unified-start-client-mapping.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/design/unified-start-client-mapping.md b/docs/design/unified-start-client-mapping.md index 5a1c360a..9f8011d4 100644 --- a/docs/design/unified-start-client-mapping.md +++ b/docs/design/unified-start-client-mapping.md @@ -65,7 +65,7 @@ an actor's handler, come from the source. | 19 | `crates/networking/p2p/rlpx/connection/server.rs:292` | `PeerConnection::unestablished` (sync) | `connection.start()` | `connection.spawn()` | | | 20 | `crates/networking/p2p/rlpx/connection/server.rs:4024` | test helper `connection()` | `server.start()` | `server.spawn()` | test | | 21 | `crates/networking/p2p/rlpx/initiator.rs:50` | `RLPxInitiator::spawn_on` (sync) | `..new(context).start_on(group)?` | `..new(context).with_options(group).spawn()?` | E | -| 22 | `crates/networking/p2p/rlpx/initiator.rs:66` | `RLPxInitiator::spawn_with_backend` (sync, returns `ActorRef`) | `state.start_with_backend(b)` | `state.with_options(b).spawn().expect("a backend alone cannot be refused")` | it fails only on options | +| 22 | `crates/networking/p2p/rlpx/initiator.rs:66` | `RLPxInitiator::spawn_with_backend` (sync, returns `ActorRef`) | `state.start_with_backend(b)` | `state.with_options(b).spawn().expect("executor construction failed")` | it fails on invalid options or an executor construction failure (for example, a thread that cannot be spawned), which bare 0.8 `start_with_backend` also panicked on | | 23 | `crates/networking/p2p/rlpx/initiator.rs:67` | same | `state.start()` | `state.spawn()` | | | 24 | `crates/networking/p2p/peer_table.rs:616` | `PeerTableServer::spawn_with_filter` (sync) | `..start()` | `..spawn()` | | | 25 | `crates/networking/p2p/peer_table.rs:635` | `PeerTableServer::spawn_on` (sync) | `.start_on(group)` | `.with_options(group).spawn()` | E | @@ -126,8 +126,12 @@ Unchanged: - **ethrex.** Site #18 is the per-connection actor that the dynamic supervisor design (#245) moves under a `DynamicSupervisor`. With unified start, its RLPx handshake becomes `Init::continue_with(Handshake)`, so no - peer-table message or `terminate_child` can run before it (gap report - 3.2.1). + peer-table message is handled before it (gap report 3.2.1). This is not a + barrier against termination: readiness commits before the continuation + runs, so a `terminate_child`, a kill or supervisor shutdown can end the + connection before or during the handshake (design section 8). The + connection's resource owners must therefore handle a peer that never + finished its handshake, as they do for one that fails it. - **ethlambda.** Site #2 is the chain actor on its own one-worker `thread_group` (0.8 replaced `Backend::Thread` with it). Unified start keeps that placement, and a `start_link` from the chain actor would From 4e49d764cdbb80077daeeb36c627d221166e8f5f Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:22:54 -0300 Subject: [PATCH 18/42] verification: MCStartCap with spawned children; README Start section for revision 2 --- verification/tla/MCStartCap.cfg | 6 +- verification/tla/MCStartCap.tla | 10 +-- verification/tla/README.md | 126 +++++++++++++++++++------------- 3 files changed, 83 insertions(+), 59 deletions(-) diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg index 0e4383ae..8d3b1de8 100644 --- a/verification/tla/MCStartCap.cfg +++ b/verification/tla/MCStartCap.cfg @@ -1,13 +1,13 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2", "c3"} - Kinds = {"link"} - Explicit = {"none"} + Kinds = {"spawn"} + Explicit = {"G"} ParentPlaces = {"G"} Cap = 3 Named = FALSE TimeoutEnabled = FALSE - ParentCanExit = TRUE + ParentCanExit = FALSE CloseEnabled = TRUE MaxMsgs = 0 ChildCanTrap = FALSE diff --git a/verification/tla/MCStartCap.tla b/verification/tla/MCStartCap.tla index 83a2ec0a..ae83b913 100644 --- a/verification/tla/MCStartCap.tla +++ b/verification/tla/MCStartCap.tla @@ -1,8 +1,8 @@ ---------------------------- MODULE MCStartCap ---------------------------- EXTENDS Start -\* Per-PR instance for group capacity: a managed parent and three linked -\* children in a group of three, so a start is refused while the slots are -\* held (until publication), a retry after a failure gets the slot, and two -\* children run in the group beside the parent. AdmitClosed and IgnoreCap -\* run here. +\* Per-PR instance for group capacity: a managed parent and three children +\* spawned into its group of three (capacity does not depend on the start +\* variant, so no edges or waiters): two children run beside the parent, a +\* third is refused while the slots are held (until publication), and gets +\* a slot after a failure is published. AdmitClosed and IgnoreCap run here. ============================================================================= diff --git a/verification/tla/README.md b/verification/tla/README.md index d9af59e5..cf8dc73f 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -17,7 +17,7 @@ this fits with Loom, Shuttle, Miri, ThreadSanitizer and mutation testing. | `DeferredReplies.tla`, `MCDeferredReplies*.tla/.cfg` | Rank 11 design model (not implemented), part 1: reply slots, send-time deadlines with shortening, deferral credits shared by a token and its first linked job, the exiting phase with unwind-local drops, a stuck phase (cleanup deadline passed while `stopped()` runs: nothing closed early, death later or never), death, and downstream request cleanup. Thirteen named negative controls. | | `DeferredJobs.tla`, `MCDeferredJobs*.tla/.cfg` | Rank 11 design model, part 2: pipe jobs with separate execution, cancellation and notification states; keyed replacement including notified jobs; optional job deadlines; a FIFO mailbox with emit credits and the terminal message last; `ActorId` fencing. `MCDeferredJobsEmits` is the fast configuration with emits. Nine named negative controls. | | `Lifecycle.tla`, `MCLifecycle{,Live,Fence,Deep,LiveDeep,WideDeep}.tla/.cfg` | Base actor lifecycle as proposed for v0.8 ([design](../../docs/design/otp-exit-signals.md)): links, monitors, `trap_exit`, exit signals (`Killed`, ignored `Normal`, `NoProc`, parent exits), administrative stops, requests around death, `Exit`/`Down` ordering. Seventeen negative-control faults (17 safety checks, 2 liveness checks): the v0.7 exit-signal behaviours (direct stops ignoring `trap_exit`, kills ending `Kill`, unfenced `unlink`/`demonitor`, real reasons for late registrations, `stopped()` after kills and untrapped signals, late signals ignored during exit), the PR #182/#185 bugs, early exit errors, kill and panic precedence, runtime-drop reasons, early notification, a split death point and a parent exit reaching `exit_received`. Per PR: `MCLifecycle` (7.5M states, fewer reasons, where the safety controls run), `MCLifecycleFence`, `MCLifecycleLive`; weekly: `MCLifecycleWideDeep` (the former 43M-state per-PR instance), `MCLifecycleDeep`, `MCLifecycleLiveDeep`. | -| `Start.tla`, `MCStart{,Continue,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design ([design](../../docs/design/unified-start.md), not implemented): `spawn`, `start`, `start_link`, `start_monitor`, with placement inheritance, an atomic link or monitor before `started()`, atomic names, fallible init, the init timeout, a continuation run before any mailbox message, abandoned waits, and parent death and group closure between the steps of stage 1. Sixteen negative controls (15 safety, 1 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartContinue`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | +| `Start.tla`, `MCStart{,Cap,Timeout,Continue,Scope,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design revision 2 ([design](../../docs/design/unified-start.md), not implemented; TLC-only in `model-map.toml` until trace validation lands with the implementation): `spawn`, `start`, `start_link`, `start_monitor`; placement inheritance; an atomic stage-1 commit; a startup cell (one commitment: ready, failed, ignored or timed out; a waiter that consumes or abandons); death as D then publication; names held until publication; fallible init with held-signal precedence and trapping children; the continuation; S8 scope ownership. 25 negative controls (22 safety, 3 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartCap`, `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | | `TraceCohorts.tla`, `TraceNested.tla` | Policy and ownership projections of real cohort/nested Rust executions. | | `trace_check.py`, `negative_checks.py` | Replay recorded traces; deliberately wrong selections, premature resume/join and broken propagation must be rejected or produce counterexamples. | | `SupervisionInd.tla`, `CohortsInd.tla`, `NestedSupervisionInd.tla`, `CoordinatorInd.tla`, `MC*Ind.tla`, `apalache.sh` | Inductive invariants (`IndInv`) checked with Apalache; see [Inductive invariants](#inductive-invariants-apalache). | @@ -607,63 +607,87 @@ All 22 named controls find their counterexample, each in a few seconds. ## Unified start (0.9 design) -`Start.tla` backs the [unified start design](../../docs/design/unified-start.md), -written before the implementation, as the deferred-replies models were. The -design's section 15 lists what it covers and what it leaves out. - -**What is modelled.** - -- One parent: managed in group `G` or unmanaged, trapping or not, and able - to exit. -- Children started with `spawn`, `start`, `start_link` or `start_monitor`, - with an explicit or default placement and possibly one shared name. -- Stage 1 as four steps (admit, link/monitor, register, dispatch). Parent - death, group closure and a competing registration can interleave between - any two of them. -- `started()` as one step with every `Init` outcome. Signals are held until - it returns. -- Messages from others that a continuation must precede. -- A waiting starter that acknowledges, fails, times out or abandons the wait. +`Start.tla` backs the [unified start design](../../docs/design/unified-start.md) +(revision 2). It was written before the implementation, as the +deferred-replies models were. Design section 15 lists what it covers and +what it leaves out. + +**What is modelled:** + +- one parent, managed in group `G` or unmanaged, trapping or not, able to + exit; +- children started with `spawn`, `start`, `start_link` or `start_monitor`; +- stage 1 as a preparation (which may fail) and one atomic `Commit`, with a + `Lookup` observer; +- `started()` returning every `Init` outcome, with a trap flag, and held + parent exits and kills resolved by the design's precedence table; +- the startup cell: a commitment (`ready`, `failed`, `ignored`, + `timedout`) and a waiter (`waiting`, `consumed`, `abandoned`); +- death as `Die` (D) and `Publish`, with the name and the slot released at + publication and notices disposed of through the cell; +- after readiness: continuations, messages from others, a trapping child's + queued parent exit, unlinking; +- S8: owned children, the scope deadline, the generation end. + +Notices are counted per child, with their reason. **Safety invariants:** -- `InheritedPlacement`, `LinkBeforeStarted`; -- `NameUnique`, `NameReleasedAtDeath`; -- `AbortInvisible`, `AckMeansStarted`, `FailureAfterDeath`, `FailureFlushed`; -- `TimeoutContained`, `NoEdgeToDeadParent`; -- `AdmittedOnlyOpen`, `CapacityRespected`, `NoticeOnce`; -- `ContinueFirst`, `AbandonedNoticeDelivered`. - -Liveness: `ParentExitReachesChild` (`MCStartLive`). +- placement and edges: `InheritedPlacement`, `LinkBeforeStarted`, + `NoEdgeToDeadParent`; +- names: `NameUnique`, `NameHeldUntilPublished`, `NoOverlapOnReuse`, + `NameFreeBeforeObservation`; +- the commit: `AbortInvisible`; +- the startup cell: `ResultMatchesCommit`, `ReadyIsFinal`, + `TimeoutNeverKillsReady`, `ReadyOnlyWhileOpen`, `FailureAfterPublication`; +- notices: `FailureFlushed`, `TimeoutContained`, `NoticeOnce`, + `AbandonedNoticeDelivered`; +- admission: `AdmittedOnlyOpen`, `CapacityRespected`; +- `ContinueFirst`, `GenDoneMeansOwnedDone`. + +**Liveness:** `ParentExitReachesChild` and `StartAnswered` (`MCStartLive`), +and `OwnedScopeFinishes` (`MCStartScope`). | Configuration | Run | Instance | |---|---|---| -| `MCStart` | per PR | two children, every waiting variant, default placements, one name, timeouts, a closing group with room for two | -| `MCStartContinue` | per PR | one child of every variant, two messages from others, no name | -| `MCStartLive` | per PR | two linked or monitored children of a managed parent, liveness | -| `MCStartDeep` | weekly | as `MCStart`, plus every explicit placement and `spawn` (57M states, about 6 minutes on 4 workers) | +| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3, closure, abandonment, retry | +| `MCStartCap` | per PR | three linked children of a managed parent in a group of three: capacity binds, is held until publication, and a retry gets the slot | +| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | +| `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | +| `MCStartScope` | per PR | a managed parent and one linked child (owned or not), trapping, unlink, the scope deadline; liveness | +| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness | +| `MCStartDeep` | weekly | two named children, every variant and explicit placement, trapping, unlink, the scope, the timer | **Negative controls** (`START_CONTROLS` and `START_TEMPORAL_CONTROLS` in `negative_checks.py`; each checks `TypeOK` and only the named property): -| Fault | Breaks | -|---|---| -| `NoInherit` (the 0.8 `start_linked` placement) | `InheritedPlacement` | -| `LinkAfterStarted` | `LinkBeforeStarted` | -| `DoubleRegister` | `NameUnique` | -| `LateUnregister` | `NameReleasedAtDeath` | -| `AbortLeaksLink` | `AbortInvisible` | -| `EarlyFailAck` | `FailureAfterDeath` | -| `AckBeforeStarted` | `AckMeansStarted` | -| `NoMonitorFlush`, `NoLinkFlush` | `FailureFlushed` | -| `KillBeforeUnlink` | `TimeoutContained` | -| `LinkCheckLate` | `NoEdgeToDeadParent` | -| `AdmitClosed` | `AdmittedOnlyOpen` | -| `IgnoreCap` | `CapacityRespected` | -| `AbandonDropsNotice` | `AbandonedNoticeDelivered` | -| `ContinueQueued` (`MCStartContinue`) | `ContinueFirst` | -| `DropHeld` (`MCStartLive`, liveness) | `ParentExitReachesChild` | - -The trace schema `start-v1` and `TraceStart.tla` (real start, link, monitor, -name and placement events replayed against this model) belong to the -implementation PR, as the verification rule requires. +| Configuration | Fault | Breaks | +|---|---|---| +| `MCStart` | `NoInherit` (the 0.8 `start_linked` placement) | `InheritedPlacement` | +| `MCStart` | `LinkAfterStarted` | `LinkBeforeStarted` | +| `MCStart` | `DoubleRegister` | `NameUnique` | +| `MCStart` | `LateUnregister` | `NameFreeBeforeObservation` | +| `MCStart` | `UnregisterAtDeath` | `NoOverlapOnReuse` | +| `MCStart` | `SplitCommit` | `AbortInvisible` | +| `MCStart` | `EarlyFailAck` | `FailureAfterPublication` | +| `MCStart` | `AckBeforeStarted`, `ErrorAsOk` | `ResultMatchesCommit` | +| `MCStart` | `DivertAfterReady` | `ReadyIsFinal` | +| `MCStart` | `ReadyAfterClose` | `ReadyOnlyWhileOpen` | +| `MCStart` | `NoMonitorFlush`, `NoLinkFlush` | `FailureFlushed` | +| `MCStart` | `LinkCheckLate` | `NoEdgeToDeadParent` | +| `MCStart` | `AbandonDropsNotice` | `AbandonedNoticeDelivered` | +| `MCStart` | `DoubleNotice` | `NoticeOnce` | +| `MCStartCap` | `AdmitClosed` | `AdmittedOnlyOpen` | +| `MCStartCap` | `IgnoreCap` | `CapacityRespected` | +| `MCStartTimeout` | `TimerIgnoresCommit` | `TimeoutNeverKillsReady` | +| `MCStartTimeout` | `KillBeforeUnlink` | `TimeoutContained` | +| `MCStartContinue` | `ContinueQueued` | `ContinueFirst` | +| `MCStartScope` | `EarlyGenDone` | `GenDoneMeansOwnedDone` | +| `MCStartLive` (liveness) | `DropHeld` | `ParentExitReachesChild` | +| `MCStartLive` (liveness) | `NoAnswerIgnore` | `StartAnswered` | +| `MCStartScope` (liveness) | `NoScopeKill` | `OwnedScopeFinishes` | + +The trace schema `start-v1` and `TraceStart.tla` are specified in design +section 16.2: events, model actions and code linearization points. They +belong to the implementation PR, and until then the model is allowlisted as +TLC-only in `verification/model-map.toml`. From f3d9e59d9f173d10e80f4c909bef0bbbdc48bb59 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:23:20 -0300 Subject: [PATCH 19/42] verification: MCStartDeep is MCStart plus the timer and an unmanaged parent --- verification/tla/MCStartDeep.cfg | 10 +++++----- verification/tla/MCStartDeep.tla | 7 +++++-- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index 5b074282..60ab98c6 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -1,8 +1,8 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2"} - Kinds = {"spawn", "start", "link", "monitor"} - Explicit = {"none", "G", "U", "same"} + Kinds = {"link", "monitor"} + Explicit = {"none"} ParentPlaces = {"G", "U"} Cap = 3 Named = TRUE @@ -10,9 +10,9 @@ CONSTANTS ParentCanExit = TRUE CloseEnabled = TRUE MaxMsgs = 0 - ChildCanTrap = TRUE - UnlinkEnabled = TRUE - ScopeEnabled = TRUE + ChildCanTrap = FALSE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE Fault = "None" INVARIANT TypeOK INVARIANT InheritedPlacement diff --git a/verification/tla/MCStartDeep.tla b/verification/tla/MCStartDeep.tla index bdb15cae..d0dd4b3c 100644 --- a/verification/tla/MCStartDeep.tla +++ b/verification/tla/MCStartDeep.tla @@ -1,5 +1,8 @@ ---------------------------- MODULE MCStartDeep ---------------------------- EXTENDS Start -\* Weekly safety instance: two children of every variant with every -\* explicit placement, trapping, unlinking and the S8 scope. +\* Weekly safety instance: MCStart plus the init timer and an unmanaged +\* parent, so the timer races two named children's readiness, failure, +\* abandonment and publication. (Every variant with every option at two +\* children is beyond TLC; MCStartContinue, MCStartTimeout and MCStartScope +\* cover those dimensions one child at a time.) ============================================================================= From 630877e770f22df1381e79e4c3d5241e1ac3bcf4 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:23:52 -0300 Subject: [PATCH 20/42] docs: revision-2 model results, controls; ROADMAP rank 5h --- ROADMAP.md | 2 +- docs/design/unified-start.md | 51 +++++++++++++++++++++++++++++------- verification/tla/README.md | 2 +- 3 files changed, 44 insertions(+), 11 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 18e58a15..99306ab9 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -703,7 +703,7 @@ startup: OTP aborts `start_link` on any child start error, while Spawned retries a Permanent or Transient child's failed factory, panic or startup timeout during the first start (semantics §11; found reviewing #233). Behaviour change: extend the `Supervision`/`Cohorts` TLA+ models and their Apalache inductive checks (`BudgetBounded`) to the new rule, with a negative control for counting per child, and update the Kani restart-window proofs and the guide. | | 5g | One exit reason when the runtime drops an actor (v0.8) | Tasks mode reports `Kill` when the runtime drops an actor; threads mode falls back to `Panic("unexpected framework panic")` for an actor that never finishes cleanup. Pick one documented reason for both modes and test it in both. | -| 5h | Unified start API (v0.9, design in review, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (16 negative controls) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()`; `spawn` does not. They share one `StartOptions` (placement, typed name, init timeout, label, stack size, per-instance bounds; slots reserved for ranks 17 and 18a). `started()` may fail (`Init`: ready, continue, error, stop, ignore), and `Init::continue_with` runs before any message (`handle_continue`). Linked and monitored children inherit the parent's group or backend, which fixes API-review footgun 5.11. Replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and `start_linked`. Client impact: 51 mechanical sites, no behaviour change. | +| 5h | Unified start API (v0.9, design in review, revision 2, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (7 configurations, 25 negative controls) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()` and answer `Started`, `Ignored` or an error after the child's publication; `spawn` does not wait. One typed `StartOptions`. `started()` may fail, ignore or continue (`Init`; the continuation precedes messages but is no termination barrier). A per-start cell commits readiness once and independently of the waiter, and the init timer and abandonment arbitrate through it. Stage 1 is one atomic commit. Linked and monitored children inherit the parent's group or backend (API-review footgun 5.11), and linked children in the parent's system are owned by its scope. Supervisors keep the whole-attempt `startup_timeout`, and `ignore` keeps the spec as OTP. Shared result types with #245 and #256. Client impact: 51 mechanical sites, no behaviour change. | | 6 | Test kit with `ActorRef::mock()` | **Phase A implemented** in [`spawned-test`](#testing-crate-spawned-test) (unreleased): closure-backed `Mock` actors that implement `Handler` for every message, so `ActorRef` satisfies any `#[protocol]` and replaces hand-stubbed protocol refs with no production change; recording `Probe` recipients (`expect`, `expect_no_message`); bounded harness waits; fault injection (`faults::exit`, `restart_child`, mock delay/discard/crash). Unmocked messages fail loudly. **Remaining:** mocking a field typed as a concrete `ActorRef` needs an `Envelope` hook that downcasts to the concrete message and yields message and reply sender (a core change, deferred); controllable time for timers; a generic idle check; core `ChildHandle::exit(reason)` would make injected exits exact (today `Panic` and `Kill` go through a linked helper, so a panic surfaces as `Panic("panic in handler: ...")`). **Client evidence:** hand-stubbed protocols in ethlambda tests (`RecordingChain` in `column_checks.rs`, also `envelope_checks.rs`, `verdict.rs`). **Research ([actor testing](docs/research/actor-testing.md)):** three gaps against Akka/Erlang kits. (1) A synchronous behavior test kit (Akka `BehaviorTestKit`): run one handler with a test `Context` and record its effects (sends, spawns, timers, stop) without a runtime; needs a core change. (2) Manual time (Akka `ManualTime`): document Tokio's `start_paused` for async actors; thread actors need the framework-owned clock, the same seam as the [simulation backend spike](#simulation-backend-spike) and `ctx.now()`, so build one clock, not two. (3) Concrete `ActorRef` mocking stays deferred (actix's `Mocker` is the precedent); until then users hold protocol refs or `Recipient`s. Also: the idle check should report total deadlock Go-`checkdead` style (every actor idle, a request outstanding, no timer pending), and a replay-equivalence check for event-sourced actors ([event sourcing](docs/research/event-sourcing.md)). | | 6b | Shared-state and abort lints in `#[actor]` | Zero runtime cost. The macro warns when an actor's fields hold shared mutable state (`Arc>`, `Arc>`, atomics behind `Arc`, `static mut`), with an opt-out attribute for deliberate cases, and a build script check warns when the profile uses `panic = "abort"`, under which a handler panic kills the process instead of one actor. Syntactic only (it cannot see sharing behind a type alias), but it catches the common pattern, including the one LLM-written code reaches for first. Erlang gap: memory isolation. | | 7 | Request deadlock detection | **Client evidence:** requests awaited inside handlers exist (ethrex peer table), no cycle found; keep as a cheap safety net. `ActorError::WouldDeadlock` naming the cycle instead of a timeout. `ActorRef::request` takes only `&self`, so set a thread-local `CURRENT_ACTOR` only while the actor's own handler future is polled (and around the synchronous handler in thread mode); detached helpers create no edges. Keep a small wait set per actor for `join!`. Measure cost before enabling outside debug builds. **Approach chosen ([deadlock and stall detection](docs/research/deadlock-and-stall-detection.md)):** eager detection. Add a wait-for edge when a handler sends a request, walk the per-actor wait sets for a cycle, fail the request that would close it with `WouldDeadlock` naming the cycle, remove the edge on reply or timeout; a request to self fails at once (Erlang's `calling_self`). Waits on raw channels, locks or `spawn_blocking` create no edges, so it is a safety net, not a proof. Edges are exported to rank 30. | diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index e8c78894..8c26217c 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -10,8 +10,8 @@ the listed changes"). What changed and why is in [Review resolutions](#20-review-resolutions). - **Model:** [`verification/tla/Start.tla`](../../verification/tla/Start.tla), - with configurations `MCStart`, `MCStartTimeout`, `MCStartContinue`, - `MCStartScope`, `MCStartLive` and `MCStartDeep`. + with configurations `MCStart`, `MCStartCap`, `MCStartTimeout`, + `MCStartContinue`, `MCStartScope`, `MCStartLive` and `MCStartDeep`. - **Client mapping:** [unified-start-client-mapping.md](unified-start-client-mapping.md). - **Agreed shapes with other designs:** - the dynamic supervisor (#245, revision 3) and introspection v1 (#256); @@ -889,17 +889,50 @@ run one at a time): | Configuration | Run | Instance | Distinct states | Depth | Time | |---|---|---|---|---|---| -| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3 (two children and P at once), closure, abandonment, retry | RESULT_MCStart | -| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | RESULT_MCStartTimeout | -| `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | RESULT_MCStartContinue | -| `MCStartScope` | per PR | managed parent, two linked children (owned or not), trapping, unlink, scope deadline; liveness `OwnedScopeFinishes` | RESULT_MCStartScope | -| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | RESULT_MCStartLive | -| `MCStartDeep` | weekly | two named children, every variant and explicit placement, trapping, unlink, scope, timer | RESULT_MCStartDeep | +| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3 (two children and P at once), closure, abandonment, retry | 10,115,028 | 22 | 80 s | +| `MCStartCap` | per PR | three children spawned into P's group of three: capacity binds, is held until publication, a retry after a published failure gets the slot | 102,526 | 23 | 4 s | +| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | 23,664 | 12 | 2 s | +| `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | 546,846 | 14 | 6 s | +| `MCStartScope` | per PR | managed parent, one linked child (owned or not), trapping, unlink, scope deadline; liveness `OwnedScopeFinishes` | 13,848 | 14 | 2 s | +| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | 8,387 | 11 | 2 s | +| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | RESULT_DEEP | **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): -CONTROLS_TABLE +Every control produced its counterexample in 1-5 s: + +| Configuration | Fault | Reintroduces | Fails | +|---|---|---|---| +| `MCStart` | `NoInherit` | 0.8 `start_linked` placement | `InheritedPlacement` | +| `MCStart` | `LinkAfterStarted` | the edge set once `started()` runs | `LinkBeforeStarted` | +| `MCStart` | `DoubleRegister` | check-then-insert names | `NameUnique` | +| `MCStart` | `LateUnregister` | the name released after the notices | `NameFreeBeforeObservation` | +| `MCStart` | `UnregisterAtDeath` | revision 1: the name released at D, before destruction | `NoOverlapOnReuse` | +| `MCStart` | `SplitCommit` | revision 1: the name published before the rest of stage 1 | `AbortInvisible` | +| `MCStart` | `EarlyFailAck` | revision 1: the error answered at D | `FailureAfterPublication` | +| `MCStart` | `AckBeforeStarted` | `Ok` before the commit | `ResultMatchesCommit` | +| `MCStart` | `ErrorAsOk` | `ignore` answered as started | `ResultMatchesCommit` | +| `MCStart` | `DivertAfterReady` | revision 1 (P0-1): readiness keeps the start marks | `ReadyIsFinal` | +| `MCStart` | `ReadyAfterClose` | readiness committed after the group closed | `ReadyOnlyWhileOpen` | +| `MCStart` | `NoMonitorFlush` | a failed `start_monitor` also delivers `Down` | `FailureFlushed` | +| `MCStart` | `NoLinkFlush` | a failed `start_link` also delivers `Exit` | `FailureFlushed` | +| `MCStart` | `LinkCheckLate` | the commit ignores the parent's state | `NoEdgeToDeadParent` | +| `MCStart` | `AbandonDropsNotice` | an abandoned wait loses the diverted notice | `AbandonedNoticeDelivered` | +| `MCStart` | `DoubleNotice` | a notice both delivered and diverted | `NoticeOnce` | +| `MCStartCap` | `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | +| `MCStartCap` | `IgnoreCap` | a full group admits | `CapacityRespected` | +| `MCStartTimeout` | `TimerIgnoresCommit` | revision 1 (P0-2): the timer kills without checking the cell | `TimeoutNeverKillsReady` | +| `MCStartTimeout` | `KillBeforeUnlink` | the timeout's `Killed` reaches a waiting parent | `TimeoutContained` | +| `MCStartContinue` | `ContinueQueued` | the continuation queued behind earlier entries | `ContinueFirst` | +| `MCStartScope` | `EarlyGenDone` | P's generation ends before its owned children | `GenDoneMeansOwnedDone` | +| `MCStartLive` | `DropHeld` (liveness) | a held parent exit dropped | `ParentExitReachesChild` | +| `MCStartLive` | `NoAnswerIgnore` (liveness) | an ignored start never answered | `StartAnswered` | +| `MCStartScope` | `NoScopeKill` (liveness) | the scope deadline spares unlinked children | `OwnedScopeFinishes` | + +A first `IgnoreCap` run on `MCStart` found no counterexample: two children +and P never exceed Cap 3, which agy P1-8 had pointed out. That led to +`MCStartCap`, where capacity binds. **Not modelled:** diff --git a/verification/tla/README.md b/verification/tla/README.md index cf8dc73f..30a98047 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -656,7 +656,7 @@ and `OwnedScopeFinishes` (`MCStartScope`). | `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | | `MCStartScope` | per PR | a managed parent and one linked child (owned or not), trapping, unlink, the scope deadline; liveness | | `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness | -| `MCStartDeep` | weekly | two named children, every variant and explicit placement, trapping, unlink, the scope, the timer | +| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | **Negative controls** (`START_CONTROLS` and `START_TEMPORAL_CONTROLS` in `negative_checks.py`; each checks `TypeOK` and only the named property): From 9938860a00c09b5a76ae2e3c566d1408e6bc1317 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:31:34 -0300 Subject: [PATCH 21/42] docs: MCStartDeep result (revision 2) --- docs/design/unified-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 8c26217c..409eb6c0 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -895,7 +895,7 @@ run one at a time): | `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | 546,846 | 14 | 6 s | | `MCStartScope` | per PR | managed parent, one linked child (owned or not), trapping, unlink, scope deadline; liveness `OwnedScopeFinishes` | 13,848 | 14 | 2 s | | `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | 8,387 | 11 | 2 s | -| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | RESULT_DEEP | +| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | 55,814,368 | 22 | 7 min 17 s | **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): From 1051e69fa89b27dc83af8e98af05a9f4f0396e5a Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:52:16 -0300 Subject: [PATCH 22/42] verification: Start.tla revision 3 (parent publication, pre-D reason, polite scope stop, owner gate, commit history, notice obligations) --- verification/tla/MCStart.cfg | 26 +- verification/tla/MCStartCap.cfg | 28 +- verification/tla/MCStartCap.tla | 9 +- verification/tla/MCStartContinue.cfg | 28 +- verification/tla/MCStartDeep.cfg | 26 +- verification/tla/MCStartLive.cfg | 26 +- verification/tla/MCStartScope.cfg | 28 +- verification/tla/MCStartScope.tla | 9 +- verification/tla/MCStartTimeout.cfg | 28 +- verification/tla/MCStartTimeout.tla | 8 +- verification/tla/Start.tla | 792 +++++++++++++++------------ 11 files changed, 495 insertions(+), 513 deletions(-) diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 99a657bc..cea56d72 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -7,32 +7,14 @@ CONSTANTS Cap = 3 Named = TRUE TimeoutEnabled = FALSE + KillEnabled = FALSE + AdminStopEnabled = FALSE ParentCanExit = TRUE CloseEnabled = TRUE + OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg index 8d3b1de8..a23041b4 100644 --- a/verification/tla/MCStartCap.cfg +++ b/verification/tla/MCStartCap.cfg @@ -7,32 +7,14 @@ CONSTANTS Cap = 3 Named = FALSE TimeoutEnabled = FALSE - ParentCanExit = FALSE + KillEnabled = FALSE + AdminStopEnabled = FALSE + ParentCanExit = TRUE CloseEnabled = TRUE + OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartCap.tla b/verification/tla/MCStartCap.tla index ae83b913..b471feae 100644 --- a/verification/tla/MCStartCap.tla +++ b/verification/tla/MCStartCap.tla @@ -1,8 +1,7 @@ ---------------------------- MODULE MCStartCap ---------------------------- EXTENDS Start -\* Per-PR instance for group capacity: a managed parent and three children -\* spawned into its group of three (capacity does not depend on the start -\* variant, so no edges or waiters): two children run beside the parent, a -\* third is refused while the slots are held (until publication), and gets -\* a slot after a failure is published. AdmitClosed and IgnoreCap run here. +\* Per-PR instance for capacity: a managed parent and three children +\* spawned into its group of three: slots (P's included) are held until +\* release (P's until its publication), a start is refused while they are +\* held, and a retry gets a slot after a release. ============================================================================= diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg index 0d2a2bb0..9ad62c03 100644 --- a/verification/tla/MCStartContinue.cfg +++ b/verification/tla/MCStartContinue.cfg @@ -2,37 +2,19 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1"} Kinds = {"spawn", "start", "link", "monitor"} - Explicit = {"none", "G", "U", "same"} + Explicit = {"none", "G", "G2", "U", "same"} ParentPlaces = {"G", "U"} Cap = 2 Named = FALSE TimeoutEnabled = TRUE + KillEnabled = FALSE + AdminStopEnabled = FALSE ParentCanExit = TRUE CloseEnabled = TRUE + OwnerCloseEnabled = FALSE MaxMsgs = 2 ChildCanTrap = TRUE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index 60ab98c6..98815466 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -7,32 +7,14 @@ CONSTANTS Cap = 3 Named = TRUE TimeoutEnabled = TRUE + KillEnabled = FALSE + AdminStopEnabled = FALSE ParentCanExit = TRUE CloseEnabled = TRUE + OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index 6bda12ee..e73274ec 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -7,34 +7,16 @@ CONSTANTS Cap = 2 Named = FALSE TimeoutEnabled = TRUE + KillEnabled = FALSE + AdminStopEnabled = FALSE ParentCanExit = TRUE CloseEnabled = FALSE + OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = TRUE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned PROPERTY ParentExitReachesChild PROPERTY StartAnswered diff --git a/verification/tla/MCStartScope.cfg b/verification/tla/MCStartScope.cfg index c3293990..bfbb9ea9 100644 --- a/verification/tla/MCStartScope.cfg +++ b/verification/tla/MCStartScope.cfg @@ -2,38 +2,20 @@ SPECIFICATION LiveSpec CONSTANTS Kids = {"c1"} Kinds = {"link"} - Explicit = {"none", "U"} + Explicit = {"none", "G2", "U"} ParentPlaces = {"G"} Cap = 3 Named = FALSE TimeoutEnabled = FALSE + KillEnabled = FALSE + AdminStopEnabled = FALSE ParentCanExit = TRUE CloseEnabled = FALSE + OwnerCloseEnabled = TRUE MaxMsgs = 0 ChildCanTrap = TRUE UnlinkEnabled = TRUE ScopeEnabled = TRUE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned PROPERTY OwnedScopeFinishes diff --git a/verification/tla/MCStartScope.tla b/verification/tla/MCStartScope.tla index a11054ac..222bffc0 100644 --- a/verification/tla/MCStartScope.tla +++ b/verification/tla/MCStartScope.tla @@ -1,7 +1,8 @@ ---------------------------- MODULE MCStartScope ---------------------------- EXTENDS Start -\* Per-PR instance for S8 (linked children owned by the parent's scope): -\* a managed parent and one linked child, in its group (owned) or unmanaged -\* (not owned), which may trap or unlink; the scope deadline and the generation end. Safety and -\* OwnedScopeFinishes. +\* Per-PR instance for S8: a managed parent and one linked child, in its +\* group, in another group of its system (both owned) or unmanaged (not +\* owned), which may trap or unlink; P's scope may close; the polite +\* Shutdown at P's publication, the scope deadline, the generation end. +\* Safety and OwnedScopeFinishes. ============================================================================= diff --git a/verification/tla/MCStartTimeout.cfg b/verification/tla/MCStartTimeout.cfg index cc0c7308..a6db364d 100644 --- a/verification/tla/MCStartTimeout.cfg +++ b/verification/tla/MCStartTimeout.cfg @@ -4,35 +4,17 @@ CONSTANTS Kinds = {"start", "link", "monitor"} Explicit = {"none"} ParentPlaces = {"G", "U"} - Cap = 3 + Cap = 2 Named = FALSE TimeoutEnabled = TRUE + KillEnabled = TRUE + AdminStopEnabled = TRUE ParentCanExit = TRUE CloseEnabled = TRUE + OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK -INVARIANT InheritedPlacement -INVARIANT LinkBeforeStarted -INVARIANT NameUnique -INVARIANT NameHeldUntilPublished -INVARIANT NoOverlapOnReuse -INVARIANT NameFreeBeforeObservation -INVARIANT AbortInvisible -INVARIANT ResultMatchesCommit -INVARIANT ReadyIsFinal -INVARIANT TimeoutNeverKillsReady -INVARIANT ReadyOnlyWhileOpen -INVARIANT FailureAfterPublication -INVARIANT FailureFlushed -INVARIANT TimeoutContained -INVARIANT NoEdgeToDeadParent -INVARIANT AdmittedOnlyOpen -INVARIANT CapacityRespected -INVARIANT NoticeOnce -INVARIANT AbandonedNoticeDelivered -INVARIANT ContinueFirst -INVARIANT GenDoneMeansOwnedDone +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartTimeout.tla b/verification/tla/MCStartTimeout.tla index 19b741f5..5bdfdb1d 100644 --- a/verification/tla/MCStartTimeout.tla +++ b/verification/tla/MCStartTimeout.tla @@ -1,6 +1,8 @@ ---------------------------- MODULE MCStartTimeout ---------------------------- EXTENDS Start -\* Per-PR instance for the startup cell's races: one child of every waiting -\* variant of a managed or unmanaged parent; the init timer racing readiness, -\* failure, abandonment and publication; group closure. +\* Per-PR instance for the startup commitment's races: one child of every +\* waiting variant of a managed or unmanaged parent; the init timer, an +\* outside kill (also after started() resolved, before D) and an +\* administrative stop during started(), racing readiness, failure, +\* abandonment, the cleanup deadline and publication; group closure. ============================================================================= diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 1eb3e2af..b0b84ee2 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -1,161 +1,169 @@ ------------------------------- MODULE Start ------------------------------- EXTENDS Naturals, FiniteSets (***************************************************************************) -(* Unified start API, revision 2 (0.9; design docs/design/unified-start.md).*) +(* Unified start API, revision 3 (0.9; design docs/design/unified-start.md).*) (* *) -(* One parent actor P (managed in group G or unmanaged; it may trap exits *) -(* and exit) starts children with spawn, start, start_link or *) -(* start_monitor, each with an explicit or default placement and possibly *) -(* the one shared name. *) +(* One parent actor P (managed in group G or unmanaged; it may trap exits) *) +(* starts children with spawn, start, start_link or start_monitor, each *) +(* with a default or explicit placement (G, G2 = another group of P's *) +(* system, U = unmanaged, same = placement_of) and possibly the one shared *) +(* name. *) (* *) -(* Stage 1 is one commit (design section 3): placement admission, the *) -(* parent's state, the link or monitor edge and the name are decided under *) -(* the commit locks together, so a refused start is never observable. A *) -(* dispatch failure happens before the commit (PrepareFail). *) +(* Stage 1 is one Commit under the design's lock order: P's scope gate (for*) +(* owned children), the target group, P's exit state, P's edge table, the *) +(* registry. A refused start is never observable. *) (* *) -(* Every waiting start owns a startup cell (design section 5): one *) -(* terminal commitment (ready, failed, ignored, timedout) made exactly *) -(* once, by the child at the end of started() or by the init timer, and a *) -(* waiter state (waiting, abandoned, consumed). Readiness clears the start *) -(* marks on the edges, so a later death is an ordinary death. A failed *) -(* start's notice is diverted to the waiter at publication and consumed by *) -(* the error, or handed to P if the wait was abandoned. *) +(* Every start has an immutable startup commitment (ready, failed, *) +(* ignored, timedout; commit0 is its first value and must never change). *) +(* Waiting starts also have a waiter (waiting, consumed, abandoned). *) (* *) -(* Death is three steps, as in otp-exit-signals: D (Die: edges closed and *) -(* their notices taken), destruction, publication (Publish: name and slot *) -(* released, then notices, then the waiter). Destruction is folded into *) -(* the gap between Die and Publish; a waiter may give up at its bounded *) -(* cleanup deadline (pend = TRUE) before publication. *) +(* Death is D (Die: edges close, notices taken), destruction, then *) +(* Release (name and slot) and Publish (notices disposed of through the *) +(* commitment and the waiter; answers become final). The exit reason can *) +(* still change before D (a kill), and errors report the final reason. A *) +(* waiter may answer before publication only after its cleanup deadline *) +(* (dl), with pend = TRUE. *) (* *) -(* A linked child placed in P's group is owned by P's scope (S8): P's *) -(* generation is done only when the owned children are published; the *) -(* scope deadline kills owned children that outlive P's exit (unlinked or *) -(* not). *) +(* P's own death is D (PDie: edges close, its link notices taken) then *) +(* publication (PPublish: notices and the scope's polite Shutdown reach *) +(* the children; P's slot is released). P's generation is done only after *) +(* P's publication and every owned child's. *) (***************************************************************************) CONSTANTS - Kids, \* child names - Kinds, \* subset of {"spawn","start","link","monitor"} - Explicit, \* subset of {"none","G","U","same"} ("same" = placement_of) - ParentPlaces, \* subset of {"G","U"} - Cap, \* live actors G admits (P included); held until publication - Named, \* children may request the one shared name - TimeoutEnabled, \* the init timer may fire - ParentCanExit, \* P may exit (Normal or Panic) - CloseEnabled, \* G may close (system shutdown) - MaxMsgs, \* messages others send to each child - ChildCanTrap, \* started() may set trap_exit - UnlinkEnabled, \* a ready linked child may unlink from P - ScopeEnabled, \* model P's generation end and its scope deadline (S8) - Fault \* "None" or a negative control (README, Unified start) + Kids, \* child names + Kinds, \* subset of {"spawn","start","link","monitor"} + Explicit, \* subset of {"none","G","G2","U","same"} + ParentPlaces, \* subset of {"G","U"} + Cap, \* live actors G admits (P included); held until release + Named, \* children may request the one shared name + TimeoutEnabled, \* the init timer may fire + KillEnabled, \* an outside kill (ChildHandle::kill) may hit a child + AdminStopEnabled, \* an administrative stop may hit a child in started() + ParentCanExit, \* P may exit (Normal or Panic) + CloseEnabled, \* G may close (system shutdown) + OwnerCloseEnabled,\* P's scope may close (its supervisor's shutdown began) + MaxMsgs, \* messages others send to each child + ChildCanTrap, \* started() may set trap_exit + UnlinkEnabled, \* a ready linked child may unlink from P + ScopeEnabled, \* model P's generation end and its scope deadline (S8) + Fault \* "None" or a negative control (README, Unified start) None == "none" NonNormal == {"Panic", "Killed", "Shutdown"} Reasons == {"Normal"} \cup NonNormal Phases == {"idle", "begun", "reg1", "starting", "ready", "exiting", "dead", - "published", "aborted"} + "released", "published", "aborted"} Commits == {None, "ready", "failed", "ignored", "timedout"} +FailCommits == {"failed", "ignored", "timedout"} Waits == {"none", "waiting", "abandoned", "consumed"} Results == {"none", "pending", "ok", "failed", "ignored", "timeout", "closed", "capacity", "already", "parentexited", "system"} -Outcomes == {"ready", "continue", "panic", "stop", "error", "ignore"} +Outcomes == {"ready", "continue", "panic", "stopN", "stopS", "error", "ignore"} +Alive(p) == p \in {"run", "exiting"} \* P's mailbox still admits VARIABLES - pst, pplace, ptrap, pwhy, pcause, gopen, gdone, + pst, pplace, ptrap, pwhy, pcause, gopen, ownerOpen, gdone, sfired, pn, kind, expl, wname, - phase, place, admOpen, owned, readyOpen, + phase, place, admOpen, owned, ownOpenAt, readyOpen, edgeWhenRun, edge, mon, linkEver, monEver, - began, killed, skill, pex, qpex, pexAbn, ctrap, - commit, why, wait, res, pend, - lnote, mnote, div, divEver, + began, killed, skill, xkill, astop, pex, qpex, pexAbn, ctrap, + commit, commit0, initR, why, wait, res, resR, pend, dl, + lnote, mnote, pubAlive, div, divEver, handed, lost, cont, msgs, early, - reg, seen, + reg, seen, blamed, slots, exitN, downN, exitR, downR -pV == <> +pV == <> oV == <> -sV == <> +sV == <> eV == <> -xV == <> -cV == <> -nV == <> +xV == <> +cV == <> +nV == <> mV == <> -rV == <> +rV == <> dV == <> vars == <> Waiting(k) == k \in {"start", "link", "monitor"} -Live(c) == phase[c] \in {"reg1", "starting", "ready", "exiting", "dead"} -LiveInG == Cardinality({c \in Kids : place[c] = "G" /\ Live(c)}) - + (IF pplace = "G" /\ pst # "dead" THEN 1 ELSE 0) +LiveP == {"starting", "ready", "exiting", "dead"} \* holds name and slot +PSlot == IF pplace = "G" /\ pst # "published" /\ ~(Fault = "ParentSlotAtD" /\ pst = "dead") + THEN 1 ELSE 0 +Used == Cardinality(slots) + PSlot Expected(c) == CASE expl[c] = "same" -> pplace - [] expl[c] \in {"G", "U"} -> expl[c] + [] expl[c] \in {"G", "G2", "U"} -> expl[c] [] kind[c] \in {"link", "monitor"} -> pplace [] OTHER -> "U" Target(c) == IF expl[c] = None /\ kind[c] \in {"link", "monitor"} /\ Fault = "NoInherit" THEN "U" ELSE Expected(c) +IsOwned(c, t) == kind[c] = "link" /\ pplace = "G" /\ t \in {"G", "G2"} TypeOK == - /\ pst \in {"run", "exiting", "dead"} /\ pplace \in {"G", "U"} + /\ pst \in {"run", "exiting", "dead", "published"} /\ pplace \in {"G", "U"} /\ ptrap \in BOOLEAN /\ pwhy \in Reasons \cup {None} - /\ pcause \in Kids \cup {None} /\ gopen \in BOOLEAN /\ gdone \in BOOLEAN + /\ pcause \in Kids \cup {None} /\ gopen \in BOOLEAN /\ ownerOpen \in BOOLEAN + /\ gdone \in BOOLEAN /\ sfired \in BOOLEAN /\ pn \in [Kids -> Reasons \cup {None}] /\ kind \in [Kids -> {"spawn", "start", "link", "monitor", None}] - /\ expl \in [Kids -> {"G", "U", "same", None}] + /\ expl \in [Kids -> {"G", "G2", "U", "same", None}] /\ wname \in [Kids -> BOOLEAN] - /\ phase \in [Kids -> Phases] /\ place \in [Kids -> {"G", "U", None}] + /\ phase \in [Kids -> Phases] /\ place \in [Kids -> {"G", "G2", "U", None}] /\ admOpen \in [Kids -> BOOLEAN] /\ owned \in [Kids -> BOOLEAN] - /\ readyOpen \in [Kids -> BOOLEAN] + /\ ownOpenAt \in [Kids -> BOOLEAN] /\ readyOpen \in [Kids -> BOOLEAN] + /\ edgeWhenRun \in [Kids -> BOOLEAN] /\ edge \in [Kids -> BOOLEAN] /\ mon \in [Kids -> BOOLEAN] /\ linkEver \in [Kids -> BOOLEAN] /\ monEver \in [Kids -> BOOLEAN] /\ began \in [Kids -> BOOLEAN] /\ killed \in [Kids -> BOOLEAN] - /\ skill \in [Kids -> BOOLEAN] + /\ skill \in [Kids -> BOOLEAN] /\ xkill \in [Kids -> BOOLEAN] + /\ astop \in [Kids -> BOOLEAN] /\ pex \in [Kids -> Reasons \cup {None}] /\ qpex \in [Kids -> Reasons \cup {None}] /\ pexAbn \in [Kids -> BOOLEAN] /\ ctrap \in [Kids -> BOOLEAN] - /\ commit \in [Kids -> Commits] /\ why \in [Kids -> Reasons \cup {None}] + /\ commit \in [Kids -> Commits] /\ commit0 \in [Kids -> Commits] + /\ initR \in [Kids -> Reasons \cup {None}] /\ why \in [Kids -> Reasons \cup {None}] /\ wait \in [Kids -> Waits] /\ res \in [Kids -> Results] - /\ pend \in [Kids -> BOOLEAN] + /\ resR \in [Kids -> Reasons \cup {None}] + /\ pend \in [Kids -> BOOLEAN] /\ dl \in [Kids -> BOOLEAN] /\ lnote \in [Kids -> BOOLEAN] /\ mnote \in [Kids -> BOOLEAN] + /\ pubAlive \in [Kids -> BOOLEAN] /\ div \in [Kids -> Reasons \cup {None}] /\ divEver \in [Kids -> BOOLEAN] + /\ handed \in [Kids -> BOOLEAN] /\ lost \in [Kids -> BOOLEAN] /\ cont \in [Kids -> BOOLEAN] /\ msgs \in [Kids -> 0..MaxMsgs] /\ early \in [Kids -> BOOLEAN] /\ reg \subseteq Kids /\ seen \in [Kids -> BOOLEAN] + /\ blamed \subseteq Kids /\ slots \subseteq Kids /\ exitN \in [Kids -> 0..3] /\ downN \in [Kids -> 0..3] /\ exitR \in [Kids -> Reasons \cup {None}] /\ downR \in [Kids -> Reasons \cup {None}] +F(v) == [c \in Kids |-> v] Init == /\ pst = "run" /\ pplace \in ParentPlaces /\ ptrap \in BOOLEAN - /\ pwhy = None /\ pcause = None /\ gopen = TRUE /\ gdone = FALSE - /\ kind = [c \in Kids |-> None] /\ expl = [c \in Kids |-> None] - /\ wname = [c \in Kids |-> FALSE] - /\ phase = [c \in Kids |-> "idle"] /\ place = [c \in Kids |-> None] - /\ admOpen = [c \in Kids |-> FALSE] /\ owned = [c \in Kids |-> FALSE] - /\ readyOpen = [c \in Kids |-> FALSE] - /\ edge = [c \in Kids |-> FALSE] /\ mon = [c \in Kids |-> FALSE] - /\ linkEver = [c \in Kids |-> FALSE] /\ monEver = [c \in Kids |-> FALSE] - /\ began = [c \in Kids |-> FALSE] /\ killed = [c \in Kids |-> FALSE] - /\ skill = [c \in Kids |-> FALSE] - /\ pex = [c \in Kids |-> None] /\ qpex = [c \in Kids |-> None] - /\ pexAbn = [c \in Kids |-> FALSE] /\ ctrap = [c \in Kids |-> FALSE] - /\ commit = [c \in Kids |-> None] /\ why = [c \in Kids |-> None] - /\ wait = [c \in Kids |-> "none"] /\ res = [c \in Kids |-> "none"] - /\ pend = [c \in Kids |-> FALSE] - /\ lnote = [c \in Kids |-> FALSE] /\ mnote = [c \in Kids |-> FALSE] - /\ div = [c \in Kids |-> None] /\ divEver = [c \in Kids |-> FALSE] - /\ cont = [c \in Kids |-> FALSE] /\ msgs = [c \in Kids |-> 0] - /\ early = [c \in Kids |-> FALSE] - /\ reg = {} /\ seen = [c \in Kids |-> FALSE] - /\ exitN = [c \in Kids |-> 0] /\ downN = [c \in Kids |-> 0] - /\ exitR = [c \in Kids |-> None] /\ downR = [c \in Kids |-> None] + /\ pwhy = None /\ pcause = None /\ gopen = TRUE /\ ownerOpen = TRUE + /\ gdone = FALSE /\ sfired = FALSE /\ pn = F(None) + /\ kind = F(None) /\ expl = F(None) /\ wname = F(FALSE) + /\ phase = F("idle") /\ place = F(None) /\ admOpen = F(FALSE) + /\ owned = F(FALSE) /\ ownOpenAt = F(FALSE) /\ readyOpen = F(FALSE) + /\ edgeWhenRun = F(FALSE) + /\ edge = F(FALSE) /\ mon = F(FALSE) /\ linkEver = F(FALSE) /\ monEver = F(FALSE) + /\ began = F(FALSE) /\ killed = F(FALSE) /\ skill = F(FALSE) /\ xkill = F(FALSE) + /\ astop = F(FALSE) /\ pex = F(None) /\ qpex = F(None) /\ pexAbn = F(FALSE) + /\ ctrap = F(FALSE) + /\ commit = F(None) /\ commit0 = F(None) /\ initR = F(None) /\ why = F(None) + /\ wait = F("none") /\ res = F("none") /\ resR = F(None) /\ pend = F(FALSE) + /\ dl = F(FALSE) + /\ lnote = F(FALSE) /\ mnote = F(FALSE) /\ pubAlive = F(FALSE) + /\ div = F(None) /\ divEver = F(FALSE) /\ handed = F(FALSE) /\ lost = F(FALSE) + /\ cont = F(FALSE) /\ msgs = F(0) /\ early = F(FALSE) + /\ reg = {} /\ seen = F(FALSE) /\ blamed = {} /\ slots = {} + /\ exitN = F(0) /\ downN = F(0) /\ exitR = F(None) /\ downR = F(None) ----------------------------------------------------------------------------- -(* Notices reaching P. *) +(* Notices reaching P (only while P's mailbox admits: before P's D). *) -\* A link notice: a trapping P gets an Exit; a non-trapping running P exits -\* on a non-Normal reason. DeliverLink(c, r) == - IF ptrap + IF ~Alive(pst) THEN UNCHANGED <> + ELSE IF ptrap THEN /\ exitN' = [exitN EXCEPT ![c] = @ + 1] /\ exitR' = [exitR EXCEPT ![c] = r] /\ UNCHANGED <> @@ -165,8 +173,14 @@ DeliverLink(c, r) == ELSE UNCHANGED <> DeliverDown(c, r) == - /\ downN' = [downN EXCEPT ![c] = @ + 1] - /\ downR' = [downR EXCEPT ![c] = r] + IF ~Alive(pst) THEN UNCHANGED <> + ELSE /\ downN' = [downN EXCEPT ![c] = @ + 1] + /\ downR' = [downR EXCEPT ![c] = r] + +\* The startup commitment: set once; commit0 keeps the first value. +SetCommit(c, v) == + /\ commit' = [commit EXCEPT ![c] = v] + /\ commit0' = IF commit0[c] = None THEN [commit0 EXCEPT ![c] = v] ELSE commit0 ----------------------------------------------------------------------------- (* Stage 1, in the caller. *) @@ -178,90 +192,91 @@ Begin(c) == /\ expl' = [expl EXCEPT ![c] = IF e = "none" THEN None ELSE e] /\ wname' = [wname EXCEPT ![c] = n] /\ phase' = [phase EXCEPT ![c] = "begun"] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* Preparation (executor resources, state) fails before anything is visible. PrepareFail(c) == /\ phase[c] = "begun" /\ phase' = [phase EXCEPT ![c] = "aborted"] /\ res' = [res EXCEPT ![c] = "system"] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* Why the commit refuses, or None. Order of the checks does not matter: all -\* are read under the commit locks. Refusal(c) == LET t == Target(c) IN - CASE t = "G" /\ ~gopen /\ Fault # "AdmitClosed" -> "closed" - [] t = "G" /\ LiveInG >= Cap /\ Fault # "IgnoreCap" -> "capacity" - [] kind[c] \in {"link", "monitor"} /\ pst # "run" /\ Fault # "LinkCheckLate" + CASE IsOwned(c, t) /\ ~ownerOpen /\ Fault # "OwnerUnchecked" -> "closed" + [] t = "G" /\ ~gopen /\ Fault # "AdmitClosed" -> "closed" + [] t = "G" /\ Used >= Cap /\ Fault # "IgnoreCap" -> "capacity" + [] kind[c] \in {"link", "monitor"} /\ pst # "run" /\ ~(Fault = "LinkCheckLate" /\ pst # "published") -> "parentexited" [] wname[c] /\ reg # {} /\ Fault # "DoubleRegister" -> "already" [] OTHER -> None -\* The stage-1 commit: everything at once, or nothing. +\* Who a refusal was caused by (a competing start's observation). +Blame(r) == CASE r = "capacity" -> slots [] r = "already" -> reg [] OTHER -> {} + +Admit(c, t, lk, mn) == + /\ phase' = [phase EXCEPT ![c] = "starting"] + /\ place' = [place EXCEPT ![c] = t] + /\ admOpen' = [admOpen EXCEPT ![c] = gopen \/ t # "G"] + /\ owned' = [owned EXCEPT ![c] = IsOwned(c, t)] + /\ ownOpenAt' = [ownOpenAt EXCEPT ![c] = ownerOpen] + /\ edgeWhenRun' = [edgeWhenRun EXCEPT ![c] = pst = "run"] + /\ edge' = [edge EXCEPT ![c] = lk] + /\ mon' = [mon EXCEPT ![c] = mn] + /\ linkEver' = [linkEver EXCEPT ![c] = lk] + /\ monEver' = [monEver EXCEPT ![c] = mn] + /\ slots' = IF t = "G" THEN slots \cup {c} ELSE slots + /\ wait' = [wait EXCEPT ![c] = IF Waiting(kind[c]) THEN "waiting" ELSE "none"] + /\ res' = [res EXCEPT ![c] = IF Waiting(kind[c]) THEN "pending" ELSE "ok"] + Commit(c) == /\ phase[c] = "begun" /\ Fault # "SplitCommit" \/ ~wname[c] /\ LET t == Target(c) r == Refusal(c) - lk == kind[c] = "link" /\ Fault # "LinkAfterStarted" - mn == kind[c] = "monitor" IN IF r # None THEN /\ phase' = [phase EXCEPT ![c] = "aborted"] /\ res' = [res EXCEPT ![c] = r] - /\ UNCHANGED <> - ELSE /\ phase' = [phase EXCEPT ![c] = "starting"] - /\ place' = [place EXCEPT ![c] = t] - /\ admOpen' = [admOpen EXCEPT ![c] = gopen] - /\ owned' = [owned EXCEPT ![c] = kind[c] = "link" /\ pplace = "G" /\ t = "G"] - /\ edge' = [edge EXCEPT ![c] = lk] - /\ mon' = [mon EXCEPT ![c] = mn] - /\ linkEver' = [linkEver EXCEPT ![c] = lk] - /\ monEver' = [monEver EXCEPT ![c] = mn] + /\ blamed' = blamed \cup Blame(r) + /\ UNCHANGED <> + ELSE /\ Admit(c, t, kind[c] = "link" /\ Fault # "LinkAfterStarted", + kind[c] = "monitor") /\ reg' = IF wname[c] THEN reg \cup {c} ELSE reg - /\ wait' = [wait EXCEPT ![c] = IF Waiting(kind[c]) THEN "waiting" ELSE "none"] - /\ res' = [res EXCEPT ![c] = IF Waiting(kind[c]) THEN "pending" ELSE "ok"] - /\ UNCHANGED <> + /\ UNCHANGED blamed + /\ UNCHANGED <> -\* SplitCommit (negative control): the name is published before the rest of -\* the commit, which may still refuse. +\* SplitCommit (negative control): the name is published before the rest. RegisterFirst(c) == /\ Fault = "SplitCommit" /\ phase[c] = "begun" /\ wname[c] /\ reg = {} /\ reg' = reg \cup {c} /\ phase' = [phase EXCEPT ![c] = "reg1"] - /\ UNCHANGED <> + /\ UNCHANGED <> CommitRest(c) == /\ phase[c] = "reg1" /\ LET t == Target(c) - r == CASE t = "G" /\ (~gopen \/ LiveInG >= Cap) -> "capacity" + r == CASE t = "G" /\ (~gopen \/ Used >= Cap) -> "capacity" [] kind[c] \in {"link", "monitor"} /\ pst # "run" -> "parentexited" [] OTHER -> None IN IF r # None THEN /\ phase' = [phase EXCEPT ![c] = "aborted"] /\ res' = [res EXCEPT ![c] = r] /\ reg' = reg \ {c} - /\ UNCHANGED <> - ELSE /\ phase' = [phase EXCEPT ![c] = "starting"] - /\ place' = [place EXCEPT ![c] = t] - /\ admOpen' = [admOpen EXCEPT ![c] = gopen] - /\ owned' = [owned EXCEPT ![c] = kind[c] = "link" /\ pplace = "G" /\ t = "G"] - /\ edge' = [edge EXCEPT ![c] = kind[c] = "link"] - /\ mon' = [mon EXCEPT ![c] = kind[c] = "monitor"] - /\ linkEver' = [linkEver EXCEPT ![c] = kind[c] = "link"] - /\ monEver' = [monEver EXCEPT ![c] = kind[c] = "monitor"] - /\ wait' = [wait EXCEPT ![c] = IF Waiting(kind[c]) THEN "waiting" ELSE "none"] - /\ res' = [res EXCEPT ![c] = IF Waiting(kind[c]) THEN "pending" ELSE "ok"] - /\ UNCHANGED reg - /\ UNCHANGED <> - -\* Anyone resolves the name (whereis). + /\ blamed' = blamed \cup Blame(r) + /\ UNCHANGED <> + ELSE /\ Admit(c, t, kind[c] = "link", kind[c] = "monitor") + /\ UNCHANGED <> + /\ UNCHANGED <> + Lookup == - /\ \E c \in reg : - /\ ~seen[c] - /\ seen' = [seen EXCEPT ![c] = TRUE] - /\ UNCHANGED <> + /\ \E c \in reg : ~seen[c] /\ seen' = [seen EXCEPT ![c] = TRUE] + /\ UNCHANGED <> ----------------------------------------------------------------------------- (* Stage 2, on the child. *) @@ -271,12 +286,11 @@ Exit(c, r) == /\ why' = [why EXCEPT ![c] = r] StartedBegin(c) == - /\ phase[c] = "starting" /\ ~began[c] /\ ~killed[c] /\ ~skill[c] + /\ phase[c] = "starting" /\ ~began[c] /\ ~killed[c] /\ ~skill[c] /\ ~xkill[c] /\ began' = [began EXCEPT ![c] = TRUE] - /\ UNCHANGED <> -\* LinkAfterStarted (negative control): the link is set once started() runs. LateLink(c) == /\ Fault = "LinkAfterStarted" /\ kind[c] = "link" /\ began[c] /\ ~linkEver[c] @@ -285,52 +299,66 @@ LateLink(c) == /\ linkEver' = [linkEver EXCEPT ![c] = TRUE] /\ UNCHANGED <> -\* A kill (init timer or scope deadline) before started() began. +\* An outside kill or an administrative stop (design 2.3 rows 2-3). +OutsideKill(c) == + /\ KillEnabled /\ ~xkill[c] /\ phase[c] \in {"starting", "ready", "exiting"} + /\ xkill' = [xkill EXCEPT ![c] = TRUE] + /\ UNCHANGED <> +AdminStop(c) == + /\ AdminStopEnabled /\ ~astop[c] /\ phase[c] = "starting" + /\ astop' = [astop EXCEPT ![c] = TRUE] + /\ UNCHANGED <> + +AnyKill(c) == killed[c] \/ skill[c] \/ xkill[c] + KilledEarly(c) == - /\ phase[c] = "starting" /\ ~began[c] /\ (killed[c] \/ skill[c]) + /\ phase[c] = "starting" /\ ~began[c] /\ AnyKill(c) /\ Exit(c, "Killed") - /\ commit' = IF commit[c] = None THEN [commit EXCEPT ![c] = "failed"] ELSE commit - /\ UNCHANGED <> - -\* started() returns (or panics); held signals are resolved now, with the -\* trap flag it leaves (design section 5.3, precedence): -\* 1. the init timer committed timedout, or a kill: exit Killed; -\* 2. a non-Normal held parent exit and the child does not trap: exit -\* with it (a held Normal parent exit is dropped); -\* 3. a panic, Init::stop (non-Normal), Init::error / ignore (Normal); -\* 4. ready: commit, unless the group closed (exit Shutdown). A held parent -\* exit to a trapping child is queued behind the continuation (E3). + /\ initR' = [initR EXCEPT ![c] = "Killed"] + /\ IF commit[c] = None THEN SetCommit(c, "failed") ELSE UNCHANGED <> + /\ UNCHANGED <> + +\* started() returns; design 2.3 precedence. No failed row runs stopped(). StartedEnd(c) == /\ phase[c] = "starting" /\ began[c] - /\ \E o \in Outcomes, t \in (IF ChildCanTrap THEN BOOLEAN ELSE {FALSE}): - LET sigOut == pex[c] \in NonNormal /\ ~t /\ Fault # "DropHeld" + /\ \E o \in Outcomes, t \in (IF ChildCanTrap THEN BOOLEAN ELSE {FALSE}), + first \in {"admin", "signal"}: + LET sig == pex[c] \in NonNormal /\ ~t /\ Fault # "DropHeld" + row3 == astop[c] \/ sig + r3 == IF astop[c] /\ (~sig \/ first = "admin") THEN "Shutdown" ELSE pex[c] + closed == (place[c] = "G" /\ ~gopen) \/ (owned[c] /\ ~ownerOpen) failWith(r) == /\ Exit(c, r) - /\ commit' = IF commit[c] = None - THEN [commit EXCEPT ![c] = "failed"] ELSE commit + /\ initR' = [initR EXCEPT ![c] = r] + /\ IF commit[c] = None THEN SetCommit(c, "failed") + ELSE UNCHANGED <> /\ UNCHANGED <> IN /\ ctrap' = [ctrap EXCEPT ![c] = t] /\ pex' = [pex EXCEPT ![c] = None] - /\ IF killed[c] \/ skill[c] THEN failWith("Killed") - ELSE IF sigOut THEN failWith(pex[c]) + /\ IF AnyKill(c) THEN failWith("Killed") + ELSE IF row3 THEN failWith(r3) ELSE IF o = "panic" THEN failWith("Panic") - ELSE IF o = "stop" THEN failWith("Shutdown") - ELSE IF o \in {"error", "ignore"} + ELSE IF o = "stopS" THEN failWith("Shutdown") + ELSE IF o \in {"stopN", "error"} THEN failWith("Normal") + ELSE IF o = "ignore" THEN /\ Exit(c, "Normal") - /\ commit' = [commit EXCEPT ![c] = IF o = "ignore" THEN "ignored" ELSE "failed"] + /\ initR' = [initR EXCEPT ![c] = "Normal"] + /\ SetCommit(c, "ignored") /\ UNCHANGED <> - ELSE IF place[c] = "G" /\ ~gopen /\ Fault # "ReadyAfterClose" + ELSE IF closed /\ Fault # "ReadyAfterClose" THEN failWith("Shutdown") ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] - /\ commit' = [commit EXCEPT ![c] = "ready"] - /\ readyOpen' = [readyOpen EXCEPT ![c] = gopen \/ place[c] # "G"] + /\ SetCommit(c, "ready") + /\ readyOpen' = [readyOpen EXCEPT ![c] = ~closed] /\ cont' = [cont EXCEPT ![c] = o = "continue"] /\ qpex' = [qpex EXCEPT ![c] = IF t THEN pex[c] ELSE None] - /\ UNCHANGED why - /\ UNCHANGED <> + /\ UNCHANGED <> + /\ UNCHANGED <> -\* Others (a peer that found the name, the caller after the ack) send. Send(c) == /\ phase[c] \in {"starting", "ready"} /\ msgs[c] < MaxMsgs /\ msgs' = [msgs EXCEPT ![c] = msgs[c] + 1] @@ -348,230 +376,291 @@ Handle(c) == /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] /\ UNCHANGED <> -\* A queued parent exit to a trapping child ends it, Normal included (E3); -\* it is a mailbox entry, so it waits for the continuation. HandleQueuedExit(c) == /\ phase[c] = "ready" /\ qpex[c] # None /\ ~cont[c] \/ Fault = "ContinueQueued" /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] /\ qpex' = [qpex EXCEPT ![c] = None] /\ Exit(c, qpex[c]) - /\ UNCHANGED <> + /\ UNCHANGED <> -\* A parent exit reaching a ready, non-trapping child (Normal is ignored). ApplyParentExit(c) == /\ phase[c] = "ready" /\ pex[c] # None /\ pex' = [pex EXCEPT ![c] = None] /\ IF pex[c] \in NonNormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> - /\ UNCHANGED <> + /\ UNCHANGED <> -\* After readiness the child may stop or crash on its own. ReadyExit(c) == /\ phase[c] = "ready" /\ \E r \in {"Normal", "Panic"} : Exit(c, r) - /\ UNCHANGED <> + /\ UNCHANGED <> -\* A kill reaching a ready child (the scope deadline; under -\* TimerIgnoresCommit also the init timer). KillReady(c) == - /\ phase[c] = "ready" /\ (killed[c] \/ skill[c]) + /\ phase[c] = "ready" /\ AnyKill(c) /\ Exit(c, "Killed") - /\ UNCHANGED <> + /\ UNCHANGED <> + +\* Before D the reason can still change: a kill makes it Killed (L3). +KillExiting(c) == + /\ phase[c] = "exiting" /\ AnyKill(c) /\ why[c] # "Killed" + /\ why' = [why EXCEPT ![c] = "Killed"] + /\ UNCHANGED <> -\* A ready linked child unlinks from P (ctx.unlink); ownership stays (S8). ChildUnlink(c) == /\ UnlinkEnabled /\ phase[c] = "ready" /\ edge[c] /\ edge' = [edge EXCEPT ![c] = FALSE] /\ UNCHANGED <> -\* D: the edges close and their notices are taken (delivered at publication). +\* D: the edges close and their notices are taken; the reason is final. Die(c) == /\ phase[c] = "exiting" /\ phase' = [phase EXCEPT ![c] = "dead"] - /\ lnote' = [lnote EXCEPT ![c] = edge[c] /\ pst # "dead"] - /\ mnote' = [mnote EXCEPT ![c] = mon[c] /\ pst # "dead"] + /\ lnote' = [lnote EXCEPT ![c] = edge[c] /\ Alive(pst)] + /\ mnote' = [mnote EXCEPT ![c] = mon[c] /\ Alive(pst)] /\ edge' = [edge EXCEPT ![c] = FALSE] /\ mon' = [mon EXCEPT ![c] = FALSE] /\ reg' = IF Fault = "UnregisterAtDeath" THEN reg \ {c} ELSE reg - /\ UNCHANGED <> - -\* Publication (after destruction): the name and the slot are released, -\* then each taken notice is disposed of by the startup cell: a failed start -\* whose waiter still waits (or consumed the error) diverts or drops it, -\* except that a non-trapping P still exits on a non-Normal init failure -\* (OTP: the link kills the caller); otherwise it is an ordinary notice. -Publish(c) == + /\ slots' = IF Fault = "EarlySlotRelease" THEN slots \ {c} ELSE slots + /\ UNCHANGED <> + +\* After destruction: the name and the slot are released. +Release(c) == /\ phase[c] = "dead" - /\ phase' = [phase EXCEPT ![c] = "published"] + /\ phase' = [phase EXCEPT ![c] = "released"] /\ reg' = IF Fault = "LateUnregister" THEN reg ELSE reg \ {c} + /\ slots' = slots \ {c} + /\ UNCHANGED <> + +\* Publication: each taken notice is disposed of by the commitment and the +\* waiter (design 5.2); answers become final. +Publish(c) == + /\ phase[c] = "released" + /\ phase' = [phase EXCEPT ![c] = "published"] + /\ pubAlive' = [pubAlive EXCEPT ![c] = Alive(pst)] /\ LET r == why[c] - failedStart == (commit[c] \in {"failed", "ignored", "timedout"} - \/ (Fault = "DivertAfterReady" /\ commit[c] = "ready")) - /\ wait[c] \in {"waiting", "consumed"} - keepTimeout == commit[c] = "timedout" /\ Fault = "KillBeforeUnlink" - ordinary == ~failedStart \/ keepTimeout - ln == lnote[c] /\ pst # "dead" - mn == mnote[c] /\ pst # "dead" - divert == failedStart /\ wait[c] = "waiting" - /\ ((ln /\ ptrap) \/ mn) /\ ~keepTimeout - IN /\ div' = IF divert THEN [div EXCEPT ![c] = r] ELSE div - /\ divEver' = IF divert THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver - /\ IF ln /\ (ordinary \/ (ptrap /\ Fault = "NoLinkFlush") - \/ (ptrap /\ divert /\ Fault = "DoubleNotice")) + fs == commit0[c] \in FailCommits + \/ (Fault = "MarkKeptAfterReady" /\ commit0[c] = "ready") + ln == lnote[c] /\ Alive(pst) + mn == mnote[c] /\ Alive(pst) + w == wait[c] + ordinary == ~fs \/ w \in {"none", "abandoned"} + \/ (Fault = "KillBeforeUnlink" /\ commit0[c] = "timedout") + divertL == ln /\ ~ordinary /\ w = "waiting" + /\ (ptrap \/ commit0[c] = "timedout") + divertM == mn /\ ~ordinary /\ w = "waiting" + applyNow == ln /\ ~ordinary /\ ~ptrap /\ commit0[c] # "timedout" + /\ (r # "Normal" \/ Fault = "KillOnNormalFailure") + /\ Fault # "NoParentKillOnFailure" + dbl == Fault = "DoubleNotice" /\ divertL /\ ptrap + b1 == ln /\ (ordinary \/ (ptrap /\ Fault = "NoLinkFlush") \/ dbl) + b2 == ~b1 /\ applyNow /\ pst = "run" + b3 == ~b1 /\ ~b2 /\ mn /\ (ordinary \/ Fault = "NoMonitorFlush") + legit == (commit0[c] \in FailCommits /\ w = "consumed") + \/ (ln /\ ~ptrap /\ (r = "Normal" \/ pst # "run")) + IN /\ lost' = [lost EXCEPT ![c] = lost[c] + \/ ((ln \/ mn) /\ ~(b1 \/ b2 \/ b3 \/ divertL \/ divertM) /\ ~legit)] + /\ div' = IF divertL \/ divertM THEN [div EXCEPT ![c] = r] ELSE div + /\ divEver' = IF divertL \/ divertM THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver + /\ IF ln /\ (ordinary \/ (ptrap /\ Fault = "NoLinkFlush") \/ dbl) THEN /\ DeliverLink(c, r) /\ UNCHANGED <> - ELSE IF ln /\ ~ptrap /\ commit[c] = "failed" /\ r # "Normal" /\ pst = "run" + ELSE IF applyNow /\ pst = "run" THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c /\ UNCHANGED dV ELSE IF mn /\ (ordinary \/ Fault = "NoMonitorFlush") THEN /\ DeliverDown(c, r) /\ UNCHANGED <> ELSE UNCHANGED <> - /\ UNCHANGED <> + /\ UNCHANGED <> -\* LateUnregister (negative control): the name is released only after the -\* notices. LateRelease(c) == /\ Fault = "LateUnregister" /\ phase[c] = "published" /\ c \in reg /\ reg' = reg \ {c} - /\ UNCHANGED <> + /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* The init timer and the waiter. *) +(* The init timer, the cleanup deadline and the waiter. *) -\* The framework-owned init timer: it commits timedout only if nothing was -\* committed, then kills. The start edges stay; their notice is disposed of -\* at publication by the waiter state (contained while waiting or consumed). +\* The framework-owned init timer commits timedout only while nothing is +\* committed. RewriteCommit / TimerIgnoresCommit are negative controls. Timeout(c) == - /\ TimeoutEnabled - /\ Waiting(kind[c]) \/ kind[c] = "spawn" - /\ ~killed[c] + /\ TimeoutEnabled /\ kind[c] # None /\ ~killed[c] /\ \/ commit[c] = None /\ phase[c] = "starting" - \/ Fault = "TimerIgnoresCommit" /\ phase[c] \in {"starting", "ready"} - /\ commit' = IF commit[c] = None THEN [commit EXCEPT ![c] = "timedout"] ELSE commit + \/ Fault \in {"TimerIgnoresCommit", "RewriteCommit"} /\ phase[c] \in {"starting", "ready"} + /\ IF commit[c] = None \/ Fault = "RewriteCommit" + THEN SetCommit(c, "timedout") ELSE UNCHANGED <> /\ killed' = [killed EXCEPT ![c] = TRUE] - /\ UNCHANGED <> + /\ UNCHANGED <> + +\* The bounded wait for publication expires (measured from the commitment). +CleanupDeadline(c) == + /\ commit0[c] \in FailCommits /\ ~dl[c] + /\ phase[c] \in {"starting", "exiting", "dead"} + /\ dl' = [dl EXCEPT ![c] = TRUE] + /\ UNCHANGED <> AckOk(c) == /\ wait[c] = "waiting" - /\ \/ commit[c] = "ready" + /\ \/ commit0[c] = "ready" \/ Fault = "AckBeforeStarted" /\ commit[c] = None /\ phase[c] = "starting" - \/ Fault = "ErrorAsOk" /\ commit[c] = "ignored" + \/ Fault = "ErrorAsOk" /\ commit0[c] = "ignored" /\ res' = [res EXCEPT ![c] = "ok"] /\ wait' = [wait EXCEPT ![c] = "consumed"] - /\ UNCHANGED <> + /\ div' = [div EXCEPT ![c] = None] + /\ lost' = [lost EXCEPT ![c] = lost[c] \/ div[c] # None] + /\ UNCHANGED <> -\* A failed, ignored or timed-out start answers after publication, or at its -\* bounded cleanup deadline with the cleanup still pending (pend). The error -\* consumes a diverted notice. +\* An error or ignore answers after publication (final reason), or after the +\* cleanup deadline with the cleanup pending (provisional reason). AckFail(c) == /\ wait[c] = "waiting" - /\ \/ commit[c] \in {"failed", "ignored", "timedout"} - \/ Fault = "DivertAfterReady" /\ commit[c] = "ready" /\ phase[c] = "published" + /\ \/ commit0[c] \in FailCommits + \/ Fault = "AnswerReadyAsFailure" /\ commit0[c] = "ready" /\ phase[c] = "published" + /\ ~(Fault = "NoAnswerIgnore" /\ commit0[c] = "ignored") /\ \E late \in BOOLEAN : - /\ IF late THEN phase[c] \in {"exiting", "dead"} - \/ (phase[c] = "starting" /\ commit[c] = "timedout") + /\ IF late THEN phase[c] \in {"starting", "exiting", "dead", "released"} + /\ (dl[c] \/ Fault = "EarlyPending") ELSE \/ phase[c] = "published" \/ Fault = "EarlyFailAck" /\ phase[c] = "dead" /\ pend' = [pend EXCEPT ![c] = late] - /\ ~(Fault = "NoAnswerIgnore" /\ commit[c] = "ignored") - /\ res' = [res EXCEPT ![c] = CASE commit[c] = "ignored" -> "ignored" - [] commit[c] = "timedout" -> "timeout" + /\ resR' = [resR EXCEPT ![c] = + IF ~late /\ Fault # "FrozenReason" THEN why[c] ELSE initR[c]] + /\ res' = [res EXCEPT ![c] = CASE commit0[c] = "ignored" -> "ignored" + [] commit0[c] = "timedout" -> "timeout" [] OTHER -> "failed"] /\ wait' = [wait EXCEPT ![c] = "consumed"] /\ div' = [div EXCEPT ![c] = None] - /\ UNCHANGED <> + /\ UNCHANGED <> -\* The Starting is dropped: the child keeps running; a diverted notice goes -\* to P as an ordinary one; later notices are ordinary. +\* The Starting is dropped: a diverted notice goes to P as an ordinary one. Abandon(c) == /\ wait[c] = "waiting" /\ wait' = [wait EXCEPT ![c] = "abandoned"] /\ div' = [div EXCEPT ![c] = None] /\ LET r == div[c] - give == r # None /\ Fault # "AbandonDropsNotice" - IN IF give /\ kind[c] = "link" THEN DeliverLink(c, r) /\ UNCHANGED <> - ELSE IF give /\ kind[c] = "monitor" - THEN DeliverDown(c, r) /\ UNCHANGED <> - ELSE UNCHANGED <> - /\ UNCHANGED <> + give == r # None /\ Alive(pst) + drop == Fault = "AbandonDropsNotice" + IN /\ handed' = [handed EXCEPT ![c] = give] + /\ lost' = [lost EXCEPT ![c] = lost[c] \/ (give /\ drop)] + /\ IF give /\ ~drop /\ kind[c] = "link" + THEN DeliverLink(c, r) /\ UNCHANGED <> + ELSE IF give /\ ~drop /\ kind[c] = "monitor" + THEN DeliverDown(c, r) /\ UNCHANGED <> + ELSE UNCHANGED <> + /\ UNCHANGED <> ----------------------------------------------------------------------------- -(* The parent, its scope and the group. *) +(* The parent, its scope and the groups. *) PExit == /\ ParentCanExit /\ pst = "run" /\ \E r \in {"Normal", "Panic"} : pst' = "exiting" /\ pwhy' = r - /\ UNCHANGED <> + /\ UNCHANGED <> + +OwnerClose == + /\ OwnerCloseEnabled /\ ownerOpen + /\ ownerOpen' = FALSE + /\ UNCHANGED <> -\* P's D: each linked child gets a parent exit, held during started(), -\* queued behind the continuation for a trapping ready child, applied -\* otherwise; P's monitors are cancelled. +\* P's D: its link notices are taken; edges and monitors close. PDie == /\ pst = "exiting" /\ pst' = "dead" - /\ pex' = [c \in Kids |-> IF edge[c] /\ ~(phase[c] = "ready" /\ ctrap[c]) - THEN pwhy ELSE pex[c]] - /\ qpex' = [c \in Kids |-> IF edge[c] /\ phase[c] = "ready" /\ ctrap[c] - THEN pwhy ELSE qpex[c]] - /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (edge[c] /\ pwhy \in NonNormal)] - /\ edge' = [c \in Kids |-> FALSE] - /\ mon' = [c \in Kids |-> FALSE] - /\ UNCHANGED <> - -\* S8: once P's exit began, the scope deadline kills every owned child -\* still alive (linked or not). + /\ pn' = [c \in Kids |-> IF edge[c] THEN pwhy ELSE pn[c]] + /\ edge' = F(FALSE) + /\ mon' = F(FALSE) + /\ UNCHANGED <> + +\* P's publication: the parent exits reach the linked children, and the +\* scope sends every live owned child a polite Shutdown (S8; E3 parent +\* signal); P's slot is released. +PPublish == + /\ pst = "dead" + /\ pst' = "published" + /\ LET sc(c) == ScopeEnabled /\ owned[c] /\ phase[c] \in {"starting", "ready"} + /\ Fault # "NoPoliteStop" + rr(c) == IF pn[c] \in NonNormal THEN pn[c] + ELSE IF sc(c) THEN "Shutdown" ELSE pn[c] + trapReady(c) == phase[c] = "ready" /\ ctrap[c] + IN /\ pex' = [c \in Kids |-> IF rr(c) # None /\ phase[c] \in {"starting", "ready"} + /\ ~trapReady(c) THEN rr(c) ELSE pex[c]] + /\ qpex' = [c \in Kids |-> IF rr(c) # None /\ trapReady(c) THEN rr(c) ELSE qpex[c]] + /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (rr(c) \in NonNormal + /\ phase[c] \in {"starting", "ready"})] + /\ pn' = F(None) + /\ UNCHANGED <> + +\* S8: from P's exit begin, the scope deadline (a timer, not assumed fair) +\* kills every live owned child directly, exiting ones included. ScopeDeadline == - /\ ScopeEnabled /\ pplace = "G" /\ pst # "run" - /\ \E c \in Kids : owned[c] /\ phase[c] \in {"starting", "ready"} /\ ~skill[c] - /\ (edge[c] \/ Fault # "NoScopeKill") - /\ skill' = [c \in Kids |-> skill[c] \/ (owned[c] /\ phase[c] \in {"starting", "ready"} - /\ (edge[c] \/ Fault # "NoScopeKill"))] - /\ UNCHANGED <> - -\* P's generation is done: P past D and every owned child published. + /\ ScopeEnabled /\ pplace = "G" /\ pst # "run" /\ ~sfired + /\ sfired' = TRUE + /\ skill' = [c \in Kids |-> skill[c] \/ (owned[c] + /\ phase[c] \in (IF Fault = "NoScopeKill" THEN {"starting", "ready"} + ELSE {"starting", "ready", "exiting"}))] + /\ UNCHANGED <> + +\* P's generation is done: P published and every owned child published. GenDone == - /\ ScopeEnabled /\ pplace = "G" /\ pst = "dead" /\ ~gdone + /\ ScopeEnabled /\ pplace = "G" /\ ~gdone + /\ pst = "published" \/ (Fault = "GenDoneAtParentD" /\ pst = "dead") /\ \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} \/ Fault = "EarlyGenDone" /\ gdone' = TRUE - /\ UNCHANGED <> + /\ UNCHANGED <> GClose == /\ CloseEnabled /\ gopen /\ gopen' = FALSE - /\ UNCHANGED <> + /\ UNCHANGED <> ChildStep(c) == \/ Begin(c) \/ PrepareFail(c) \/ Commit(c) \/ RegisterFirst(c) \/ CommitRest(c) - \/ StartedBegin(c) \/ LateLink(c) \/ KilledEarly(c) \/ StartedEnd(c) + \/ StartedBegin(c) \/ LateLink(c) \/ OutsideKill(c) \/ AdminStop(c) + \/ KilledEarly(c) \/ StartedEnd(c) \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ HandleQueuedExit(c) - \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ ChildUnlink(c) - \/ Die(c) \/ Publish(c) \/ LateRelease(c) - \/ Timeout(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) + \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ KillExiting(c) + \/ ChildUnlink(c) \/ Die(c) \/ Release(c) \/ Publish(c) \/ LateRelease(c) + \/ Timeout(c) \/ CleanupDeadline(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) -Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ PDie - \/ ScopeDeadline \/ GenDone \/ GClose +Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ OwnerClose \/ PDie + \/ PPublish \/ ScopeDeadline \/ GenDone \/ GClose -\* Liveness assumes the child's own steps, publication, the waiter's answer, -\* P's death and the scope deadline complete (started() and destructors -\* return). +\* Liveness assumes the child's own steps, its death and publication, the +\* waiter's answer, and P's D and publication complete. The scope deadline +\* and the cleanup deadline are not assumed to fire. Progress(c) == \/ StartedBegin(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ RunContinue(c) \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ KillReady(c) - \/ Die(c) \/ Publish(c) \/ AckOk(c) + \/ Die(c) \/ Release(c) \/ Publish(c) \/ AckOk(c) Fairness == /\ \A c \in Kids : /\ WF_vars(Progress(c)) /\ WF_vars(Commit(c)) /\ SF_vars(AckFail(c)) - /\ WF_vars(PDie) /\ WF_vars(ScopeDeadline) /\ WF_vars(GenDone) + /\ WF_vars(PDie) /\ WF_vars(PPublish) /\ WF_vars(GenDone) Spec == Init /\ [][Next]_vars LiveSpec == Spec /\ Fairness @@ -581,102 +670,119 @@ LiveSpec == Spec /\ Fairness InheritedPlacement == \A c \in Kids : place[c] # None => place[c] = Expected(c) -\* The edge really exists before started() begins. LinkBeforeStarted == \A c \in Kids : /\ (began[c] /\ kind[c] = "link") => linkEver[c] /\ (began[c] /\ kind[c] = "monitor") => monEver[c] -NameUnique == Cardinality(reg) <= 1 - -\* The name is held from the commit until publication. -NameHeldUntilPublished == \A c \in reg : phase[c] \in {"reg1", "starting", "ready", "exiting", "dead"} +\* An edge is committed only while P runs (not after its exit began). +EdgeOnlyToRunningParent == \A c \in Kids : (linkEver[c] \/ monEver[c]) => edgeWhenRun[c] +NoEdgeToDeadParent == \A c \in Kids : (edge[c] \/ mon[c]) => Alive(pst) -\* A new holder never overlaps an old holder's destruction. +NameUnique == Cardinality(reg) <= 1 +\* Every holder is between its commit and its release ... +NameHeldUntilReleased == \A c \in reg : phase[c] \in {"reg1"} \cup LiveP +\* ... and an admitted named child keeps the name until its release. +NameKeptWhileLive == \A c \in Kids : (wname[c] /\ phase[c] \in LiveP) => c \in reg +\* A new holder never overlaps an old holder that is not yet released. NoOverlapOnReuse == - \A c \in reg : \A d \in Kids \ {c} : wname[d] => phase[d] # "dead" - -\* No notice or final error about c is observable while c holds the name. + \A c \in reg : \A d \in Kids \ {c} : wname[d] => phase[d] \notin LiveP +\* No notice, parent kill or final answer about c while c holds the name. NameFreeBeforeObservation == \A c \in Kids : - (exitN[c] + downN[c] > 0 \/ (res[c] \in {"failed", "ignored", "timeout"} /\ ~pend[c])) + (exitN[c] + downN[c] > 0 \/ pcause = c + \/ (res[c] \in {"failed", "ignored", "timeout"} /\ ~pend[c])) => c \notin reg -\* A refused start is never observable. +\* A refused start is never observable, also not through a competitor's +\* refusal. AbortInvisible == \A c \in Kids : phase[c] = "aborted" => /\ ~edge[c] /\ ~mon[c] /\ ~linkEver[c] /\ ~monEver[c] /\ c \notin reg - /\ ~seen[c] /\ ~began[c] /\ place[c] = None + /\ ~seen[c] /\ c \notin blamed /\ c \notin slots /\ ~began[c] /\ place[c] = None /\ exitN[c] = 0 /\ downN[c] = 0 /\ pcause # c -\* The answer is the committed outcome. +\* The commitment never changes once made. +CommitImmutable == \A c \in Kids : commit0[c] # None => commit[c] = commit0[c] + +\* The answer is the first commitment. ResultMatchesCommit == \A c \in Kids : - /\ res[c] = "ok" /\ Waiting(kind[c]) => commit[c] = "ready" - /\ res[c] = "failed" => commit[c] = "failed" - /\ res[c] = "ignored" => commit[c] = "ignored" - /\ res[c] = "timeout" => commit[c] = "timedout" + /\ res[c] = "ok" /\ Waiting(kind[c]) => commit0[c] = "ready" + /\ res[c] = "failed" => commit0[c] = "failed" + /\ res[c] = "ignored" => commit0[c] = "ignored" + /\ res[c] = "timeout" => commit0[c] = "timedout" -\* Readiness is final: a ready child's later death is never a start error. -ReadyIsFinal == \A c \in Kids : commit[c] = "ready" => res[c] \notin {"failed", "ignored", "timeout"} +ReadyIsFinal == \A c \in Kids : commit0[c] = "ready" => res[c] \notin {"failed", "ignored", "timeout"} -\* The init timer never kills a child that committed ready. -TimeoutNeverKillsReady == \A c \in Kids : commit[c] = "ready" => ~killed[c] +TimeoutNeverKillsReady == \A c \in Kids : commit0[c] = "ready" => ~killed[c] -\* Managed readiness commits only while admission is open. -ReadyOnlyWhileOpen == \A c \in Kids : commit[c] = "ready" => readyOpen[c] +ReadyOnlyWhileOpen == \A c \in Kids : commit0[c] = "ready" => readyOpen[c] -\* Without a pending cleanup, a start error is answered after publication. +\* An answer without pending comes after publication and reports the final +\* exit reason; a pending answer only after the cleanup deadline. FailureAfterPublication == \A c \in Kids : (res[c] \in {"failed", "ignored", "timeout"} /\ ~pend[c]) => phase[c] = "published" +ReportedReasonFinal == + \A c \in Kids : (res[c] = "failed" /\ ~pend[c]) => resR[c] = why[c] +PendingOnlyAfterDeadline == \A c \in Kids : pend[c] => dl[c] -\* A failed start_monitor delivers no Down; a failed start_link no Exit to a -\* trapping parent. +\* A failed start's notice is consumed: no Down; no Exit to a trapping P. FailureFlushed == \A c \in Kids : res[c] \in {"failed", "ignored", "timeout"} => /\ kind[c] = "monitor" => downN[c] = 0 /\ (kind[c] = "link" /\ ptrap) => exitN[c] = 0 +\* ... but a non-Normal init failure still ends a non-trapping P (OTP) ... +FailureEndsNonTrappingParent == + \A c \in Kids : + (kind[c] = "link" /\ ~ptrap /\ commit0[c] = "failed" /\ why[c] \in NonNormal + /\ phase[c] = "published" /\ lnote[c] /\ pubAlive[c]) => pst # "run" +\* ... and a Normal one (error, ignore, stop(Normal)) never does. +NormalFailureSparesParent == + \A c \in Kids : + (kind[c] = "link" /\ (commit0[c] = "ignored" \/ (commit0[c] = "failed" /\ why[c] = "Normal"))) + => pcause # c -\* An init timeout reaches P only if the wait was abandoned. TimeoutContained == - \A c \in Kids : (commit[c] = "timedout" /\ wait[c] # "abandoned") => + \A c \in Kids : (commit0[c] = "timedout" /\ wait[c] # "abandoned") => /\ pcause # c /\ exitN[c] = 0 /\ downN[c] = 0 -NoEdgeToDeadParent == \A c \in Kids : (edge[c] \/ mon[c]) => pst # "dead" - AdmittedOnlyOpen == \A c \in Kids : place[c] = "G" => admOpen[c] +OwnedOnlyWhileOwnerOpen == \A c \in Kids : owned[c] => ownOpenAt[c] -CapacityRespected == LiveInG <= Cap +\* Slots (P's included) are held until release, and never exceed Cap. +CapacityRespected == Used <= Cap +ParentSlotHeld == (pplace = "G" /\ pst \in {"run", "exiting", "dead"}) => PSlot = 1 +SlotHeldUntilReleased == \A c \in Kids : (place[c] = "G" /\ phase[c] \in LiveP) => c \in slots NoticeOnce == \A c \in Kids : exitN[c] <= 1 /\ downN[c] <= 1 -\* A notice diverted to an abandoned wait reaches P, with the child's reason. +\* No notice owed to P is ever dropped. +NoLostNotice == \A c \in Kids : ~lost[c] + +\* A notice handed on by an abandoned wait reaches P with its kind and reason. AbandonedNoticeDelivered == - \A c \in Kids : (wait[c] = "abandoned" /\ divEver[c]) => - /\ kind[c] = "link" => exitN[c] >= 1 /\ exitR[c] = why[c] + \A c \in Kids : handed[c] => + /\ (kind[c] = "link" /\ ptrap) => exitN[c] >= 1 /\ exitR[c] = why[c] + /\ (kind[c] = "link" /\ ~ptrap /\ why[c] \in NonNormal) => pst # "run" /\ kind[c] = "monitor" => downN[c] >= 1 /\ downR[c] = why[c] ContinueFirst == \A c \in Kids : ~early[c] -\* P's generation is done only after its owned children. GenDoneMeansOwnedDone == - gdone => \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} + gdone => /\ pst = "published" + /\ \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} -Safety == - /\ InheritedPlacement /\ LinkBeforeStarted /\ NameUnique - /\ NameHeldUntilPublished /\ NoOverlapOnReuse /\ NameFreeBeforeObservation - /\ AbortInvisible /\ ResultMatchesCommit /\ ReadyIsFinal - /\ TimeoutNeverKillsReady /\ ReadyOnlyWhileOpen /\ FailureAfterPublication - /\ FailureFlushed /\ TimeoutContained /\ NoEdgeToDeadParent - /\ AdmittedOnlyOpen /\ CapacityRespected /\ NoticeOnce - /\ AbandonedNoticeDelivered /\ ContinueFirst /\ GenDoneMeansOwnedDone +\* When the scope deadline fires, every live owned child is killed. +DeadlineKillsAllOwned == + sfired => \A c \in Kids : (owned[c] /\ phase[c] \in {"starting", "ready", "exiting"}) + => skill[c] (* Liveness. *) -\* A child linked to a parent that dies with a non-Normal reason ends. ParentExitReachesChild == \A c \in Kids : pexAbn[c] ~> phase[c] \in {"published", "aborted"} -\* Every waiting start is answered or abandoned. StartAnswered == \A c \in Kids : wait[c] = "waiting" ~> wait[c] # "waiting" -\* A managed parent's generation finishes once it died. -OwnedScopeFinishes == (ScopeEnabled /\ pplace = "G" /\ pst = "dead") ~> gdone +\* Once P is published, its generation ends without the scope deadline +\* (the polite Shutdown suffices when children cooperate). +OwnedScopeFinishes == (ScopeEnabled /\ pplace = "G" /\ pst = "published") ~> gdone ============================================================================= From 0623c30e1b10f59d80a968335cc21c6cde2eb8c6 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 15:57:28 -0300 Subject: [PATCH 23/42] verification: MCStartCap with a parent that stays --- docs/design/unified-start.md | 546 ++++++++++++++++++++------------ verification/tla/MCStartCap.cfg | 2 +- verification/tla/MCStartCap.tla | 4 +- 3 files changed, 355 insertions(+), 197 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 409eb6c0..cdbc9745 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1,13 +1,13 @@ # Unified start API (0.9): design contract -Status: design in review, revision 2, not implemented. Target release 0.9. +Status: design in review, revision 3, not implemented. Target release 0.9. The implementation follows after two independent reviews. It is designed against the 0.8 API as #240 ships it: exit signals, `try_link`, `NoProc`, and the 0.8 API-review cleanups (`send_request`, `#[non_exhaustive]` builders). -Revision 2 answers both round-one reviews (Codex and agy, both "approve after -the listed changes"). What changed and why is in -[Review resolutions](#20-review-resolutions). +Revision 2 answered both round-one reviews. Revision 3 answers both round-two +reviews (Codex and agy, both "approve after the listed changes", no P0). What +changed and why is in [Review resolutions](#20-review-resolutions). - **Model:** [`verification/tla/Start.tla`](../../verification/tla/Start.tla), with configurations `MCStart`, `MCStartCap`, `MCStartTimeout`, @@ -105,8 +105,10 @@ impl Starting { } /// OTP {ok, Pid} | ignore. Shared with the dynamic supervisor (section 18). +/// `pending` is Some when the ignoring instance was not yet published at +/// the cleanup deadline (section 5.5): it still holds its name and slot. #[must_use] -pub enum StartOutcome { Started(T), Ignored } +pub enum StartOutcome { Started(T), Ignored { pending: Option } } impl StartOutcome { pub fn started(self) -> Option; pub fn is_ignored(&self) -> bool; @@ -169,38 +171,51 @@ returned. links should see, as `{stop, R}`. **Resolution when `started()` returns.** Signals received during `started()` -are held (0.8 phase 0, otp-exit-signals E2/E3). They are resolved with the -trap flag that `started()` leaves, in this precedence. The first matching row -decides both the starter's result and the final exit reason. - -| # | Condition | Exit reason | `stopped()` | Startup cell | +are held (0.8 phase 0, otp-exit-signals E2/E3) and resolved with the trap flag +that `started()` leaves. The first matching row below decides the **startup +decision**: the immutable commitment (section 5.1) and the result's kind and +diagnostic. It also sets the **requested exit reason**. + +The **final exit reason** is fixed only at D, by 0.8's precedence rules +(otp-exit-signals "Reason precedence until D", L3). A kill, for example the +scope deadline or `ChildHandle::kill()`, that arrives between the resolution +and D still makes it `Killed`. An answer given after publication reports the +final reason. An answer given at the cleanup deadline, before D, reports the +requested reason as provisional (`pending: Some`). + +No failed row runs `stopped()`. OTP never calls `terminate/2` when `init/1` +did not return `{ok, ..}`, and cleanup before readiness belongs in +`started()` (agy F2, option A). A failed start therefore reaches D without a +`stopped()` phase. Its reason can still be escalated by a kill before D. + +| # | Condition | Requested reason | `stopped()` | Startup decision | |---|---|---|---|---| | 1 | the init timer committed `timedout` (section 6) | `Killed` | no | `timedout` → `InitTimeout` | -| 2 | a kill arrived (`kill()`, `exit(Kill)`, the S8 scope deadline) | `Killed` | no | failed → `InitExited { Killed }` | -| 3 | the child does not trap, and a held non-`Normal` signal (parent exit, link notice, exit signal) arrived; the first in arrival order wins | that reason | no | failed → `InitExited { r }` | -| 4 | `started()` panicked | `Panic("panic in started(): ..")` | no | failed → `InitExited { Panic }` | -| 5 | `ctx.stop*(r)` was called in `started()` | `r` | yes (a self-exit, L1) | failed → `InitExited { r }` | -| 6 | `Init::stop(r)` | `r` | no | failed → `InitExited { r }` | +| 2 | a kill arrived (`kill()`, `exit(Kill)`, the S8 scope deadline) | `Killed` | no | failed → `InitExited` | +| 3 | an administrative stop (`ChildHandle::stop*`, `shutdown`) arrived, or the child does not trap and a held non-`Normal` signal (parent exit, link notice, exit signal) arrived; the first of these in arrival order wins | its reason | no | failed → `InitExited` | +| 4 | `started()` panicked | `Panic("panic in started(): ..")` | no | failed → `InitExited` | +| 5 | `ctx.stop*(r)` was called in `started()` | `r` | **no** (as `{stop, r}`) | failed → `InitExited` | +| 6 | `Init::stop(r)` | `r` | no | failed → `InitExited` | | 7 | `Err(e)` / `Init::error(e)` | `Normal` | no | failed → `InitFailed { cause: e }` | | 8 | `Init::ignore()` | `Normal` | no | ignored → `Ok(Ignored)` | -| 9 | `Init::ready()` / `continue_with(m)`, but a managed scope's admission closed | `Shutdown` | no | failed → `InitExited { Shutdown }` | +| 9 | `Init::ready()` / `continue_with(m)`, but the child's admission closed: its group, or for an owned child its owner's scope | `Shutdown` | no | failed → `InitExited` | | 10 | `Init::ready()` / `continue_with(m)` | runs | later | **ready** → `Ok(Started(..))` | - **Held signals in rows 9-10.** A held `Normal` signal to a non-trapping - child is dropped (E2). For a child that traps, every held non-kill signal is - queued as a mailbox entry in arrival order. That includes a parent exit, + child is dropped (E2). For a child that traps, every held trappable signal + is queued as a mailbox entry in arrival order. That includes a parent exit, even a `Normal` one (E3: it ends the child when handled). The child - commits ready, runs its continuation first, then handles the queued entries. - This is OTP's behaviour for an `init/1` that sets `trap_exit` and returns - `{continue, C}`. + commits ready, runs its continuation first, then handles the queued + entries. This is OTP's behaviour for an `init/1` that sets `trap_exit` and + returns `{continue, C}`. Administrative stops and kills are untrappable + (rows 2-3). - **A superseded diagnostic.** When a higher row wins over a returned - `Init::error(e)`, `e` is logged at debug level with the actor id. The - result is the winning row's. -- **Cooperative deviation.** OTP applies a signal when it arrives, so a - signal before `process_flag(trap_exit, true)` kills a non-trapping process - immediately. Spawned reads the flag when `started()` returns. This is - 0.8's documented Rust deviation (phase 0, R: no preemption). Section 11 - labels it as such. + `Init::error(e)`, `e` is logged at debug level with the actor id. +- **Cooperative deviation.** OTP applies a signal when it arrives; Spawned + reads the trap flag when `started()` returns. This is 0.8's documented + phase-0 deviation (R: no preemption). +- **Behaviour change from 0.8.** A `ctx.stop()` inside `started()` no longer + runs `stopped()`. ### 2.4 `StartOptions` @@ -246,6 +261,21 @@ returns `InvalidOptions`: These rules are the same in both modes. +**Names** (restored from revision 1; Codex R10): + +```rust +pub struct Name { /* the key; PhantomData A> */ } +impl Name { + pub const fn new(key: &'static str) -> Self; // usable in a static + pub fn from_string(key: String) -> Self; + pub fn key(&self) -> &str; + pub fn whereis(&self) -> Option>; // None if free or held by another type +} +impl From<&'static str> for Name; +impl From for Name; +impl Clone for Name; // no A: Clone bound +``` + **Typing.** `StartOptions` is typed by the actor, and `name` takes a `Name`. Passing a `Name` to an `A` start is a compile error (trybuild `name_type_must_match_actor`). `Name` and `StartOptions` are `Clone` @@ -281,9 +311,15 @@ pub enum StartError { - **`pending`.** It is `None` when the child was published before the answer: dead, state dropped, notices sent, name and slot free. It is - `Some(handle)` when the bounded wait for publication expired first - (section 6). The handle is instance-bound: `wait_exit` reaches the real - end, and the name and slot stay held until then. + `Some(handle)` when the cleanup deadline expired first (section 5.5). The + handle is instance-bound: `wait_exit` reaches the real end, and the name + and slot stay held until then. +- **`reason`.** With `pending: None` it is the final exit reason, the one + `wait_exit`, `Down` and `Exit` report. With `pending: Some` it is the + requested reason, and the final one comes from `pending.wait_exit()` + (Codex R3). +- **A stuck `ignore`** answers `Ok(Ignored { pending: Some(handle) })` (Codex + R4). - **`ignore`** is the `Ok(StartOutcome::Ignored)` outcome, not an error, as OTP returns `ignore` beside `{ok, Pid}` (Codex 9). - **Removed:** `SystemError::StartupFailed`. `ready()` returns these @@ -291,56 +327,90 @@ pub enum StartError { ## 3. Stage 1: one commit -Stage 1 runs inside the call, before any `.await`, in two parts. - -1. **Prepare (invisible).** Validate the options; build the actor's shared - state, mailbox, startup cell and edges in private memory; and acquire the - executor resources. For `Backend::Thread` and threads mode, spawn the - thread parked on a start latch. For an async placement, reserve the task - slot. A failure returns `System` or `InvalidOptions`, and the state is - dropped outside every framework lock. Nothing was visible. -2. **Commit (atomic).** Take, in this order: - 1. the target group's admission lock; - 2. the parent's exit-request lock, the one an exit begin takes; - 3. the parent's link or monitor table lock; - 4. the registry lock. - - Under all of them, check in any order: - - the group is open and below capacity (`Closed`, `Capacity`); +Stage 1 runs inside the call, before any `.await`, in two parts. The atomic +protocol continues through readiness (section 5.1) under the same global lock +order (Codex R1, agy F4). + +1. **Prepare (invisible).** + - Validate the options. + - Build the actor's shared state in private memory: mailbox, startup + result, waiter cell, and the child's own link or monitor table with the + record of its parent edge. The child's tables are still unreachable, so + no shared lock is taken (agy F4.1). + - Acquire executor resources. For `Backend::Thread` and threads mode, spawn + the thread parked on a start latch. For an async placement, reserve the + task slot. + + A failure returns `System` or `InvalidOptions`. The prepared state is + dropped outside every framework lock, so nothing was ever visible. +2. **Commit (atomic).** Take the locks in the global order below: + 1. the owner's admission gate (owned children only, 4.1), and the + system's admission gate for any managed child; + 2. the target group's admission lock; + 3. the parent's exit-request lock; + 4. the parent's link or monitor table; + 5. the registry. + + Under all of them, check: + - the owner's scope admits (`Closed`); + - the target group is open (`Closed`) and has a free slot (`Capacity`); - the parent's exit has not begun (`ParentExited`); - - the name is free (`AlreadyStarted`). - - On success, apply everything before releasing: - - the admission slot; - - the parent's edge, created with the startup cell's mark; + - the name is free (`AlreadyStarted`); + - for a supervised attempt, its absolute deadline has not passed (section + 9). + + On success, apply, still under the locks: + - the slot; + - the attachment to the owner's scope (membership, section 4.1); + - the parent's half of the edge; - the name; - the actor's entry in introspection's actor table. - Release in reverse order, then open the start latch, which cannot fail. - On a refusal nothing was applied: release the locks, drop the prepared - state and stop the parked thread. + Release the locks in reverse order, then open the start latch, which + cannot fail. On a refusal nothing was applied: release the locks, drop the + prepared state, and stop the parked thread. -**Guarantees.** A refused start is never observable: no `whereis` can return -it, no `Capacity` refusal is caused by it, and no edge, notice or hook -exists. Every admission becomes visible at one instant. This is the -`spawn_opt([link, monitor])` atomicity. Model: `AbortInvisible`, with a -`Lookup` observer; control `SplitCommit`, which registers the name before the -rest of the commit. +**The global lock order** (each path takes a subsequence, outer first): -**Lock-order obligations for the implementation.** The order above is global. -No 0.8 path takes a registry lock and then a link-table, exit-request or -group lock, and the implementation keeps it so. A Loom test covers a commit -racing the parent's exit begin and a concurrent commit of the same name. +| # | Lock | Taken by | +|---|---|---| +| 1 | owner scope admission gate (`Admission`), then the system's | commit; the readiness commit (5.1); scope or system shutdown closing admission | +| 2 | one group's admission state, and never two group locks at once | commit (the target group); release at publication | +| 3 | an actor's exit-request lock | commit (the parent); exit begin; D | +| 4 | link and monitor tables: one, or two in `Arc::as_ptr` order (`link.rs` `with_pair`) | commit (the parent's table only, since the child's is private); ordinary `link`, `unlink`, `monitor`; D | +| 5 | the registry | commit; release at publication; `whereis` | +| 6 | a startup cell | the readiness commit; the timer; publication's disposition; the waiter; abandonment | + +Rules: + +- **Scope state.** A `Scope`'s own state lock is never held while a group, + exit-request, table or registry lock is taken (0.8's `Scope` methods take + the group first; they stay leaves). +- **Delivery outside the locks.** A notice, a signal or a kill decided under a + lock is delivered after the lock is released. Delivery takes the target's + mailbox and exit-request locks, which are rank 3. Nothing decides under + rank 3 or 4 and then takes rank 1 or 2. +- **Cross-group starts.** An actor in `G_p` starting a child in `G_c`, while + an actor in `G_c` starts one in `G_p`, holds its owner gate and then one + group lock. Gates are per scope, and at most one group lock is held at a + time, so no cycle exists. +- **Loom.** The test covers: a commit racing the parent's exit begin; two + commits of one name; a readiness commit racing scope closure; the timer + racing the readiness commit; and two cross-group commits. + +**Guarantees.** A refused start is never observable: no `whereis` returns it, +no competing start is refused because of it, and no edge, notice or hook +exists. Every admission becomes visible at one instant. This is the +`spawn_opt([link, monitor])` atomicity. Model: `AbortInvisible`, which checks +lookups and competitors' refusals (control `SplitCommit`), and +`OwnedOnlyWhileOwnerOpen` (control `OwnerUnchecked`). -**Bare `spawn()`** is infallible in its signature. An executor construction -failure in the prepare step panics with the `SystemError`, as -`std::thread::spawn` does. `with_options(..).spawn()` returns it instead. +**Bare `spawn()`** panics with the `SystemError` if executor construction +fails in the prepare step, as `std::thread::spawn` does. +`with_options(..).spawn()` returns it instead. **Forgotten `.await`.** Stage 1 is eager, so `x.start();` without `.await` -still starts the actor, and the dropped `Starting` abandons the wait -(section 5.4). In the migration a 0.8 statement `x.start();` therefore keeps -working. It also warns under `#[must_use]`, which the clients' clippy -`-D warnings` gate turns into an error. +still starts the actor, and the dropped `Starting` abandons the wait (5.4). ## 4. Placement and inheritance @@ -371,127 +441,196 @@ and would silently move 20 ethrex call sites into groups. ### 4.1 S8: linked children owned by the parent's scope -Ownership is decided separately from inheritance, by this rule: - -- **Owned.** A `start_link` child is owned by its parent's scope when the - parent is managed and the child is admitted to a group of the parent's - `ActorSystem`. That covers an inherited group, an explicit group of the same - system, and a pinned shard. -- **Not owned.** A `start_monitor` child, a child placed unmanaged, and a - child in another system. It is linked only, as in OTP, and is owned by its - own placement: the other system's shutdown, or nobody (S6). - -**What ownership means.** - -- **System shutdown.** It reaches an owned child through its parent. The - parent's turn in the reverse admission order (S2) ends when the parent and - its owned descendants are published, or at the grace deadline. -- **Supervised parents.** The generation counts as finished (V3) only when - its owned descendants are published too. A replacement never overlaps - them. -- **Unlink.** Ownership persists after `unlink`. That is not OTP link - semantics; it is scope ownership, justified by Rust resource lifetimes. A - descendant may hold resources (sockets, files, a native slot) that the - parent's replacement or the system's shutdown report must see released, as - for 0.8's registered helpers. - -**Deadlines.** One interval, the parent's row in the E3 deadline table, -covers the whole scope: - -- `shutdown_timeout` from the parent's signal when the supervisor or system - stops it; -- `shutdown_timeout` from the exit begin when the parent exits on its own - (5 s unmanaged default). - -At its expiry the scope deadline sends a kill to every still-alive member, -the parent and each owned descendant, linked or not. Each member is killed -directly; nothing waits for the parent's D. One more interval later, the -scope is stuck, with the existing V3/E3 consequences. - -A parent that never reaches D does not delay the descendants' kill. System -grace expiry likewise kills every remaining actor of every scope directly -(S2, S3). A descendant still linked normally ends earlier, by its parent -exit at the parent's D (E3). - -Model: `MCStartScope`, with invariant `GenDoneMeansOwnedDone` (control -`EarlyGenDone`) and liveness `OwnedScopeFinishes` (control `NoScopeKill`: the -deadline spares unlinked descendants). +**Ownership** is decided separately from inheritance. A `start_link` child is +owned by its parent's scope when the parent is managed and the child is +admitted to a group of the parent's `ActorSystem`. + +- **Owned:** an inherited group, an explicit group of the same system, or a + pinned shard. +- **Not owned:** a `start_monitor` child, a child placed unmanaged, or a + child in another system. These are linked only, as in OTP. +- **When ownership is attached:** in the commit, under the owner's admission + gate (section 3), so a closed owner refuses (`Closed`). +- **Nested ownership.** An owned child that starts its own linked children + owns them in turn. Its scope is a member of its parent's scope, so + ownership is transitive. + +**When the parent exits** (Codex R2, agy F3): + +1. **At P's exit begin:** P's scope closes for new owned children (they are + refused as `ParentExited` or `Closed`), and the scope deadline is armed + (below). +2. **At P's D:** P's edges close, and the link notices to its linked children + are taken. +3. **At P's publication** (after P's destruction): the parent exits reach + the linked children (E3). The scope also sends every live owned member + that is not already exiting a polite parent `Shutdown` signal, which ends + it through E3. That covers unlinked members, and linked ones that a + `Normal` parent exit would leave running. Releasing P's slot comes at this + point too, not at D. +4. **P's generation is done** (V3, the scope's join) only when P and every + owned member are published and their owned work has finished. That is + 0.8's scope completion: helpers and jobs joined, as for any scope. + +The polite `Shutdown` is not OTP link semantics; a gen_server's linked worker +survives a `normal` parent exit. It is the consequence of scope ownership, as +a supervisor's shutdown of its children is. Its justification is Rust's +resource lifetimes: a member may hold resources (sockets, files, a native +slot) that the parent's replacement or the system's report must see released. + +**Deadlines** (Codex R7). All deadlines are absolute instants, and composing +two deadlines takes the earlier. Anchors: + +| Event | Deadline armed | At expiry | +|---|---|---| +| P's exit begins: by a supervisor or system stop, a signal, or on its own | P's **scope deadline** = the anchor of P's E3 row + P's `shutdown_timeout` (5 s unmanaged default) | every live member of P's scope, transitively, is killed directly, exiting ones included (agy F3). P itself follows its own E3 row: killed if the supervisor or system stopped it, never if it exited on its own (no relabel) | +| the scope deadline expires | the **stuck deadline** = the scope deadline + the same interval | the scope is stuck (V3, E3) | +| a member's own exit begins on its own | its E3 row 4 cleanup interval; for a member, the earlier of that and its owner's scope deadline | stuck (row 4), or killed by the owner's scope deadline if that comes first | +| a nested owned scope's parent exits | its own scope deadline = the earlier of its own anchor + interval and every ancestor's scope deadline | as above | + +No later transition extends a deadline: a deadline only moves earlier. The +reason stays final after D, and a kill after D does nothing. P stuck before +its D does not delay anything, since the scope deadline is anchored at P's +exit begin. A member that is never told to stop (P stuck in `stopped()`, so +no publication) is still killed at the scope deadline. System grace expiry +kills every remaining actor of every scope directly (S2, S3). + +**Revised E3 table rows**, written into `otp-exit-signals.md` and +`semantics.md` by the implementation PR: + +- **row 1 (supervisor shutdown):** unchanged for the child, and applies to + the child's whole scope; +- **row 4 (exits on its own):** stuck after one interval, unchanged for the + actor itself; its owned members are killed at that same instant (the scope + deadline), then the scope is stuck one interval later. + +**Model** (`MCStartScope`). One linked child of a managed parent, owned +(group `G` or `G2`) or not (unmanaged), trapping or not, which may unlink. +P's scope may close. The model covers P's D and its publication with the +polite `Shutdown`, a scope deadline armed from P's exit begin that reaches +exiting members, and the generation end. Properties and controls: + +- `OwnedOnlyWhileOwnerOpen`, control `OwnerUnchecked`; +- `GenDoneMeansOwnedDone` (P published and every member published), + controls `EarlyGenDone` and `GenDoneAtParentD`; +- `DeadlineKillsAllOwned`, control `NoScopeKill`, which spares exiting + members; +- `ParentSlotHeld`, control `ParentSlotAtD`; +- liveness `OwnedScopeFinishes` without assuming the scope deadline, control + `NoPoliteStop`. + +The parent-child-grandchild case is not modelled. Transitive membership +means the root scope's deadline acts on one member set, and the integration +and sim tests in 16.1 use three levels. ## 5. The startup cell and `start_link`/`start_monitor` ### 5.1 The cell -Each start that has a waiter (`start`, `start_link`, `start_monitor`), and -each start with an init timer, gets one cell. Its two parts: +**Storage for every start** (Codex R5). Every actor's shared state, on every +verb including plain `spawn`, holds a `StartupResult`: -```text -commitment: Pending -> Ready | Failed(cause, reason) | Ignored | TimedOut (exactly once) -waiter: Waiting -> Consumed | Abandoned (exactly once) -``` +- an atomic state word: `Pending`, `Ready`, `Failed`, `Ignored` or + `TimedOut`; +- a `OnceLock` with the diagnostic (`InitFailed` cause) and the requested + reason; +- the commitment instant, which anchors the cleanup deadline. + +It is written once and never changes. It costs one word and one uninitialised +`OnceLock` in the `Arc` that `spawn` already allocates. The run loop only +writes it once at the end of `started()`, so nothing is added on send or +dispatch (benchmark gate, 16.1). + +**The waiter cell** exists only for a waiting start +(`start`/`start_link`/`start_monitor`) or a timed one. It holds the waiter +state (`Waiting`, then `Consumed` or `Abandoned`), the diverted notice slot, +the init timer, and the start marks the edges refer to. It is separate from +the result, so `ready()` never depends on a waiter. -The cell has one lock. The edges created at the commit carry the cell's mark. +**The readiness commit** runs at the end of `started()` when row 10 of 2.3 +applies. It takes, in the order of section 3: -**Ready.** The child commits it at the end of `started()` under the cell -lock, when row 10 of 2.3 applies and (for managed actors) admission is still -open. The same critical section: +1. the child's admission gate (its group's, and for an owned child, its + owner's); +2. the cell lock. -1. commits `Ready`; -2. disarms the init timer; -3. clears the start marks on the edges; +Under both, it checks that admission is open, then: + +1. CASes `Pending → Ready` (the timer's CAS cannot win after this); +2. disarms the timer; +3. sets the waiter cell to "ready", which makes every later notice of this + instance ordinary (the edges' start marks are read through the cell); 4. publishes the result to the waiter and to `ready()`. -Only then does the child run its continuation or dispatch. Readiness is -final: a later death of the child is an ordinary death with an ordinary -`Exit`/`Down`, whether or not the waiter has polled yet (agy P0-1, Codex 1). -OTP sends `init_ack` before entering the loop (`gen_server.erl:2185-2201`). -Model: `ReadyIsFinal` (control `DivertAfterReady`), `ResultMatchesCommit`, +Only after releasing does the child run its continuation or dispatch. Scope +or group closure takes the same admission gate, so it either precedes the +commit (row 9) or follows it (the child is ready and is stopped later). Model: `ReadyOnlyWhileOpen` (control `ReadyAfterClose`). -**Failed / Ignored.** The child commits it at the end of `started()` (rows -2-9), or at a kill before `started()` began. +**Readiness is final** (agy P0-1, Codex 1). A later death of the child is an +ordinary death with an ordinary `Exit`/`Down`, whether or not the waiter has +polled. OTP sends `init_ack` before entering the loop +(`gen_server.erl:2185-2201`). Model: -**TimedOut.** The framework-owned init timer commits it (section 6). +- `CommitImmutable`: a first-commit history `commit0` that no action may + change (control `RewriteCommit`); +- `ReadyIsFinal` (control `AnswerReadyAsFailure`); +- `NoLostNotice` (control `MarkKeptAfterReady`: readiness leaves the marks, + so a post-ready notice is diverted and lost). -Exactly one of these wins the `Pending` state. In particular, the timer can -win only while the cell is `Pending`, and the child's readiness commit can -win only while the timer has not committed (`TimeoutNeverKillsReady`, control -`TimerIgnoresCommit`; agy P0-2, Codex 2). +**Failed / Ignored** are committed by the child at the end of `started()` +(rows 2-9), or at a kill before `started()` began. **TimedOut** is committed +by the framework-owned init timer (section 6). Exactly one CAS wins `Pending`: +`TimeoutNeverKillsReady` (control `TimerIgnoresCommit`). ### 5.2 Edges, death and publication -The child's death has three steps, as in otp-exit-signals: +The child's death has four steps (otp-exit-signals phases 4-6, with +publication split for the trace mapping; Codex R8): -1. **D.** The edges close and their notices are taken. +1. **D.** The final exit reason is fixed. The edges close and their notices + are taken. 2. **Destruction.** Queued messages and the state `A` are dropped, outside framework locks. -3. **Publication**, in this order: - 1. release the name; - 2. release the admission slot; - 3. dispose of each taken notice through the cell; - 4. publish `wait_exit`; - 5. answer the waiter. - -The waiter answers only from publication (agy P0-3, Codex 5), so a retry never -overlaps the old instance's `Drop`. This is OTP's completed-failure boundary, -since `sync_start` awaits the `'DOWN'`. Model: `FailureAfterPublication` -(control `EarlyFailAck`), `NoOverlapOnReuse` (control `UnregisterAtDeath`), -`NameFreeBeforeObservation` (control `LateUnregister`). - -**Disposition of a taken notice** under the cell lock at publication: - -| Cell | Waiter | Trapping parent (`Exit`) | Non-trapping parent | Monitor (`Down`) | +3. **Release.** The name is released (registry lock), then the admission + slot (group lock). +4. **Publication.** Under the cell lock, each taken notice is disposed of by + the commitment and the waiter state (table below). Deliveries happen + after the lock is released. Then `wait_exit` is published, and answers + become available. + +The waiter answers only from publication, so a retry never overlaps the old +instance's `Drop` and the reported reason is final. Model: + +- `FailureAfterPublication` (control `EarlyFailAck`); +- `ReportedReasonFinal` (control `FrozenReason`); +- `NameKeptWhileLive` and `NoOverlapOnReuse` (control `UnregisterAtDeath`); +- `NameFreeBeforeObservation` (control `LateUnregister`); +- `SlotHeldUntilReleased` (control `EarlySlotRelease`). + +**Disposition of a taken notice at publication** (agy F1 fixes the +`TimedOut` rows): + +| Commitment | Waiter | Trapping parent (`Exit`) | Non-trapping parent | Monitor (`Down`) | |---|---|---|---|---| -| `Ready` (any waiter) | any | delivered | applied if non-`Normal` (X3) | delivered | -| `Failed`/`Ignored` | `Waiting` | diverted to the cell, consumed by the error | **applied if non-`Normal`**: the parent exits after its current handler (OTP: the link kills the caller) | diverted, consumed | -| `Failed`/`Ignored` | `Consumed` (bounded wait expired) | dropped | applied if non-`Normal` | dropped | -| `TimedOut` | `Waiting` / `Consumed` | dropped | **not applied** | dropped | -| any non-`Ready` | `Abandoned` | delivered | applied if non-`Normal` | delivered | - -The timeout rows are OTP's `kill_flush_EXIT`, which unlinks before the kill. -Spawned keeps the edge and suppresses its notice instead, which is the same -observably, and the waiter decides at publication. Model: `FailureFlushed` -(controls `NoLinkFlush`, `NoMonitorFlush`), `TimeoutContained` (control -`KillBeforeUnlink`), `NoticeOnce` (control `DoubleNotice`). +| `Ready` | any | delivered | applied if non-`Normal` (X3) | delivered | +| `Failed`/`Ignored` | `Waiting` | diverted to the cell; the error consumes it, abandonment hands it on | **applied if non-`Normal`**: the parent exits after its current handler (OTP: the link kills the caller); `Normal` ignored | diverted (as for `Exit`) | +| `TimedOut` | `Waiting` | diverted; `InitTimeout` consumes it, abandonment hands it on | **diverted**: `InitTimeout` consumes it, so P is not killed (`kill_flush_EXIT`); abandonment hands it on, so a non-trapping P then dies `Killed` | diverted | +| `Failed`/`Ignored`/`TimedOut` | `Consumed` (answered at the cleanup deadline) | dropped | `Failed` non-`Normal`: applied; otherwise dropped | dropped | +| any non-`Ready` | `Abandoned` or none | delivered | applied if non-`Normal` | delivered | + +A notice is never lost: every taken notice is delivered, diverted, applied, +or suppressed by a consumed failure. Model: `NoLostNotice` (the model +records any other drop), `FailureFlushed` (controls `NoLinkFlush`, +`NoMonitorFlush`), `FailureEndsNonTrappingParent` (control +`NoParentKillOnFailure`), `NormalFailureSparesParent` (control +`KillOnNormalFailure`), `TimeoutContained` (control `KillBeforeUnlink`), +and `NoticeOnce` (control `DoubleNotice`). + +**The parent's own death** follows the same steps (Codex R2): its link +notices to children are taken at P's D, delivered at P's publication, and +P's slot is released at P's publication. Model: `ParentSlotHeld` (control +`ParentSlotAtD`), `ParentExitReachesChild`. ### 5.3 Parent state @@ -514,42 +653,61 @@ observably, and the waiter decides at publication. Model: `FailureFlushed` ### 5.4 Abandoning the wait Dropping the `Starting` sets the waiter to `Abandoned` under the cell lock. An -abandoned wait never consumes anything: +abandoned wait never consumes anything. -- **Already diverted.** A notice already diverted to the cell is handed to - the parent as an ordinary notice, exactly once (`AbandonedNoticeDelivered`, - which checks the kind and the reason; control `AbandonDropsNotice`). -- **Later.** A later notice is ordinary (table above). -- **Timer.** The init timer is unaffected (section 6). +- **Already diverted.** A diverted notice is handed to the parent as an + ordinary notice, exactly once, after the lock is released. This applies to + every commitment and both trap settings, including a `TimedOut` notice to a + non-trapping parent, which then dies `Killed`. +- **Later.** A later notice is ordinary (5.2). +- **The timer and the result** are unaffected. -An abandoned `start` is `spawn`. +Model: `AbandonedNoticeDelivered`, which checks the kind, the reason and the +parent's exit for a non-trapping parent (control `AbandonDropsNotice`), and +`NoLostNotice`. An abandoned `start` is `spawn`. ### 5.5 Answering -| Committed | Waiter answers | -|---|---| -| `Ready` | at once: `Ok(Started(..))` | -| `Ignored` | after publication: `Ok(Ignored)` | -| `Failed` | after publication: `InitFailed`/`InitExited { pending: None }` | -| `TimedOut` | after publication: `InitTimeout { pending: None }` | -| any failure, publication not reached by the bounded wait | at the bound: `pending: Some(handle)` | - -The bounded wait is the placement's cleanup deadline: 5 s unmanaged, or the -group's or spec's otherwise. `ready()` reads the same cell and answers the -same value, at any time and any number of times (agy P1-6). Model: -`StartAnswered` (liveness; control `NoAnswerIgnore`). +| Committed | Waiter answers | `ready()` answers | +|---|---|---| +| `Ready` | at once: `Ok(Started(..))` | `Ok(Started(()))` | +| `Ignored` | after publication: `Ok(Ignored { pending: None })`; at the cleanup deadline: `Ok(Ignored { pending: Some(h) })` | the same | +| `Failed` | after publication: `InitFailed`/`InitExited { reason: final, pending: None }`; at the cleanup deadline: `{ reason: requested, pending: Some(h) }` | the same | +| `TimedOut` | after publication: `InitTimeout { pending: None }`; at the cleanup deadline: `{ pending: Some(h) }` | the same | + +**The cleanup deadline** is one absolute instant per instance (Codex R5, +R6): the commitment instant plus the placement's cleanup interval (5 s +unmanaged; otherwise the group's or the spec's). Every observer shares it. +Before it, an observer of a failed, ignored or timed-out instance waits for +publication. After it, an observer answers with `pending: Some` if +publication has not happened, and `pending: None` once it has. + +**Repeated observations** sample the cleanup status: the commitment kind and +the diagnostic never change. A `ready()` after publication answers +`pending: None` even if an earlier observer got `Some`. Only the waiter +consumes its notice. + +Model: + +- `PendingOnlyAfterDeadline`, with an explicit `CleanupDeadline` step that + is not assumed to fire (control `EarlyPending`); +- `ResultMatchesCommit` against `commit0` (controls `AckBeforeStarted`, + `ErrorAsOk`); +- `StartAnswered` (liveness, control `NoAnswerIgnore`). ## 6. Init timeout `init_timeout(d)` arms a framework-owned timer at the commit, only when the -option is set. The waiter does not own it, so it runs whether or not anyone +option is set. The timer is stored in the waiter cell, or for a timed +`spawn`, in a cell created for it. The waiter does not own it, so it runs whether or not anyone polls, and abandonment does not change it (Codex 2). At expiry, under the cell lock: -- **Cell `Pending`:** commit `TimedOut`, then send a kill. The start edges - stay. Their notice is disposed of at publication by the waiter: suppressed - while waiting or consumed, ordinary if abandoned (5.2). +- **Commitment `Pending`:** commit `TimedOut`, then send a kill. The start + edges stay. Their notice is disposed of at publication by the waiter: + diverted while waiting (consumed by `InitTimeout`, handed on if the wait is + later abandoned), dropped once consumed, ordinary if abandoned first (5.2). - **Already committed:** nothing happens (`TimeoutNeverKillsReady`). **Stuck policy (explicit, agy P1-5).** The kill is applied cooperatively, diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg index a23041b4..9d50acad 100644 --- a/verification/tla/MCStartCap.cfg +++ b/verification/tla/MCStartCap.cfg @@ -9,7 +9,7 @@ CONSTANTS TimeoutEnabled = FALSE KillEnabled = FALSE AdminStopEnabled = FALSE - ParentCanExit = TRUE + ParentCanExit = FALSE CloseEnabled = TRUE OwnerCloseEnabled = FALSE MaxMsgs = 0 diff --git a/verification/tla/MCStartCap.tla b/verification/tla/MCStartCap.tla index b471feae..3ab77831 100644 --- a/verification/tla/MCStartCap.tla +++ b/verification/tla/MCStartCap.tla @@ -2,6 +2,6 @@ EXTENDS Start \* Per-PR instance for capacity: a managed parent and three children \* spawned into its group of three: slots (P's included) are held until -\* release (P's until its publication), a start is refused while they are -\* held, and a retry gets a slot after a release. +\* release, a start is refused while they are held, and a retry gets a +\* slot after a release. (P's own slot is checked in MCStartScope.) ============================================================================= From ef4457e42bcbb26ec2265d9334842a5d60a01851 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:00:01 -0300 Subject: [PATCH 24/42] verification: NormalFailureSparesParent keys on the final reason --- verification/tla/Start.tla | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index b0b84ee2..46a94290 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -737,10 +737,11 @@ FailureEndsNonTrappingParent == \A c \in Kids : (kind[c] = "link" /\ ~ptrap /\ commit0[c] = "failed" /\ why[c] \in NonNormal /\ phase[c] = "published" /\ lnote[c] /\ pubAlive[c]) => pst # "run" -\* ... and a Normal one (error, ignore, stop(Normal)) never does. +\* ... and one whose final reason is Normal (error, ignore, stop(Normal), +\* not escalated by a kill before D) never does. NormalFailureSparesParent == \A c \in Kids : - (kind[c] = "link" /\ (commit0[c] = "ignored" \/ (commit0[c] = "failed" /\ why[c] = "Normal"))) + (kind[c] = "link" /\ commit0[c] \in {"failed", "ignored"} /\ why[c] = "Normal") => pcause # c TimeoutContained == From 5fd2d0500c6c2909da37214718e987b231965d9b Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:04:25 -0300 Subject: [PATCH 25/42] verification: MCStart without group closure (covered in Cap and Timeout) --- docs/design/unified-start.md | 131 ++++++++++++++++++++++++----------- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStart.tla | 7 +- 3 files changed, 96 insertions(+), 44 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index cdbc9745..9b80c8e2 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -799,11 +799,22 @@ ChildSpec::new(id, &group, |fctx| async { Ok(MyActor::new(..)) }) // or Ok(Chi outcome. - **`startup_timeout` stays on `ChildSpec`** (agy P1-7, Codex 3). It bounds the whole attempt: factory, stage 1 and `started()`. The default is 5 s - (V7), and `Timeout::Infinity` is allowed. The actor's init timer gets the - remaining budget at the commit, or the smaller of that and - `options.init_timeout`. Expiry during the factory cancels it (V7, unchanged); - expiry after the commit is the cell's `TimedOut`. The dynamic supervisor - bounds construction and `started()` together the same way. + (V7), and `Timeout::Infinity` is allowed. The dynamic supervisor bounds + construction and `started()` together the same way. The attempt carries + one **absolute deadline**, taken when the attempt begins (Codex R9, agy + F5): + - **during the factory:** expiry cancels it (V7, unchanged) and gives + `StartupTimeout { pending: None }`; + - **in the prepare step and the commit:** the deadline is rechecked under + the commit locks (section 3). If it has passed, the commit refuses: the + prepared state is dropped, nothing is visible, and the attempt fails + with `StartupTimeout { pending: None }`. No zero-duration `init_timeout` + is ever built, so this is never `InvalidOptions`; + - **after the commit:** the instance's init timer is armed at the same + absolute deadline, or at `options.init_timeout`'s deadline if that is + earlier. Its expiry is the cell's `TimedOut`, reported to the supervisor + as `Start(InitTimeout { pending })` and treated as a startup timeout + under V1/V7. - **Placement.** The positional group stays: a supervised child is always managed, in the supervisor's system. A `group` in `options` replaces it; a `backend` is `InvalidOptions` at `add()`. @@ -830,6 +841,17 @@ ChildSpec::new(id, &group, |fctx| async { Ok(MyActor::new(..)) }) // or Ok(Chi stamped (V5) and adds nothing. - **Waiting.** `SupervisedChild::wait_ready()` and `current()` return `ChildUnavailable(ChildState::Ignored)` at once rather than waiting. + - **The controller loop** (agy F6). An `Ignored` child's controller stays + parked. If supervisor shutdown comes, it reports done at once. If a cohort + round comes, it answers the round's stop and clean barriers (`CStop`, + `CClean`) at once, since it has no generation, then waits on the round's + resume (`CResume`) and starts its next attempt in declaration order. + - **A stuck `ignore`** (Codex R4). The ignoring instance is observed as + `Ignored` only once it is published. Until then its slot stays `Stopping` + and owned, under V3's stuck rule: cleanup deadline, then stuck. A cohort + barrier treats it as not yet clean, so no replacement overlaps it. The + spec becomes `Ignored` only after publication. A direct waiter gets + `Ok(Ignored { pending: Some(h) })` at the cleanup deadline (5.5). - **Model and traces.** `Supervision.tla` and `Cohorts.tla` gain the `Ignored` state and the cohort rule, and traces gain an `Ignore(c, g)` event between `Attempt` and the round's barrier, with `trace_check.py` @@ -942,15 +964,18 @@ Revision 2 corrects the mapping on two points: The implementation PR writes these into `semantics.md`, with the "Checked by" lines from section 16. -- **L1 (changed).** `stopped()` runs after a ready commit, or after a - `ctx.stop*` called in `started()`; never after a failed or ignored init. +- **L1 (changed).** `stopped()` runs only after a ready commit; never after + a failed or ignored init, including a `ctx.stop*` called in `started()` (as + OTP skips `terminate/2` when `init/1` fails). - **L2 (changed).** Readiness exists for every placement. It is the startup cell's `Ready`: committed by the child at the end of `started()`, while admission is open, before any continuation or dispatch. It is final. `ready()` and the waiting starts read the same committed value. -- **L8 (new), Start.** Stage 1 is one commit (section 3), and a refusal is - never observable. The waiting starts answer per 5.5. Errors come after - publication or carry `pending`. +- **L8 (new), Start.** Stage 1 is one commit under the global lock order, + and a refusal is never observable (section 3). Every instance has an + immutable startup result (5.1). The waiting starts and `ready()` answer per + 5.5: errors come after publication with the final reason, or after the + cleanup deadline with `pending` and the requested reason. - **L9 (new), Init resolution.** The precedence table in 2.3, including trapping children. - **L10 (new), Continuation.** It precedes every mailbox entry. It is not a @@ -959,12 +984,19 @@ lines from section 16. child's `started()`, on every placement. A start through an exiting or dead parent is refused. - **X3 (changed).** The parent row reads "`start_link` caller". -- **X8 (new), Startup notices.** The disposition table in 5.2. The init - timeout never reaches the parent unless the wait was abandoned. An - abandoned wait delivers a diverted notice exactly once. +- **X8 (new), Startup notices.** The disposition table in 5.2: + - no taken notice is ever lost; + - the init timeout never reaches the parent unless the wait was abandoned; + - an abandoned wait delivers a diverted notice exactly once; + - a parent's notices to its children are delivered at the parent's + publication. - **S6 (changed).** `spawn`/`start` are unmanaged unless given a placement. - **S7 (new), Inheritance.** Section 4, with no fallback. -- **S8 (new), Scope ownership.** Section 4.1. +- **S8 (new), Scope ownership.** Section 4.1: ownership is attached under + the owner's gate. At the parent's publication, its owned members get a + polite `Shutdown`. The scope deadline is anchored at the parent's exit + begin and kills members directly. The generation ends after the parent and + every member are published. The E3 rows are revised. - **S9 (new), Start-time names.** Committed atomically, unique, held until publication, released before any notice. - **T5 (new), Init timeout.** A framework-owned timer that loses to a ready @@ -1121,18 +1153,24 @@ guarantee breaks. | X5 parent exited | `NoEdgeToDeadParent` | `start_link_through_dead_context_is_refused`; `start_link_from_stopped_is_refused` | `parent_stops_during_child_started` | | | | X8 disposition | `FailureFlushed`, `NoticeOnce`, `TimeoutContained` | `failed_start_link_consumes_exit_of_trapping_parent`; `failed_start_link_kills_nontrapping_parent_after_handler`; `failed_start_monitor_consumes_down`; `init_timeout_never_reaches_the_parent` | `failure_races_ack` | no notice for a start that returned an error; at most one per child | | | X8 abandonment | `AbandonedNoticeDelivered` | `abandoned_start_monitor_still_delivers_down`; `abandoned_start_link_delivers_exit`; `abandoned_start_with_timeout_delivers_killed` | `abandon_races_publication` | | | -| L8 failure after publication (P0-3) | `FailureAfterPublication`, `NoOverlapOnReuse` | `state_drop_finishes_before_failure_answer_and_name_reuse` (`Mock::drop_flag`); `hung_drop_answers_with_pending_handle_at_the_bound` | `retry_never_overlaps_old_drop` | no two live states of one name | | +| L8 failure after publication (P0-3) | `FailureAfterPublication`, `NoOverlapOnReuse`, `ReportedReasonFinal`, `PendingOnlyAfterDeadline` | `state_drop_finishes_before_failure_answer_and_name_reuse` (`Mock::drop_flag`); `hung_drop_answers_with_pending_handle_at_the_bound`; `kill_between_resolution_and_d_reports_killed` (round 2) | `retry_never_overlaps_old_drop`; `kill_races_d_after_failed_init` | no two live states of one name; the reported reason equals `wait_exit`'s | | +| Stuck `ignore` (Codex R4) | `FailureAfterPublication`, `PendingOnlyAfterDeadline` | `hung_drop_after_ignore_answers_ignored_with_pending`; `supervised_stuck_ignore_keeps_the_slot_stopping` | | | | +| `ready()` storage (Codex R5) | `CommitImmutable` | `spawn_ready_reports_init_failed_with_its_diagnostic`; `ready_samples_pending_then_published` (two observers on either side of publication) | | | | +| Parent publication (Codex R2) | `ParentSlotHeld`, `GenDoneMeansOwnedDone` | `parent_hung_in_drop_keeps_children_unnotified_and_slot_held`; `generation_waits_for_parent_publication` | | | | +| S8 polite stop (agy F3) | `OwnedScopeFinishes`, `DeadlineKillsAllOwned` | `owned_unlinked_child_gets_shutdown_at_parent_publication`; `owned_nontrapping_child_stops_on_normal_parent_exit`; `scope_deadline_kills_a_member_stuck_in_stopped`; `nested_scope_deadline_is_the_earliest`; three-level tree | `parent_stuck_before_d_does_not_delay_member_kill` | | | +| Owner gate (Codex R1) | `OwnedOnlyWhileOwnerOpen`, `ReadyOnlyWhileOpen` | `closed_owner_refuses_an_open_target_group`; `owner_closure_racing_ready_commit` | | | | +| Attempt deadline (Codex R9) | | `startup_deadline_expiring_in_prepare_is_startup_timeout_not_invalid_options` | | | | | T5 timer vs readiness (P0-2) | `TimeoutNeverKillsReady`, `ResultMatchesCommit` | `init_timeout_racing_ready_agrees_with_ready()` (failpoint at the commit); `spawn_init_timeout_kills_and_ready_reports_it`; `stuck_started_keeps_name_and_reports_pending` | `timeout_races_ready_commit` | timeout options generated; a ready child never ends `Killed` by its timer | | | `ready()` parity (agy P1-6) | `ResultMatchesCommit` | `ready_returns_the_same_result_as_the_waiter` for `Started`, `Ignored`, `InitFailed`, `InitExited`, `InitTimeout`, via `spawn` and `Starting::actor_ref()` | | | | | S7 inheritance | `InheritedPlacement` | `link_and_monitor_inherit_{async_group,thread_group,pinned_shard,backend_async,backend_blocking,backend_thread}`; `plain_start_never_inherits`; `placement_of_opts_in` | | placement oracle | `threads_options_have_no_group_or_backend` | | S7 closed/full | `AdmittedOnlyOpen`, `CapacityRespected` | `inherited_start_into_closing_group_fails_closed`; `..._full_thread_group_fails_capacity`; `capacity_released_only_at_publication` | `group_shutdown_during_start` | | | | S8 ownership | `GenDoneMeansOwnedDone`, `OwnedScopeFinishes` | `system_shutdown_stops_owned_children_with_their_parent`; `supervised_generation_waits_for_owned_children`; `unlinked_owned_child_is_killed_at_the_scope_deadline`; `stuck_parent_does_not_delay_descendant_kill`; `foreign_system_and_unmanaged_children_are_not_owned` | `parent_shutdown_racing_child_start` | S1-S4/V3 oracles see owned children | | -| S9 names | `NameUnique`, `NameHeldUntilPublished`, `NameFreeBeforeObservation` | `name_clash_returns_already_started_with_holder` (the holder kept alive across both calls); `failed_named_start_can_be_retried_at_once`; `name_released_before_down_and_exit`; `typed_name_whereis`; `unregister_then_reregister_keeps_ownership`; `cross_type_name_clash_reports_none_holder` | `name_registration_race` | one live holder | `name_type_must_match_actor` | +| S9 names | `NameUnique`, `NameHeldUntilReleased`, `NameKeptWhileLive`, `NameFreeBeforeObservation` | `name_clash_returns_already_started_with_holder` (the holder kept alive across both calls); `failed_named_start_can_be_retried_at_once`; `name_released_before_down_and_exit`; `typed_name_whereis`; `unregister_then_reregister_keeps_ownership`; `cross_type_actor_name_clash_reports_the_holder_handle`; `non_actor_holder_reports_none` | `name_registration_race` | one live holder | `name_type_must_match_actor` | | V1/V5/V7 supervisors | `Supervision`/`Cohorts` `Ignored` (impl PR) | `factory_ignore_and_init_ignore_keep_the_spec`; `ignored_child_restarted_in_one_for_all_round`; `ignored_child_restarted_in_rest_for_one_round`; `temporary_ignore_is_removed`; `wait_ready_on_ignored_returns_unavailable`; `startup_timeout_bounds_a_hung_factory`; `startup_timeout_infinity`; `restart_name_clash_is_a_failed_attempt` (traced, `trace_check.py`) | | V1 oracle treats `Ignored` | | | Options | | `zero_timeout_and_zero_bounds_are_invalid_options` (both modes); `stack_size_on_pooled_placement_is_invalid`; `spawn_panics_on_executor_failure` (failpoint) | | | `start_options_struct_literal`; `start_options_clone_without_actor_clone` (ui-pass) | | API removal | | | | | `start_on_removed`, `start_with_backend_removed`, `start_linked_removed`, `execution_group_start_removed`, `start_monitor_requires_down_handler` | | No cost on send/dispatch | | | | | benchmark gate: the default send, request and dispatch benchmarks within noise of 0.8 (`benchmarks/`, the gate introspection #256 uses); `future_layout` sizes unchanged for the run loop | -| Lock order | | Loom: `commit_races_parent_exit_begin`, `two_commits_one_name`, `ready_commit_races_timer` (the cell's CAS) | | | | +| Lock order | | Loom: `commit_races_parent_exit_begin`, `two_commits_one_name`, `ready_commit_races_timer` (the cell's CAS), `ready_commit_races_scope_close`, `cross_group_commits` | | | | **Placements.** Every integration test runs in: @@ -1148,36 +1186,46 @@ They use a per-placement macro, as `semantics_gaps` does. `concurrency/tests/start_trace.rs` records every scenario in 16.1 on `Async`, `async_group` and `thread_group`. `TraceStart.tla` replays each trace as a behaviour of `Start.tla`, and `trace_check.py` gains schema `start-v1`. - -Every event names the code point at which it is recorded, which is its -linearization point. +Every event names the code point at which it is recorded (its +linearization point), and every event has exactly one consumable model +transition (Codex R8). | Event | Fields | Model action | Linearization point (code) | |---|---|---|---| -| `SConfig` | parent placement, parent trap flag, Cap | `Init` | trace start | +| `SConfig` | parent placement, parent trap flag, Cap, owner gate | `Init` | trace start | | `SBegin` | c, kind, explicit, named | `Begin` | entry of the start call | | `SPrepareFail` | c, error | `PrepareFail` | the prepare step's error return | | `SCommit` | c, place, edge kind, named, owned | `Commit` | inside the commit section, all locks held | -| `SRefuse` | c, reason | `Commit` (refusal) | inside the commit section | -| `SLookup` | c | `Lookup` | `whereis` under the registry read lock (test probes only) | +| `SRefuse` | c, reason, blamed holder | `Commit` (refusal) | inside the commit section | +| `SLookup` | name, hit c or miss | `Lookup` (hit) or a stuttering step whose guard requires the name free (miss) | `whereis`, under the registry lock (test probes) | | `SStartedBegin` | c | `StartedBegin` | before the first poll of `started()` | -| `SResolve` | c, outcome, trap flag, held signals, committed cell | `StartedEnd` / `KilledEarly` | under the cell lock at the end of `started()` | -| `STimer` | c, won | `Timeout` | under the cell lock in the timer | -| `SContinue` / `SHandle` | c | `RunContinue` / `Handle` | the start of the turn | -| `SExit` | c, reason | `Exit` (via `ReadyExit`, `KillReady`, `ApplyParentExit`, `HandleQueuedExit`) | exit begin (0.8 phase 2) | +| `SAdminStop`, `SKill` | c | `AdminStop`, `OutsideKill` | the signal's arrival (exit-request lock) | +| `SResolve` | c, outcome, trap flag, held signals, commitment, requested reason | `StartedEnd` / `KilledEarly` | under the cell lock at the end of `started()`; every pre-readiness exit is recorded here and never as `SExit` (agy F8) | +| `STimer` | c, won or lost | `Timeout` (won) or a stuttering step whose guard requires a commitment (lost) | under the cell lock in the timer | +| `SDeadline` | c | `CleanupDeadline` | the cleanup-deadline timer | +| `SContinue` / `SHandle` | c | `RunContinue` / `Handle` / `HandleQueuedExit` | the start of the turn | +| `SExit` | c, reason | `ReadyExit`, `KillReady`, `ApplyParentExit`, `HandleQueuedExit` (post-readiness only) | exit begin (0.8 phase 2) | +| `SReason` | c, new reason | `KillExiting` | a pre-D reason change (exit-request lock) | | `SUnlink` | c | `ChildUnlink` | the unlink fence | -| `SDie` | c, taken link note, taken monitor note | `Die` | D (0.8 phase 4) | -| `SPublish` | c, name released, slot released, disposition per note (delivered, diverted, applied, dropped) | `Publish` | publication (0.8 phase 6), under the cell lock | -| `SAnswer` | c, result, pending | `AckOk` / `AckFail` | under the cell lock when the waiter takes the result | +| `SDie` | c, final reason, taken notes | `Die` | D (0.8 phase 4) | +| `SRelease` | c, name released, slot released | `Release` | after destruction; registry then group lock | +| `SPublish` | c, disposition per note (delivered, diverted, applied, dropped) | `Publish` | publication, under the cell lock | +| `SAnswer` | c, result, reason, pending | `AckOk` / `AckFail` | under the cell lock when the waiter takes the result | +| `SReady` | c, observed result | a stuttering step whose guard requires the commitment (and publication or the deadline) to match | `ready()` returning | | `SAbandon` | c, handed-on notice | `Abandon` | `Drop` of `Starting`, under the cell lock | -| `PExitBegin` | reason | `PExit` | the parent's exit begin | -| `PDie` | per child: queued or held parent exit | `PDie` | the parent's D | -| `PNotice` | c, kind, reason | (part of `Publish`/`Abandon`) | enqueue into the parent's mailbox, or the signal applied to it | +| `PExitBegin`, `PDie`, `PPublish` | reason; per child: link note taken; per child: notice and polite Shutdown | `PExit`, `PDie`, `PPublish` | the parent's exit begin, D and publication | +| `POwnerClose` | | `OwnerClose` | the owner's gate closing | +| `PNotice` | c, kind, reason | part of `Publish`/`Abandon` | enqueue into the parent's mailbox, or the signal applied to it | | `GClose`, `GScopeKill`, `GGenDone` | | `GClose`, `ScopeDeadline`, `GenDone` | group close; the scope deadline; the generation-end commit | -- **Complete consumption.** Replay must consume every event, and it may not - invent observable events: every notice the parent sees has its own - `PNotice` and must match the disposition recorded in `SPublish`. +- **Interleavings.** `SRelease` and `SPublish` are separate events because + the release and the disposition happen under different locks. A trace in + which C1's name is released, C2 commits the same name, and then C1 + publishes replays faithfully. A control moves `SRelease` after the + competing `SCommit` and must be rejected. +- **Complete consumption.** Replay must consume every event without inventing + observable events. Every notice the parent sees has its own `PNotice`, + which must match the disposition in `SPublish` or `SAbandon`. - **Fault-only actions** (`RegisterFirst`, `CommitRest`, `LateLink`, `LateRelease`) are excluded from positive coverage. `trace_coverage.py` requires every other action to be reached. @@ -1185,10 +1233,13 @@ linearization point. trace, which the checker must reject: - move `SCommit` after `SStartedBegin`; - give two children overlapping names; - - change a managed child's `SCommit` place to unmanaged; - - add a `PNotice` after an `SAnswer` error for a trapping parent; - - move `SAnswer` (error) before `SPublish`; - - add an `STimer won` after an `SResolve` that committed ready. + - move a `SRelease` after a competing `SCommit`; + - change a managed child's place to unmanaged; + - add a `PNotice` after an error answer to a trapping parent; + - move an error `SAnswer` before `SPublish` without an `SDeadline`; + - add an `STimer won` after a ready `SResolve`; + - change an error answer's reason to the requested one after an `SReason`; + - drop a diverted notice on `SAbandon`. ### 16.3 Simulator diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index cea56d72..fdb1d556 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -10,7 +10,7 @@ CONSTANTS KillEnabled = FALSE AdminStopEnabled = FALSE ParentCanExit = TRUE - CloseEnabled = TRUE + CloseEnabled = FALSE OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index 486597b9..75885872 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -2,7 +2,8 @@ EXTENDS Start \* Per-PR safety instance (design docs/design/unified-start.md, section 15): \* two children racing for one name in a group with room for both and the -\* parent, start_link and start_monitor with their default placement, a -\* managed parent that may exit, group closure, abandoned -\* waits, failure and retry (the init timer is in MCStartTimeout). Most safety controls run here. +\* parent; start_link and start_monitor with their default placement; a +\* managed parent that may exit (D, then publication); abandonment, failure, +\* retry and pre-D escalation-free deaths. Group closure is in MCStartCap +\* and MCStartTimeout; the init timer in MCStartTimeout. ============================================================================= From 8fc94f7178bfb7dcfd7e49bd8db9866cfc24ac4d Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:07:46 -0300 Subject: [PATCH 26/42] verification: revision-3 Start controls --- verification/tla/negative_checks.py | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 9bebbe24..f6a16b12 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -246,31 +246,43 @@ def named_temporal(module, fault, prop, tlc, constants=None): START_CONTROLS = [ ("MCStart", "NoInherit", "InheritedPlacement"), ("MCStart", "LinkAfterStarted", "LinkBeforeStarted"), + ("MCStart", "LinkCheckLate", "EdgeOnlyToRunningParent"), ("MCStart", "DoubleRegister", "NameUnique"), - ("MCStart", "LateUnregister", "NameFreeBeforeObservation"), + ("MCStart", "UnregisterAtDeath", "NameKeptWhileLive"), ("MCStart", "UnregisterAtDeath", "NoOverlapOnReuse"), + ("MCStart", "LateUnregister", "NameFreeBeforeObservation"), ("MCStart", "SplitCommit", "AbortInvisible"), ("MCStart", "EarlyFailAck", "FailureAfterPublication"), + ("MCStart", "EarlyPending", "PendingOnlyAfterDeadline"), ("MCStart", "AckBeforeStarted", "ResultMatchesCommit"), ("MCStart", "ErrorAsOk", "ResultMatchesCommit"), - ("MCStart", "DivertAfterReady", "ReadyIsFinal"), - ("MCStart", "ReadyAfterClose", "ReadyOnlyWhileOpen"), + ("MCStart", "AnswerReadyAsFailure", "ReadyIsFinal"), + ("MCStart", "MarkKeptAfterReady", "NoLostNotice"), ("MCStart", "NoMonitorFlush", "FailureFlushed"), ("MCStart", "NoLinkFlush", "FailureFlushed"), - ("MCStart", "LinkCheckLate", "NoEdgeToDeadParent"), - ("MCStartCap", "AdmitClosed", "AdmittedOnlyOpen"), - ("MCStartCap", "IgnoreCap", "CapacityRespected"), + ("MCStart", "NoParentKillOnFailure", "FailureEndsNonTrappingParent"), + ("MCStart", "KillOnNormalFailure", "NormalFailureSparesParent"), ("MCStart", "AbandonDropsNotice", "AbandonedNoticeDelivered"), ("MCStart", "DoubleNotice", "NoticeOnce"), + ("MCStartTimeout", "RewriteCommit", "CommitImmutable"), ("MCStartTimeout", "TimerIgnoresCommit", "TimeoutNeverKillsReady"), ("MCStartTimeout", "KillBeforeUnlink", "TimeoutContained"), + ("MCStartTimeout", "ReadyAfterClose", "ReadyOnlyWhileOpen"), + ("MCStartTimeout", "FrozenReason", "ReportedReasonFinal"), + ("MCStartCap", "AdmitClosed", "AdmittedOnlyOpen"), + ("MCStartCap", "IgnoreCap", "CapacityRespected"), + ("MCStartCap", "EarlySlotRelease", "SlotHeldUntilReleased"), ("MCStartContinue", "ContinueQueued", "ContinueFirst"), ("MCStartScope", "EarlyGenDone", "GenDoneMeansOwnedDone"), + ("MCStartScope", "GenDoneAtParentD", "GenDoneMeansOwnedDone"), + ("MCStartScope", "NoScopeKill", "DeadlineKillsAllOwned"), + ("MCStartScope", "OwnerUnchecked", "OwnedOnlyWhileOwnerOpen"), + ("MCStartScope", "ParentSlotAtD", "ParentSlotHeld"), ] START_TEMPORAL_CONTROLS = [ ("MCStartLive", "DropHeld", "ParentExitReachesChild"), ("MCStartLive", "NoAnswerIgnore", "StartAnswered"), - ("MCStartScope", "NoScopeKill", "OwnedScopeFinishes"), + ("MCStartScope", "NoPoliteStop", "OwnedScopeFinishes"), ] From 1c91de34595dabcdfc274a8c8dbd1449976e72d5 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:10:05 -0300 Subject: [PATCH 27/42] verification: MCStart (one child) and MCStartNames per PR; the 30M instance becomes MCStartDeep --- docs/design/unified-start.md | 24 ++++++++++++++++++++++++ verification/tla/MCStart.cfg | 4 ++-- verification/tla/MCStart.tla | 10 +++++----- verification/tla/MCStartDeep.cfg | 6 +++--- verification/tla/MCStartDeep.tla | 9 ++++----- verification/tla/MCStartNames.cfg | 20 ++++++++++++++++++++ verification/tla/MCStartNames.tla | 6 ++++++ verification/tla/negative_checks.py | 10 +++++----- 8 files changed, 69 insertions(+), 20 deletions(-) create mode 100644 verification/tla/MCStartNames.cfg create mode 100644 verification/tla/MCStartNames.tla diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 9b80c8e2..49ba707c 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1376,3 +1376,27 @@ and the model. | agy P2-11 | `NoticeOnce` had no control and used sets: true | Counters, control `DoubleNotice`; `MCStartContinue` runs every explicit placement | | agy P2-12 | | `init_fail/3` cited; `Init::stop(Normal)` documented (2.3) | | agy P2-13 | | All six tests added (16.1) | + +### Round 2 (revision 3) + +Both reviews: approve after the listed changes, no P0. Codex confirmed the +rejection of agy P1-10. Each finding was checked against the design, the +model and the 0.8 code. + +| Finding | Claim checked | Resolution | +|---|---|---| +| Codex R1 = agy F4: the atomic protocol stops before readiness and scope admission; the lock hierarchy is incomplete | **True.** The readiness commit's locks were unspecified. Owner-scope admission was not checked, so a closed owner could start an owned child in another open group. `with_pair` and cross-group scope locks were unordered | **Accepted.** One global lock order, owner gate first and cell last, covers the commit, the readiness commit, the timer, release and publication. The child's table is filled in prepare, so the commit locks only the parent's table. At most one group lock is held at a time, and `Scope` state stays a leaf (section 3). Ownership is attached under the owner's gate. Modelled: `ownerOpen`, a second group `G2`, `OwnedOnlyWhileOwnerOpen` (control `OwnerUnchecked`), and readiness checks the owner gate (`ReadyOnlyWhileOpen`). Loom tests listed | +| Codex R3 = agy F2: the precedence table freezes the final reason | **True.** A `ctx.stop(r)` then a kill during `stopped()` must end `Killed` | **Accepted, agy's option A.** No failed row runs `stopped()`, `ctx.stop*` in `started()` included, as OTP never calls `terminate` after a failed `init`. The table now decides the immutable startup decision and the requested reason; the final reason is fixed at D by 0.8's precedence, and post-publication answers report it. Administrative stops get row 3. Modelled: `KillExiting` (pre-D escalation), `ReportedReasonFinal` (control `FrozenReason`) | +| Codex R2: parent D treated as publication | **True.** `PDie` delivered notices and released P's slot at D | **Accepted.** P has D and publication. Its notices to children and its slot release come at publication, and the generation ends only after P's publication and every member's (section 4.1, 5.2). Modelled: `PPublish`, `ParentSlotHeld` (control `ParentSlotAtD`), `GenDoneMeansOwnedDone` requiring P published (controls `EarlyGenDone`, `GenDoneAtParentD`) | +| agy F3: owned children get no polite stop | **True** for unlinked members, and for non-trapping linked ones under a `Normal` parent exit. `ScopeDeadline` missed `exiting` | **Accepted.** At P's publication the scope sends every live owned member a polite parent `Shutdown` (E3). The scope deadline reaches `exiting` members. Modelled: liveness `OwnedScopeFinishes` without assuming the scope deadline (control `NoPoliteStop`), `DeadlineKillsAllOwned` (control `NoScopeKill`, which spares `exiting`) | +| Codex R6 = agy F1: controls narrower than claimed; `TimedOut` notice dropped while waiting; publish-then-abandon loses a non-trapping parent's kill | **True.** Revision 2 dropped a waiting `TimedOut` notice for a non-trapping parent, so abandonment after publication lost it. `ReadyIsFinal` used the current commitment, and `DivertAfterReady` mixed two bugs | **Accepted.** `TimedOut` with a waiter is diverted for both trap settings; the `InitTimeout` answer consumes it, and abandonment hands it on. Each control now targets one property: `commit0` history with `CommitImmutable` (control `RewriteCommit`); `NoLostNotice`, which the model sets on any non-legitimate drop (control `MarkKeptAfterReady`); `ReadyIsFinal` (control `AnswerReadyAsFailure`); `AbandonedNoticeDelivered` keyed on the hand-off, checking a non-trapping parent's exit; the explicit `CleanupDeadline` with `PendingOnlyAfterDeadline` (control `EarlyPending`); `FailureEndsNonTrappingParent` and `NormalFailureSparesParent` (controls `NoParentKillOnFailure`, `KillOnNormalFailure`); `EdgeOnlyToRunningParent` (control `LinkCheckLate`); `NameKeptWhileLive`; `SlotHeldUntilReleased` (control `EarlySlotRelease`); refusal-blame history in `AbortInvisible` | +| Codex R4: stuck `ignore` | **True** | **Accepted.** `StartOutcome::Ignored { pending }`. `Ok(Ignored)` without pending needs publication; a supervised stuck ignore keeps its slot `Stopping` under V3 until published (5.5, 9) | +| Codex R5: `ready()` storage and repeated observation | **True** | **Accepted.** A `StartupResult` in every actor's shared state, plain `spawn` included, separate from the waiter cell. The cleanup deadline is one absolute instant from the commitment; repeated observations sample cleanup status while the commitment and diagnostic stay immutable (5.1, 5.5) | +| Codex R7: deadline composition | **True** | **Accepted.** Absolute deadlines that compose by minimum and never extend; the scope deadline is anchored at P's exit begin; nested scopes take the earliest ancestor deadline; the E3 rows are revised explicitly (4.1). A parent stuck before D is modelled (the deadline is enabled from exit begin). The three-level case is argued and tested, not modelled | +| Codex R8: publication trace mapping | **True** | **Accepted.** `Release` and `Publish` are separate model actions and events. Every recorded event has a consumable transition: lookup hit and miss, timer won and lost, `ready()` observations, cleanup deadline, pre-D reason changes. New trace controls cover the exposed orderings (16.2) | +| Codex R9 = agy F5 (P2): attempt deadline during prepare | **True** | **Accepted.** An absolute attempt deadline is rechecked in the commit; expiry is `StartupTimeout`, never a zero `init_timeout` (9) | +| Codex R10 (P2): name API | **True** | **Accepted.** `Name` API restored (2.4); the cross-type actor clash test expects the holder's handle | +| agy F6 (P2): ignored controller in cohort rounds | **True** | **Accepted.** The parked controller answers the stop and clean barriers at once and waits for resume (9) | +| agy F7 (P2): model gaps | **True** | **Accepted.** `stop(Normal)` and `stop(Shutdown)` outcomes; `NormalFailureSparesParent`; `NoOverlapOnReuse` over all live phases; `commit0` | +| agy F8 (P2): `SExit` vs `SResolve` | **True** | **Accepted.** Pre-readiness exits are recorded only as `SResolve` (16.2) | + diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index fdb1d556..0db9e198 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -1,11 +1,11 @@ SPECIFICATION Spec CONSTANTS - Kids = {"c1", "c2"} + Kids = {"c1"} Kinds = {"link", "monitor"} Explicit = {"none"} ParentPlaces = {"G"} Cap = 3 - Named = TRUE + Named = FALSE TimeoutEnabled = FALSE KillEnabled = FALSE AdminStopEnabled = FALSE diff --git a/verification/tla/MCStart.tla b/verification/tla/MCStart.tla index 75885872..993c169a 100644 --- a/verification/tla/MCStart.tla +++ b/verification/tla/MCStart.tla @@ -1,9 +1,9 @@ ---------------------------- MODULE MCStart ---------------------------- EXTENDS Start \* Per-PR safety instance (design docs/design/unified-start.md, section 15): -\* two children racing for one name in a group with room for both and the -\* parent; start_link and start_monitor with their default placement; a -\* managed parent that may exit (D, then publication); abandonment, failure, -\* retry and pre-D escalation-free deaths. Group closure is in MCStartCap -\* and MCStartTimeout; the init timer in MCStartTimeout. +\* one child started with start_link or start_monitor by a managed parent +\* that may trap and exit (D, then publication): every init outcome, the +\* disposition of every notice, abandonment, the cleanup deadline. Most +\* notice controls run here; names in MCStartNames, closure in MCStartCap and +\* MCStartTimeout, the timer in MCStartTimeout. ============================================================================= diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index 98815466..fdb1d556 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -3,14 +3,14 @@ CONSTANTS Kids = {"c1", "c2"} Kinds = {"link", "monitor"} Explicit = {"none"} - ParentPlaces = {"G", "U"} + ParentPlaces = {"G"} Cap = 3 Named = TRUE - TimeoutEnabled = TRUE + TimeoutEnabled = FALSE KillEnabled = FALSE AdminStopEnabled = FALSE ParentCanExit = TRUE - CloseEnabled = TRUE + CloseEnabled = FALSE OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE diff --git a/verification/tla/MCStartDeep.tla b/verification/tla/MCStartDeep.tla index d0dd4b3c..be5a7fd2 100644 --- a/verification/tla/MCStartDeep.tla +++ b/verification/tla/MCStartDeep.tla @@ -1,8 +1,7 @@ ---------------------------- MODULE MCStartDeep ---------------------------- EXTENDS Start -\* Weekly safety instance: MCStart plus the init timer and an unmanaged -\* parent, so the timer races two named children's readiness, failure, -\* abandonment and publication. (Every variant with every option at two -\* children is beyond TLC; MCStartContinue, MCStartTimeout and MCStartScope -\* cover those dimensions one child at a time.) +\* Weekly safety instance (30M states, about 5 minutes on 4 workers): two +\* named children started with start_link or start_monitor by a managed +\* parent that may exit, abandonment, failure, retry: the per-PR MCStart and +\* MCStartNames combined. ============================================================================= diff --git a/verification/tla/MCStartNames.cfg b/verification/tla/MCStartNames.cfg new file mode 100644 index 00000000..dcaa711a --- /dev/null +++ b/verification/tla/MCStartNames.cfg @@ -0,0 +1,20 @@ +SPECIFICATION Spec +CONSTANTS + Kids = {"c1", "c2"} + Kinds = {"start"} + Explicit = {"none"} + ParentPlaces = {"G"} + Cap = 3 + Named = TRUE + TimeoutEnabled = FALSE + KillEnabled = FALSE + AdminStopEnabled = FALSE + ParentCanExit = FALSE + CloseEnabled = FALSE + OwnerCloseEnabled = FALSE + MaxMsgs = 0 + ChildCanTrap = FALSE + UnlinkEnabled = FALSE + ScopeEnabled = FALSE + Fault = "None" +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartNames.tla b/verification/tla/MCStartNames.tla new file mode 100644 index 00000000..57909094 --- /dev/null +++ b/verification/tla/MCStartNames.tla @@ -0,0 +1,6 @@ +---------------------------- MODULE MCStartNames ---------------------------- +EXTENDS Start +\* Per-PR instance for names and the commit: two children started with +\* start, racing for one name in a group with room for both and the parent; +\* lookups, refusals, release, retry. +============================================================================= diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index f6a16b12..096b6237 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -247,11 +247,11 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStart", "NoInherit", "InheritedPlacement"), ("MCStart", "LinkAfterStarted", "LinkBeforeStarted"), ("MCStart", "LinkCheckLate", "EdgeOnlyToRunningParent"), - ("MCStart", "DoubleRegister", "NameUnique"), - ("MCStart", "UnregisterAtDeath", "NameKeptWhileLive"), - ("MCStart", "UnregisterAtDeath", "NoOverlapOnReuse"), - ("MCStart", "LateUnregister", "NameFreeBeforeObservation"), - ("MCStart", "SplitCommit", "AbortInvisible"), + ("MCStartNames", "DoubleRegister", "NameUnique"), + ("MCStartNames", "UnregisterAtDeath", "NameKeptWhileLive"), + ("MCStartNames", "UnregisterAtDeath", "NoOverlapOnReuse"), + ("MCStartNames", "LateUnregister", "NameFreeBeforeObservation"), + ("MCStartNames", "SplitCommit", "AbortInvisible"), ("MCStart", "EarlyFailAck", "FailureAfterPublication"), ("MCStart", "EarlyPending", "PendingOnlyAfterDeadline"), ("MCStart", "AckBeforeStarted", "ResultMatchesCommit"), From 6615acabc0e4550d4ca340a577692c7a6c32e1dd Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:11:07 -0300 Subject: [PATCH 28/42] docs: revision 3 design sections, model section and TLA README --- docs/design/unified-start.md | 154 ++++++++++++++--------------------- verification/tla/README.md | 104 +++++++++-------------- 2 files changed, 102 insertions(+), 156 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 49ba707c..a741f310 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -10,8 +10,10 @@ reviews (Codex and agy, both "approve after the listed changes", no P0). What changed and why is in [Review resolutions](#20-review-resolutions). - **Model:** [`verification/tla/Start.tla`](../../verification/tla/Start.tla), - with configurations `MCStart`, `MCStartCap`, `MCStartTimeout`, - `MCStartContinue`, `MCStartScope`, `MCStartLive` and `MCStartDeep`. + with configurations `MCStart`, `MCStartNames`, `MCStartCap`, + `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` and + `MCStartDeep`. It is TLC-only in `verification/model-map.toml` until the + implementation adds trace validation. - **Client mapping:** [unified-start-client-mapping.md](unified-start-client-mapping.md). - **Agreed shapes with other designs:** - the dynamic supervisor (#245, revision 3) and introspection v1 (#256); @@ -1017,119 +1019,87 @@ lines from section 16. ## 15. The model -[`Start.tla`](../../verification/tla/Start.tla), revision 2: - -- **Parent.** One parent P: managed in group `G` or unmanaged, trapping or - not, able to exit. -- **Children.** Started with `spawn`, `start`, `start_link` or - `start_monitor`, with a default or explicit placement and possibly the - shared name. -- **Stage 1.** Prepare failure, then the atomic `Commit`; a `Lookup` - observer resolves the name at any time. -- **End of `started()`.** One step with every `Init` outcome, a trap flag, - held parent exits and kills, resolved by the 2.3 precedence. -- **The startup cell.** The commitment and the waiter state, with `AckOk`, - `AckFail` (after publication, or at the bounded wait with `pend`), - `Abandon`, and the init `Timeout`. -- **Death.** `Die` (D) and `Publish`, with destruction between them; group - capacity is released at publication. -- **After readiness.** Ready children may exit, be killed or unlink; a - continuation and messages from others; a trapping child's queued parent - exit. -- **S8.** Owned children, the scope deadline, and the generation end. -- **Notices.** Counted per child (`exitN`, `downN`) with their reason, so a - duplicate delivery is visible. +[`Start.tla`](../../verification/tla/Start.tla), revision 3: + +- **The parent** P is managed in group `G` or unmanaged, trapping or not. It + may exit, then reach its D (`PDie`) and its publication (`PPublish`). Its + scope's admission gate may close. +- **Children** are started with `spawn`, `start`, `start_link` or + `start_monitor`. Each has a default or explicit placement (`G`, `G2` + another group of P's system, unmanaged, `same`) and possibly the shared + name. +- **Stage 1** is a preparation failure or one atomic `Commit`, with a + `Lookup` observer and a record of which holders a competitor's refusal + blamed. +- **`started()`** resolves every `Init` outcome, with a trap flag, held + parent exits, outside kills and administrative stops, by the 2.3 + precedence. A kill before D can escalate the reason (`KillExiting`). +- **The startup result** has a first commitment `commit0` that must never + change. The waiter answers (`AckOk`, `AckFail` after publication or after + an explicit `CleanupDeadline`) or abandons. The init timer is + framework-owned. +- **The child's death** is `Die` (D, reason final), `Release` (name and + slot), then `Publish` (notices disposed of; the model flags any illegitimate + drop as lost). +- **After readiness:** continuations, messages from others, a trapping + child's queued parent exit, unlinking. +- **S8:** owned children, the owner gate, the polite `Shutdown` at P's + publication, the scope deadline (not assumed to fire), and the generation + end. +- **Notices** are counted per child, with their reason. **Safety invariants:** | Invariant | Meaning | |---|---| | `InheritedPlacement` | explicit wins; link/monitor take the parent's; otherwise unmanaged | -| `LinkBeforeStarted` | the edge was really created (at the commit) before `started()` began | -| `NameUnique` | at most one holder | -| `NameHeldUntilPublished` | a holder is between its commit and its publication | -| `NoOverlapOnReuse` | no holder while another named child is between D and publication | -| `NameFreeBeforeObservation` | no notice or final error while the child still holds the name | -| `AbortInvisible` | a refused start: no edge, name, lookup, slot, hook or notice | -| `ResultMatchesCommit` | the answer is the committed outcome | -| `ReadyIsFinal` | a ready commit is never answered as a start error | -| `TimeoutNeverKillsReady` | the init timer never kills a ready child | -| `ReadyOnlyWhileOpen` | managed readiness commits only while admission is open | -| `FailureAfterPublication` | an error without `pending` comes after publication | -| `FailureFlushed` | a failed start delivers no `Down`, and no `Exit` to a trapping parent | +| `LinkBeforeStarted` | the edge was really created at the commit, before `started()` | +| `EdgeOnlyToRunningParent`, `NoEdgeToDeadParent` | edges are committed only while P runs, and none outlives P's D | +| `NameUnique`, `NameHeldUntilReleased`, `NameKeptWhileLive` | one holder; held exactly from the commit to the release | +| `NoOverlapOnReuse` | no new holder while another named child is still live | +| `NameFreeBeforeObservation` | no notice, parent exit or final answer while the child holds the name | +| `AbortInvisible` | a refused start: no edge, name, slot, lookup, competitor blame, hook or notice | +| `CommitImmutable` | the commitment never changes | +| `ResultMatchesCommit`, `ReadyIsFinal` | answers match the first commitment; ready is never answered as an error | +| `TimeoutNeverKillsReady`, `ReadyOnlyWhileOpen` | the timer loses to readiness; readiness only while admission (group and owner) is open | +| `FailureAfterPublication`, `ReportedReasonFinal`, `PendingOnlyAfterDeadline` | a final answer comes after publication, with the final reason; a pending one only after the cleanup deadline | +| `FailureFlushed`, `FailureEndsNonTrappingParent`, `NormalFailureSparesParent` | a failed start's notice is consumed, yet a non-`Normal` one still ends a non-trapping P, and a `Normal` one never does | | `TimeoutContained` | an init timeout reaches P only after an abandoned wait | -| `NoEdgeToDeadParent` | no edge outlives P's D | -| `AdmittedOnlyOpen`, `CapacityRespected` | admission; slots held until publication | -| `NoticeOnce` | at most one `Exit` and one `Down` per child (counters) | -| `AbandonedNoticeDelivered` | an abandoned wait hands the diverted notice to P with its kind and reason | +| `AdmittedOnlyOpen`, `OwnedOnlyWhileOwnerOpen` | admission by the target group and the owner gate | +| `CapacityRespected`, `SlotHeldUntilReleased`, `ParentSlotHeld` | slots (P's included) are held until release (P's until its publication) | +| `NoticeOnce`, `NoLostNotice`, `AbandonedNoticeDelivered` | at most one of each notice; none dropped illegitimately; an abandoned wait hands it on with its kind and reason, ending a non-trapping P | | `ContinueFirst` | no mailbox entry before the continuation | -| `GenDoneMeansOwnedDone` | P's generation ends only after its owned children | +| `GenDoneMeansOwnedDone`, `DeadlineKillsAllOwned` | the generation ends after P's and every member's publication; the scope deadline kills every live member, exiting ones included | -**Liveness** (weak fairness on the child's own steps, publication, P's D and -the scope deadline; strong fairness on the answer): +**Liveness** (weak fairness on the child's own steps, its death, release and +publication, the waiter's answer, and P's D and publication; strong +fairness on the error answer; the scope deadline and the cleanup deadline +are not assumed to fire): | Property | Meaning | |---|---| | `ParentExitReachesChild` | a non-`Normal` parent exit ends a linked child, also when held during `started()` | | `StartAnswered` | every waiting start is answered or abandoned | -| `OwnedScopeFinishes` | a managed parent's generation ends after it dies | +| `OwnedScopeFinishes` | once P is published, its generation ends through the polite `Shutdown` alone | -**Configurations and results** (TLC 2.19, fede, 4 workers, `nice -n 19`, -run one at a time): +**Configurations and results** (TLC 2.19, fede, 4 workers, `nice -n 19`, run +one at a time; every number is from a run that ended with TLC's own "No +error has been found"): -| Configuration | Run | Instance | Distinct states | Depth | Time | -|---|---|---|---|---|---| -| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3 (two children and P at once), closure, abandonment, retry | 10,115,028 | 22 | 80 s | -| `MCStartCap` | per PR | three children spawned into P's group of three: capacity binds, is held until publication, a retry after a published failure gets the slot | 102,526 | 23 | 4 s | -| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | 23,664 | 12 | 2 s | -| `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | 546,846 | 14 | 6 s | -| `MCStartScope` | per PR | managed parent, one linked child (owned or not), trapping, unlink, scope deadline; liveness `OwnedScopeFinishes` | 13,848 | 14 | 2 s | -| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | 8,387 | 11 | 2 s | -| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | 55,814,368 | 22 | 7 min 17 s | +RESULTS_TABLE **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): -Every control produced its counterexample in 1-5 s: - -| Configuration | Fault | Reintroduces | Fails | -|---|---|---|---| -| `MCStart` | `NoInherit` | 0.8 `start_linked` placement | `InheritedPlacement` | -| `MCStart` | `LinkAfterStarted` | the edge set once `started()` runs | `LinkBeforeStarted` | -| `MCStart` | `DoubleRegister` | check-then-insert names | `NameUnique` | -| `MCStart` | `LateUnregister` | the name released after the notices | `NameFreeBeforeObservation` | -| `MCStart` | `UnregisterAtDeath` | revision 1: the name released at D, before destruction | `NoOverlapOnReuse` | -| `MCStart` | `SplitCommit` | revision 1: the name published before the rest of stage 1 | `AbortInvisible` | -| `MCStart` | `EarlyFailAck` | revision 1: the error answered at D | `FailureAfterPublication` | -| `MCStart` | `AckBeforeStarted` | `Ok` before the commit | `ResultMatchesCommit` | -| `MCStart` | `ErrorAsOk` | `ignore` answered as started | `ResultMatchesCommit` | -| `MCStart` | `DivertAfterReady` | revision 1 (P0-1): readiness keeps the start marks | `ReadyIsFinal` | -| `MCStart` | `ReadyAfterClose` | readiness committed after the group closed | `ReadyOnlyWhileOpen` | -| `MCStart` | `NoMonitorFlush` | a failed `start_monitor` also delivers `Down` | `FailureFlushed` | -| `MCStart` | `NoLinkFlush` | a failed `start_link` also delivers `Exit` | `FailureFlushed` | -| `MCStart` | `LinkCheckLate` | the commit ignores the parent's state | `NoEdgeToDeadParent` | -| `MCStart` | `AbandonDropsNotice` | an abandoned wait loses the diverted notice | `AbandonedNoticeDelivered` | -| `MCStart` | `DoubleNotice` | a notice both delivered and diverted | `NoticeOnce` | -| `MCStartCap` | `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | -| `MCStartCap` | `IgnoreCap` | a full group admits | `CapacityRespected` | -| `MCStartTimeout` | `TimerIgnoresCommit` | revision 1 (P0-2): the timer kills without checking the cell | `TimeoutNeverKillsReady` | -| `MCStartTimeout` | `KillBeforeUnlink` | the timeout's `Killed` reaches a waiting parent | `TimeoutContained` | -| `MCStartContinue` | `ContinueQueued` | the continuation queued behind earlier entries | `ContinueFirst` | -| `MCStartScope` | `EarlyGenDone` | P's generation ends before its owned children | `GenDoneMeansOwnedDone` | -| `MCStartLive` | `DropHeld` (liveness) | a held parent exit dropped | `ParentExitReachesChild` | -| `MCStartLive` | `NoAnswerIgnore` (liveness) | an ignored start never answered | `StartAnswered` | -| `MCStartScope` | `NoScopeKill` (liveness) | the scope deadline spares unlinked children | `OwnedScopeFinishes` | - -A first `IgnoreCap` run on `MCStart` found no counterexample: two children -and P never exceed Cap 3, which agy P1-8 had pointed out. That led to -`MCStartCap`, where capacity binds. +CONTROLS_TABLE **Not modelled:** -- the lock implementation, which is covered by the Loom test in section 16; -- more than one parent; -- time (the bounded wait is a nondeterministic choice); -- destruction as a separate step (it lies between `Die` and `Publish`); +- the lock implementation (Loom tests, section 16); +- a parent-child-grandchild chain (transitive membership is one member set; + the integration and sim tests use three levels); +- time (deadlines are nondeterministic steps); +- destruction as its own step (it lies between `Die` and `Release`); - the supervisor's `ignore` and `startup_timeout` rules, which change `Supervision.tla` and `Cohorts.tla` in the implementation PR (section 9). diff --git a/verification/tla/README.md b/verification/tla/README.md index 30a98047..f8b8bcaa 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -17,7 +17,7 @@ this fits with Loom, Shuttle, Miri, ThreadSanitizer and mutation testing. | `DeferredReplies.tla`, `MCDeferredReplies*.tla/.cfg` | Rank 11 design model (not implemented), part 1: reply slots, send-time deadlines with shortening, deferral credits shared by a token and its first linked job, the exiting phase with unwind-local drops, a stuck phase (cleanup deadline passed while `stopped()` runs: nothing closed early, death later or never), death, and downstream request cleanup. Thirteen named negative controls. | | `DeferredJobs.tla`, `MCDeferredJobs*.tla/.cfg` | Rank 11 design model, part 2: pipe jobs with separate execution, cancellation and notification states; keyed replacement including notified jobs; optional job deadlines; a FIFO mailbox with emit credits and the terminal message last; `ActorId` fencing. `MCDeferredJobsEmits` is the fast configuration with emits. Nine named negative controls. | | `Lifecycle.tla`, `MCLifecycle{,Live,Fence,Deep,LiveDeep,WideDeep}.tla/.cfg` | Base actor lifecycle as proposed for v0.8 ([design](../../docs/design/otp-exit-signals.md)): links, monitors, `trap_exit`, exit signals (`Killed`, ignored `Normal`, `NoProc`, parent exits), administrative stops, requests around death, `Exit`/`Down` ordering. Seventeen negative-control faults (17 safety checks, 2 liveness checks): the v0.7 exit-signal behaviours (direct stops ignoring `trap_exit`, kills ending `Kill`, unfenced `unlink`/`demonitor`, real reasons for late registrations, `stopped()` after kills and untrapped signals, late signals ignored during exit), the PR #182/#185 bugs, early exit errors, kill and panic precedence, runtime-drop reasons, early notification, a split death point and a parent exit reaching `exit_received`. Per PR: `MCLifecycle` (7.5M states, fewer reasons, where the safety controls run), `MCLifecycleFence`, `MCLifecycleLive`; weekly: `MCLifecycleWideDeep` (the former 43M-state per-PR instance), `MCLifecycleDeep`, `MCLifecycleLiveDeep`. | -| `Start.tla`, `MCStart{,Cap,Timeout,Continue,Scope,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design revision 2 ([design](../../docs/design/unified-start.md), not implemented; TLC-only in `model-map.toml` until trace validation lands with the implementation): `spawn`, `start`, `start_link`, `start_monitor`; placement inheritance; an atomic stage-1 commit; a startup cell (one commitment: ready, failed, ignored or timed out; a waiter that consumes or abandons); death as D then publication; names held until publication; fallible init with held-signal precedence and trapping children; the continuation; S8 scope ownership. 25 negative controls (22 safety, 3 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartCap`, `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | +| `Start.tla`, `MCStart{,Names,Cap,Timeout,Continue,Scope,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design revision 3 ([design](../../docs/design/unified-start.md), not implemented; TLC-only in `model-map.toml` until trace validation lands with the implementation): `spawn`, `start`, `start_link`, `start_monitor`; placement inheritance; an atomic stage-1 commit with the owner gate; an immutable startup commitment and a waiter; death as D, release, then publication, for the child and the parent; the final reason fixed at D; fallible init with held-signal precedence; the continuation; S8 scope ownership with the polite Shutdown and the scope deadline. 37 negative controls (34 safety, 3 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartNames`, `MCStartCap`, `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | | `TraceCohorts.tla`, `TraceNested.tla` | Policy and ownership projections of real cohort/nested Rust executions. | | `trace_check.py`, `negative_checks.py` | Replay recorded traces; deliberately wrong selections, premature resume/join and broken propagation must be rejected or produce counterexamples. | | `SupervisionInd.tla`, `CohortsInd.tla`, `NestedSupervisionInd.tla`, `CoordinatorInd.tla`, `MC*Ind.tla`, `apalache.sh` | Inductive invariants (`IndInv`) checked with Apalache; see [Inductive invariants](#inductive-invariants-apalache). | @@ -608,86 +608,62 @@ All 22 named controls find their counterexample, each in a few seconds. ## Unified start (0.9 design) `Start.tla` backs the [unified start design](../../docs/design/unified-start.md) -(revision 2). It was written before the implementation, as the -deferred-replies models were. Design section 15 lists what it covers and -what it leaves out. - -**What is modelled:** - -- one parent, managed in group `G` or unmanaged, trapping or not, able to - exit; -- children started with `spawn`, `start`, `start_link` or `start_monitor`; -- stage 1 as a preparation (which may fail) and one atomic `Commit`, with a - `Lookup` observer; -- `started()` returning every `Init` outcome, with a trap flag, and held - parent exits and kills resolved by the design's precedence table; -- the startup cell: a commitment (`ready`, `failed`, `ignored`, - `timedout`) and a waiter (`waiting`, `consumed`, `abandoned`); -- death as `Die` (D) and `Publish`, with the name and the slot released at - publication and notices disposed of through the cell; -- after readiness: continuations, messages from others, a trapping child's - queued parent exit, unlinking; -- S8: owned children, the scope deadline, the generation end. - -Notices are counted per child, with their reason. - -**Safety invariants:** - -- placement and edges: `InheritedPlacement`, `LinkBeforeStarted`, - `NoEdgeToDeadParent`; -- names: `NameUnique`, `NameHeldUntilPublished`, `NoOverlapOnReuse`, - `NameFreeBeforeObservation`; -- the commit: `AbortInvisible`; -- the startup cell: `ResultMatchesCommit`, `ReadyIsFinal`, - `TimeoutNeverKillsReady`, `ReadyOnlyWhileOpen`, `FailureAfterPublication`; -- notices: `FailureFlushed`, `TimeoutContained`, `NoticeOnce`, - `AbandonedNoticeDelivered`; -- admission: `AdmittedOnlyOpen`, `CapacityRespected`; -- `ContinueFirst`, `GenDoneMeansOwnedDone`. - -**Liveness:** `ParentExitReachesChild` and `StartAnswered` (`MCStartLive`), -and `OwnedScopeFinishes` (`MCStartScope`). +(revision 3). It was written before the implementation, as the +deferred-replies models were. Design section 15 lists what it covers, its +invariants and what it leaves out. It is TLC-only in +`verification/model-map.toml` until the implementation adds trace +validation (design section 16.2). | Configuration | Run | Instance | |---|---|---| -| `MCStart` | per PR | two named children, link/monitor, managed parent, Cap 3, closure, abandonment, retry | -| `MCStartCap` | per PR | three linked children of a managed parent in a group of three: capacity binds, is held until publication, and a retry gets the slot | -| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent, the init timer | +| `MCStart` | per PR | one child (`start_link`, `start_monitor`) of a managed parent that may trap and exit: every init outcome, every notice disposition, abandonment, the cleanup deadline | +| `MCStartNames` | per PR | two named `start` children in a group with room for both and the parent: lookups, refusals, release, retry | +| `MCStartCap` | per PR | three children spawned into P's group of three: capacity binds, slots are held until release, a retry gets the slot | +| `MCStartTimeout` | per PR | one child of each waiting variant, managed or unmanaged parent; the init timer, outside kills (also before D), administrative stops, group closure | | `MCStartContinue` | per PR | one child of every variant with every explicit placement, two messages, trapping | -| `MCStartScope` | per PR | a managed parent and one linked child (owned or not), trapping, unlink, the scope deadline; liveness | -| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness | -| `MCStartDeep` | weekly | `MCStart` plus the init timer and an unmanaged parent | +| `MCStartScope` | per PR | a managed parent and one linked child (owned in `G` or `G2`, or unmanaged), trapping, unlink, owner closure; P's D and publication, the polite Shutdown, the scope deadline; liveness `OwnedScopeFinishes` | +| `MCStartLive` | per PR | one child of each waiting variant, trapping; liveness `ParentExitReachesChild`, `StartAnswered` | +| `MCStartDeep` | weekly | `MCStart` and `MCStartNames` combined: two named link/monitor children of an exiting parent | **Negative controls** (`START_CONTROLS` and `START_TEMPORAL_CONTROLS` in `negative_checks.py`; each checks `TypeOK` and only the named property): | Configuration | Fault | Breaks | |---|---|---| -| `MCStart` | `NoInherit` (the 0.8 `start_linked` placement) | `InheritedPlacement` | +| `MCStart` | `NoInherit` | `InheritedPlacement` | | `MCStart` | `LinkAfterStarted` | `LinkBeforeStarted` | -| `MCStart` | `DoubleRegister` | `NameUnique` | -| `MCStart` | `LateUnregister` | `NameFreeBeforeObservation` | -| `MCStart` | `UnregisterAtDeath` | `NoOverlapOnReuse` | -| `MCStart` | `SplitCommit` | `AbortInvisible` | +| `MCStart` | `LinkCheckLate` | `EdgeOnlyToRunningParent` | +| `MCStartNames` | `DoubleRegister` | `NameUnique` | +| `MCStartNames` | `UnregisterAtDeath` | `NameKeptWhileLive` | +| `MCStartNames` | `UnregisterAtDeath` | `NoOverlapOnReuse` | +| `MCStartNames` | `LateUnregister` | `NameFreeBeforeObservation` | +| `MCStartNames` | `SplitCommit` | `AbortInvisible` | | `MCStart` | `EarlyFailAck` | `FailureAfterPublication` | -| `MCStart` | `AckBeforeStarted`, `ErrorAsOk` | `ResultMatchesCommit` | -| `MCStart` | `DivertAfterReady` | `ReadyIsFinal` | -| `MCStart` | `ReadyAfterClose` | `ReadyOnlyWhileOpen` | -| `MCStart` | `NoMonitorFlush`, `NoLinkFlush` | `FailureFlushed` | -| `MCStart` | `LinkCheckLate` | `NoEdgeToDeadParent` | +| `MCStart` | `EarlyPending` | `PendingOnlyAfterDeadline` | +| `MCStart` | `AckBeforeStarted` | `ResultMatchesCommit` | +| `MCStart` | `ErrorAsOk` | `ResultMatchesCommit` | +| `MCStart` | `AnswerReadyAsFailure` | `ReadyIsFinal` | +| `MCStart` | `MarkKeptAfterReady` | `NoLostNotice` | +| `MCStart` | `NoMonitorFlush` | `FailureFlushed` | +| `MCStart` | `NoLinkFlush` | `FailureFlushed` | +| `MCStart` | `NoParentKillOnFailure` | `FailureEndsNonTrappingParent` | +| `MCStart` | `KillOnNormalFailure` | `NormalFailureSparesParent` | | `MCStart` | `AbandonDropsNotice` | `AbandonedNoticeDelivered` | | `MCStart` | `DoubleNotice` | `NoticeOnce` | -| `MCStartCap` | `AdmitClosed` | `AdmittedOnlyOpen` | -| `MCStartCap` | `IgnoreCap` | `CapacityRespected` | +| `MCStartTimeout` | `RewriteCommit` | `CommitImmutable` | | `MCStartTimeout` | `TimerIgnoresCommit` | `TimeoutNeverKillsReady` | | `MCStartTimeout` | `KillBeforeUnlink` | `TimeoutContained` | +| `MCStartTimeout` | `ReadyAfterClose` | `ReadyOnlyWhileOpen` | +| `MCStartTimeout` | `FrozenReason` | `ReportedReasonFinal` | +| `MCStartCap` | `AdmitClosed` | `AdmittedOnlyOpen` | +| `MCStartCap` | `IgnoreCap` | `CapacityRespected` | +| `MCStartCap` | `EarlySlotRelease` | `SlotHeldUntilReleased` | | `MCStartContinue` | `ContinueQueued` | `ContinueFirst` | | `MCStartScope` | `EarlyGenDone` | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `GenDoneAtParentD` | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `NoScopeKill` | `DeadlineKillsAllOwned` | +| `MCStartScope` | `OwnerUnchecked` | `OwnedOnlyWhileOwnerOpen` | +| `MCStartScope` | `ParentSlotAtD` | `ParentSlotHeld` | | `MCStartLive` (liveness) | `DropHeld` | `ParentExitReachesChild` | | `MCStartLive` (liveness) | `NoAnswerIgnore` | `StartAnswered` | -| `MCStartScope` (liveness) | `NoScopeKill` | `OwnedScopeFinishes` | - -The trace schema `start-v1` and `TraceStart.tla` are specified in design -section 16.2: events, model actions and code linearization points. They -belong to the implementation PR, and until then the model is allowlisted as -TLC-only in `verification/model-map.toml`. +| `MCStartScope` (liveness) | `NoPoliteStop` | `OwnedScopeFinishes` | From 6f6f868ffefcacfdd88991d75b3db060a3ab3d2b Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:12:22 -0300 Subject: [PATCH 29/42] verification: MCStartNames with group closure (makes SplitCommit non-vacuous) --- verification/tla/MCStartNames.cfg | 2 +- verification/tla/MCStartNames.tla | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/verification/tla/MCStartNames.cfg b/verification/tla/MCStartNames.cfg index dcaa711a..3a80bde3 100644 --- a/verification/tla/MCStartNames.cfg +++ b/verification/tla/MCStartNames.cfg @@ -10,7 +10,7 @@ CONSTANTS KillEnabled = FALSE AdminStopEnabled = FALSE ParentCanExit = FALSE - CloseEnabled = FALSE + CloseEnabled = TRUE OwnerCloseEnabled = FALSE MaxMsgs = 0 ChildCanTrap = FALSE diff --git a/verification/tla/MCStartNames.tla b/verification/tla/MCStartNames.tla index 57909094..acfa16a2 100644 --- a/verification/tla/MCStartNames.tla +++ b/verification/tla/MCStartNames.tla @@ -2,5 +2,5 @@ EXTENDS Start \* Per-PR instance for names and the commit: two children started with \* start, racing for one name in a group with room for both and the parent; -\* lookups, refusals, release, retry. +\* lookups, refusals (also by group closure), release, retry. ============================================================================= From e086ae526bc64ff1d6390f0dd306a50904c2f0f5 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:13:23 -0300 Subject: [PATCH 30/42] verification: CommitRest reuses the refusal rules; MCStartNames places children in G --- verification/tla/MCStartNames.cfg | 2 +- verification/tla/MCStartNames.tla | 2 +- verification/tla/Start.tla | 11 ++++++----- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/verification/tla/MCStartNames.cfg b/verification/tla/MCStartNames.cfg index 3a80bde3..4cd0e646 100644 --- a/verification/tla/MCStartNames.cfg +++ b/verification/tla/MCStartNames.cfg @@ -2,7 +2,7 @@ SPECIFICATION Spec CONSTANTS Kids = {"c1", "c2"} Kinds = {"start"} - Explicit = {"none"} + Explicit = {"G"} ParentPlaces = {"G"} Cap = 3 Named = TRUE diff --git a/verification/tla/MCStartNames.tla b/verification/tla/MCStartNames.tla index acfa16a2..3347d290 100644 --- a/verification/tla/MCStartNames.tla +++ b/verification/tla/MCStartNames.tla @@ -1,6 +1,6 @@ ---------------------------- MODULE MCStartNames ---------------------------- EXTENDS Start \* Per-PR instance for names and the commit: two children started with -\* start, racing for one name in a group with room for both and the parent; +\* start into P's group (room for both and P), racing for one name; \* lookups, refusals (also by group closure), release, retry. ============================================================================= diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 46a94290..e2b9b7a8 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -203,15 +203,18 @@ PrepareFail(c) == eV, xV, commit, commit0, initR, why, wait, resR, pend, dl, nV, mV, rV, dV>> -Refusal(c) == +RefusalNoName(c) == LET t == Target(c) IN CASE IsOwned(c, t) /\ ~ownerOpen /\ Fault # "OwnerUnchecked" -> "closed" [] t = "G" /\ ~gopen /\ Fault # "AdmitClosed" -> "closed" [] t = "G" /\ Used >= Cap /\ Fault # "IgnoreCap" -> "capacity" [] kind[c] \in {"link", "monitor"} /\ pst # "run" /\ ~(Fault = "LinkCheckLate" /\ pst # "published") -> "parentexited" - [] wname[c] /\ reg # {} /\ Fault # "DoubleRegister" -> "already" [] OTHER -> None +Refusal(c) == + IF RefusalNoName(c) # None THEN RefusalNoName(c) + ELSE IF wname[c] /\ reg # {} /\ Fault # "DoubleRegister" THEN "already" + ELSE None \* Who a refusal was caused by (a competing start's observation). Blame(r) == CASE r = "capacity" -> slots [] r = "already" -> reg [] OTHER -> {} @@ -260,9 +263,7 @@ RegisterFirst(c) == CommitRest(c) == /\ phase[c] = "reg1" /\ LET t == Target(c) - r == CASE t = "G" /\ (~gopen \/ Used >= Cap) -> "capacity" - [] kind[c] \in {"link", "monitor"} /\ pst # "run" -> "parentexited" - [] OTHER -> None + r == RefusalNoName(c) IN IF r # None THEN /\ phase' = [phase EXCEPT ![c] = "aborted"] /\ res' = [res EXCEPT ![c] = r] From 4322fae426d0a3b20e2bc4bf079b0fe20ca0c941 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:14:25 -0300 Subject: [PATCH 31/42] docs: revision 3 controls table, planted bugs, CHANGELOG, ROADMAP --- CHANGELOG.md | 20 ++++++++------ ROADMAP.md | 2 +- docs/design/unified-start.md | 52 ++++++++++++++++++++++++++++++++++-- 3 files changed, 63 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f1eb47b..02b5383d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,16 +2,20 @@ ## Unreleased -- Design (0.9, in review, revision 2): a unified start API +- Design (0.9, in review, revision 3): a unified start API (`docs/design/unified-start.md`, rank 5h). OTP's `start`, `start_link` and `start_monitor` wait for `started()`, and `spawn` does not. All four take one - typed `StartOptions`. `started()` may fail, ignore or continue (`Init`). A - per-start cell commits readiness once and independently of the waiter, and - the init timer and abandonment arbitrate through it. Errors are answered - after publication. Linked children are owned by the parent's scope. It - replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and - `start_linked`. A new TLA+ model, `Start.tla`, has six configurations and - 25 negative controls, and a client mapping covers 51 sites. + typed `StartOptions`. `started()` may fail, ignore or continue (`Init`), and a + failed init never runs `stopped()`, as OTP. Every instance has an immutable + startup commitment, and readiness is committed by the child, independently + of the waiter. Errors are answered after publication with the final exit + reason. One global lock order covers the commit, readiness and scope + admission. Linked children in the parent's system are owned by its scope, + with a polite Shutdown at the parent's publication and absolute, + composing deadlines. It replaces `start_on`, `start_with_backend`, + `ExecutionGroup::start` and `start_linked`. A new TLA+ model, `Start.tla`, + has eight configurations and 37 negative controls, and a client mapping + covers 51 sites. - ROADMAP: a ranked determinism backlog (D1-D10) under priority tier 3, from an audit of what escapes the simulator in Spawned, Ethrex and Ethlambda ([docs/research/determinism-audit.md](docs/research/determinism-audit.md)). diff --git a/ROADMAP.md b/ROADMAP.md index 99306ab9..ddfaff0e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -703,7 +703,7 @@ startup: OTP aborts `start_link` on any child start error, while Spawned retries a Permanent or Transient child's failed factory, panic or startup timeout during the first start (semantics §11; found reviewing #233). Behaviour change: extend the `Supervision`/`Cohorts` TLA+ models and their Apalache inductive checks (`BudgetBounded`) to the new rule, with a negative control for counting per child, and update the Kani restart-window proofs and the guide. | | 5g | One exit reason when the runtime drops an actor (v0.8) | Tasks mode reports `Kill` when the runtime drops an actor; threads mode falls back to `Panic("unexpected framework panic")` for an actor that never finishes cleanup. Pick one documented reason for both modes and test it in both. | -| 5h | Unified start API (v0.9, design in review, revision 2, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (7 configurations, 25 negative controls) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()` and answer `Started`, `Ignored` or an error after the child's publication; `spawn` does not wait. One typed `StartOptions`. `started()` may fail, ignore or continue (`Init`; the continuation precedes messages but is no termination barrier). A per-start cell commits readiness once and independently of the waiter, and the init timer and abandonment arbitrate through it. Stage 1 is one atomic commit. Linked and monitored children inherit the parent's group or backend (API-review footgun 5.11), and linked children in the parent's system are owned by its scope. Supervisors keep the whole-attempt `startup_timeout`, and `ignore` keeps the spec as OTP. Shared result types with #245 and #256. Client impact: 51 mechanical sites, no behaviour change. | +| 5h | Unified start API (v0.9, design in review, revision 3, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (8 configurations, 37 negative controls; TLC-only until implemented) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()` and answer `Started`, `Ignored` or an error after the child's publication (with the final exit reason); `spawn` does not wait. One typed `StartOptions`. `started()` may fail, ignore or continue (`Init`); a failed init never runs `stopped()`. An immutable startup commitment per instance; readiness committed by the child under the global lock order; the init timer and abandonment arbitrate through the commitment. Linked and monitored children inherit the parent's group or backend (API-review footgun 5.11); linked children in the parent's system are owned by its scope (polite Shutdown at the parent's publication, absolute composing deadlines). Supervisors keep the whole-attempt `startup_timeout`, and `ignore` keeps the spec as OTP. Shared result types with #245 and #256. Client impact: 51 mechanical sites, no behaviour change. | | 6 | Test kit with `ActorRef::mock()` | **Phase A implemented** in [`spawned-test`](#testing-crate-spawned-test) (unreleased): closure-backed `Mock` actors that implement `Handler` for every message, so `ActorRef` satisfies any `#[protocol]` and replaces hand-stubbed protocol refs with no production change; recording `Probe` recipients (`expect`, `expect_no_message`); bounded harness waits; fault injection (`faults::exit`, `restart_child`, mock delay/discard/crash). Unmocked messages fail loudly. **Remaining:** mocking a field typed as a concrete `ActorRef` needs an `Envelope` hook that downcasts to the concrete message and yields message and reply sender (a core change, deferred); controllable time for timers; a generic idle check; core `ChildHandle::exit(reason)` would make injected exits exact (today `Panic` and `Kill` go through a linked helper, so a panic surfaces as `Panic("panic in handler: ...")`). **Client evidence:** hand-stubbed protocols in ethlambda tests (`RecordingChain` in `column_checks.rs`, also `envelope_checks.rs`, `verdict.rs`). **Research ([actor testing](docs/research/actor-testing.md)):** three gaps against Akka/Erlang kits. (1) A synchronous behavior test kit (Akka `BehaviorTestKit`): run one handler with a test `Context` and record its effects (sends, spawns, timers, stop) without a runtime; needs a core change. (2) Manual time (Akka `ManualTime`): document Tokio's `start_paused` for async actors; thread actors need the framework-owned clock, the same seam as the [simulation backend spike](#simulation-backend-spike) and `ctx.now()`, so build one clock, not two. (3) Concrete `ActorRef` mocking stays deferred (actix's `Mocker` is the precedent); until then users hold protocol refs or `Recipient`s. Also: the idle check should report total deadlock Go-`checkdead` style (every actor idle, a request outstanding, no timer pending), and a replay-equivalence check for event-sourced actors ([event sourcing](docs/research/event-sourcing.md)). | | 6b | Shared-state and abort lints in `#[actor]` | Zero runtime cost. The macro warns when an actor's fields hold shared mutable state (`Arc>`, `Arc>`, atomics behind `Arc`, `static mut`), with an opt-out attribute for deliberate cases, and a build script check warns when the profile uses `panic = "abort"`, under which a handler panic kills the process instead of one actor. Syntactic only (it cannot see sharing behind a type alias), but it catches the common pattern, including the one LLM-written code reaches for first. Erlang gap: memory isolation. | | 7 | Request deadlock detection | **Client evidence:** requests awaited inside handlers exist (ethrex peer table), no cycle found; keep as a cheap safety net. `ActorError::WouldDeadlock` naming the cycle instead of a timeout. `ActorRef::request` takes only `&self`, so set a thread-local `CURRENT_ACTOR` only while the actor's own handler future is polled (and around the synchronous handler in thread mode); detached helpers create no edges. Keep a small wait set per actor for `join!`. Measure cost before enabling outside debug builds. **Approach chosen ([deadlock and stall detection](docs/research/deadlock-and-stall-detection.md)):** eager detection. Add a wait-for edge when a handler sends a request, walk the per-actor wait sets for a cycle, fail the request that would close it with `WouldDeadlock` naming the cycle, remove the edge on reply or timeout; a request to self fails at once (Erlang's `calling_self`). Waits on raw channels, locks or `spawn_blocking` create no edges, so it is a safety net, not a proof. Edges are exported to rank 30. | diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index a741f310..fccbf819 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1091,7 +1091,52 @@ RESULTS_TABLE **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): -CONTROLS_TABLE +Every control produced its counterexample in 1-12 s: + +| Configuration | Fault | Reintroduces | Fails | +|---|---|---|---| +| `MCStart` | `NoInherit` | 0.8 `start_linked` placement | `InheritedPlacement` | +| `MCStart` | `LinkAfterStarted` | the edge set once `started()` runs | `LinkBeforeStarted` | +| `MCStart` | `LinkCheckLate` | the commit ignores the parent's exit | `EdgeOnlyToRunningParent` | +| `MCStartNames` | `DoubleRegister` | check-then-insert names | `NameUnique` | +| `MCStartNames` | `UnregisterAtDeath` | the name released at D, before destruction | `NameKeptWhileLive` | +| `MCStartNames` | `UnregisterAtDeath` | the name released at D, before destruction | `NoOverlapOnReuse` | +| `MCStartNames` | `LateUnregister` | the name released after the notices | `NameFreeBeforeObservation` | +| `MCStartNames` | `SplitCommit` | the name published before the rest of the commit | `AbortInvisible` | +| `MCStart` | `EarlyFailAck` | an error answered before publication without pending | `FailureAfterPublication` | +| `MCStart` | `EarlyPending` | a pending answer before the cleanup deadline | `PendingOnlyAfterDeadline` | +| `MCStart` | `AckBeforeStarted` | `Ok` before the commitment | `ResultMatchesCommit` | +| `MCStart` | `ErrorAsOk` | `ignore` answered as started | `ResultMatchesCommit` | +| `MCStart` | `AnswerReadyAsFailure` | a ready instance answered as an error | `ReadyIsFinal` | +| `MCStart` | `MarkKeptAfterReady` | readiness keeps the start marks, so a post-ready notice is lost | `NoLostNotice` | +| `MCStart` | `NoMonitorFlush` | a failed `start_monitor` also delivers `Down` | `FailureFlushed` | +| `MCStart` | `NoLinkFlush` | a failed `start_link` also delivers `Exit` | `FailureFlushed` | +| `MCStart` | `NoParentKillOnFailure` | a non-`Normal` init failure spares a non-trapping parent | `FailureEndsNonTrappingParent` | +| `MCStart` | `KillOnNormalFailure` | a `Normal` init failure kills a non-trapping parent | `NormalFailureSparesParent` | +| `MCStart` | `AbandonDropsNotice` | an abandoned wait loses the diverted notice | `AbandonedNoticeDelivered` | +| `MCStart` | `DoubleNotice` | a notice both delivered and diverted | `NoticeOnce` | +| `MCStartTimeout` | `RewriteCommit` | the timer rewrites a ready commitment | `CommitImmutable` | +| `MCStartTimeout` | `TimerIgnoresCommit` | the timer kills without checking the commitment | `TimeoutNeverKillsReady` | +| `MCStartTimeout` | `KillBeforeUnlink` | the timeout's `Killed` reaches a waiting parent | `TimeoutContained` | +| `MCStartTimeout` | `ReadyAfterClose` | readiness after admission closed | `ReadyOnlyWhileOpen` | +| `MCStartTimeout` | `FrozenReason` | the answer reports the requested reason after publication | `ReportedReasonFinal` | +| `MCStartCap` | `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | +| `MCStartCap` | `IgnoreCap` | a full group admits | `CapacityRespected` | +| `MCStartCap` | `EarlySlotRelease` | the slot released at D | `SlotHeldUntilReleased` | +| `MCStartContinue` | `ContinueQueued` | the continuation queued behind earlier entries | `ContinueFirst` | +| `MCStartScope` | `EarlyGenDone` | the generation ends before its members | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `GenDoneAtParentD` | the generation ends at P's D | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `NoScopeKill` | the scope deadline spares exiting members | `DeadlineKillsAllOwned` | +| `MCStartScope` | `OwnerUnchecked` | the commit ignores the owner's gate | `OwnedOnlyWhileOwnerOpen` | +| `MCStartScope` | `ParentSlotAtD` | P's slot released at its D | `ParentSlotHeld` | +| `MCStartLive` | `DropHeld` (liveness) | a held parent exit dropped | `ParentExitReachesChild` | +| `MCStartLive` | `NoAnswerIgnore` (liveness) | an ignored start never answered | `StartAnswered` | +| `MCStartScope` | `NoPoliteStop` (liveness) | no Shutdown to owned members at P's publication | `OwnedScopeFinishes` | + +In round 2, `SplitCommit` first found no counterexample in `MCStartNames`: +`start` children were unmanaged and nothing refused them. The names +instance now places them in `G` with closure, and `CommitRest` reuses the +full refusal rules. A control that cannot fail is reported, not counted. **Not modelled:** @@ -1247,6 +1292,9 @@ Each is caught by at least two layers: | `start-name-released-at-death` | name released at D (before destruction) | `state_drop_finishes_before_failure_answer_and_name_reuse`, trace `NoOverlapOnReuse` | | `start-mark-not-cleared-at-ready` | readiness leaves the start marks | `ready_then_immediate_crash_with_unpolled_waiter_*`, `start_sim::ready_commit_races_post_ready_death` | | `start-timer-without-cas` | the timer kills without checking the cell | `init_timeout_racing_ready_agrees_with_ready`, Loom `ready_commit_races_timer` | +| `start-timedout-notice-dropped` | a waiting `TimedOut` notice is dropped at publication, so a later abandonment loses a non-trapping parent's kill (round 2, agy F1) | `abandoned_start_with_timeout_delivers_killed` (abandon after publication), `start_sim::abandon_races_publication`, trace control "drop a diverted notice on `SAbandon`" | +| `start-reason-frozen-at-resolution` | the answer reports the requested reason after a pre-D kill (round 2, Codex R3) | `kill_between_resolution_and_d_reports_killed`, `start_sim::kill_races_d_after_failed_init` | +| `start-parent-notices-at-d` | a parent's notices to children delivered, and its slot released, at its D instead of its publication (round 2, Codex R2) | `parent_hung_in_drop_keeps_children_unnotified_and_slot_held`, trace `ParentSlotHeld` | ### 16.6 Coverage matrix @@ -1295,7 +1343,7 @@ branch, which its own author maintains. | Item | Agreed shape | Unified start | #245 dynamic supervisor | #256 introspection | |---|---|---|---|---| -| Start result | `StartOutcome { Started(T), Ignored }` | `T = ActorRef` or `(ActorRef, MonitorRef)` | `StartOutcome>` (rename from `StartOutcome`) | n/a | +| Start result | `StartOutcome { Started(T), Ignored { pending: Option } }` | `T = ActorRef` or `(ActorRef, MonitorRef)` | `StartOutcome>` (rename from `StartOutcome`); `Ignored` gains `pending` for a stuck ignoring instance (round 2, Codex R4) | n/a | | `Started` | `{ actor, child }` | n/a | **drop `data: A::StartData`** (deferred, section 17) | n/a | | Factory result | `ChildStart { Actor(A), Ignore }`, `From` | static `ChildSpec` | as proposed | n/a | | `ignore` | static: spec kept, `ChildState::Ignored`, restarted in later cohort rounds, Temporary removed; dynamic: nothing stored (first start) or removed (restart), no `ChildState` variant needed | section 9 | as proposed | `ChildState::Ignored` appears only for static children; counted in `specs` and `workers`, not `active` | From b3976f1ed25fe67ee447337d116f13ff5e4abfb1 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:20:50 -0300 Subject: [PATCH 32/42] docs: revision 3 TLC results (all with TLC's verdict) --- docs/design/unified-start.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index fccbf819..ebf32040 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1086,7 +1086,16 @@ are not assumed to fire): one at a time; every number is from a run that ended with TLC's own "No error has been found"): -RESULTS_TABLE +| Configuration | Run | Instance | Distinct states | Depth | Time | +|---|---|---|---|---|---| +| `MCStart` | per PR | one link/monitor child of a managed parent that may trap and exit: every init outcome and notice disposition, abandonment, the cleanup deadline | 7,086 | 13 | 1 s | +| `MCStartNames` | per PR | two named `start` children in P's group, closure, lookups, refusals, retry | 703,268 | 22 | 10 s | +| `MCStartCap` | per PR | three children spawned into P's group of three | 628,622 | 26 | 11 s | +| `MCStartTimeout` | per PR | one child of each waiting variant; init timer, outside kills (also before D), administrative stops, closure; managed or unmanaged parent | 258,868 | 17 | 4 s | +| `MCStartContinue` | per PR | one child of every variant and explicit placement, two messages, trapping | 2,017,292 | 17 | 19 s | +| `MCStartScope` | per PR | S8: one linked child owned (`G`, `G2`) or not, trapping, unlink, owner closure, P's D and publication, polite Shutdown, scope deadline; liveness `OwnedScopeFinishes` | 221,072 | 17 | 15 s | +| `MCStartLive` | per PR | liveness `ParentExitReachesChild`, `StartAnswered` | 29,834 | 14 | 4 s | +| `MCStartDeep` | weekly | two named link/monitor children of an exiting parent (`MCStart` and `MCStartNames` combined) | 30,230,014 | 24 | 5 min 19 s | **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): From 4825b3887fcbc2a3d0026d315b6237ac644048e5 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:37:29 -0300 Subject: [PATCH 33/42] fix(design): one rule for a pre-D kill vs the startup decision (round 3: Codex 1, agy F1) A non-ready resolution and D are one critical section, so the startup decision and its reason are final; a kill after it does nothing, a kill before it is row 2. Ready children escalate atomically on a pre-D kill. Model: atomic ResolveD, kill escalation at arrival, invariants IgnoredExitsNormal and KillBeforeDIsKilled with controls SplitResolutionD and SkipEscalation (FrozenReason became vacuous and is retired). --- docs/design/unified-start.md | 55 ++++++++++++++----- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStartCap.cfg | 2 +- verification/tla/MCStartContinue.cfg | 2 +- verification/tla/MCStartDeep.cfg | 2 +- verification/tla/MCStartLive.cfg | 2 +- verification/tla/MCStartNames.cfg | 2 +- verification/tla/MCStartScope.cfg | 2 +- verification/tla/MCStartTimeout.cfg | 2 +- verification/tla/Start.tla | 81 ++++++++++++++++++---------- verification/tla/negative_checks.py | 3 +- 11 files changed, 104 insertions(+), 51 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index ebf32040..e1229d4d 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -178,17 +178,40 @@ that `started()` leaves. The first matching row below decides the **startup decision**: the immutable commitment (section 5.1) and the result's kind and diagnostic. It also sets the **requested exit reason**. -The **final exit reason** is fixed only at D, by 0.8's precedence rules -(otp-exit-signals "Reason precedence until D", L3). A kill, for example the -scope deadline or `ChildHandle::kill()`, that arrives between the resolution -and D still makes it `Killed`. An answer given after publication reports the -final reason. An answer given at the cleanup deadline, before D, reports the -requested reason as provisional (`pending: Some`). - No failed row runs `stopped()`. OTP never calls `terminate/2` when `init/1` did not return `{ok, ..}`, and cleanup before readiness belongs in -`started()` (agy F2, option A). A failed start therefore reaches D without a -`stopped()` phase. Its reason can still be escalated by a kill before D. +`started()` (round 2: agy F2, option A). + +**One rule for kills and the startup decision** (round 3: Codex 1, agy F1). +A non-ready resolution and D happen in **one critical section**. Under the +child's exit-request lock (rank 3 of the lock order), the resolution: + +1. reads the held signals and the returned outcome; +2. commits the decision; +3. fixes the reason; +4. runs D: the edges close and their notices are taken. + +Nothing runs in between, so no signal can arrive there: + +- **A kill that arrives before the resolution** is row 2. The decision is + then failed `InitExited { Killed }`, never `Ignored` or `InitFailed`. +- **A kill that arrives after the resolution** finds the actor past D and + does nothing (D3, X5). + +The startup decision therefore wins, and its requested reason **is** the +final reason. The starter's answer, `wait_exit`, and every linked parent's +`Exit` or `Down` all carry the same reason: + +- `Ok(Ignored)` and `InitFailed` always mean the child ended `Normal`; +- a non-trapping linked parent is never killed by an ignored or `Normal` + failed start. + +A **ready** child keeps 0.8's rule: its exit (with `stopped()`) runs until D, +and a kill that arrives before D makes the reason `Killed` at once (L3). + +The model checks this with `IgnoredExitsNormal` (control `SplitResolutionD`, +which reopens the window) and `KillBeforeDIsKilled` (control +`SkipEscalation`, a pre-D kill that does not escalate). | # | Condition | Requested reason | `stopped()` | Startup decision | |---|---|---|---|---| @@ -316,10 +339,12 @@ pub enum StartError { `Some(handle)` when the cleanup deadline expired first (section 5.5). The handle is instance-bound: `wait_exit` reaches the real end, and the name and slot stay held until then. -- **`reason`.** With `pending: None` it is the final exit reason, the one - `wait_exit`, `Down` and `Exit` report. With `pending: Some` it is the - requested reason, and the final one comes from `pending.wait_exit()` - (Codex R3). +- **`reason`.** It is always the final exit reason, the one `wait_exit`, + `Down` and `Exit` report. A non-ready resolution and D are one step + (section 2.3), so the reason is fixed when the decision is. With + `pending: Some`, only destruction and publication are still outstanding. + A timed-out child still inside `started()` reports `Killed`, the kill it + was sent. - **A stuck `ignore`** answers `Ok(Ignored { pending: Some(handle) })` (Codex R4). - **`ignore`** is the `Ok(StartOutcome::Ignored)` outcome, not an error, as @@ -591,7 +616,9 @@ The child's death has four steps (otp-exit-signals phases 4-6, with publication split for the trace mapping; Codex R8): 1. **D.** The final exit reason is fixed. The edges close and their notices - are taken. + are taken. For a non-ready resolution, D is part of the resolution's + critical section (2.3); for a ready child it follows `stopped()`, as in + 0.8. 2. **Destruction.** Queued messages and the state `A` are dropped, outside framework locks. 3. **Release.** The name is released (registry lock), then the admission diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 0db9e198..2380f374 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg index 9d50acad..0dd714bd 100644 --- a/verification/tla/MCStartCap.cfg +++ b/verification/tla/MCStartCap.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg index 9ad62c03..b748dc78 100644 --- a/verification/tla/MCStartContinue.cfg +++ b/verification/tla/MCStartContinue.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index fdb1d556..750d80a2 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index e73274ec..3fcde597 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -17,6 +17,6 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned PROPERTY ParentExitReachesChild PROPERTY StartAnswered diff --git a/verification/tla/MCStartNames.cfg b/verification/tla/MCStartNames.cfg index 4cd0e646..cdaa8030 100644 --- a/verification/tla/MCStartNames.cfg +++ b/verification/tla/MCStartNames.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/MCStartScope.cfg b/verification/tla/MCStartScope.cfg index bfbb9ea9..7fef1313 100644 --- a/verification/tla/MCStartScope.cfg +++ b/verification/tla/MCStartScope.cfg @@ -17,5 +17,5 @@ CONSTANTS UnlinkEnabled = TRUE ScopeEnabled = TRUE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned PROPERTY OwnedScopeFinishes diff --git a/verification/tla/MCStartTimeout.cfg b/verification/tla/MCStartTimeout.cfg index a6db364d..e408d74d 100644 --- a/verification/tla/MCStartTimeout.cfg +++ b/verification/tla/MCStartTimeout.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index e2b9b7a8..a8831dea 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -286,6 +286,27 @@ Exit(c, r) == /\ phase' = [phase EXCEPT ![c] = "exiting"] /\ why' = [why EXCEPT ![c] = r] +\* A ready child's own exit: a kill that already arrived wins (L3). +ExitK(c, r) == Exit(c, IF killed[c] \/ skill[c] \/ xkill[c] THEN "Killed" ELSE r) + +\* A non-ready resolution and D are one critical section (design 2.3): no +\* stopped() runs, so no signal can arrive in between and the requested +\* reason is final. SplitResolutionD (negative control) reopens the window. +DieEffects(c) == + /\ lnote' = [lnote EXCEPT ![c] = edge[c] /\ Alive(pst)] + /\ mnote' = [mnote EXCEPT ![c] = mon[c] /\ Alive(pst)] + /\ edge' = [edge EXCEPT ![c] = FALSE] + /\ mon' = [mon EXCEPT ![c] = FALSE] + /\ reg' = IF Fault = "UnregisterAtDeath" THEN reg \ {c} ELSE reg + /\ slots' = IF Fault = "EarlySlotRelease" THEN slots \ {c} ELSE slots +ResolveD(c, r) == + /\ why' = [why EXCEPT ![c] = r] + /\ IF Fault = "SplitResolutionD" + THEN /\ phase' = [phase EXCEPT ![c] = "exiting"] + /\ UNCHANGED <> + ELSE /\ phase' = [phase EXCEPT ![c] = "dead"] + /\ DieEffects(c) + StartedBegin(c) == /\ phase[c] = "starting" /\ ~began[c] /\ ~killed[c] /\ ~skill[c] /\ ~xkill[c] /\ began' = [began EXCEPT ![c] = TRUE] @@ -304,8 +325,11 @@ LateLink(c) == OutsideKill(c) == /\ KillEnabled /\ ~xkill[c] /\ phase[c] \in {"starting", "ready", "exiting"} /\ xkill' = [xkill EXCEPT ![c] = TRUE] + \* Before D a kill makes the reason Killed at once (L3). + /\ why' = IF phase[c] = "exiting" /\ Fault # "SkipEscalation" + THEN [why EXCEPT ![c] = "Killed"] ELSE why /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, nV, mV, rV, dV>> AdminStop(c) == /\ AdminStopEnabled /\ ~astop[c] /\ phase[c] = "starting" /\ astop' = [astop EXCEPT ![c] = TRUE] @@ -316,11 +340,12 @@ AnyKill(c) == killed[c] \/ skill[c] \/ xkill[c] KilledEarly(c) == /\ phase[c] = "starting" /\ ~began[c] /\ AnyKill(c) - /\ Exit(c, "Killed") + /\ ResolveD(c, "Killed") /\ initR' = [initR EXCEPT ![c] = "Killed"] /\ IF commit[c] = None THEN SetCommit(c, "failed") ELSE UNCHANGED <> /\ UNCHANGED <> + linkEver, monEver, xV, wait, res, resR, pend, dl, + pubAlive, div, divEver, handed, lost, mV, seen, blamed, dV>> \* started() returns; design 2.3 precedence. No failed row runs stopped(). StartedEnd(c) == @@ -331,7 +356,7 @@ StartedEnd(c) == row3 == astop[c] \/ sig r3 == IF astop[c] /\ (~sig \/ first = "admin") THEN "Shutdown" ELSE pex[c] closed == (place[c] = "G" /\ ~gopen) \/ (owned[c] /\ ~ownerOpen) - failWith(r) == /\ Exit(c, r) + failWith(r) == /\ ResolveD(c, r) /\ initR' = [initR EXCEPT ![c] = r] /\ IF commit[c] = None THEN SetCommit(c, "failed") ELSE UNCHANGED <> @@ -344,7 +369,7 @@ StartedEnd(c) == ELSE IF o = "stopS" THEN failWith("Shutdown") ELSE IF o \in {"stopN", "error"} THEN failWith("Normal") ELSE IF o = "ignore" - THEN /\ Exit(c, "Normal") + THEN /\ ResolveD(c, "Normal") /\ initR' = [initR EXCEPT ![c] = "Normal"] /\ SetCommit(c, "ignored") /\ UNCHANGED <> @@ -355,10 +380,11 @@ StartedEnd(c) == /\ readyOpen' = [readyOpen EXCEPT ![c] = ~closed] /\ cont' = [cont EXCEPT ![c] = o = "continue"] /\ qpex' = [qpex EXCEPT ![c] = IF t THEN pex[c] ELSE None] - /\ UNCHANGED <> - /\ UNCHANGED <> + /\ UNCHANGED <> + wait, res, resR, pend, dl, pubAlive, div, divEver, handed, lost, + msgs, early, seen, blamed, dV>> Send(c) == /\ phase[c] \in {"starting", "ready"} /\ msgs[c] < MaxMsgs @@ -382,7 +408,7 @@ HandleQueuedExit(c) == /\ ~cont[c] \/ Fault = "ContinueQueued" /\ early' = [early EXCEPT ![c] = early[c] \/ cont[c]] /\ qpex' = [qpex EXCEPT ![c] = None] - /\ Exit(c, qpex[c]) + /\ ExitK(c, qpex[c]) /\ UNCHANGED <> @@ -390,14 +416,14 @@ HandleQueuedExit(c) == ApplyParentExit(c) == /\ phase[c] = "ready" /\ pex[c] # None /\ pex' = [pex EXCEPT ![c] = None] - /\ IF pex[c] \in NonNormal THEN Exit(c, pex[c]) ELSE UNCHANGED <> + /\ IF pex[c] \in NonNormal THEN ExitK(c, pex[c]) ELSE UNCHANGED <> /\ UNCHANGED <> ReadyExit(c) == /\ phase[c] = "ready" - /\ \E r \in {"Normal", "Panic"} : Exit(c, r) + /\ \E r \in {"Normal", "Panic"} : ExitK(c, r) /\ UNCHANGED <> @@ -407,13 +433,6 @@ KillReady(c) == /\ UNCHANGED <> -\* Before D the reason can still change: a kill makes it Killed (L3). -KillExiting(c) == - /\ phase[c] = "exiting" /\ AnyKill(c) /\ why[c] # "Killed" - /\ why' = [why EXCEPT ![c] = "Killed"] - /\ UNCHANGED <> - ChildUnlink(c) == /\ UnlinkEnabled /\ phase[c] = "ready" /\ edge[c] /\ edge' = [edge EXCEPT ![c] = FALSE] @@ -423,12 +442,7 @@ ChildUnlink(c) == Die(c) == /\ phase[c] = "exiting" /\ phase' = [phase EXCEPT ![c] = "dead"] - /\ lnote' = [lnote EXCEPT ![c] = edge[c] /\ Alive(pst)] - /\ mnote' = [mnote EXCEPT ![c] = mon[c] /\ Alive(pst)] - /\ edge' = [edge EXCEPT ![c] = FALSE] - /\ mon' = [mon EXCEPT ![c] = FALSE] - /\ reg' = IF Fault = "UnregisterAtDeath" THEN reg \ {c} ELSE reg - /\ slots' = IF Fault = "EarlySlotRelease" THEN slots \ {c} ELSE slots + /\ DieEffects(c) /\ UNCHANGED <> @@ -537,8 +551,7 @@ AckFail(c) == ELSE \/ phase[c] = "published" \/ Fault = "EarlyFailAck" /\ phase[c] = "dead" /\ pend' = [pend EXCEPT ![c] = late] - /\ resR' = [resR EXCEPT ![c] = - IF ~late /\ Fault # "FrozenReason" THEN why[c] ELSE initR[c]] + /\ resR' = [resR EXCEPT ![c] = IF why[c] = None THEN "Killed" ELSE why[c]] /\ res' = [res EXCEPT ![c] = CASE commit0[c] = "ignored" -> "ignored" [] commit0[c] = "timedout" -> "timeout" [] OTHER -> "failed"] @@ -619,9 +632,11 @@ ScopeDeadline == /\ skill' = [c \in Kids |-> skill[c] \/ (owned[c] /\ phase[c] \in (IF Fault = "NoScopeKill" THEN {"starting", "ready"} ELSE {"starting", "ready", "exiting"}))] + /\ why' = [c \in Kids |-> IF owned[c] /\ phase[c] = "exiting" /\ Fault \notin {"NoScopeKill", "SkipEscalation"} + THEN "Killed" ELSE why[c]] /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, nV, mV, rV, dV>> \* P's generation is done: P published and every owned child published. GenDone == @@ -644,7 +659,7 @@ ChildStep(c) == \/ StartedBegin(c) \/ LateLink(c) \/ OutsideKill(c) \/ AdminStop(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ HandleQueuedExit(c) - \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ KillExiting(c) + \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ ChildUnlink(c) \/ Die(c) \/ Release(c) \/ Publish(c) \/ LateRelease(c) \/ Timeout(c) \/ CleanupDeadline(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) @@ -728,6 +743,16 @@ ReportedReasonFinal == \A c \in Kids : (res[c] = "failed" /\ ~pend[c]) => resR[c] = why[c] PendingOnlyAfterDeadline == \A c \in Kids : pend[c] => dl[c] +\* A kill that arrived before D makes the final reason Killed. +KillBeforeDIsKilled == + \A c \in Kids : ((killed[c] \/ skill[c] \/ xkill[c]) /\ phase[c] \in {"dead", "released", "published"}) + => why[c] = "Killed" +\* An ignored start ends Normal: no later signal can change it (the +\* resolution and D are one step). +IgnoredExitsNormal == + \A c \in Kids : (commit0[c] = "ignored" /\ phase[c] \in {"dead", "released", "published"}) + => why[c] = "Normal" + \* A failed start's notice is consumed: no Down; no Exit to a trapping P. FailureFlushed == \A c \in Kids : res[c] \in {"failed", "ignored", "timeout"} => diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 096b6237..846a1863 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -268,7 +268,8 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStartTimeout", "TimerIgnoresCommit", "TimeoutNeverKillsReady"), ("MCStartTimeout", "KillBeforeUnlink", "TimeoutContained"), ("MCStartTimeout", "ReadyAfterClose", "ReadyOnlyWhileOpen"), - ("MCStartTimeout", "FrozenReason", "ReportedReasonFinal"), + ("MCStartTimeout", "SkipEscalation", "KillBeforeDIsKilled"), + ("MCStartTimeout", "SplitResolutionD", "IgnoredExitsNormal"), ("MCStartCap", "AdmitClosed", "AdmittedOnlyOpen"), ("MCStartCap", "IgnoreCap", "CapacityRespected"), ("MCStartCap", "EarlySlotRelease", "SlotHeldUntilReleased"), From 16f29052f6c8f783ff1979377547851c1e45c953 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:38:38 -0300 Subject: [PATCH 34/42] wip: release split and delivery step (model) --- verification/tla/Start.tla | 88 ++++++++++++++++++++++++-------------- 1 file changed, 55 insertions(+), 33 deletions(-) diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index a8831dea..f3827471 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -52,7 +52,7 @@ None == "none" NonNormal == {"Panic", "Killed", "Shutdown"} Reasons == {"Normal"} \cup NonNormal Phases == {"idle", "begun", "reg1", "starting", "ready", "exiting", "dead", - "released", "published", "aborted"} + "namerel", "released", "published", "aborted"} Commits == {None, "ready", "failed", "ignored", "timedout"} FailCommits == {"failed", "ignored", "timedout"} Waits == {"none", "waiting", "abandoned", "consumed"} @@ -71,7 +71,7 @@ VARIABLES lnote, mnote, pubAlive, div, divEver, handed, lost, cont, msgs, early, reg, seen, blamed, slots, - exitN, downN, exitR, downR + exitN, downN, exitR, downR, obN, obR, dlvOk pV == <> oV == <> @@ -82,7 +82,7 @@ cV == <> nV == <> mV == <> rV == <> -dV == <> +dV == <> vars == <> Waiting(k) == k \in {"start", "link", "monitor"} @@ -135,6 +135,8 @@ TypeOK == /\ blamed \subseteq Kids /\ slots \subseteq Kids /\ exitN \in [Kids -> 0..3] /\ downN \in [Kids -> 0..3] /\ exitR \in [Kids -> Reasons \cup {None}] /\ downR \in [Kids -> Reasons \cup {None}] + /\ obN \in [Kids -> 0..3] /\ obR \in [Kids -> Reasons \cup {None}] + /\ dlvOk \in [Kids -> BOOLEAN] F(v) == [c \in Kids |-> v] Init == @@ -157,6 +159,7 @@ Init == /\ cont = F(FALSE) /\ msgs = F(0) /\ early = F(FALSE) /\ reg = {} /\ seen = F(FALSE) /\ blamed = {} /\ slots = {} /\ exitN = F(0) /\ downN = F(0) /\ exitR = F(None) /\ downR = F(None) + /\ obN = F(0) /\ obR = F(None) /\ dlvOk = F(FALSE) ----------------------------------------------------------------------------- (* Notices reaching P (only while P's mailbox admits: before P's D). *) @@ -275,8 +278,9 @@ CommitRest(c) == /\ UNCHANGED <> +\* Any number of lookups; a repeated one is a stuttering step. Lookup == - /\ \E c \in reg : ~seen[c] /\ seen' = [seen EXCEPT ![c] = TRUE] + /\ \E c \in reg : seen' = [seen EXCEPT ![c] = TRUE] /\ UNCHANGED <> ----------------------------------------------------------------------------- @@ -447,14 +451,20 @@ Die(c) == linkEver, monEver, xV, cV, pubAlive, div, divEver, handed, lost, mV, seen, blamed, dV>> -\* After destruction: the name and the slot are released. -Release(c) == +\* After destruction: the name is released (registry lock), then the slot +\* (group lock); two atomic points, two trace events (design 5.2, 16.2). +ReleaseName(c) == /\ phase[c] = "dead" - /\ phase' = [phase EXCEPT ![c] = "released"] + /\ phase' = [phase EXCEPT ![c] = "namerel"] /\ reg' = IF Fault = "LateUnregister" THEN reg ELSE reg \ {c} + /\ UNCHANGED <> +ReleaseSlot(c) == + /\ phase[c] = "namerel" + /\ phase' = [phase EXCEPT ![c] = "released"] /\ slots' = slots \ {c} /\ UNCHANGED <> + eV, xV, cV, nV, mV, reg, seen, blamed, dV>> \* Publication: each taken notice is disposed of by the commitment and the \* waiter (design 5.2); answers become final. @@ -486,17 +496,30 @@ Publish(c) == \/ ((ln \/ mn) /\ ~(b1 \/ b2 \/ b3 \/ divertL \/ divertM) /\ ~legit)] /\ div' = IF divertL \/ divertM THEN [div EXCEPT ![c] = r] ELSE div /\ divEver' = IF divertL \/ divertM THEN [divEver EXCEPT ![c] = TRUE] ELSE divEver - /\ IF ln /\ (ordinary \/ (ptrap /\ Fault = "NoLinkFlush") \/ dbl) - THEN /\ DeliverLink(c, r) /\ UNCHANGED <> - ELSE IF applyNow /\ pst = "run" - THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c - /\ UNCHANGED dV - ELSE IF mn /\ (ordinary \/ Fault = "NoMonitorFlush") - THEN /\ DeliverDown(c, r) /\ UNCHANGED <> - ELSE UNCHANGED <> - /\ UNCHANGED <> + \* The disposition is decided here; delivery is a later step. + /\ obN' = IF b1 \/ b2 \/ b3 THEN [obN EXCEPT ![c] = @ + 1] ELSE obN + /\ obR' = IF b1 \/ b2 \/ b3 THEN [obR EXCEPT ![c] = r] ELSE obR + /\ UNCHANGED <> + +\* Delivery of a disposed notice to P (its mailbox, or the signal applied +\* to a non-trapping P); a separate event (PNotice) from the disposition. +Deliver(c) == + /\ obN[c] > 0 + /\ obN' = [obN EXCEPT ![c] = @ - 1] + /\ dlvOk' = [dlvOk EXCEPT ![c] = Alive(pst)] + /\ LET r == obR[c] + force == Fault = "KillOnNormalFailure" /\ ~ptrap /\ kind[c] = "link" + /\ commit0[c] \in {"failed", "ignored"} /\ pst = "run" + IN IF kind[c] = "monitor" + THEN DeliverDown(c, r) /\ UNCHANGED <> + ELSE IF force + THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c + /\ UNCHANGED <> + ELSE DeliverLink(c, r) /\ UNCHANGED <> + /\ UNCHANGED <> LateRelease(c) == /\ Fault = "LateUnregister" /\ phase[c] = "published" /\ c \in reg @@ -521,7 +544,7 @@ Timeout(c) == \* The bounded wait for publication expires (measured from the commitment). CleanupDeadline(c) == /\ commit0[c] \in FailCommits /\ ~dl[c] - /\ phase[c] \in {"starting", "exiting", "dead"} + /\ phase[c] \in {"starting", "exiting", "dead", "namerel"} /\ dl' = [dl EXCEPT ![c] = TRUE] /\ UNCHANGED <> @@ -546,7 +569,7 @@ AckFail(c) == \/ Fault = "AnswerReadyAsFailure" /\ commit0[c] = "ready" /\ phase[c] = "published" /\ ~(Fault = "NoAnswerIgnore" /\ commit0[c] = "ignored") /\ \E late \in BOOLEAN : - /\ IF late THEN phase[c] \in {"starting", "exiting", "dead", "released"} + /\ IF late THEN phase[c] \in {"starting", "exiting", "dead", "namerel", "released"} /\ (dl[c] \/ Fault = "EarlyPending") ELSE \/ phase[c] = "published" \/ Fault = "EarlyFailAck" /\ phase[c] = "dead" @@ -570,14 +593,12 @@ Abandon(c) == drop == Fault = "AbandonDropsNotice" IN /\ handed' = [handed EXCEPT ![c] = give] /\ lost' = [lost EXCEPT ![c] = lost[c] \/ (give /\ drop)] - /\ IF give /\ ~drop /\ kind[c] = "link" - THEN DeliverLink(c, r) /\ UNCHANGED <> - ELSE IF give /\ ~drop /\ kind[c] = "monitor" - THEN DeliverDown(c, r) /\ UNCHANGED <> - ELSE UNCHANGED <> - /\ UNCHANGED <> + lnote, mnote, pubAlive, divEver, mV, rV, + exitN, downN, exitR, downR, dlvOk>> ----------------------------------------------------------------------------- (* The parent, its scope and the groups. *) @@ -660,7 +681,8 @@ ChildStep(c) == \/ KilledEarly(c) \/ StartedEnd(c) \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) - \/ ChildUnlink(c) \/ Die(c) \/ Release(c) \/ Publish(c) \/ LateRelease(c) + \/ ChildUnlink(c) \/ Die(c) \/ ReleaseName(c) \/ ReleaseSlot(c) \/ Publish(c) \/ Deliver(c) + \/ LateRelease(c) \/ Timeout(c) \/ CleanupDeadline(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ OwnerClose \/ PDie @@ -672,7 +694,7 @@ Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ OwnerClose \/ PDie Progress(c) == \/ StartedBegin(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ RunContinue(c) \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ KillReady(c) - \/ Die(c) \/ Release(c) \/ Publish(c) \/ AckOk(c) + \/ Die(c) \/ ReleaseName(c) \/ ReleaseSlot(c) \/ Publish(c) \/ Deliver(c) \/ AckOk(c) Fairness == /\ \A c \in Kids : /\ WF_vars(Progress(c)) /\ WF_vars(Commit(c)) /\ SF_vars(AckFail(c)) @@ -762,7 +784,7 @@ FailureFlushed == FailureEndsNonTrappingParent == \A c \in Kids : (kind[c] = "link" /\ ~ptrap /\ commit0[c] = "failed" /\ why[c] \in NonNormal - /\ phase[c] = "published" /\ lnote[c] /\ pubAlive[c]) => pst # "run" + /\ phase[c] = "published" /\ lnote[c] /\ pubAlive[c] /\ obN[c] = 0) => pst # "run" \* ... and one whose final reason is Normal (error, ignore, stop(Normal), \* not escalated by a kill before D) never does. NormalFailureSparesParent == @@ -780,7 +802,7 @@ OwnedOnlyWhileOwnerOpen == \A c \in Kids : owned[c] => ownOpenAt[c] \* Slots (P's included) are held until release, and never exceed Cap. CapacityRespected == Used <= Cap ParentSlotHeld == (pplace = "G" /\ pst \in {"run", "exiting", "dead"}) => PSlot = 1 -SlotHeldUntilReleased == \A c \in Kids : (place[c] = "G" /\ phase[c] \in LiveP) => c \in slots +SlotHeldUntilReleased == \A c \in Kids : (place[c] = "G" /\ phase[c] \in LiveP \cup {"namerel"}) => c \in slots NoticeOnce == \A c \in Kids : exitN[c] <= 1 /\ downN[c] <= 1 @@ -789,7 +811,7 @@ NoLostNotice == \A c \in Kids : ~lost[c] \* A notice handed on by an abandoned wait reaches P with its kind and reason. AbandonedNoticeDelivered == - \A c \in Kids : handed[c] => + \A c \in Kids : (handed[c] /\ obN[c] = 0 /\ dlvOk[c]) => /\ (kind[c] = "link" /\ ptrap) => exitN[c] >= 1 /\ exitR[c] = why[c] /\ (kind[c] = "link" /\ ~ptrap /\ why[c] \in NonNormal) => pst # "run" /\ kind[c] = "monitor" => downN[c] >= 1 /\ downR[c] = why[c] From 226176fbbaf8b473a4a4af0ef61a16ee477893c7 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:39:06 -0300 Subject: [PATCH 35/42] fix(design): name and slot release are separate atomic points; delivery is its own step (round 3: Codex 2) Model: ReleaseName and ReleaseSlot, repeatable Lookup, an outbox drained by Deliver (PNotice); design 5.2 and the 16.2 mapping with the delivery refinement rule and new trace controls. --- docs/design/unified-start.md | 47 ++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index e1229d4d..01929be6 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -621,12 +621,20 @@ publication split for the trace mapping; Codex R8): 0.8. 2. **Destruction.** Queued messages and the state `A` are dropped, outside framework locks. -3. **Release.** The name is released (registry lock), then the admission - slot (group lock). -4. **Publication.** Under the cell lock, each taken notice is disposed of by - the commitment and the waiter state (table below). Deliveries happen - after the lock is released. Then `wait_exit` is published, and answers - become available. +3. **Release of the name**, under the registry lock: one atomic point + (`SReleaseName`). +4. **Release of the slot**, under the group lock: a second atomic point + (`SReleaseSlot`). The two are separate because they are separate locks. + A competing start can commit the freed name between them, and still be + refused for capacity (round 3, Codex 2). +5. **Publication.** Under the cell lock, each taken notice is disposed of by + the commitment and the waiter state (table below) (`SPublish`). Then + `wait_exit` is published, and answers become available. +6. **Delivery.** Each notice publication disposed of as "deliver" or "apply" + is delivered after the cell lock is released, one `PNotice` per notice. + Other events may interleave between a disposition and its delivery. If P + passed its D in between, the notice is dropped, since P's mailbox no + longer admits it. The waiter answers only from publication, so a retry never overlaps the old instance's `Drop` and the reported reason is final. Model: @@ -1248,7 +1256,7 @@ transition (Codex R8). | `SPrepareFail` | c, error | `PrepareFail` | the prepare step's error return | | `SCommit` | c, place, edge kind, named, owned | `Commit` | inside the commit section, all locks held | | `SRefuse` | c, reason, blamed holder | `Commit` (refusal) | inside the commit section | -| `SLookup` | name, hit c or miss | `Lookup` (hit) or a stuttering step whose guard requires the name free (miss) | `whereis`, under the registry lock (test probes) | +| `SLookup` | name, hit c or miss | `Lookup` (hit; any number of times, a repeat is a stuttering step) or a stuttering step whose guard requires the name free (miss) | `whereis`, under the registry lock (test probes) | | `SStartedBegin` | c | `StartedBegin` | before the first poll of `started()` | | `SAdminStop`, `SKill` | c | `AdminStop`, `OutsideKill` | the signal's arrival (exit-request lock) | | `SResolve` | c, outcome, trap flag, held signals, commitment, requested reason | `StartedEnd` / `KilledEarly` | under the cell lock at the end of `started()`; every pre-readiness exit is recorded here and never as `SExit` (agy F8) | @@ -1259,21 +1267,27 @@ transition (Codex R8). | `SReason` | c, new reason | `KillExiting` | a pre-D reason change (exit-request lock) | | `SUnlink` | c | `ChildUnlink` | the unlink fence | | `SDie` | c, final reason, taken notes | `Die` | D (0.8 phase 4) | -| `SRelease` | c, name released, slot released | `Release` | after destruction; registry then group lock | +| `SReleaseName` | c | `ReleaseName` | after destruction, under the registry lock | +| `SReleaseSlot` | c | `ReleaseSlot` | after the name, under the group lock | | `SPublish` | c, disposition per note (delivered, diverted, applied, dropped) | `Publish` | publication, under the cell lock | | `SAnswer` | c, result, reason, pending | `AckOk` / `AckFail` | under the cell lock when the waiter takes the result | | `SReady` | c, observed result | a stuttering step whose guard requires the commitment (and publication or the deadline) to match | `ready()` returning | | `SAbandon` | c, handed-on notice | `Abandon` | `Drop` of `Starting`, under the cell lock | | `PExitBegin`, `PDie`, `PPublish` | reason; per child: link note taken; per child: notice and polite Shutdown | `PExit`, `PDie`, `PPublish` | the parent's exit begin, D and publication | | `POwnerClose` | | `OwnerClose` | the owner's gate closing | -| `PNotice` | c, kind, reason | part of `Publish`/`Abandon` | enqueue into the parent's mailbox, or the signal applied to it | +| `PNotice` | c, kind, reason | `Deliver` | enqueue into the parent's mailbox, or the signal applied to it, after the cell lock is released | | `GClose`, `GScopeKill`, `GGenDone` | | `GClose`, `ScopeDeadline`, `GenDone` | group close; the scope deadline; the generation-end commit | -- **Interleavings.** `SRelease` and `SPublish` are separate events because - the release and the disposition happen under different locks. A trace in - which C1's name is released, C2 commits the same name, and then C1 - publishes replays faithfully. A control moves `SRelease` after the - competing `SCommit` and must be rejected. +- **Interleavings.** Name release, slot release, disposition and delivery are + four events, because they happen under four different locks or after + them. A trace in which C1 releases its name, C2 commits the same name, and + C1 then releases its slot and publishes replays faithfully. +- **Refinement rule for delivery.** Each `SPublish` or `SAbandon` adds the + deliveries it disposed of to c's pending list. Each `PNotice` for c + consumes the oldest pending delivery, which must match its kind and + reason. At the end of a trace no pending delivery may remain unless the + parent passed its D, in which case the replay requires the trace to show + P's `PDie` before the missing `PNotice`. - **Complete consumption.** Replay must consume every event without inventing observable events. Every notice the parent sees has its own `PNotice`, which must match the disposition in `SPublish` or `SAbandon`. @@ -1284,7 +1298,10 @@ transition (Codex R8). trace, which the checker must reject: - move `SCommit` after `SStartedBegin`; - give two children overlapping names; - - move a `SRelease` after a competing `SCommit`; + - move a `SReleaseName` after a competing `SCommit` of the same name; + - move a `SReleaseSlot` before its `SReleaseName`; + - add a `PNotice` with no pending delivery, or a second one for a single + delivery; - change a managed child's place to unmanaged; - add a `PNotice` after an error answer to a trapping parent; - move an error `SAnswer` before `SPublish` without an `SDeadline`; From 44f8a7878db31db13a922c5b54e5eb6ab40a9931 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:39:53 -0300 Subject: [PATCH 36/42] fix(design): generation completion waits for owned work, not publication alone (round 3: Codex 3) Model: WorkDoneP/WorkDoneC; GenDone and GenDoneMeansOwnedDone require P's and every member's owned work; control GenDoneIgnoresWork. --- docs/design/unified-start.md | 11 ++++-- verification/tla/Start.tla | 57 +++++++++++++++++++---------- verification/tla/negative_checks.py | 1 + 3 files changed, 47 insertions(+), 22 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 01929be6..c6bcefe6 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -497,7 +497,10 @@ admitted to a group of the parent's `ActorSystem`. point too, not at D. 4. **P's generation is done** (V3, the scope's join) only when P and every owned member are published and their owned work has finished. That is - 0.8's scope completion: helpers and jobs joined, as for any scope. + 0.8's scope completion: helpers and jobs joined after death (rank 11 + joins jobs after D, bounded by the cleanup deadline), as for any scope. + The model represents that work explicitly (`WorkDoneP`, `WorkDoneC`), so + publication alone never completes a generation. The polite `Shutdown` is not OTP link semantics; a gen_server's linked worker survives a `normal` parent exit. It is the consequence of scope ownership, as @@ -538,8 +541,10 @@ polite `Shutdown`, a scope deadline armed from P's exit begin that reaches exiting members, and the generation end. Properties and controls: - `OwnedOnlyWhileOwnerOpen`, control `OwnerUnchecked`; -- `GenDoneMeansOwnedDone` (P published and every member published), - controls `EarlyGenDone` and `GenDoneAtParentD`; +- `GenDoneMeansOwnedDone`: P published, every member published, and their + owned work (helpers, jobs) finished. Controls: `EarlyGenDone` (members + not published), `GenDoneAtParentD` (P only at its D) and + `GenDoneIgnoresWork` (owned work still running; round 3, Codex 3); - `DeadlineKillsAllOwned`, control `NoScopeKill`, which spares exiting members; - `ParentSlotHeld`, control `ParentSlotAtD`; diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index f3827471..5086af4a 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -62,7 +62,9 @@ Outcomes == {"ready", "continue", "panic", "stopN", "stopS", "error", "ignore"} Alive(p) == p \in {"run", "exiting"} \* P's mailbox still admits VARIABLES - pst, pplace, ptrap, pwhy, pcause, gopen, ownerOpen, gdone, sfired, pn, + pst, pplace, ptrap, pwhy, pcause, gopen, ownerOpen, gdone, sfired, pn, pwork, cwork, + pwork, \* P's owned work (helpers, jobs) finished after its publication + cwork, \* [Kids -> that child's owned work finished after its publication] kind, expl, wname, phase, place, admOpen, owned, ownOpenAt, readyOpen, edgeWhenRun, edge, mon, linkEver, monEver, @@ -73,7 +75,7 @@ VARIABLES reg, seen, blamed, slots, exitN, downN, exitR, downR, obN, obR, dlvOk -pV == <> +pV == <> oV == <> sV == <> eV == <> @@ -106,6 +108,7 @@ TypeOK == /\ ptrap \in BOOLEAN /\ pwhy \in Reasons \cup {None} /\ pcause \in Kids \cup {None} /\ gopen \in BOOLEAN /\ ownerOpen \in BOOLEAN /\ gdone \in BOOLEAN /\ sfired \in BOOLEAN /\ pn \in [Kids -> Reasons \cup {None}] + /\ pwork \in BOOLEAN /\ cwork \in [Kids -> BOOLEAN] /\ kind \in [Kids -> {"spawn", "start", "link", "monitor", None}] /\ expl \in [Kids -> {"G", "G2", "U", "same", None}] /\ wname \in [Kids -> BOOLEAN] @@ -142,7 +145,7 @@ F(v) == [c \in Kids |-> v] Init == /\ pst = "run" /\ pplace \in ParentPlaces /\ ptrap \in BOOLEAN /\ pwhy = None /\ pcause = None /\ gopen = TRUE /\ ownerOpen = TRUE - /\ gdone = FALSE /\ sfired = FALSE /\ pn = F(None) + /\ gdone = FALSE /\ sfired = FALSE /\ pn = F(None) /\ pwork = FALSE /\ cwork = F(FALSE) /\ kind = F(None) /\ expl = F(None) /\ wname = F(FALSE) /\ phase = F("idle") /\ place = F(None) /\ admOpen = F(FALSE) /\ owned = F(FALSE) /\ ownOpenAt = F(FALSE) /\ readyOpen = F(FALSE) @@ -518,7 +521,7 @@ Deliver(c) == THEN /\ pst' = "exiting" /\ pwhy' = r /\ pcause' = c /\ UNCHANGED <> ELSE DeliverLink(c, r) /\ UNCHANGED <> - /\ UNCHANGED <> LateRelease(c) == @@ -606,13 +609,13 @@ Abandon(c) == PExit == /\ ParentCanExit /\ pst = "run" /\ \E r \in {"Normal", "Panic"} : pst' = "exiting" /\ pwhy' = r - /\ UNCHANGED <> OwnerClose == /\ OwnerCloseEnabled /\ ownerOpen /\ ownerOpen' = FALSE - /\ UNCHANGED <> \* P's D: its link notices are taken; edges and monitors close. @@ -622,7 +625,7 @@ PDie == /\ pn' = [c \in Kids |-> IF edge[c] THEN pwhy ELSE pn[c]] /\ edge' = F(FALSE) /\ mon' = F(FALSE) - /\ UNCHANGED <> \* P's publication: the parent exits reach the linked children, and the @@ -642,7 +645,7 @@ PPublish == /\ pexAbn' = [c \in Kids |-> pexAbn[c] \/ (rr(c) \in NonNormal /\ phase[c] \in {"starting", "ready"})] /\ pn' = F(None) - /\ UNCHANGED <> \* S8: from P's exit begin, the scope deadline (a timer, not assumed fair) @@ -655,24 +658,40 @@ ScopeDeadline == ELSE {"starting", "ready", "exiting"}))] /\ why' = [c \in Kids |-> IF owned[c] /\ phase[c] = "exiting" /\ Fault \notin {"NoScopeKill", "SkipEscalation"} THEN "Killed" ELSE why[c]] - /\ UNCHANGED <> -\* P's generation is done: P published and every owned child published. +\* Owned work (helpers, jobs) is joined after publication (0.8 scope +\* completion); not assumed instantaneous. +WorkDoneP == + /\ pst = "published" /\ ~pwork + /\ pwork' = TRUE + /\ UNCHANGED <> +WorkDoneC(c) == + /\ phase[c] = "published" /\ ~cwork[c] + /\ cwork' = [cwork EXCEPT ![c] = TRUE] + /\ UNCHANGED <> + +\* P's generation is done: P and every owned member published, and their +\* owned work finished (the scope's join, design 4.1). GenDone == /\ ScopeEnabled /\ pplace = "G" /\ ~gdone /\ pst = "published" \/ (Fault = "GenDoneAtParentD" /\ pst = "dead") - /\ \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} + /\ pwork \/ Fault \in {"GenDoneIgnoresWork", "GenDoneAtParentD"} + /\ \A c \in Kids : owned[c] => phase[c] = "aborted" + \/ (phase[c] = "published" /\ (cwork[c] \/ Fault = "GenDoneIgnoresWork")) \/ Fault = "EarlyGenDone" /\ gdone' = TRUE - /\ UNCHANGED <> GClose == /\ CloseEnabled /\ gopen /\ gopen' = FALSE - /\ UNCHANGED <> ChildStep(c) == @@ -682,11 +701,11 @@ ChildStep(c) == \/ Send(c) \/ RunContinue(c) \/ Handle(c) \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ ReadyExit(c) \/ KillReady(c) \/ ChildUnlink(c) \/ Die(c) \/ ReleaseName(c) \/ ReleaseSlot(c) \/ Publish(c) \/ Deliver(c) - \/ LateRelease(c) + \/ LateRelease(c) \/ WorkDoneC(c) \/ Timeout(c) \/ CleanupDeadline(c) \/ AckOk(c) \/ AckFail(c) \/ Abandon(c) Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ OwnerClose \/ PDie - \/ PPublish \/ ScopeDeadline \/ GenDone \/ GClose + \/ PPublish \/ WorkDoneP \/ ScopeDeadline \/ GenDone \/ GClose \* Liveness assumes the child's own steps, its death and publication, the \* waiter's answer, and P's D and publication complete. The scope deadline @@ -694,11 +713,11 @@ Next == (\E c \in Kids : ChildStep(c)) \/ Lookup \/ PExit \/ OwnerClose \/ PDie Progress(c) == \/ StartedBegin(c) \/ KilledEarly(c) \/ StartedEnd(c) \/ RunContinue(c) \/ HandleQueuedExit(c) \/ ApplyParentExit(c) \/ KillReady(c) - \/ Die(c) \/ ReleaseName(c) \/ ReleaseSlot(c) \/ Publish(c) \/ Deliver(c) \/ AckOk(c) + \/ Die(c) \/ ReleaseName(c) \/ ReleaseSlot(c) \/ Publish(c) \/ Deliver(c) \/ AckOk(c) \/ WorkDoneC(c) Fairness == /\ \A c \in Kids : /\ WF_vars(Progress(c)) /\ WF_vars(Commit(c)) /\ SF_vars(AckFail(c)) - /\ WF_vars(PDie) /\ WF_vars(PPublish) /\ WF_vars(GenDone) + /\ WF_vars(PDie) /\ WF_vars(PPublish) /\ WF_vars(WorkDoneP) /\ WF_vars(GenDone) Spec == Init /\ [][Next]_vars LiveSpec == Spec /\ Fairness @@ -819,8 +838,8 @@ AbandonedNoticeDelivered == ContinueFirst == \A c \in Kids : ~early[c] GenDoneMeansOwnedDone == - gdone => /\ pst = "published" - /\ \A c \in Kids : owned[c] => phase[c] \in {"published", "aborted"} + gdone => /\ pst = "published" /\ pwork + /\ \A c \in Kids : owned[c] => phase[c] = "aborted" \/ (phase[c] = "published" /\ cwork[c]) \* When the scope deadline fires, every live owned child is killed. DeadlineKillsAllOwned == diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 846a1863..3120685a 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -276,6 +276,7 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStartContinue", "ContinueQueued", "ContinueFirst"), ("MCStartScope", "EarlyGenDone", "GenDoneMeansOwnedDone"), ("MCStartScope", "GenDoneAtParentD", "GenDoneMeansOwnedDone"), + ("MCStartScope", "GenDoneIgnoresWork", "GenDoneMeansOwnedDone"), ("MCStartScope", "NoScopeKill", "DeadlineKillsAllOwned"), ("MCStartScope", "OwnerUnchecked", "OwnedOnlyWhileOwnerOpen"), ("MCStartScope", "ParentSlotAtD", "ParentSlotHeld"), From 999cc225fca13fe3786b7ac286b81c78e246f964 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:41:15 -0300 Subject: [PATCH 37/42] wip: owner gate history (model) --- verification/tla/MCStart.cfg | 2 +- verification/tla/MCStartCap.cfg | 2 +- verification/tla/MCStartContinue.cfg | 2 +- verification/tla/MCStartDeep.cfg | 2 +- verification/tla/MCStartLive.cfg | 2 +- verification/tla/MCStartNames.cfg | 2 +- verification/tla/MCStartScope.cfg | 2 +- verification/tla/MCStartTimeout.cfg | 2 +- verification/tla/Start.tla | 75 ++++++++++++++++++---------- verification/tla/negative_checks.py | 1 + 10 files changed, 58 insertions(+), 34 deletions(-) diff --git a/verification/tla/MCStart.cfg b/verification/tla/MCStart.cfg index 2380f374..7cb84c1d 100644 --- a/verification/tla/MCStart.cfg +++ b/verification/tla/MCStart.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/MCStartCap.cfg b/verification/tla/MCStartCap.cfg index 0dd714bd..2895f017 100644 --- a/verification/tla/MCStartCap.cfg +++ b/verification/tla/MCStartCap.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/MCStartContinue.cfg b/verification/tla/MCStartContinue.cfg index b748dc78..b64be8b8 100644 --- a/verification/tla/MCStartContinue.cfg +++ b/verification/tla/MCStartContinue.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/MCStartDeep.cfg b/verification/tla/MCStartDeep.cfg index 750d80a2..32182632 100644 --- a/verification/tla/MCStartDeep.cfg +++ b/verification/tla/MCStartDeep.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/MCStartLive.cfg b/verification/tla/MCStartLive.cfg index 3fcde597..f3d31d93 100644 --- a/verification/tla/MCStartLive.cfg +++ b/verification/tla/MCStartLive.cfg @@ -17,6 +17,6 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned PROPERTY ParentExitReachesChild PROPERTY StartAnswered diff --git a/verification/tla/MCStartNames.cfg b/verification/tla/MCStartNames.cfg index cdaa8030..f1083abc 100644 --- a/verification/tla/MCStartNames.cfg +++ b/verification/tla/MCStartNames.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/MCStartScope.cfg b/verification/tla/MCStartScope.cfg index 7fef1313..330a894c 100644 --- a/verification/tla/MCStartScope.cfg +++ b/verification/tla/MCStartScope.cfg @@ -17,5 +17,5 @@ CONSTANTS UnlinkEnabled = TRUE ScopeEnabled = TRUE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned PROPERTY OwnedScopeFinishes diff --git a/verification/tla/MCStartTimeout.cfg b/verification/tla/MCStartTimeout.cfg index e408d74d..bd21be39 100644 --- a/verification/tla/MCStartTimeout.cfg +++ b/verification/tla/MCStartTimeout.cfg @@ -17,4 +17,4 @@ CONSTANTS UnlinkEnabled = FALSE ScopeEnabled = FALSE Fault = "None" -INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone DeadlineKillsAllOwned +INVARIANT TypeOK InheritedPlacement LinkBeforeStarted EdgeOnlyToRunningParent NoEdgeToDeadParent NameUnique NameHeldUntilReleased NameKeptWhileLive NoOverlapOnReuse NameFreeBeforeObservation AbortInvisible CommitImmutable ResultMatchesCommit ReadyIsFinal TimeoutNeverKillsReady ReadyOnlyWhileOpen FailureAfterPublication ReportedReasonFinal KillBeforeDIsKilled IgnoredExitsNormal PendingOnlyAfterDeadline FailureFlushed FailureEndsNonTrappingParent NormalFailureSparesParent TimeoutContained AdmittedOnlyOpen OwnedOnlyWhileOwnerOpen CapacityRespected ParentSlotHeld SlotHeldUntilReleased NoticeOnce NoLostNotice AbandonedNoticeDelivered ContinueFirst GenDoneMeansOwnedDone OwnerRefusalOnlyOnShutdown DeadlineKillsAllOwned diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index 5086af4a..f6e6ecf8 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -69,7 +69,9 @@ VARIABLES phase, place, admOpen, owned, ownOpenAt, readyOpen, edgeWhenRun, edge, mon, linkEver, monEver, began, killed, skill, xkill, astop, pex, qpex, pexAbn, ctrap, - commit, commit0, initR, why, wait, res, resR, pend, dl, + commit, commit0, initR, why, wait, res, resR, pend, dl, oref, closedBy, + oref, \* [Kids -> readiness refused because the owner gate was closed] + closedBy, \* who closed the owner gate: "none", "ancestor" (or "parent" under a fault) lnote, mnote, pubAlive, div, divEver, handed, lost, cont, msgs, early, reg, seen, blamed, slots, @@ -80,7 +82,7 @@ oV == <> sV == <> eV == <> xV == <> -cV == <> +cV == <> nV == <> mV == <> rV == <> @@ -128,6 +130,7 @@ TypeOK == /\ wait \in [Kids -> Waits] /\ res \in [Kids -> Results] /\ resR \in [Kids -> Reasons \cup {None}] /\ pend \in [Kids -> BOOLEAN] /\ dl \in [Kids -> BOOLEAN] + /\ oref \in [Kids -> BOOLEAN] /\ closedBy \in {"none", "ancestor", "parent"} /\ lnote \in [Kids -> BOOLEAN] /\ mnote \in [Kids -> BOOLEAN] /\ pubAlive \in [Kids -> BOOLEAN] /\ div \in [Kids -> Reasons \cup {None}] /\ divEver \in [Kids -> BOOLEAN] @@ -156,7 +159,7 @@ Init == /\ ctrap = F(FALSE) /\ commit = F(None) /\ commit0 = F(None) /\ initR = F(None) /\ why = F(None) /\ wait = F("none") /\ res = F("none") /\ resR = F(None) /\ pend = F(FALSE) - /\ dl = F(FALSE) + /\ dl = F(FALSE) /\ oref = F(FALSE) /\ closedBy = "none" /\ lnote = F(FALSE) /\ mnote = F(FALSE) /\ pubAlive = F(FALSE) /\ div = F(None) /\ divEver = F(FALSE) /\ handed = F(FALSE) /\ lost = F(FALSE) /\ cont = F(FALSE) /\ msgs = F(0) /\ early = F(FALSE) @@ -206,7 +209,7 @@ PrepareFail(c) == /\ phase' = [phase EXCEPT ![c] = "aborted"] /\ res' = [res EXCEPT ![c] = "system"] /\ UNCHANGED <> RefusalNoName(c) == @@ -255,7 +258,7 @@ Commit(c) == kind[c] = "monitor") /\ reg' = IF wname[c] THEN reg \cup {c} ELSE reg /\ UNCHANGED blamed - /\ UNCHANGED <> \* SplitCommit (negative control): the name is published before the rest. @@ -278,7 +281,7 @@ CommitRest(c) == /\ UNCHANGED <> ELSE /\ Admit(c, t, kind[c] = "link", kind[c] = "monitor") /\ UNCHANGED <> - /\ UNCHANGED <> \* Any number of lookups; a repeated one is a stuttering step. @@ -336,7 +339,7 @@ OutsideKill(c) == /\ why' = IF phase[c] = "exiting" /\ Fault # "SkipEscalation" THEN [why EXCEPT ![c] = "Killed"] ELSE why /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> AdminStop(c) == /\ AdminStopEnabled /\ ~astop[c] /\ phase[c] = "starting" /\ astop' = [astop EXCEPT ![c] = TRUE] @@ -351,7 +354,7 @@ KilledEarly(c) == /\ initR' = [initR EXCEPT ![c] = "Killed"] /\ IF commit[c] = None THEN SetCommit(c, "failed") ELSE UNCHANGED <> /\ UNCHANGED <> \* started() returns; design 2.3 precedence. No failed row runs stopped(). @@ -367,7 +370,7 @@ StartedEnd(c) == /\ initR' = [initR EXCEPT ![c] = r] /\ IF commit[c] = None THEN SetCommit(c, "failed") ELSE UNCHANGED <> - /\ UNCHANGED <> + /\ UNCHANGED <> IN /\ ctrap' = [ctrap EXCEPT ![c] = t] /\ pex' = [pex EXCEPT ![c] = None] /\ IF AnyKill(c) THEN failWith("Killed") @@ -379,18 +382,23 @@ StartedEnd(c) == THEN /\ ResolveD(c, "Normal") /\ initR' = [initR EXCEPT ![c] = "Normal"] /\ SetCommit(c, "ignored") - /\ UNCHANGED <> + /\ UNCHANGED <> ELSE IF closed /\ Fault # "ReadyAfterClose" - THEN failWith("Shutdown") + THEN /\ ResolveD(c, "Shutdown") + /\ initR' = [initR EXCEPT ![c] = "Shutdown"] + /\ IF commit[c] = None THEN SetCommit(c, "failed") + ELSE UNCHANGED <> + /\ oref' = [oref EXCEPT ![c] = owned[c] /\ ~ownerOpen] + /\ UNCHANGED <> ELSE /\ phase' = [phase EXCEPT ![c] = "ready"] /\ SetCommit(c, "ready") /\ readyOpen' = [readyOpen EXCEPT ![c] = ~closed] /\ cont' = [cont EXCEPT ![c] = o = "continue"] /\ qpex' = [qpex EXCEPT ![c] = IF t THEN pex[c] ELSE None] - /\ UNCHANGED <> + /\ UNCHANGED <> /\ UNCHANGED <> Send(c) == @@ -418,7 +426,7 @@ HandleQueuedExit(c) == /\ ExitK(c, qpex[c]) /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, cont, msgs, rV, dV>> ApplyParentExit(c) == /\ phase[c] = "ready" /\ pex[c] # None @@ -426,19 +434,19 @@ ApplyParentExit(c) == /\ IF pex[c] \in NonNormal THEN ExitK(c, pex[c]) ELSE UNCHANGED <> /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> ReadyExit(c) == /\ phase[c] = "ready" /\ \E r \in {"Normal", "Panic"} : ExitK(c, r) /\ UNCHANGED <> + xV, commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> KillReady(c) == /\ phase[c] = "ready" /\ AnyKill(c) /\ Exit(c, "Killed") /\ UNCHANGED <> + xV, commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> ChildUnlink(c) == /\ UnlinkEnabled /\ phase[c] = "ready" /\ edge[c] @@ -542,14 +550,14 @@ Timeout(c) == THEN SetCommit(c, "timedout") ELSE UNCHANGED <> /\ killed' = [killed EXCEPT ![c] = TRUE] /\ UNCHANGED <> + initR, why, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> \* The bounded wait for publication expires (measured from the commitment). CleanupDeadline(c) == /\ commit0[c] \in FailCommits /\ ~dl[c] /\ phase[c] \in {"starting", "exiting", "dead", "namerel"} /\ dl' = [dl EXCEPT ![c] = TRUE] - /\ UNCHANGED <> AckOk(c) == @@ -561,7 +569,7 @@ AckOk(c) == /\ wait' = [wait EXCEPT ![c] = "consumed"] /\ div' = [div EXCEPT ![c] = None] /\ lost' = [lost EXCEPT ![c] = lost[c] \/ div[c] # None] - /\ UNCHANGED <> \* An error or ignore answers after publication (final reason), or after the @@ -583,7 +591,7 @@ AckFail(c) == [] OTHER -> "failed"] /\ wait' = [wait EXCEPT ![c] = "consumed"] /\ div' = [div EXCEPT ![c] = None] - /\ UNCHANGED <> \* The Starting is dropped: a diverted notice goes to P as an ordinary one. @@ -599,24 +607,35 @@ Abandon(c) == /\ obN' = IF give /\ ~drop THEN [obN EXCEPT ![c] = @ + 1] ELSE obN /\ obR' = IF give /\ ~drop THEN [obR EXCEPT ![c] = r] ELSE obR /\ UNCHANGED <> ----------------------------------------------------------------------------- (* The parent, its scope and the groups. *) +\* P's own exit begin refuses new starts (ParentExited, under P's +\* exit-request lock) but does not close its scope's admission gate: a child +\* already in started() finishes init, then gets the parent exit (OTP). +\* ParentExitClosesOwner (negative control) closes the gate here. PExit == /\ ParentCanExit /\ pst = "run" /\ \E r \in {"Normal", "Panic"} : pst' = "exiting" /\ pwhy' = r - /\ UNCHANGED <> + /\ IF Fault = "ParentExitClosesOwner" /\ ownerOpen + THEN ownerOpen' = FALSE /\ closedBy' = "parent" + ELSE UNCHANGED <> + /\ UNCHANGED <> +\* An ancestor's (supervisor's) or the system's shutdown closes the gate (S1). OwnerClose == /\ OwnerCloseEnabled /\ ownerOpen /\ ownerOpen' = FALSE + /\ closedBy' = "ancestor" /\ UNCHANGED <> + oV, sV, eV, xV, commit, commit0, initR, why, wait, res, resR, pend, dl, oref, + nV, mV, rV, dV>> \* P's D: its link notices are taken; edges and monitors close. PDie == @@ -660,7 +679,7 @@ ScopeDeadline == THEN "Killed" ELSE why[c]] /\ UNCHANGED <> + commit, commit0, initR, wait, res, resR, pend, dl, oref, closedBy, nV, mV, rV, dV>> \* Owned work (helpers, jobs) is joined after publication (0.8 scope \* completion); not assumed instantaneous. @@ -837,6 +856,10 @@ AbandonedNoticeDelivered == ContinueFirst == \A c \in Kids : ~early[c] +\* An owned child's readiness is refused for its owner only when an ancestor +\* or the system closed the gate, never because P itself is exiting. +OwnerRefusalOnlyOnShutdown == \A c \in Kids : oref[c] => closedBy = "ancestor" + GenDoneMeansOwnedDone == gdone => /\ pst = "published" /\ pwork /\ \A c \in Kids : owned[c] => phase[c] = "aborted" \/ (phase[c] = "published" /\ cwork[c]) diff --git a/verification/tla/negative_checks.py b/verification/tla/negative_checks.py index 3120685a..44bbe3a2 100644 --- a/verification/tla/negative_checks.py +++ b/verification/tla/negative_checks.py @@ -279,6 +279,7 @@ def named_temporal(module, fault, prop, tlc, constants=None): ("MCStartScope", "GenDoneIgnoresWork", "GenDoneMeansOwnedDone"), ("MCStartScope", "NoScopeKill", "DeadlineKillsAllOwned"), ("MCStartScope", "OwnerUnchecked", "OwnedOnlyWhileOwnerOpen"), + ("MCStartScope", "ParentExitClosesOwner", "OwnerRefusalOnlyOnShutdown"), ("MCStartScope", "ParentSlotAtD", "ParentSlotHeld"), ] START_TEMPORAL_CONTROLS = [ From 0282763ba0de53ec0e3c0aaa6819ddd5fe99e1e8 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:41:57 -0300 Subject: [PATCH 38/42] fix(design): the parent's own exit refuses new starts but does not close its scope gate (round 3: agy F2) As OTP, a child in started() whose start_link parent begins exiting still finishes init (and may commit ready, trapping children run their continuation) and then gets the parent exit. Only an ancestor's or the system's shutdown closes the owner gate. Model: closedBy/oref history, invariant OwnerRefusalOnlyOnShutdown, control ParentExitClosesOwner. --- docs/design/unified-start.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index c6bcefe6..15720486 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -223,7 +223,7 @@ which reopens the window) and `KillBeforeDIsKilled` (control | 6 | `Init::stop(r)` | `r` | no | failed → `InitExited` | | 7 | `Err(e)` / `Init::error(e)` | `Normal` | no | failed → `InitFailed { cause: e }` | | 8 | `Init::ignore()` | `Normal` | no | ignored → `Ok(Ignored)` | -| 9 | `Init::ready()` / `continue_with(m)`, but the child's admission closed: its group, or for an owned child its owner's scope | `Shutdown` | no | failed → `InitExited` | +| 9 | `Init::ready()` / `continue_with(m)`, but the child's admission closed: its group, or for an owned child its owner's scope gate, which only an ancestor's or the system's shutdown closes (S1), never the parent's own exit (4.1) | `Shutdown` | no | failed → `InitExited` | | 10 | `Init::ready()` / `continue_with(m)` | runs | later | **ready** → `Ok(Started(..))` | - **Held signals in rows 9-10.** A held `Normal` signal to a non-trapping @@ -484,9 +484,15 @@ admitted to a group of the parent's `ActorSystem`. **When the parent exits** (Codex R2, agy F3): -1. **At P's exit begin:** P's scope closes for new owned children (they are - refused as `ParentExited` or `Closed`), and the scope deadline is armed - (below). +1. **At P's exit begin:** new starts through P are refused as + `ParentExited`. The check is under P's exit-request lock (rank 3), not P's + scope gate. The scope deadline is armed (below). P's scope admission gate + stays open (round 3, agy F2). A child already inside `started()` finishes + init as OTP's does, and may commit ready; it then gets P's exit at P's + publication (E3). A trapping child runs its continuation first. The gate + closes only when an ancestor supervisor or the system begins shutdown + (S1). That refuses new owned children (`Closed`) and makes the readiness + commit of a child still in `started()` fail with row 9. 2. **At P's D:** P's edges close, and the link notices to its linked children are taken. 3. **At P's publication** (after P's destruction): the parent exits reach @@ -541,6 +547,10 @@ polite `Shutdown`, a scope deadline armed from P's exit begin that reaches exiting members, and the generation end. Properties and controls: - `OwnedOnlyWhileOwnerOpen`, control `OwnerUnchecked`; +- `OwnerRefusalOnlyOnShutdown`: a readiness refusal by the owner gate + happens only after an ancestor's or the system's shutdown. Control + `ParentExitClosesOwner` closes the gate at P's own exit begin, which would + refuse a child that OTP lets finish init; - `GenDoneMeansOwnedDone`: P published, every member published, and their owned work (helpers, jobs) finished. Controls: `EarlyGenDone` (members not published), `GenDoneAtParentD` (P only at its D) and @@ -583,7 +593,8 @@ the result, so `ready()` never depends on a waiter. applies. It takes, in the order of section 3: 1. the child's admission gate (its group's, and for an owned child, its - owner's); + owner's scope gate, closed only by an ancestor's or the system's + shutdown; the parent's own exit does not close it, 4.1); 2. the cell lock. Under both, it checks that admission is open, then: From 85aca4b1b9d0afb5b45c8085de777d08af84313b Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:42:04 -0300 Subject: [PATCH 39/42] docs(design): migration row for the stopped() change on a failed init (round 3: Codex, agy) --- docs/design/unified-start.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 15720486..8c26ddd8 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -990,6 +990,7 @@ Erlang lacks). | `ChildSpec::{max_outstanding, max_native_jobs}` | `.options(StartOptions::new().max_*(..))` | same | | factory `Ok(actor)` | unchanged (`From for ChildStart`); `Ok(ChildStart::Ignore)` is new | same | | `async fn started(..) {}` | unchanged | same | +| `stopped()` ran after a `ctx.stop*()` called inside `started()` | it no longer runs after any failed init, `ctx.stop*` in `started()` included (as OTP skips `terminate/2` when `init/1` fails, 2.3) | **changes:** move initialization cleanup into `started()` itself before returning the error, or hold the resources in RAII guards whose `Drop` releases them | | `Mock::start()` / `start_on` | `Mock::spawn()` / `spawn_with` | same | | `TestSystem::start` / `start_mock` | `TestSystem::spawn` / `spawn_mock` | same | From 3315192dc32917c13717260ebbec54be1185de3b Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 16:43:05 -0300 Subject: [PATCH 40/42] docs: round-three resolutions, control tables and counts --- CHANGELOG.md | 4 ++-- ROADMAP.md | 2 +- docs/design/unified-start.md | 34 ++++++++++++++++++++++++++++------ verification/tla/README.md | 9 ++++++--- 4 files changed, 37 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 02b5383d..cb27b530 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## Unreleased -- Design (0.9, in review, revision 3): a unified start API +- Design (0.9, approved for implementation after round 3): a unified start API (`docs/design/unified-start.md`, rank 5h). OTP's `start`, `start_link` and `start_monitor` wait for `started()`, and `spawn` does not. All four take one typed `StartOptions`. `started()` may fail, ignore or continue (`Init`), and a @@ -14,7 +14,7 @@ with a polite Shutdown at the parent's publication and absolute, composing deadlines. It replaces `start_on`, `start_with_backend`, `ExecutionGroup::start` and `start_linked`. A new TLA+ model, `Start.tla`, - has eight configurations and 37 negative controls, and a client mapping + has eight configurations and 40 negative controls, and a client mapping covers 51 sites. - ROADMAP: a ranked determinism backlog (D1-D10) under priority tier 3, from an audit of what escapes the simulator in Spawned, Ethrex and Ethlambda diff --git a/ROADMAP.md b/ROADMAP.md index ddfaff0e..8dd94dcc 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -703,7 +703,7 @@ startup: OTP aborts `start_link` on any child start error, while Spawned retries a Permanent or Transient child's failed factory, panic or startup timeout during the first start (semantics §11; found reviewing #233). Behaviour change: extend the `Supervision`/`Cohorts` TLA+ models and their Apalache inductive checks (`BudgetBounded`) to the new rule, with a negative control for counting per child, and update the Kani restart-window proofs and the guide. | | 5g | One exit reason when the runtime drops an actor (v0.8) | Tasks mode reports `Kill` when the runtime drops an actor; threads mode falls back to `Panic("unexpected framework panic")` for an actor that never finishes cleanup. Pick one documented reason for both modes and test it in both. | -| 5h | Unified start API (v0.9, design in review, revision 3, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (8 configurations, 37 negative controls; TLC-only until implemented) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()` and answer `Started`, `Ignored` or an error after the child's publication (with the final exit reason); `spawn` does not wait. One typed `StartOptions`. `started()` may fail, ignore or continue (`Init`); a failed init never runs `stopped()`. An immutable startup commitment per instance; readiness committed by the child under the global lock order; the init timer and abandonment arbitrate through the commitment. Linked and monitored children inherit the parent's group or backend (API-review footgun 5.11); linked children in the parent's system are owned by its scope (polite Shutdown at the parent's publication, absolute composing deadlines). Supervisors keep the whole-attempt `startup_timeout`, and `ignore` keeps the spec as OTP. Shared result types with #245 and #256. Client impact: 51 mechanical sites, no behaviour change. | +| 5h | Unified start API (v0.9, design approved for implementation after three review rounds, PR #259) | [Design](docs/design/unified-start.md) with the `Start.tla` model (8 configurations, 40 negative controls; TLC-only until implemented) and a [client mapping](docs/design/unified-start-client-mapping.md). OTP's `start`, `start_link` and `start_monitor` wait for `started()` and answer `Started`, `Ignored` or an error after the child's publication (with the final exit reason); `spawn` does not wait. One typed `StartOptions`. `started()` may fail, ignore or continue (`Init`); a failed init never runs `stopped()`. An immutable startup commitment per instance; readiness committed by the child under the global lock order; the init timer and abandonment arbitrate through the commitment. Linked and monitored children inherit the parent's group or backend (API-review footgun 5.11); linked children in the parent's system are owned by its scope (polite Shutdown at the parent's publication, absolute composing deadlines). Supervisors keep the whole-attempt `startup_timeout`, and `ignore` keeps the spec as OTP. Shared result types with #245 and #256. Client impact: 51 mechanical sites, no behaviour change. | | 6 | Test kit with `ActorRef::mock()` | **Phase A implemented** in [`spawned-test`](#testing-crate-spawned-test) (unreleased): closure-backed `Mock` actors that implement `Handler` for every message, so `ActorRef` satisfies any `#[protocol]` and replaces hand-stubbed protocol refs with no production change; recording `Probe` recipients (`expect`, `expect_no_message`); bounded harness waits; fault injection (`faults::exit`, `restart_child`, mock delay/discard/crash). Unmocked messages fail loudly. **Remaining:** mocking a field typed as a concrete `ActorRef` needs an `Envelope` hook that downcasts to the concrete message and yields message and reply sender (a core change, deferred); controllable time for timers; a generic idle check; core `ChildHandle::exit(reason)` would make injected exits exact (today `Panic` and `Kill` go through a linked helper, so a panic surfaces as `Panic("panic in handler: ...")`). **Client evidence:** hand-stubbed protocols in ethlambda tests (`RecordingChain` in `column_checks.rs`, also `envelope_checks.rs`, `verdict.rs`). **Research ([actor testing](docs/research/actor-testing.md)):** three gaps against Akka/Erlang kits. (1) A synchronous behavior test kit (Akka `BehaviorTestKit`): run one handler with a test `Context` and record its effects (sends, spawns, timers, stop) without a runtime; needs a core change. (2) Manual time (Akka `ManualTime`): document Tokio's `start_paused` for async actors; thread actors need the framework-owned clock, the same seam as the [simulation backend spike](#simulation-backend-spike) and `ctx.now()`, so build one clock, not two. (3) Concrete `ActorRef` mocking stays deferred (actix's `Mocker` is the precedent); until then users hold protocol refs or `Recipient`s. Also: the idle check should report total deadlock Go-`checkdead` style (every actor idle, a request outstanding, no timer pending), and a replay-equivalence check for event-sourced actors ([event sourcing](docs/research/event-sourcing.md)). | | 6b | Shared-state and abort lints in `#[actor]` | Zero runtime cost. The macro warns when an actor's fields hold shared mutable state (`Arc>`, `Arc>`, atomics behind `Arc`, `static mut`), with an opt-out attribute for deliberate cases, and a build script check warns when the profile uses `panic = "abort"`, under which a handler panic kills the process instead of one actor. Syntactic only (it cannot see sharing behind a type alias), but it catches the common pattern, including the one LLM-written code reaches for first. Erlang gap: memory isolation. | | 7 | Request deadlock detection | **Client evidence:** requests awaited inside handlers exist (ethrex peer table), no cycle found; keep as a cheap safety net. `ActorError::WouldDeadlock` naming the cycle instead of a timeout. `ActorRef::request` takes only `&self`, so set a thread-local `CURRENT_ACTOR` only while the actor's own handler future is polled (and around the synchronous handler in thread mode); detached helpers create no edges. Keep a small wait set per actor for `join!`. Measure cost before enabling outside debug builds. **Approach chosen ([deadlock and stall detection](docs/research/deadlock-and-stall-detection.md)):** eager detection. Add a wait-for edge when a handler sends a request, walk the per-actor wait sets for a cycle, fail the request that would close it with `WouldDeadlock` naming the cycle, remove the edge on reply or timeout; a request to self fails at once (Erlang's `calling_self`). Waits on raw channels, locks or `spawn_blocking` create no edges, so it is a safety net, not a proof. Edges are exported to rank 30. | diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 8c26ddd8..65fee506 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1090,9 +1090,11 @@ lines from section 16. change. The waiter answers (`AckOk`, `AckFail` after publication or after an explicit `CleanupDeadline`) or abandons. The init timer is framework-owned. -- **The child's death** is `Die` (D, reason final), `Release` (name and - slot), then `Publish` (notices disposed of; the model flags any illegitimate - drop as lost). +- **The child's death** is D (reason final; for a non-ready resolution, D + is part of the resolution step), `ReleaseName`, `ReleaseSlot`, `Publish` + (notices disposed of; the model flags any illegitimate drop as lost), and + `Deliver` (one step per notice reaching P). Owned work finishes after + publication (`WorkDoneP`, `WorkDoneC`). - **After readiness:** continuations, messages from others, a trapping child's queued parent exit, unlinking. - **S8:** owned children, the owner gate, the polite `Shutdown` at P's @@ -1115,13 +1117,15 @@ lines from section 16. | `ResultMatchesCommit`, `ReadyIsFinal` | answers match the first commitment; ready is never answered as an error | | `TimeoutNeverKillsReady`, `ReadyOnlyWhileOpen` | the timer loses to readiness; readiness only while admission (group and owner) is open | | `FailureAfterPublication`, `ReportedReasonFinal`, `PendingOnlyAfterDeadline` | a final answer comes after publication, with the final reason; a pending one only after the cleanup deadline | +| `KillBeforeDIsKilled`, `IgnoredExitsNormal` | a kill that arrived before D makes the final reason `Killed`; an ignored start always ends `Normal` (its resolution and D are one step; round 3) | +| `OwnerRefusalOnlyOnShutdown` | an owned child's readiness is refused by the owner gate only after an ancestor's or the system's shutdown, never because P is exiting (round 3) | | `FailureFlushed`, `FailureEndsNonTrappingParent`, `NormalFailureSparesParent` | a failed start's notice is consumed, yet a non-`Normal` one still ends a non-trapping P, and a `Normal` one never does | | `TimeoutContained` | an init timeout reaches P only after an abandoned wait | | `AdmittedOnlyOpen`, `OwnedOnlyWhileOwnerOpen` | admission by the target group and the owner gate | | `CapacityRespected`, `SlotHeldUntilReleased`, `ParentSlotHeld` | slots (P's included) are held until release (P's until its publication) | | `NoticeOnce`, `NoLostNotice`, `AbandonedNoticeDelivered` | at most one of each notice; none dropped illegitimately; an abandoned wait hands it on with its kind and reason, ending a non-trapping P | | `ContinueFirst` | no mailbox entry before the continuation | -| `GenDoneMeansOwnedDone`, `DeadlineKillsAllOwned` | the generation ends after P's and every member's publication; the scope deadline kills every live member, exiting ones included | +| `GenDoneMeansOwnedDone`, `DeadlineKillsAllOwned` | the generation ends after P's and every member's publication and owned work; the scope deadline kills every live member, exiting ones included | **Liveness** (weak fairness on the child's own steps, its death, release and publication, the waiter's answer, and P's D and publication; strong @@ -1152,7 +1156,7 @@ error has been found"): **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time): -Every control produced its counterexample in 1-12 s: +Every control produced its counterexample (round 3 rerun): | Configuration | Fault | Reintroduces | Fails | |---|---|---|---| @@ -1180,15 +1184,18 @@ Every control produced its counterexample in 1-12 s: | `MCStartTimeout` | `TimerIgnoresCommit` | the timer kills without checking the commitment | `TimeoutNeverKillsReady` | | `MCStartTimeout` | `KillBeforeUnlink` | the timeout's `Killed` reaches a waiting parent | `TimeoutContained` | | `MCStartTimeout` | `ReadyAfterClose` | readiness after admission closed | `ReadyOnlyWhileOpen` | -| `MCStartTimeout` | `FrozenReason` | the answer reports the requested reason after publication | `ReportedReasonFinal` | +| `MCStartTimeout` | `SkipEscalation` | a kill that arrives before D does not make the reason `Killed` (round 3) | `KillBeforeDIsKilled` | +| `MCStartTimeout` | `SplitResolutionD` | revision 3: a window between a non-ready resolution and D, so a kill turns `Ignored` into `Killed` (round 3) | `IgnoredExitsNormal` | | `MCStartCap` | `AdmitClosed` | a closed group admits | `AdmittedOnlyOpen` | | `MCStartCap` | `IgnoreCap` | a full group admits | `CapacityRespected` | | `MCStartCap` | `EarlySlotRelease` | the slot released at D | `SlotHeldUntilReleased` | | `MCStartContinue` | `ContinueQueued` | the continuation queued behind earlier entries | `ContinueFirst` | | `MCStartScope` | `EarlyGenDone` | the generation ends before its members | `GenDoneMeansOwnedDone` | | `MCStartScope` | `GenDoneAtParentD` | the generation ends at P's D | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `GenDoneIgnoresWork` | the generation ends while owned work runs (round 3) | `GenDoneMeansOwnedDone` | | `MCStartScope` | `NoScopeKill` | the scope deadline spares exiting members | `DeadlineKillsAllOwned` | | `MCStartScope` | `OwnerUnchecked` | the commit ignores the owner's gate | `OwnedOnlyWhileOwnerOpen` | +| `MCStartScope` | `ParentExitClosesOwner` | P's own exit closes its scope gate, refusing a child OTP lets finish init (round 3) | `OwnerRefusalOnlyOnShutdown` | | `MCStartScope` | `ParentSlotAtD` | P's slot released at its D | `ParentSlotHeld` | | `MCStartLive` | `DropHeld` (liveness) | a held parent exit dropped | `ParentExitReachesChild` | | `MCStartLive` | `NoAnswerIgnore` (liveness) | an ignored start never answered | `StartAnswered` | @@ -1488,3 +1495,18 @@ model and the 0.8 code. | agy F7 (P2): model gaps | **True** | **Accepted.** `stop(Normal)` and `stop(Shutdown)` outcomes; `NormalFailureSparesParent`; `NoOverlapOnReuse` over all live phases; `commit0` | | agy F8 (P2): `SExit` vs `SResolve` | **True** | **Accepted.** Pre-readiness exits are recorded only as `SResolve` (16.2) | +### Round 3 (confirmation; approved for implementation after these changes) + +Both reviews: approve after the listed changes, no P0. Codex confirmed the +`stopped()` change against OTP 27.1 `gen_server.erl` and the partial +acceptance of agy P1-5. The rejection of agy P1-10 stands. Each fix is its +own commit. + +| Finding | Claim checked | Resolution | +|---|---|---| +| Codex 1 = agy F1: a pre-D kill after a non-ready resolution corrupts the outcome; a recorded pre-D kill need not affect the reason | **True.** Revision 3 had a window between a non-ready resolution and D. A kill there turned `Ok(Ignored)` or `InitFailed` into `Killed` and could kill a non-trapping parent. Separately, `KillExiting` was optional, so a pre-D kill could be skipped | **Accepted, one rule.** A non-ready resolution and D are one critical section: the decision wins, its reason is final, and a later kill finds the actor past D. A ready child's pre-D kill escalates atomically on arrival. New: `IgnoredExitsNormal` (control `SplitResolutionD`), `KillBeforeDIsKilled` (control `SkipEscalation`). `FrozenReason` became vacuous (the reason is final at the decision) and is retired | +| Codex 2: no valid release linearization point; repeated lookups; disposition vs delivery | **True** | **Accepted.** `ReleaseName` (registry lock) and `ReleaseSlot` (group lock) are separate model actions and trace events. Lookups repeat as stuttering steps. Delivery is its own `Deliver` step (`PNotice`), with a refinement rule and new trace controls (5.2, 16.2) | +| Codex 3: generation completion checks publication alone | **True** | **Accepted.** Owned work is modelled (`WorkDoneP`, `WorkDoneC`); `GenDone` and `GenDoneMeansOwnedDone` require it (control `GenDoneIgnoresWork`) | +| agy F2: P's exit closes its scope gate, so a trapping owned child cannot commit ready (OTP lets it) | **True:** the design text contradicted the model and OTP | **Accepted, OTP semantics.** P's own exit refuses new starts (`ParentExited`, under P's exit-request lock) but does not close its scope gate. A child in `started()` finishes init and then gets P's exit; only an ancestor's or the system's shutdown closes the gate. New: `OwnerRefusalOnlyOnShutdown` (control `ParentExitClosesOwner`) | +| Codex, agy: the `stopped()` change is missing from section 12 | **True** | **Accepted.** Migration row: move initialization cleanup into `started()` or RAII | + diff --git a/verification/tla/README.md b/verification/tla/README.md index f8b8bcaa..fc533acd 100644 --- a/verification/tla/README.md +++ b/verification/tla/README.md @@ -17,7 +17,7 @@ this fits with Loom, Shuttle, Miri, ThreadSanitizer and mutation testing. | `DeferredReplies.tla`, `MCDeferredReplies*.tla/.cfg` | Rank 11 design model (not implemented), part 1: reply slots, send-time deadlines with shortening, deferral credits shared by a token and its first linked job, the exiting phase with unwind-local drops, a stuck phase (cleanup deadline passed while `stopped()` runs: nothing closed early, death later or never), death, and downstream request cleanup. Thirteen named negative controls. | | `DeferredJobs.tla`, `MCDeferredJobs*.tla/.cfg` | Rank 11 design model, part 2: pipe jobs with separate execution, cancellation and notification states; keyed replacement including notified jobs; optional job deadlines; a FIFO mailbox with emit credits and the terminal message last; `ActorId` fencing. `MCDeferredJobsEmits` is the fast configuration with emits. Nine named negative controls. | | `Lifecycle.tla`, `MCLifecycle{,Live,Fence,Deep,LiveDeep,WideDeep}.tla/.cfg` | Base actor lifecycle as proposed for v0.8 ([design](../../docs/design/otp-exit-signals.md)): links, monitors, `trap_exit`, exit signals (`Killed`, ignored `Normal`, `NoProc`, parent exits), administrative stops, requests around death, `Exit`/`Down` ordering. Seventeen negative-control faults (17 safety checks, 2 liveness checks): the v0.7 exit-signal behaviours (direct stops ignoring `trap_exit`, kills ending `Kill`, unfenced `unlink`/`demonitor`, real reasons for late registrations, `stopped()` after kills and untrapped signals, late signals ignored during exit), the PR #182/#185 bugs, early exit errors, kill and panic precedence, runtime-drop reasons, early notification, a split death point and a parent exit reaching `exit_received`. Per PR: `MCLifecycle` (7.5M states, fewer reasons, where the safety controls run), `MCLifecycleFence`, `MCLifecycleLive`; weekly: `MCLifecycleWideDeep` (the former 43M-state per-PR instance), `MCLifecycleDeep`, `MCLifecycleLiveDeep`. | -| `Start.tla`, `MCStart{,Names,Cap,Timeout,Continue,Scope,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design revision 3 ([design](../../docs/design/unified-start.md), not implemented; TLC-only in `model-map.toml` until trace validation lands with the implementation): `spawn`, `start`, `start_link`, `start_monitor`; placement inheritance; an atomic stage-1 commit with the owner gate; an immutable startup commitment and a waiter; death as D, release, then publication, for the child and the parent; the final reason fixed at D; fallible init with held-signal precedence; the continuation; S8 scope ownership with the polite Shutdown and the scope deadline. 37 negative controls (34 safety, 3 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartNames`, `MCStartCap`, `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | +| `Start.tla`, `MCStart{,Names,Cap,Timeout,Continue,Scope,Live,Deep}.tla/.cfg` | Unified start API, 0.9 design (approved after three review rounds) ([design](../../docs/design/unified-start.md), not implemented; TLC-only in `model-map.toml` until trace validation lands with the implementation): `spawn`, `start`, `start_link`, `start_monitor`; placement inheritance; an atomic stage-1 commit with the owner gate; an immutable startup commitment and a waiter; death as D, release, then publication, for the child and the parent; the final reason fixed at D; fallible init with held-signal precedence; the continuation; S8 scope ownership with the polite Shutdown and the scope deadline. 40 negative controls (37 safety, 3 liveness); see [Unified start](#unified-start-09-design). Per PR: `MCStart`, `MCStartNames`, `MCStartCap`, `MCStartTimeout`, `MCStartContinue`, `MCStartScope`, `MCStartLive` (CI leg `start`); weekly: `MCStartDeep`. | | `TraceCohorts.tla`, `TraceNested.tla` | Policy and ownership projections of real cohort/nested Rust executions. | | `trace_check.py`, `negative_checks.py` | Replay recorded traces; deliberately wrong selections, premature resume/join and broken propagation must be rejected or produce counterexamples. | | `SupervisionInd.tla`, `CohortsInd.tla`, `NestedSupervisionInd.tla`, `CoordinatorInd.tla`, `MC*Ind.tla`, `apalache.sh` | Inductive invariants (`IndInv`) checked with Apalache; see [Inductive invariants](#inductive-invariants-apalache). | @@ -608,7 +608,7 @@ All 22 named controls find their counterexample, each in a few seconds. ## Unified start (0.9 design) `Start.tla` backs the [unified start design](../../docs/design/unified-start.md) -(revision 3). It was written before the implementation, as the +(approved after three review rounds). It was written before the implementation, as the deferred-replies models were. Design section 15 lists what it covers, its invariants and what it leaves out. It is TLC-only in `verification/model-map.toml` until the implementation adds trace @@ -654,15 +654,18 @@ validation (design section 16.2). | `MCStartTimeout` | `TimerIgnoresCommit` | `TimeoutNeverKillsReady` | | `MCStartTimeout` | `KillBeforeUnlink` | `TimeoutContained` | | `MCStartTimeout` | `ReadyAfterClose` | `ReadyOnlyWhileOpen` | -| `MCStartTimeout` | `FrozenReason` | `ReportedReasonFinal` | +| `MCStartTimeout` | `SkipEscalation` | `KillBeforeDIsKilled` | +| `MCStartTimeout` | `SplitResolutionD` | `IgnoredExitsNormal` | | `MCStartCap` | `AdmitClosed` | `AdmittedOnlyOpen` | | `MCStartCap` | `IgnoreCap` | `CapacityRespected` | | `MCStartCap` | `EarlySlotRelease` | `SlotHeldUntilReleased` | | `MCStartContinue` | `ContinueQueued` | `ContinueFirst` | | `MCStartScope` | `EarlyGenDone` | `GenDoneMeansOwnedDone` | | `MCStartScope` | `GenDoneAtParentD` | `GenDoneMeansOwnedDone` | +| `MCStartScope` | `GenDoneIgnoresWork` | `GenDoneMeansOwnedDone` | | `MCStartScope` | `NoScopeKill` | `DeadlineKillsAllOwned` | | `MCStartScope` | `OwnerUnchecked` | `OwnedOnlyWhileOwnerOpen` | +| `MCStartScope` | `ParentExitClosesOwner` | `OwnerRefusalOnlyOnShutdown` | | `MCStartScope` | `ParentSlotAtD` | `ParentSlotHeld` | | `MCStartLive` (liveness) | `DropHeld` | `ParentExitReachesChild` | | `MCStartLive` (liveness) | `NoAnswerIgnore` | `StartAnswered` | From c758db189cecc08def1c85eccf13a8c5d9cea0e8 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 17:16:52 -0300 Subject: [PATCH 41/42] verification: AbandonedNoticeDelivered keys on a live parent (the dropped hand-off control is non-vacuous again) --- verification/tla/Start.tla | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/verification/tla/Start.tla b/verification/tla/Start.tla index f6e6ecf8..1b07b73f 100644 --- a/verification/tla/Start.tla +++ b/verification/tla/Start.tla @@ -849,7 +849,8 @@ NoLostNotice == \A c \in Kids : ~lost[c] \* A notice handed on by an abandoned wait reaches P with its kind and reason. AbandonedNoticeDelivered == - \A c \in Kids : (handed[c] /\ obN[c] = 0 /\ dlvOk[c]) => + \* Once nothing is pending for c, a P whose mailbox still admits has it. + \A c \in Kids : (handed[c] /\ obN[c] = 0 /\ Alive(pst)) => /\ (kind[c] = "link" /\ ptrap) => exitN[c] >= 1 /\ exitR[c] = why[c] /\ (kind[c] = "link" /\ ~ptrap /\ why[c] \in NonNormal) => pst # "run" /\ kind[c] = "monitor" => downN[c] >= 1 /\ downR[c] = why[c] From d1d086b5562a961a47a23861c6993bda790daf67 Mon Sep 17 00:00:00 2001 From: Federico Carrone Date: Wed, 11 Nov 2026 19:33:49 -0300 Subject: [PATCH 42/42] docs: round-three TLC results (every config with TLC's verdict) --- docs/design/unified-start.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/design/unified-start.md b/docs/design/unified-start.md index 65fee506..5a794746 100644 --- a/docs/design/unified-start.md +++ b/docs/design/unified-start.md @@ -1144,14 +1144,14 @@ error has been found"): | Configuration | Run | Instance | Distinct states | Depth | Time | |---|---|---|---|---|---| -| `MCStart` | per PR | one link/monitor child of a managed parent that may trap and exit: every init outcome and notice disposition, abandonment, the cleanup deadline | 7,086 | 13 | 1 s | -| `MCStartNames` | per PR | two named `start` children in P's group, closure, lookups, refusals, retry | 703,268 | 22 | 10 s | -| `MCStartCap` | per PR | three children spawned into P's group of three | 628,622 | 26 | 11 s | -| `MCStartTimeout` | per PR | one child of each waiting variant; init timer, outside kills (also before D), administrative stops, closure; managed or unmanaged parent | 258,868 | 17 | 4 s | -| `MCStartContinue` | per PR | one child of every variant and explicit placement, two messages, trapping | 2,017,292 | 17 | 19 s | -| `MCStartScope` | per PR | S8: one linked child owned (`G`, `G2`) or not, trapping, unlink, owner closure, P's D and publication, polite Shutdown, scope deadline; liveness `OwnedScopeFinishes` | 221,072 | 17 | 15 s | -| `MCStartLive` | per PR | liveness `ParentExitReachesChild`, `StartAnswered` | 29,834 | 14 | 4 s | -| `MCStartDeep` | weekly | two named link/monitor children of an exiting parent (`MCStart` and `MCStartNames` combined) | 30,230,014 | 24 | 5 min 19 s | +| `MCStart` | per PR | one link/monitor child of a managed parent that may trap and exit: every init outcome and notice disposition, delivery, abandonment, the cleanup deadline | 18,810 | 17 | 1 s | +| `MCStartNames` | per PR | two named `start` children in P's group, closure, lookups, refusals, name and slot release, retry | 1,396,516 | 26 | 16 s | +| `MCStartCap` | per PR | three children spawned into P's group of three | 1,386,734 | 32 | 19 s | +| `MCStartTimeout` | per PR | one child of each waiting variant; init timer, outside kills (also before D), administrative stops, closure; managed or unmanaged parent | 374,244 | 20 | 5 s | +| `MCStartContinue` | per PR | one child of every variant and explicit placement, two messages, trapping | 4,527,484 | 20 | 30 s | +| `MCStartScope` | per PR | S8: one linked child owned (`G`, `G2`) or not, trapping, unlink, owner closure, P's D and publication, polite Shutdown, scope deadline, owned work; liveness `OwnedScopeFinishes` | 419,228 | 20 | 23 s | +| `MCStartLive` | per PR | liveness `ParentExitReachesChild`, `StartAnswered` | 68,400 | 17 | 6 s | +| `MCStartDeep` | weekly | two named link/monitor children of an exiting parent (`MCStart` and `MCStartNames` combined) | 182,156,086 | 31 | 29 min 47 s | **Negative controls** (`negative_checks.py`, each with `TypeOK` and only the named property, run one at a time):