[unbalancedparentheses]

Determinism backlog D1-D3 (ROADMAP tier 3, docs/research/determinism-audit.md). All of it applies under --cfg spawned_sim only; normal builds change only by the new spawned_rt::tasks items and the Sync bound on sim_test bodies.

D1: wall clock and OS entropy (interposition, plus strict mode)

Design: interposition (madsim's technique), with strict mode on top. A new crate, spawned-sim-libc, defines clock_gettime, gettimeofday, getrandom, getentropy, on Linux the syscall entry for SYS_getrandom (the getrandom 0.2 crate behind rand 0.8's thread_rng/OsRng), and on macOS CCRandomGenerateBytes (std's HashMap keys there). The test binary's references (std's Instant/SystemTime, std's weak-linked getrandom for HashMap keys, rand, chrono) bind to them at link time. Each one asks hooks that spawned-rt installs, and otherwise calls libc through dlsym(RTLD_NEXT).

• Clock. While a world runs on the thread (inside its block_on), Instant::now() reads sim::VIRTUAL_BOOT (1 day) plus virtual time and SystemTime::now() reads sim::VIRTUAL_EPOCH (2000-01-01) plus virtual time. A real Tokio runtime driven on the same thread outside the simulation keeps real time.
• Entropy. On any thread with a world, reads come from a per-world stream seeded by the seed. It is separate from scheduling and from spawned_rt::time::entropy.
• Fresh thread per run. run_once/explore/sim_test run each seed on a fresh OS thread. std's HashMap keys and rand::thread_rng are seeded once per thread, so this seeds them inside the run: a seed in a sweep, the same seed alone, a tape replay and a separate process see the same values. The cost is a Sync bound on bodies (MIGRATION.md).
• Simulator internals. The poll watchdog, the turn budget and Tokio's own runtime metrics around a poll read the real clock (sim::wall_now(), real_clock).
• Strict mode. SPAWNED_SIM_STRICT=1 / sim::set_strict(true) records the first direct read with its backtrace, and the next scheduling step fails the run with the caller's frames, the way a direct tokio::spawn fails today. HashMap key generation (GRND_INSECURE) is never reported.

Unsafe containment. spawned-sim-libc is #![cfg(spawned_sim)] at its root, so it compiles to an empty crate in every other build. spawned-rt depends on it only under [target.'cfg(spawned_sim)'.dependencies]. spawned-rt, spawned-concurrency, spawned-macros and spawned-test keep #![forbid(unsafe_code)] in every build. Inside the crate: deny(unsafe_op_in_unsafe_fn), a // SAFETY: on every block, hooks behind catch_unwind (a panic falls back to libc), a re-entrancy flag so a hook's own reads go to libc, and const thread-locals that are safe during thread teardown. It is published alongside spawned-rt (release.yaml and the packaging check updated).

Limits (TESTING.md):

• Only Linux/glibc (the syscall entry on x86_64/aarch64) and macOS are interposed (sim::INTERPOSED).
• getrandom 0.3+ on Linux finds the symbol with dlsym(RTLD_DEFAULT) and is not covered unless the binary exports it.
• A blocking wait with a timeout on the simulated thread computes its kernel deadline from virtual time.

D2: spawn_blocking inline

spawn_blocking (and Handle::spawn_blocking, JoinSet::spawn_blocking, actor spawn_blocking_helpers) runs its closure as the single poll of a new simulated task. That is one atomic step, never concurrent with anything else in the world. block_in_place calls its closure. A nested block_on now fails the run even when its panic lands in a task nobody joins, so a Backend::Blocking actor fails the run instead of silently never starting.

D3: timers and task sets

spawned_rt::tasks::{sleep_until, interval, interval_at, Interval, MissedTickBehavior, JoinSet, AbortHandle, JoinError} are virtual under the simulator and Tokio's otherwise, with Tokio's semantics. Under the simulator, join_next breaks ties by task id. Tokio's "timers are disabled" panic, from the root or a detached task, now fails the run with a message naming spawned_rt::tasks::{sleep, sleep_until, timeout, interval}.

Tests

• rt/tests/sim_determinism.rs:
  • The same seed gives identical Instant (absolute and steps), SystemTime, thread_rng, OsRng, getrandom and HashMap/HashSet order across runs, across a sweep versus alone, across a tape replay, and across two separate processes.
  • A different seed differs.
  • Another thread reads real time.
  • Strict mode names the caller.
• rt/tests/blocking_and_timers.rs runs on Tokio and on the simulator. Under the simulator it also checks exact virtual instants for Burst/Delay/Skip, inline spawn_blocking, the nested-block_on failure, deterministic JoinSet order, and the direct tokio::time message.
• concurrency/tests/sim.rs: spawn_blocking runs inline, and a Blocking-backend actor fails the run.
• The full simulated suite passes on macOS and Linux (fede, x86_64).

Model: no change because this changes only the simulator backend (spawned-rt) and its tests, not modelled library code.

[github-actions]

🤖 Codex Code Review

• [P2] Match Tokio’s missed-tick threshold — rt/src/tasks/sim/interval.rs:84. Tokio applies MissedTickBehavior only when a tick is over 5 ms late; this implementation applies it unconditionally. With a 10 ms period and Delay, consuming the tick scheduled for 10 ms at 11 ms schedules the next tick at 21 ms here, versus 20 ms in production. This can change actor timer and timeout ordering under simulation. Preserve fired + period within Tokio’s tolerance and add coverage for slightly late ticks.

Tests could not run because rustup attempted to write to a read-only directory.

---

Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt

[github-actions]

🤖 Claude Code Review

Review: PR #267, Determinism D1–D3

This is solid work overall. The unsafe code is well isolated in spawned-sim-libc, SAFETY comments are thorough, re-entrancy is guarded (IN_HOOK, real_clock/intercepted), and the tests cover cross-seed, replay and cross-process determinism. Most issues below are about robustness and edge cases, not correctness of the happy path.

High

1. now_nanos mirror can go stale (rt/src/tasks/sim.rs, Action::Fire arm, ~L936)

• now_nanos is updated only in the Fire branch.
• The module docs say the clock also "creeps by 1 ms" after SPAWNED_SIM_SPIN busy polls. If that path, or any other path, writes state.now directly, std::time::Instant::now() will diverge from spawned_rt::time::Instant::now() inside a run.
• Please audit every write to state.now, and route them all through one helper so the mirror can't be forgotten. For example: fn set_now(&self, state: &mut State, t: Duration).

2. HashMap-key seeding depends on construction order (Shared::new, run_on_this_thread)

• std seeds RandomState keys once per thread, on the first RandomState::new().
• interpose::install() runs in Shared::new, but the world is not in AMBIENT/CURRENT until later. Any HashMap::new() on the fresh run thread before registration takes OS keys and silently breaks D1 for the whole run. That includes one inside State, explore::Plan, tape parsing, or tracing setup.
• The tests pass today, so nothing does this yet, but it's fragile. Consider:
  • registering the ambient world before building State, or
  • having run_on_this_thread force the first RandomState::new() right after registration, with a comment explaining why.

3. spawn_blocking diagnostics regressed for long-running closures

• Previously, spawn_blocking failed fast with a clear message.
• Now a closure that blocks indefinitely hangs until the 60 s poll watchdog aborts the entire test process. Examples: std::sync::mpsc::Receiver::recv, a dedicated loop {} consumer, a Condvar::wait.
• Tokio's blocking_recv still panics fast because the poll runs in a Tokio context. std primitives do not.
• Consider a dedicated, shorter wall-clock budget for blocking-task polls, analogous to SPAWNED_SIM_TURN_BUDGET, that fails the run instead of aborting.

4. Thread-per-run drops caller thread-locals (run_planned, ~L2131)

• QUIET is propagated, but nothing else is.
• Scoped tracing subscribers (tracing::subscriber::set_default / with_default) are lost.
• Any per-thread "ambient world" config set before run_once (e.g. sim::set_turn_budget, sim::set_strict, which use current_or_ambient()) now targets the caller's thread, not the run's. Please verify this doesn't change the existing semantics of set_turn_budget when called outside a run.
• harness = false binaries calling from main drop from an 8 MiB to a 2 MiB stack. Consider builder.stack_size(...) honoring RUST_MIN_STACK, or a larger default.

Medium

5. Duplicate #[no_mangle] symbols across copies (sim-libc/Cargo.toml)

• Two semver-incompatible copies of spawned-sim-libc, or madsim alongside it, would fail at link time with duplicate clock_gettime / syscall symbols.
• Add links = "spawned_sim_libc" to the [package] section so Cargo rejects multiple copies with a clear resolver error.

6. Interposers are not async-signal-safe (sim-libc/src/lib.rs, clock_gettime)

• clock_gettime is async-signal-safe in libc. The interposed version is not: it touches TLS, clones an Arc, and in strict mode takes a Mutex and captures a Backtrace.
• A signal handler on the world thread that reads the clock (profilers, some crash handlers) could deadlock.
• At minimum, document this in the crate's limitations.

7. Sim and Tokio API surfaces diverge (rt/src/tasks/tokio/mod.rs)

• The non-sim build re-exports Tokio's full JoinSet, AbortHandle and JoinError. The sim versions are a subset: no spawn_local, spawn_on, or JoinError::id.
• Code using those methods compiles normally but breaks under --cfg spawned_sim.
• The Interval wrapper already shows the right pattern. A thin JoinSet newtype over Tokio's would keep both cfgs identical.
• Similarly, the sim Sleep is likely Unpin while tokio::time::Sleep is not, so &mut sleep in select! compiles under sim only.

8. Interval missed-tick semantics differ from Tokio (rt/src/tasks/sim/interval.rs, poll_tick)

• Tokio applies MissedTickBehavior only when now > timeout + 5ms, and otherwise uses timeout + period.
• Here, Delay and Skip always use now.
• After a clock creep (now slightly past the deadline), Delay drifts where Tokio wouldn't. Mirror the 5 ms tolerance.
• Nit: (period - late % period) as u64 silently truncates from u128.

9. JoinSet::poll_next_entry is O(n) per wake

• It re-polls every JoinHandle on each wake, which is quadratic for draining large sets.
• This is acceptable for tests, but a short comment would help. A "finished" queue keyed by task id would fix it if large sets matter.

10. MSRV

• Waker::noop() in join_set.rs (try_join_next) needs Rust 1.85.
• c"..." literals need Rust 1.77.
• Confirm these against the workspace rust-version.

11. Breaking change in normal builds

• The new F: Sync bound on sim_test / SimTest::run applies outside sim too.
• It's documented in MIGRATION.md, but it should ship in 0.8, not a 0.7.x patch.

Low / Nits

• Cargo.lock: unrelated windows-sys churn (0.48/0.52 → 0.61.2 for errno, rustix, tempfile, etc.). Intentional?
• tokio_time_panic: relies on substring-matching Tokio's message. It's covered by direct_tokio_time_names_the_replacement, which is good. Note that it also rewrites any user panic containing that substring.
• Root catch_unwind (~L1103): clearing state.foreign when it equals DIRECT_TOKIO_TIME also discards a leak recorded earlier by a detached task. That's harmless since the run fails anyway, but worth a comment.
• Shared::strict: uses std::sync::atomic::AtomicBool by full path while AtomicU64 is imported. Import it for consistency.
• syscall interposer: it now sits in front of every libc::syscall call in the binary (std's futex, statx, etc.). It's fine on x86_64/aarch64 as written, but a one-line note in the crate docs would help future readers.

---

Automated review by Claude · claude-opus-5-5 · custom prompt

[github-actions]

🤖 Gemini Code Review

This PR is very well-structured and thoroughly tested. Isolating the unsafe libc interposers into spawned-sim-libc under #![cfg(spawned_sim)] while running each harness seed on a fresh OS thread is a clean way to get deterministic Instant/SystemTime/HashMap/thread_rng behavior without sacrificing #![forbid(unsafe_code)] in the main crates.

A few small issues and suggestions below:

1. sim::JoinSet::try_join_next overwrites pending task wakers with Waker::noop()

File: rt/src/tasks/sim/join_set.rs:163-169

    pub fn try_join_next(&mut self) -> Option<Result<T, JoinError>> {
        match self.poll_join_next(&mut Context::from_waker(Waker::noop())) {
            Poll::Ready(result) => result,
            Poll::Pending => None,
        }
    }

In rt/src/tasks/sim.rs, JoinHandle::poll sets inner.waker = Some(cx.waker().clone()) whenever a task is not yet finished. Calling poll_join_next with Waker::noop() iterates over all unfinished tasks ahead of the first finished one (or all tasks if none are finished) and overwrites any previously registered waker with a no-op waker. If a caller polls join_next() in a select! or custom future and calls try_join_next() while that future is pending, the waiting task will lose its wakeup.

Checking is_finished() first avoids touching unfinished tasks:

    pub fn try_join_next(&mut self) -> Option<Result<T, JoinError>> {
        let id = *self.tasks.iter().find(|(_, h)| h.is_finished())?.0;
        let mut handle = self.tasks.remove(&id)?;
        match Pin::new(&mut handle).poll(&mut Context::from_waker(Waker::noop())) {
            Poll::Ready(result) => Some(result),
            Poll::Pending => unreachable!("finished task must be ready"),
        }
    }

2. sim::AbortHandle is missing Clone

File: rt/src/tasks/sim/join_set.rs:22-26

tokio::task::AbortHandle implements Clone, which is commonly used to store abort handles alongside task metadata or pass them into cancellation callbacks. Because sim::AbortHandle stores Box<dyn Fn() + Send + Sync>, code that clones an AbortHandle will compile in normal builds but fail under --cfg spawned_sim.

Wrapping the closures (or a small helper trait) in Arc instead of Box allows #[derive(Clone)] on sim::AbortHandle. (Optionally, exposing JoinHandle::abort_handle(&self) -> AbortHandle on sim::JoinHandle would also match Tokio's JoinHandle API.)

3. Move self.tokio() inside real_clock in Shared::run_poll

File: rt/src/tasks/sim.rs:744-750

In Shared::run_poll, let runtime = self.tokio(); is called while CURRENT is active (running() is Some) and before entering spawned_sim_libc::real_clock(|| ...). On the first poll of a run, self.tokio() lazily constructs the driverless Tokio runtime; moving let runtime = self.tokio(); inside the spawned_sim_libc::real_clock closure ensures any clock reads Tokio makes during runtime initialization are treated the same as its per-poll runtime metrics reads and cannot trigger a false positive when SPAWNED_SIM_STRICT=1 is set via the environment.

---

Automated review by Gemini · gemini-4-argon · custom prompt

[github-actions]

🤖 Grok Code Review

I'll verify a few high-risk spots in the simulator—time mirroring, JoinHandle drop, and JoinSet polling—before writing the review.

---

Automated review by Grok (xAI) · grok-4.7 · custom prompt

[github-actions]

🤖 DeepSeek Code Review

⚠️ Review failed: DeepSeek returned an empty content field (finish_reason=length, reasoning_content present=yes). Raise max_tokens or lower reasoning_effort ('none' disables thinking).

---

Automated review by DeepSeek

[github-actions]

🤖 Kimi Code Review

Overall, the simulator changes are well documented and tested. I found no blocking actor-lifecycle or synchronization issues, but a few non-blocking concerns:

• sim-libc/src/lib.rs, interposed gettimeofday: initialize or reject non-null tz. The virtual-time path returns success after writing only tv. libc also initializes the timezone structure when tz is non-null; leaving it untouched can let callers read uninitialized data. Either populate it or forward calls with non-null tz to libc.

• sim-libc/src/lib.rs, random: validate len before constructing the slice. slice::from_raw_parts_mut requires the total size not exceed isize::MAX, but this exported FFI path accepts an arbitrary size_t. Add a len > isize::MAX as usize guard and forward oversized requests to the real function. This also avoids overflowing len as ssize_t in interposed getrandom.

• rt/src/tasks/sim/join_set.rs:23-43: simulated AbortHandle lacks Clone. Tokio’s AbortHandle, reexported in normal builds, is cloneable. Backend-agnostic code that clones the handle will therefore compile normally but fail under spawned_sim. Storing the closures in Arc rather than Box would make Clone straightforward.

• rt/src/tasks/sim/join_set.rs:139-157: draining a JoinSet is potentially O(n²). Every join_next scans and polls handles from the beginning of the BTreeMap. This is acceptable for small simulation sets, but could become expensive for larger client workloads; a finished-task queue plus set waker would avoid repeatedly polling unfinished tasks.

The fresh-thread harness, watchdog use of the real clock, and propagation of detached-task timer/nested-block_on failures look particularly well handled.

---

Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt