From 98aa9ee7711af477538c21e61d1f003c6b7192a3 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 22:56:47 +0530 Subject: [PATCH 1/3] UTS: make AWAIT_STATE subscribe before checking the current state writing-test-specs.md "State Transitions" told implementations to check whether the condition already holds and only then wait for state change events. With asynchronous event delivery a transition that happens between the check and the subscription is missed, so the helper hangs until its timeout. Subscribe first, then check, complete exactly once, and remove the listener on every exit path. The observable semantics of AWAIT_STATE are unchanged. The translator-skills guide already required this and labelled it a divergence from writing-test-specs.md; drop that label and cite the updated section instead. --- uts/docs/translator-skills/writing-translator-skills.md | 4 ++-- uts/docs/writing-test-specs.md | 8 +++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index eaebbc7b3..550d0fa4d 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -561,7 +561,7 @@ Every rule below is a MUST unless marked otherwise. ### 5.7 Waits must catch events -- **`AWAIT_STATE` must be event-latched and race-free**: subscribe first, then check the current state, and complete exactly once (whichever of the listener and the immediate check wins). Checking before subscribing leaves a window in which the transition is missed. ably-java's `awaitState` does this, with an atomic single-winner resume. *Divergence from `writing-test-specs.md`* ("State Transitions" lists check, then wait): the observable semantics are the same, but a check-then-subscribe implementation misses a transition that happens between the two calls, so subscribe first. The helper also removes its listener on success, timeout and cancellation; leaked listeners keep firing into later tests. +- **`AWAIT_STATE` must be event-latched and race-free**: subscribe first, then check the current state, and complete exactly once (whichever of the listener and the immediate check wins), as `writing-test-specs.md` "State Transitions" requires. Checking before subscribing leaves a window in which the transition is missed. ably-java's `awaitState` does this, with an atomic single-winner resume. The helper also removes its listener on success, timeout and cancellation; leaked listeners keep firing into later tests. - **A state wait fails fast on FAILED.** If the connection enters FAILED while waiting for another state, fail immediately with the error reason instead of running out the timeout (ably-cocoa's integration `awaitState`). Don't treat a state that is already FAILED at the start of the wait as a failure when the test has just requested a transition out of it (`connect()`, or `authorize()` from FAILED per RTC8c); latch the next transition instead. - **Transient states use record-and-verify** (`writing-test-specs.md` "Verifying Transient States"): record all state changes, let the cycle finish, then assert `CONTAINS_IN_ORDER`. Even a correct latch misses a state entered and left before the call. Don't insert an intermediate `AWAIT_STATE disconnected`. - **Polls return the settled value.** For a value-assigned `x = poll_until(…)`, the helper returns the value and later assertions use it. Don't re-read afterwards: an eventually-consistent store can return fewer items on the second read. @@ -603,7 +603,7 @@ Fill in the last column for your language; the skill MUST map every construct it |---|---|---|---|---| | `AWAIT expr` | Await an async call | `try await op`, or a continuation-bridged helper for callback APIs | `.await()` on a future, or `suspendCancellableCoroutine` around a callback | | | `f = op()` … later `AWAIT f` (incl. `AWAIT f FAILS WITH error`) | Deferred future: start an operation, act, then await it | `let t = Task { try await op() }` … `try await t.value` (objects module notes) | Hold the `CompletableFuture` (``val getFuture = channel.`object`.get()``) … `getFuture.await()` (generated tests, e.g. `RealtimeObjectTest`) | | -| `AWAIT_STATE x.state == ConnectionState.connected` (also `ChannelState.attached`, bare `CONNECTED`) | Proceed if already true, otherwise wait for the state event; fail on timeout (`writing-test-specs.md` "State Transitions") | `awaitConnectionState(client, .connected)` (unit); `await awaitState(client, .connected)` (integration) | `awaitState(client, ConnectionState.connected)` | | +| `AWAIT_STATE x.state == ConnectionState.connected` (also `ChannelState.attached`, bare `CONNECTED`) | Subscribe, then proceed if already true, otherwise wait for the state event; fail on timeout (`writing-test-specs.md` "State Transitions"; [5.7](#57-waits-must-catch-events)) | `awaitConnectionState(client, .connected)` (unit); `await awaitState(client, .connected)` (integration) | `awaitState(client, ConnectionState.connected)` | | | `… WITH timeout: N seconds` (`Ns`, `N second`) | Bound on the wait; wall-clock at integration tier | `timeout:` parameter on helpers; no per-call timeout on native `await` (note it in a comment) | `awaitState(client, state, 10.seconds)`; `withRealTimeout` for integration futures | | | `AWAIT UNTIL `, `WAIT_FOR ` *(undocumented)* | Wait until the condition holds | — | — | | | `AWAIT_ERROR future` *(undocumented)* | Await a future expected to fail; bind its error | — | — | | diff --git a/uts/docs/writing-test-specs.md b/uts/docs/writing-test-specs.md index aa403e6b2..d20f238f8 100644 --- a/uts/docs/writing-test-specs.md +++ b/uts/docs/writing-test-specs.md @@ -650,9 +650,11 @@ AWAIT_STATE client.connection.state == ConnectionState.connecting ``` This means implementations should: -- Check if condition is already true -> proceed -- Otherwise wait for state change events with timeout -- Fail if timeout expires +- Subscribe to state change events first, then check whether the condition is already true, + and proceed on whichever happens first, exactly once (checking before subscribing leaves a + window in which a transition is missed) +- Otherwise wait for a state change event that satisfies the condition, with a timeout +- Fail if the timeout expires, and remove the listener on success, timeout or cancellation In **integration and proxy** specs these timeouts are **wall-clock time** (the test waits on a real server — see the *Timeout Strategy* section in `docs/integration-testing.md`); in unit specs they may be From 643a2bbc87f401823ca73cc093bc5461e63816e0 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 22:57:04 +0530 Subject: [PATCH 2/3] UTS: drain all queued work before negative assertions writing-derived-tests.md said a single event-loop yield is sufficient before a "prove a negative" assertion. That holds for one JavaScript macrotask queue, but where an SDK chains work across dispatch queues, threads or several loop iterations one yield leaves work pending and the negative assertion passes falsely. Both existing non-JS SDKs needed explicit drains. State the rule as "process_pending_events() drains all already-queued work", keep one yield as the JavaScript rendering, and point at an explicit drain or the objects control-signal quiescence pattern elsewhere. Align the README's process_pending_events() definition. The translator-skills guide labelled this a divergence from writing-derived-tests.md; drop that label and cite the updated rule. --- uts/README.md | 5 +++-- uts/docs/translator-skills/writing-translator-skills.md | 2 +- uts/docs/writing-derived-tests.md | 2 +- uts/docs/writing-test-specs.md | 3 ++- 4 files changed, 7 insertions(+), 5 deletions(-) diff --git a/uts/README.md b/uts/README.md index 252312331..d70c99ce6 100644 --- a/uts/README.md +++ b/uts/README.md @@ -142,8 +142,9 @@ Pseudocode maps to language idioms rather than prescribing exact syntax: unit tier before asserting, especially for negative assertions ("nothing happened"). Involves no real delay and is never rendered as a timed sleep; implementations define a shared helper using the platform's zero-delay yield (e.g. `flushAsync()` awaiting a `setImmediate` in - ably-js, or `runCurrent()` on a kotlinx-coroutines test scheduler) — see the timer guidance - in [docs/writing-derived-tests.md](docs/writing-derived-tests.md). + ably-js, or `runCurrent()` on a kotlinx-coroutines test scheduler), repeated or replaced by an + explicit queue drain where the SDK chains work across queues, threads or loop iterations — see + the timer guidance in [docs/writing-derived-tests.md](docs/writing-derived-tests.md). - **Language-inapplicable inputs**: a test input that cannot be constructed in a given language (e.g. a non-string map key in JavaScript, where object keys are always coerced to strings; or a `null` argument where the SDK's signature makes null indistinguishable from "omitted") makes diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index 550d0fa4d..b74bb0874 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -190,7 +190,7 @@ Where the SDK already has a test-options type, add the hooks to it, and implemen Most flakiness in both existing SDKs came from concurrency, not from translation. Before writing the skill, know: - Which thread or queue SDK callbacks fire on, and whether listeners can run concurrently with the test body. -- How to drain the SDK's internal work queue (this becomes `process_pending_events()`). One zero-delay yield (e.g. one `await asyncio.sleep(0)`) may not drain chained callbacks; loop until the queue is empty if you can observe it. *Divergence from `writing-derived-tests.md`* ("No real timers in unit tests"), which says one event-loop yield suffices for a negative assertion: that holds for a single JavaScript macrotask queue, but not where the SDK chains work across queues, threads or loop iterations (both existing SDKs needed explicit queue drains). Use one yield only where you have shown it drains everything. +- How to drain the SDK's internal work queue (this becomes `process_pending_events()`). One zero-delay yield (e.g. one `await asyncio.sleep(0)`) may not drain chained callbacks; loop until the queue is empty if you can observe it (`writing-derived-tests.md` "No real timers in unit tests"). One yield is usually enough for a single JavaScript macrotask queue (unless the SDK re-queues work from within a callback), but not where the SDK chains work across queues, threads or loop iterations (both existing SDKs needed explicit queue drains). Use one yield only where you have shown it drains everything. - Whether the test runner virtualises time (kotlinx `runTest` does; this breaks real-network timeouts), and, for async runners, the plugin mode and event-loop scope that suite-level fixtures need (e.g. pytest-asyncio). - **Whether the runner runs tests in parallel** (xUnit does by default; pytest-xdist and NUnit `[Parallelizable]` when enabled; Swift Testing unless `.serialized`). If any injected mock, fake clock or log sink is process-global, serialise the UTS suites or make the hooks per-client, and say which in the skill. ably-cocoa marks its core realtime/rest and integration suites `@Suite(.serialized)`; its objects unit suites, which inject nothing process-global, are plain parallel `struct`s. - Whether the language enforces concurrency safety at compile time (Swift 6 `Sendable` drove ably-cocoa's capture types). diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index 3f3c952cc..97e1decd0 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -297,7 +297,7 @@ Unit tests must not use real timers (`setTimeout`, `setInterval`, `sleep`, `dela - **For time-dependent SDK behaviour** (timeouts, retries, heartbeats): use fake timers that replace the SDK's timer API and can be advanced deterministically. - **For waiting on async event delivery** (mock message propagation, promise settlement): yield to the event loop with a zero-delay mechanism like `setImmediate`, `process.nextTick`, or equivalent. Define a `flushAsync()` helper and use it everywhere instead of `setTimeout(resolve, N)`. This is the rendering of the spec's `process_pending_events()` convention (see `uts/README.md`). -- **For "prove a negative" assertions** (confirming something did NOT happen): a single event-loop yield is sufficient — if the event hasn't fired after one pass through the macrotask queue, it won't fire from the current stimulus. +- **For "prove a negative" assertions** (confirming something did NOT happen): call `process_pending_events()` to drain all already-queued work, then assert. In a single-queue runtime (JavaScript) one event-loop yield usually does this, provided the SDK does not re-queue work from within a callback — if the event hasn't fired after one pass through the macrotask queue, it won't fire from the current stimulus. Where the SDK chains work across queues, threads or loop iterations, one yield is not enough and gives false passes: drain until the SDK's internal work is quiescent (e.g. loop until its queue is empty), or use a control-signal quiescence check as in [`objects/helpers/standard_test_pool.md`](../objects/helpers/standard_test_pool.md#negative-assertion-quiescence) "Negative-assertion quiescence". The only acceptable use of a real timer is a **safety timeout on test execution** — a long deadline (e.g. 5 seconds) that fails the test if an expected event never arrives, preventing the test from hanging indefinitely. This is a test-level safeguard, not a delay mechanism. diff --git a/uts/docs/writing-test-specs.md b/uts/docs/writing-test-specs.md index d20f238f8..15679c69a 100644 --- a/uts/docs/writing-test-specs.md +++ b/uts/docs/writing-test-specs.md @@ -843,7 +843,8 @@ wall-clock* in `writing-derived-tests.md` for the traps on both sides). For a **negative assertion** at unit tier — proving something did *not* happen — there is nothing to poll for: use `process_pending_events()` (see the pseudocode conventions in -`uts/README.md`) to let already-queued events settle, then assert. Never a fixed `WAIT`. +`uts/README.md`) to drain all already-queued work, then assert (see *No real timers in unit +tests* in `writing-derived-tests.md` for SDKs that chain work across queues). Never a fixed `WAIT`. ```pseudo # Good - negative assertion: settle pending events, then assert nothing happened From 3f4019fad4566f6f19c5cbdde3419bc3339b27d5 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 22:57:21 +0530 Subject: [PATCH 3/3] UTS: judge UTS specs against the features spec from the same revision writing-derived-tests.md 2a told translators to fetch the features spec from GitHub main. A UTS spec read from one commit and judged against a features spec at main HEAD can be misclassified as wrong (or right) whenever both changed together, which happens routinely in this repo. Require the features spec to be read from the same revision as the UTS spec: the same local clone, or GitHub at that commit; main only when the UTS spec itself came from main. The existing links stay as browsing links. The translator-skills guide labelled reading from the local clone a divergence from writing-derived-tests.md; turn it into a plain rule that cites 2a. --- uts/docs/translator-skills/writing-translator-skills.md | 4 ++-- uts/docs/writing-derived-tests.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index b74bb0874..49341711e 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -897,11 +897,11 @@ Examples of the pattern, as of 2026-10-07 (spec `12540dcf`): Pick **one** source for docs and specs, and pin it. This guide recommends **reading everything from the local spec clone** the module directory came from: - the UTS docs (`/../docs/writing-derived-tests.md`, `proxy.md`); -- in evaluate mode, the features specs the decision tree needs (`/../../specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. +- in evaluate mode, the features specs the decision tree needs (`/../../specifications/features.md`, `objects-features.md`, `protocol.md`), rather than fetching them from GitHub. Record the clone's HEAD SHA, whether it is dirty and which files are; a translated spec that is locally modified is handled as in 9.1. You MAY warn when the clone is behind `origin/main`. -- *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. +- Read the features specs from the same clone as the UTS specs: the UTS spec and the features spec it is judged against must come from one revision (`writing-derived-tests.md` Phase 2, "2a. Is the UTS spec wrong?"), and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. - *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md` from GitHub `main`, and tell the model to "fetch" the features spec with no pinned source, while reading specs from the local clone, so the two can skew. --- diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index 97e1decd0..0c6a3be50 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -77,13 +77,13 @@ A test failure has exactly three possible causes. Work through them in order: #### 2a. Is the UTS spec wrong? -Compare the UTS spec's claim against the relevant Ably **features spec** — the ultimate authority. Every spec lives under the repo's [`specifications/`](https://github.com/ably/specification/tree/main/specifications) directory; the raw, fetchable base is `https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/`. Fetch the one for the module under test: +Compare the UTS spec's claim against the relevant Ably **features spec** — the ultimate authority. Every spec lives under the repo's [`specifications/`](https://github.com/ably/specification/tree/main/specifications) directory. **Read it from the same revision as the UTS spec you are translating**: from the same local clone, or from GitHub at that commit (`https://raw.githubusercontent.com/ably/specification//specifications/`). If the UTS spec came from `main`, resolve `main` to its SHA once and read both from that SHA. A UTS spec and the features spec it is judged against must come from one revision, because features-spec and UTS changes often land in the same commit; comparing against a different revision can misclassify a spec error. The features spec for each module (links point at `main` for browsing; substitute the SHA when reading): - Core / Realtime / REST — [`features.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/features.md) - LiveObjects (`objects`) — [`objects-features.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/objects-features.md) - Chat — [`chat-features.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/chat-features.md) -Related authorities in the **same** `specifications/` directory back specific areas — e.g. [`protocol.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/protocol.md) (wire protocol) and [`encryption.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/encryption.md) — fetch them too when a test touches that area. +Related authorities in the **same** `specifications/` directory back specific areas — e.g. [`protocol.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/protocol.md) (wire protocol) and [`encryption.md`](https://raw.githubusercontent.com/ably/specification/refs/heads/main/specifications/encryption.md) — read them too, from the same revision, when a test touches that area. A UTS spec can be wrong two ways: it **contradicts the features spec**, or it is **internally inconsistent** (e.g. a replayed serial that doesn't match the value its own harness produces, or a test title that names a state its body never injects). Either way the **spec is the source of truth and must be fixed there** — do **not** silently rewrite the test to match the features spec and move on. Quietly "fixing the test" hides the spec bug, leaves the spec wrong for every other SDK to re-hit, and defeats the point of a single source of truth. Instead: