diff --git a/uts/README.md b/uts/README.md index 8cc4d9b9d..1a7fe23d8 100644 --- a/uts/README.md +++ b/uts/README.md @@ -17,6 +17,7 @@ uts/ │ │ ├── push/ # RSH — push admin │ │ └── types/ # T* — type definitions │ └── integration/ # REST integration tests (Ably sandbox) +│ └── proxy/ # Proxy-based fault injection tests ├── realtime/ │ ├── unit/ # Realtime unit tests (mocked WebSocket) │ │ ├── helpers/ @@ -29,13 +30,23 @@ uts/ │ │ └── presence/ # RTP — realtime presence │ └── integration/ # Realtime integration tests │ ├── proxy/ # Proxy-based fault injection tests +│ ├── auth/, channels/, connection/, presence/ # Direct sandbox tests, by area │ └── *.md # Direct sandbox tests +├── objects/ +│ ├── helpers/ +│ │ └── standard_test_pool.md # Standard LiveObjects test pool and fixtures +│ ├── unit/ # LiveObjects unit tests (mocked WebSocket) +│ └── integration/ # LiveObjects integration tests +│ └── proxy/ # Proxy-based fault injection tests ├── docs/ # Guides and reference │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests +│ ├── writing-uts-spec-translator-skills.md # What a uts-to- translator skill and its UTS infra must contain │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix +├── skills/ # Agent skills (Claude Code, Codex) +│ └── uts-to-lang-skill-creator/ # Builds or upgrades a uts-to- skill and its UTS infra (SKILL.md, references/, assets/, scripts/) └── README.md # This file ``` @@ -43,13 +54,19 @@ uts/ | Category | Count | Description | |----------|-------|-------------| -| REST unit | 40 | Mocked HTTP client tests | -| REST integration | 11 | Ably sandbox tests | +| REST unit | 41 | Mocked HTTP client tests | +| REST integration (direct) | 11 | Ably sandbox tests | +| REST integration (proxy) | 1 | Fault injection via Go proxy | | Realtime unit | 54 | Mocked WebSocket tests | | Realtime integration (direct) | 13 | Direct sandbox tests | | Realtime integration (proxy) | 7 | Fault injection via Go proxy | -| Helper specs | 4 | Mock infrastructure definitions | -| **Total** | **129** | | +| Objects unit | 15 | LiveObjects, mocked WebSocket | +| Objects integration (direct) | 3 | LiveObjects sandbox tests | +| Objects integration (proxy) | 1 | LiveObjects fault injection | +| Helper specs | 4 | Mock and fixture infrastructure definitions | +| **Total** | **150** | | + +For current counts, `find uts/ -name '*.md'` is authoritative (it also lists `objects/PLAN.md`, which is not a spec). ## Three Test Tiers @@ -138,9 +155,26 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations +- **[Writing UTS Spec Translator Skills](docs/writing-uts-spec-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its UTS infrastructure: infra smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync, upgrading an existing skill +- **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that builds or upgrades an SDK repo's `uts-to-` skill and the UTS test infrastructure (UTS infra) it runs on. Each run starts with Orient: it checks the repo against the whitelist in [`assets/eligibility.json`](skills/uts-to-lang-skill-creator/assets/eligibility.json) (Ably Pub/Sub SDK repositories only), detects any existing `uts-to-*` skill, UTS infra and UTS-tagged tests, classifies the repo's state (S0–S4), and detects its REST and Realtime clients, which set the scope: a REST-only or realtime-only SDK gets a partial skill. It then recommends Create (design, build and test the UTS infra, generate the skill, validate it without generating tests) or Upgrade/Fix (diff-driven for a skill it built, or a full gap audit against the guide), and always asks whether to add LiveObjects (`uts/objects`) support. See [Installing the skill creator](#installing-the-skill-creator) - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs +### Installing the skill creator + +Clone this repo to `~/.ably/specification` and link the skill into Claude Code and Codex: + +```sh +git clone https://github.com/ably/specification.git ~/.ably/specification +mkdir -p ~/.claude/skills ~/.agents/skills +ln -sfn ~/.ably/specification/uts/skills/uts-to-lang-skill-creator ~/.claude/skills/uts-to-lang-skill-creator # Claude Code +ln -sfn ~/.ably/specification/uts/skills/uts-to-lang-skill-creator ~/.agents/skills/uts-to-lang-skill-creator # Codex +``` + +The skill offers to keep that clone up to date each time it runs. If either path already holds a copied folder rather than a link, delete it first. On Windows, copy the folder to both paths instead of linking it. + +Then, in the SDK repo, run `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex) on an Opus-class model. + ## Go Test Proxy The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection. diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index e5dd1f321..d505eadf3 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -13,6 +13,8 @@ The process has two phases: Not every situation has an existing implementation. Tests may be written ahead of the implementation (test-first development), or for a new SDK that doesn't yet exist. In those cases, only the translation phase applies. +To automate translation for one SDK with an agent skill (`uts-to-`), see [Writing UTS Spec Translator Skills](writing-uts-spec-translator-skills.md). That guide packages this document's rules into a repeatable procedure. This document remains the authority on translation and evaluation semantics. + --- ## Phase 1: Translation diff --git a/uts/docs/writing-uts-spec-translator-skills.md b/uts/docs/writing-uts-spec-translator-skills.md new file mode 100644 index 000000000..655e8be83 --- /dev/null +++ b/uts/docs/writing-uts-spec-translator-skills.md @@ -0,0 +1,1171 @@ +# Writing UTS Spec Translator Skills + +This guide explains how to build `uts-to-` (for example `uts-to-python`, `uts-to-csharp` or `uts-to-go`): a per-language agent skill, for Claude Code and Codex, that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building the skill in their own repo. Two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), appear only as examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)); consulting them is a last resort ([Reference implementations (last resort)](#reference-implementations-last-resort)). + +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, or to audit and upgrade an existing one, use the [`uts-to-lang-skill-creator` skill](../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building or upgrading the skill and its UTS infra; this guide remains the authority on what the skill must contain. + +**Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. + +- **The UTS infra and the skill are one deliverable.** The skill translates onto a UTS infra built in the same repo from its existing test setup, and it isn't complete until that UTS infra's smoke tests and self-tests are green, in CI, at every tier the skill supports ([section 2](#2-the-uts-infrastructure)). +- **Easy to use.** One command (`/uts-to- []` in Claude Code, `$uts-to- []` in Codex; with no module, the skill asks which one) and four questions; the skill does the rest. +- **Context-complete.** Every run gathers the full context: the spec, the translation manual, the README of the UTS test infrastructure (the UTS infra) and the tier's infra sources (derived by the resolver), the module notes and the resolver output. A good test doesn't depend on anyone's memory. +- **Tests are reproducible artifacts.** The UTS spec plus the skill is the source of truth; the generated test is an output. When a translation is wrong or the spec changes, regenerate it ([section 9](#9-keeping-in-sync-with-spec-changes)) rather than hand-maintain it. +- **Fix the cause, then regenerate.** Fix the skill, the module notes or the UTS infra, never only the generated file ([section 5](#5-translation-rules)); only the per-test state listed in [9.2](#92-a-re-sync-mode-should) survives regeneration. + +**How to read this guide.** Requirements use three levels: + +- **MUST**: required for a skill that conforms to this guide. Most MUSTs protect translation fidelity (without them a skill produces tests that silently check less than the spec); the rest protect repeatability and reviewability. +- **SHOULD**: strongly recommended. +- **MAY**: optional, useful in some SDKs. + +Where a section's heading states a level, a lowercase "must" or "should" inside it takes that level unless marked otherwise. + +## Contents + +1. [Purpose and scope](#1-purpose-and-scope) +2. [The UTS infrastructure](#2-the-uts-infrastructure) +3. [Skill anatomy](#3-skill-anatomy) +4. [The workflow the skill must implement](#4-the-workflow-the-skill-must-implement) +5. [Translation rules](#5-translation-rules) +6. [Pseudocode construct catalogue](#6-pseudocode-construct-catalogue) +7. [Evaluation and deviations](#7-evaluation-and-deviations) +8. [Deterministic tooling](#8-deterministic-tooling) +9. [Keeping in sync with spec changes](#9-keeping-in-sync-with-spec-changes) +10. [Verification and CI](#10-verification-and-ci) +11. [Final report format](#11-final-report-format) +12. [Lessons learned](#12-lessons-learned) +13. [Checklist for a new `uts-to-` skill](#13-checklist-for-a-new-uts-to-lang-skill) +- [Appendix: Existing skills (background and patterns to avoid)](#appendix-existing-skills-background-and-patterns-to-avoid) + +--- + +## 1. Purpose and scope + +### What a translator skill is + +A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that an agent (Claude Code or Codex) follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a UTS infra in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). Translator skills, and the procedure that builds them, are for Ably Pub/Sub SDK repositories. The skill creator accepts only the repositories on its whitelist (current and legacy names, in [`assets/eligibility.json`](../skills/uts-to-lang-skill-creator/assets/eligibility.json)); adding a new SDK repository means adding it there. + +### How it relates to the other UTS docs + +| Document | Owns | +|---|---| +| [`uts/README.md`](../README.md) | Pseudocode conventions: identifier naming, absent values, property access, enum values, `poll_until_success`, `process_pending_events()`, language-inapplicable inputs | +| [`writing-test-specs.md`](writing-test-specs.md) | The pseudocode reference: Test IDs, mock patterns, `AWAIT_STATE`, `poll_until`, record-and-verify, error pattern, unique channel names | +| [`writing-derived-tests.md`](writing-derived-tests.md) | Translation and evaluation semantics: faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | +| [`integration-testing.md`](integration-testing.md), [`proxy.md`](proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | +| **This guide** | How to package all of the above into a repeatable, scripted agent skill (Claude Code and Codex) for one SDK | +| Existing skills and their UTS infra ([Reference implementations](#reference-implementations-last-resort)) | Nothing normative: a last-resort, read-only reference for patterns when creating a skill, below every document above | + +`writing-derived-tests.md` is the **source of truth for semantics**. This guide never redefines what a faithful translation is, or how a deviation is classified and recorded; it says how a skill enforces those rules. If the two ever disagree, `writing-derived-tests.md` wins. + +### What the skill owns, and what it defers + +The skill owns the infra mechanics: where each test file goes and what it is called, the pseudocode → SDK/test-framework mapping table, per-module API notes (ably-js-shaped pseudocode → your SDK's API), the language rendering of each test pattern (skip idiom, fail-fast idiom), and the build, run, lint and audit commands. It defers semantics to the documents in the table above. + +A skill SHOULD keep only the language rendering of a rule and link to the source for its meaning. Inline copies of semantic rules drift. Both existing skills replaced their inline `deviations.md` format with a pointer to the manual for this reason, although they still inline the decision tree and test patterns. + +### Model tier + +Creating a skill and its UTS infra, and running the skill, are long, multi-file reasoning tasks in which a silent omission costs far more than compute. Use the most capable model tier available (Claude Opus or an equivalent Opus-class model). "Opus-class" means that tier, whatever its current version, not a particular release; in another tool (for example Codex), it means that tool's most capable tier, recorded by name. + +| Task | Level | Why | +|---|---|---| +| Creating the skill and building its UTS infra (the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | +| Translating and evaluating with the finished skill (`/uts-to-` or `$uts-to-`) | **SHOULD** | The resolver and the audit ([section 8](#8-deterministic-tooling)) make paths, names, ID coverage and assertion counts mechanical. Wait predicates and timeouts, verbatim comments, setup fidelity and failure diagnosis ([7](#7-evaluation-and-deviations), [8.3](#83-review-checklist-must-after-the-audit)) remain model judgement | +| Sub-agents spawned during either (review, validation) | **The level of the task they serve** | A reviewer weaker than the author misses what the author missed. A cheaper model MAY be used only for a clearly mechanical step whose result a script verifies (for example, running the audit over the corpus and collecting its JSON), if at all | + +The deterministic scripts reduce the dependence on the model; they don't remove it. State the model tier in the skill, pin it only where a tool supports that ([3.3](#33-frontmatter-and-arguments)), and record the model used in every run's final report ([section 11](#11-final-report-format)). + +--- + +## 2. The UTS infrastructure + +The UTS infra is part of the deliverable, not an optional prerequisite: a skill can't make translation mechanical if the UTS infra it translates onto doesn't exist or doesn't keep its contracts. + +**Design the UTS infra from the repo's existing test setup.** For every capability in 2.1–2.7, first find what the SDK's native tests already have, then reuse, wrap, extend or build new (2.2, "Reuse before you build"). Build the UTS infra, and its tests, before writing the skill. In both existing SDKs this took several iterations, and the UTS infra co-evolved with the first real translation run. + +**Scope follows the SDK's capabilities.** The UTS infra and the skill cover the clients the SDK has: a REST client, and a Realtime client with its connection, channels and presence. An SDK without a Realtime client (a REST-only SDK) gets a REST-only UTS infra and a partial skill: the `realtime` and `objects` modules (apart from the few REST-only tests filed under `uts/realtime`), and every infra capability or checklist item that needs a Realtime client, are not applicable (capability absent), not gaps. The REST-only forms of the sandbox and proxy smoke tests publish over REST and serve a REST request through an HTTP rule. A realtime-only SDK is the mirror case: the `rest` module is not applicable, but the UTS infra still needs the HTTP hook and `mock_http`, because the Realtime client uses HTTP too. Where an SDK exposes its clients through separate doors (for example server and device factories), the skill constructs clients through one chosen side, or through each side as an infra parameter, and the scope follows the clients that side reaches. When the SDK gains a client, the skill is extended (9.5). + +### 2.1 SDK test hooks (MUST) + +The unit tier needs these injection points. Use the SDK's existing seams first (an internal option, an interface the production path already resolves, internals visible to tests). + +| Hook | Purpose | Examples | +|---|---|---| +| WebSocket transport factory | Install `MockWebSocket` | ably-java `DebugOptions.webSocketEngineFactory`; ably-cocoa `testOptions.transportFactory` | +| HTTP client/executor | Install `MockHttpClient` | ably-java `DebugOptions.httpEngine`; ably-cocoa `testOptions.httpExecutor` | +| Clock / timer source | `enable_fake_timers()`, `ADVANCE_TIME` | ably-java `DebugOptions.clock` (a `Clock` abstraction retrofitted across `Auth`, `Presence`, `Hosts`, `WebSocketTransport`); ably-cocoa `testOptions.timeProvider` | +| Reachability / network monitor | Keep the unit tier hermetic; drive `MockNetworkListener` (`realtime/unit/connection/network_change_test.md`) | ably-cocoa `testOptions.reachabilityClass` | +| Randomness | Jitter (`get_jitter_coefficient()`, `backoff_jitter_test.md`), fallback-host shuffle (RTN17j, `fallback_hosts_test.md`), where a spec in scope needs deterministic values (most of the corpus asserts ranges, and `backoff_jitter_test.md` makes mocking the generator optional) | ably-cocoa `testOptions.jitterCoefficientGenerator`, `testOptions.shuffleArray` | + +Mocks and the fake clock are injected at client construction, so the UTS infra must install them before the client is created. + +**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when a retry ran on wall-clock time through `FakeClock.waitOn`, which does a real timed wait.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. Two workable shapes: route every SDK delay (`sleep`, `call_later`, timeouts) through the clock hook, so `ADVANCE_TIME` runs them; or run unit tests on a virtual-time event loop whose `time()` the fake clock controls, in which case the wall-clock timeout wrapper must not use the loop's clock (it measures virtual time). + +If no existing seam reaches a behaviour a spec in scope needs, and the UTS infra can't reach it by wrapping or extending, add the smallest hook that does, with the SDK owner's approval. Hooks are the only production-code change this work makes: minimal, with test-only visibility, and the default path unchanged. Everything else (wrapping, extending doubles, helpers) lives in the UTS infra. The SDK's native tests are never changed to fit the UTS infra or the hooks; their failures are reported. Native test-support code changes only where a hook's interface forces it, listed in and approved with the hook's proposal. Without control of the transport, HTTP and clock, the unit tier is impossible. + +### 2.2 A shared test library implementing the helper specs (MUST) + +Implement each helper spec **once**, as a native library that every derived test calls. Derived tests never hand-roll mock setup, protocol messages or serials. + +| Helper spec | Must implement | Watch for | +|---|---|---| +| `rest/unit/helpers/mock_http.md` | `MockHttpClient`, `PendingConnection`, `PendingRequest`, `install_mock` / `uninstall_mock`, `reset()`, connection-level vs request-level failures | The injection mechanism is implementation-specific. The corpus still uses a legacy API that the helper spec doesn't define and `writing-test-specs.md` "Common Mistakes to Avoid" deprecates: the queue calls (`queue_response`, `queue_responses`, `queue_response_for_host`, `queue_response_for_url`, `queue_timeout`, `queue_delayed_response`; in `rest/unit/fallback.md`, `request.md`, `rest_client.md`) and the mock-owned request log `mock_http.captured_requests` (in `fallback.md`, `request_endpoint.md`, `request.md`, `rest_client.md`): implement them, or map them onto `onRequest` handlers and a local capture list in the skill | +| `realtime/unit/helpers/mock_websocket.md` | `MockWebSocket`, handler and await styles, raw-frame hooks (`onTextDataFrame` / `onBinaryDataFrame`), `respond_with_*`, `send_to_client`, `send_to_client_and_close`, `simulate_disconnect`, ping frames, templates (`CONNECTED_MESSAGE`, `CLOSED_MESSAGE`, `DISCONNECTED_MESSAGE`, `HEARTBEAT_MESSAGE`, `ERROR_MESSAGE(code, message)`, `PING_MESSAGE(id)`) | See the notes below the table | +| `realtime/unit/helpers/mock_vcdiff.md` | `MockVCDiffEncoder`, `MockVCDiffDecoder`, `FailingMockVCDiffDecoder` | Deterministic, not a real codec | +| `objects/helpers/standard_test_pool.md` | The standard pool, `STANDARD_POOL_OBJECTS`, `setup_synced_channel` (and the no-ack variant), every `build_*` builder, serial helpers (`ack_serial`, `remote_serial`, `below_ack_serial`, `SITE_CODE`, `POOL_SERIAL`), quiescence helper, `provision_objects_via_rest` | Serials compare as strings, so hand-written literals sort wrongly. REST provisioning uses the V2 format with the singular `/object` path, and any REST client used must be closed | +| *(no helper spec)* | `MockNetworkListener` with `simulate_network_lost()` / `simulate_network_available()`, installed via `install_mock` | Used by `realtime/unit/connection/network_change_test.md`; needs the network-monitor hook (2.1) | + +`mock_websocket.md` notes: + +- `close()` MUST call `onClose` asynchronously; `respond_with_success` MUST complete the connection first, then deliver CONNECTED asynchronously. +- `mock_ws.active_connection` is used in the helper spec's own examples but isn't declared in its interface. `connect_attempts`, `last_connect_url` and `create_mock_websocket()` aren't declared in any helper spec ([6.6](#66-mocks-fixtures-and-uts-infra)). Support them anyway. +- The corpus also uses an undocumented alternative API, listed in [6.6](#66-mocks-fixtures-and-uts-infra); support it too. Its `send_to_client_raw(json)` sends an untyped frame (`forwards_compatibility_test.md`), which even a typed SDK's UTS infra must be able to produce. +- State, in the infra README and the skill, whether handler and await styles can be mixed for one event type, and on which thread handlers run. (ably-java's can't be mixed: a set handler consumes the event, so the matching `await_*` never returns; handlers run synchronously on the SDK's thread, so they must not block or await.) + +The library also needs: + +- **Wait helpers:** an event-latched `AWAIT_STATE`, value-returning `poll_until`, `poll_until_success`, `process_pending_events()`, and (for virtual-time runners) a wall-clock timeout wrapper. See [section 5](#5-translation-rules). +- **Thread-safe capture** for lists appended from SDK callbacks (ably-java `CopyOnWriteArrayList`; ably-cocoa `Captured`). +- **A log sink** for "an error is logged" assertions. +- **An in-order subsequence assertion** for `CONTAINS_IN_ORDER`. +- **Caller-attributed failures:** helpers that time out or receive an unexpected error report the failure at the calling test line (Swift `sourceLocation:`, pytest `__tracebackhide__`, C# caller-info attributes), not inside the helper. +- **Per-tier smoke tests and helper self-tests:** see [2.7](#27-infra-smoke-tests-and-self-tests-must). + +**Placement.** Separate shared test-support code (used by both the SDK's native tests and the UTS ports, importing no test framework) from port-only infra code (doubles and fixtures used only by UTS ports); keep port-only code with the ports. Module-specific helpers (for example the `standard_test_pool.md` implementation) live next to that module's suites; the shared library stays module-agnostic. For a plugin-backed module such as LiveObjects, provide a client-options builder in the module helpers that registers the plugin, so tests never wire it by hand. + +**Infra layout (MUST for the tier split; SHOULD for the names).** One infra library (a test-support module, target or package) that every target hosting UTS tests consumes as a test dependency, organised by tier, with its smoke tests and self-tests in a separate tree of the same shape, and the generated tests kept separate from both: + +``` +/ # the infra library, e.g. infra/ (SHOULD); one per SDK + README.md # what exists, the layout, how to run each tier's infra tests, Known gaps (3.1) + # used by more than one tier: waits and polls, capture, log sink, assertContainsInOrder, + # caller attribution, deviation gates (MAY be grouped in common/) + unit/ # MockWebSocket, MockHttpClient, fake clock, optional mocks (vcdiff, network), + # message templates, mock-based client factories, the unit base or fixture + integration/ # SandboxApp, sandbox client factories, the integration base or fixture + # (guaranteed teardown), token helpers + proxy/ # ProxyManager, ProxySession, rule builders, the proxy base +/ # smoke tests and helper self-tests (2.7), in the same tier tree (the mapping's testsRoot): + unit/ integration/ integration/proxy/ + # the mapping's target directories (3.2), spec sub-directories mirrored; outside both + unit// integration/standard// integration/proxy// + # next to each module's suites, depending on , never the reverse; + # a module's smoke test lives beside its helpers (outside every target dir) + # and is selected by the unit tier's infra command +``` + +- **Placement by use.** A file goes in the lowest directory that every tier using it can see: used by one tier, that tier's directory; by integration and proxy, `integration/`; by unit and integration (or every tier), the root. Dependencies point only up the tree: `integration/proxy/` may use `integration/` and the root; `unit/` and `integration/` never reference each other. +- **Inside a tier**, capability sub-folders (`http/`, `websocket/`) MAY group files when the tier has many. Never at the top level, where they hide which tier a file serves. +- **Names follow folders.** Namespaces, packages or modules follow the folders at least down to the tier (C# `…Infra.Unit`, Kotlin `…infra.integration.proxy`, a Python package or a Go package per directory). Where the language has one namespace per target (Swift), the folders are organisation only, and type names must be unique target-wide. A tier name must not shadow a type the tests use ([8.1](#81-resolver-resolve_utspy-must), collision detection). +- **Infra tests** go in a separate tree beside the UTS infra, with the same tier folders, so the infra root holds library code only (recommended). `/tests/` is allowed instead; the source derivation then excludes `tests/`. Either way they stay outside every generated-test directory (2.7). +- **The resolver derives each tier's helper sources from this layout** ([3.2](#32-the-mapping-file)): the root's own files (and `common/`), plus every tier directory on the path to the tier, each recursively but without a nested tier directory. Unit is `unit/`; integration is `integration/` without `integration/proxy/`; proxy is `integration/` without `integration/proxy/`, plus `integration/proxy/`. An infra-tests directory is never a source. A UTS infra not yet organised by tier falls back to the whole root until it is restructured ([9.5](#95-upgrading-an-existing-skill-should)). + +Where the SDK already has a test-options type, add the hooks to it, and implement the SDK's own transport and HTTP abstractions rather than patching internals. Two examples of this shape: ably-java's `:uts` Gradle module (`infra/`, `infra/unit/`, `infra/integration/`, `infra/integration/proxy/`, packages following the folders, with the smoke tests in the module's test source set, in the same tier tree), and ably-cocoa's `Test/UTS/infra/` (the same tiers as folders in one SPM target, with the generated tests beside `infra/`, never inside it). + +**Document each fixture helper's scope** in the module notes: which client messages it auto-answers, which spec branches it doesn't wire (a DETACH reply, fake timers), the local pattern tests use for those gaps, and any option tweaks the gap needs. (ably-java: the fake-clock variant of `setupSyncedChannel` needs a large `maxIdleInterval`, or virtual-time advances trip the idle timer.) + +**Reuse before you build.** If the SDK's native tests already have transport doubles, a fake clock or sandbox helpers, assess whether to reuse, wrap or extend them against the helper specs' semantics before writing new ones; never change their behaviour for the native tests that use them. Existing doubles bound to another test framework, or built for real-network interception rather than the helper specs' in-process semantics, are better left alone: build pseudocode-shaped doubles on the same SDK hooks instead, and extract shared plumbing (such as sandbox retries) for both. (One SDK kept its legacy doubles for its native suite and built new framework-free UTS doubles on the same transport and HTTP seams.) + +**Build the UTS infra to look like the pseudocode.** Callback-style mocks with the spec's handler names, and a client factory that mirrors `Realtime(options: …)`, turn translation into transliteration. ably-java's first generation run replaced a "wrap every callback in a coroutine" rule with callback-style mocks for exactly this reason. + +### 2.3 Sandbox provisioning and fixtures (MUST for integration tiers) + +- A `SandboxApp` helper that creates an app from `ably-common/test-resources/test-app-setup.json` against `sandbox.realtime.ably-nonprod.net` and deletes it best-effort. **Retry only idempotent requests** (fetching the app setup); never retry `POST /apps`, which can create duplicate apps when the response is lost. Check the HTTP status before parsing the body. +- `ably-common` as a submodule. It also provides `msgpack_test_fixtures.json` (`rest/unit/encoding/msgpack_interop.md`) and `encoding.json` (`load_fixtures`, `rest/unit/encoding/message_encoding.md`). +- A way to point the client at the sandbox. The docs use `endpoint: "nonprod:sandbox"`; an SDK without `endpoint` sets `realtimeHost`/`restHost` instead. Explicit hosts disable fallbacks: REC2c6 for `realtimeHost`/`restHost`, REC2c2 for an `endpoint` hostname. + +### 2.4 The proxy (MUST for the proxy tier) + +- A `ProxyManager` that downloads, caches and starts [`ably/uts-proxy`](https://github.com/ably/uts-proxy). Pin the version, and check that the pinned release publishes a binary for every developer OS (its GitHub release assets). An environment override pointing at a locally built binary (`UTS_PROXY_LOCAL_PATH` in the UTS infra of both existing skills; ably-java also accepts `-Duts.proxy.localPath`) is useful; keep it a configuration override, not a test gate. Operationally, the manager also verifies the downloaded archive (a checksum), serialises the download across concurrent test processes, waits for the proxy's control endpoint to report healthy before returning, reaps the proxy process when the test process exits, and handles the port: proxy suites share one control port, so run them in one process (one fork or worker), or give each worker its own port. +- A native `ProxySession` implementing [`proxy.md`](proxy.md): `create_proxy_session`, `add_rules`, `trigger_action`, `get_log`, `close`, plus rule-builder helpers and a typed event log. +- **The `action` inside a frame `match` (`ws_frame_to_client` / `ws_frame_to_server`) must be a JSON string**: a name (`"ATTACHED"`) or a numeric string (`"11"`), never a number (the proxy rejects a number with HTTP 400). uts-proxy (as of v0.3.0) resolves names only up to `AUTH` (17), so use numeric strings for `OBJECT` (`"19"`), `OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`); an unresolvable name silently never matches. This applies only to `match.action`: a rule's own `action` is an object (`{"type": "suppress"}`), and the `action` inside an injected message stays a number. Rule builders take an integer and stringify it, as the UTS infra of both existing skills does. +- Proxy tests always use JSON: the proxy supports text frames only (`integration-testing.md`, "Protocol Variants"). +- **Auth through the proxy.** The proxy hop is plain HTTP/WS (`tls: false`), and RSA1 and RSC18 say Basic Auth over HTTP without TLS results in an error (`writing-test-specs.md` "RSC18 only applies to Basic auth configurations": checked at construction). `proxy.md`, `integration-testing.md` and `writing-test-specs.md` nonetheless show `key: api_key` with `tls: false`. In practice, ably-cocoa authenticates proxied clients with a locally signed `TokenRequest`, as does ably-java where the spec observes auth; ably-java's objects proxy suite uses `key` over the plain hop, which ably-java permits. Token auth is the option valid under RSA1 for every SDK; whichever you use, say why in a comment. +- Platform gating where the proxy can't run (ably-cocoa wraps proxy files in `#if os(macOS)`). + +### 2.5 Build and CI wiring (MUST) + +- Test tasks or filters that select each tier (unit, direct integration, proxy), each with a single-class/single-test filter, and, per tier, the complete infra-test command ([2.7](#27-infra-smoke-tests-and-self-tests-must)). Check the repo's existing test commands and jobs: a bare `pytest`, `dotnet test` on the solution, or a Gradle `test` that aggregates every module also collects the new UTS infra and UTS tests (including network tiers). Exclude them there (`testpaths`/`--ignore`, a solution filter, a separate project or task), or include them deliberately in that job as their one CI home. +- **Every suite runs in exactly one CI job per platform it targets, and nothing runs green without executing** (MUST). Splitting by tier (unit in the fast check job; integration and proxy in a networked job) is recommended (SHOULD): ably-java does this, while ably-cocoa runs all UTS tiers from one job definition, one job per platform (an iOS / tvOS / macOS matrix; proxy suites compile only on macOS). +- Integration jobs need network access, the `ably-common` submodule and the proxy binary. + +### 2.6 Understand the SDK's concurrency model (MUST) + +Most flakiness in both existing SDKs came from concurrency, not from translation. Before writing the skill, know: + +- Which thread or queue SDK callbacks fire on, and whether listeners can run concurrently with the test body. +- How to drain the SDK's internal work queue (this becomes `process_pending_events()`). One zero-delay yield (e.g. one `await asyncio.sleep(0)`) may not drain chained callbacks; loop until the queue is empty if you can observe it. *Divergence from `writing-derived-tests.md`* ("No real timers in unit tests"), which says one event-loop yield suffices for a negative assertion: that holds for a single JavaScript macrotask queue, but not where the SDK chains work across queues, threads or loop iterations (both existing SDKs needed explicit queue drains). Use one yield only where you have shown it drains everything. +- Whether the test runner virtualises time (kotlinx `runTest` does; this breaks real-network timeouts), and, for async runners, the plugin mode and event-loop scope that suite-level fixtures need (e.g. pytest-asyncio). +- **Whether the runner runs tests in parallel** (xUnit does by default; pytest-xdist and NUnit `[Parallelizable]` when enabled; Swift Testing unless `.serialized`). If any injected mock, fake clock or log sink is process-global, serialise the UTS suites or make the hooks per-client, and say which in the skill. ably-cocoa marks its core realtime/rest and integration suites `@Suite(.serialized)`; its objects unit suites, which inject nothing process-global, are plain parallel `struct`s. +- Whether the language enforces concurrency safety at compile time (Swift 6 `Sendable` drove ably-cocoa's capture types). +- Where white-box unit specs can see internals (Swift `@testable import`, Objective-C private headers, C# `InternalsVisibleTo`, Kotlin `internal` within the owning module). This decides **where tests live**. + +### 2.7 Infra smoke tests and self-tests (MUST) + +The UTS infra has two kinds of test of its own, both permanent and both required: + +- **Per-tier smoke tests:** end-to-end wiring of one tier through the real SDK hooks (unit with mocks and the fake clock; direct sandbox; proxy). +- **Helper self-tests:** contract tests that each infra helper obeys its helper spec, or this guide's rule for it. + +**Why both.** Spec-derived tests can't be the UTS infra's acceptance gate. They are regenerated, they may legitimately be red (a UTS spec error), and a failing one has to be triaged as SDK, spec or UTS infra. A smoke-test or self-test failure is unambiguously a UTS infra or environment fault. Scenario-only smoke tests aren't enough: they follow a happy path, so a helper that breaks its contract (a mock `close()` that notifies synchronously, a declared mock event that is never emitted, a state wait that samples) still passes, and the defect surfaces later as a flaky spec-derived test. Both existing SDKs hit this ([Patterns to avoid](#patterns-to-avoid)). + +**Rules (MUST):** + +- **Permanent.** Don't retire them when spec-derived tests cover a tier. +- **In CI, ungated.** They run, ungated, in whichever CI job runs their tier ([2.5](#25-build-and-ci-wiring-must)); no environment variable may skip them (compile-time platform gating per 2.4 aside). +- **SDK-blocked objects smoke test.** Where the SDK exposes the LiveObjects public API but doesn't implement it yet, an objects smoke-test assertion that fails only on an SDK not-implemented path (the UTS infra compiles, every self-test is green, and the error names the SDK symbol) may be listed in Known gaps as "SDK-blocked", kept running under a strict expected-failure marker, and leaves the smoke-test checklist item unmet; the objects tier is then translate-only until the assertion passes. +- **Untagged and structural.** No `UTS:` tag; each is named after the capability it proves; each passes on its own under the single-class filter. +- **Never copied.** The skill never copies their shape and never names them as reference tests. Its Infra reference ([3.4](#34-recommended-outlines-should)) may cite a smoke test as a wiring example only (client construction, suite fixtures, teardown). +- **Separate.** They live in their own tree beside the UTS infra, organised by the same tiers (the common self-tests, `unit/`, `integration/`, `integration/proxy/`; layout in 2.2), or (allowed) under `/tests/`. Either way they are outside every directory the mapping gives the resolver as a target, so generated tests can't collide with or overwrite them, and never among the helper sources the resolver derives. +- **Selectable.** An infra test's tier is its folder. One complete command per tier runs that tier's smoke tests and self-tests (plus the common self-tests). The mapping declares it once, as `testCommand` with a `{tier}` placeholder, and the tree as `testsRoot` ([3.2](#32-the-mapping-file)); the skill's preflight runs it ([4](#4-the-workflow-the-skill-must-implement), step F). +- **Explicit.** Each item below is an explicit assertion. One scenario test may cover several items; an item covered only implicitly doesn't count. Every event type the UTS infra declares is emitted and asserted (or the type is removed). A helper-spec contract point the UTS infra doesn't meet is listed in the infra README's Known gaps ([3.1](#31-layout)), not left untested. + +The minimum set per tier follows. A tier the skill doesn't support needs none of its entries. + +#### Common (every tier) + +- Self-tests: + - `assertContainsInOrder` has subsequence semantics (interleaving passes, a repeated state passes, wrong order fails) + - `poll_until` returns the settled value and aborts on an error + - `poll_until_success` rethrows the last error on timeout + - The wall-clock timeout wrapper bounds a wait, including under a virtual-time runner + - Capture lists are safe under concurrent appends + - A failing helper reports the caller's line + - Every state-wait helper (unit and integration) latches a state entered and left within one scheduling tick and fails fast on FAILED + - Teardown closes clients and cancels timers when the body throws, and, from a state where the SDK's `close()` can't reach CLOSED ([5.4](#54-setup-and-teardown)), doesn't wait for CLOSED + +#### Unit, realtime (mock WebSocket and fake clock) + +- Smoke test: + - A factory-built client with the mock installed reaches CONNECTED via `respond_with_success` and `CONNECTED_MESSAGE`, and a template field (e.g. the connection id) reaches the client + - The connection attempt exposes host, port, TLS and query params + - A client→server frame (ATTACH, with a non-ASCII channel name) is captured and decoded + - A server→client frame (ATTACHED) changes state + - `simulate_disconnect` yields DISCONNECTED, observed by record-and-verify + - A timer-driven retry doesn't happen before `ADVANCE_TIME` and does after it + - The await style drives refused attempts through to SUSPENDED +- Self-tests: + - Mock `close()` calls `onClose` asynchronously + - CONNECTED is delivered after the connection completes + - A raw untyped frame can be sent + - WS-level timeout and DNS-error outcomes exist + - Handler and await styles both work, and the documented mixing rule holds + - Every declared mock event type is emitted, in order + - `process_pending_events()` runs a queued callback with no real delay + +#### Unit, fake clock + +- Smoke test: (driven by the realtime smoke test) +- Self-tests: + - One `ADVANCE_TIME` runs cascaded work to quiescence (zero-delay reschedules, timers created mid-advance) + - The fake clock never waits on the real clock (the SDK's timed blocking waits and the async runtime's timers follow it) + - A poll deadline still expires with the fake clock installed + +#### Unit, REST (mock HTTP) + +- Smoke test: + - A request is captured with method, path, headers and body + - `respond_with` status, body and headers are decoded by the SDK + - A realtime-driven HTTP request (e.g. token auth via `authUrl`) is served by the HTTP mock while the WS mock serves the connection +- Self-tests: + - Connection-level and request-level failures are distinguishable + - Handler and await styles both work + +#### Unit, optional mocks + +- Smoke test: + - `MockNetworkListener` (if the SDK has a network monitor) and `MockVCDiff*` (if it supports deltas) each drive the SDK once + +#### Unit, objects (each module with a fixture helper) + +- Smoke test: + - `setup_synced_channel` (or its sanctioned stand-in) completes and one pool object reads its seeded value +- Self-tests: + - Serial helpers sort as the helper spec requires + +#### Direct sandbox + +- Smoke test: + - An app is provisioned once per suite (or per test where 5.4 allows it, disclosed) and deleted in teardown, and its keys look right + - A client connects to the sandbox (the recorded state sequence ends CONNECTED) + - Attach, subscribe and ack-awaited publish on a unique channel, with messages received in order + - A REST read (history) returns them via `poll_until` + - It runs once per protocol variant + - Every wait is wall-clock and bounded +- Self-tests: + - `SandboxApp` creates and deletes an app, and never retries `POST /apps` + +#### Proxy + +- Smoke test: + - The pinned proxy (or the local override) is fetched, verified, started and healthy + - A pass-through session connects a client with the chosen proxy auth + - The typed log shows `ws_connect` and a server→client CONNECTED frame + - `trigger_action(disconnect)` is followed by a recorded DISCONNECTED, then recovery + - A frame-match rule with a string `match.action` (e.g. `"11"`) fires once, then the client recovers + - Sessions and clients close in teardown, even on failure +- Self-tests: + - Rule builders stringify `match.action` + - A session with `rules:` only defaults its endpoint to the sandbox ([6.7](#67-notes-on-the-catalogue)) + +--- + +## 3. Skill anatomy + +### 3.1 Layout + +``` +/ # .claude/skills/uts-to-/ (linked from .agents/skills/uts-to-/) +├── SKILL.md # the procedure +├── uts-package-mapping.json # spec module → target dir per tier, + notes pointer; only what can't be derived +├── scripts/ +│ ├── resolve_uts.py # deterministic selection +│ ├── audit_translation.py # deterministic faithfulness audit +│ ├── scan_constructs.py # corpus construct scanner (SHOULD, section 6) +│ └── ensure_spec_clone.py # spec-clone sync, copied from the skill creator (SHOULD, 9.4) +└── references/ + └── -mapping.md # module notes (e.g. objects-mapping.md) +``` + +| File | Purpose | Level | +|---|---|---| +| `SKILL.md` | The two-phase procedure, construct table, a file template per tier, evaluation patterns, tier wiring. Keep it module-generic: the core realtime/rest mapping may live here; plugin or typed-module specifics go in notes | MUST | +| `uts-package-mapping.json` | Maps each source module (`rest`, `realtime`, `objects`) and tier (`unit`, `integration`, `proxy`) to one target directory, plus an optional `notes` path; declares the UTS infra (root, README, infra-tests tree and the one infra-test command); nothing the resolver can derive (3.2) | MUST | +| `scripts/resolve_uts.py` | Validates the module dir, reads the mapping, lists specs and derives file/class/package names; prints one JSON object | MUST | +| `scripts/audit_translation.py` | Compares one spec file with its generated test file; prints one JSON object | MUST | +| `scripts/scan_constructs.py` | The corpus construct scanner ([section 6](#6-pseudocode-construct-catalogue)) | SHOULD | +| `scripts/ensure_spec_clone.py` | A copy of the skill creator's spec-clone sync ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)) | SHOULD | +| `references/-mapping.md` | Maps ably-js-shaped pseudocode to your SDK's API for one module; may override parts of the generic flow | MUST wherever the SDK surface diverges from the pseudocode (in practice `objects` for every typed SDK) | + +Both existing skills use Python for scripts. Any language works, but the scripts MUST run on every developer platform (see [8.1](#81-resolver-resolve_utspy-must) and [section 10](#10-verification-and-ci)). + +**Install for both Claude Code and Codex (SHOULD).** Claude Code loads project skills from `.claude/skills/`; Codex loads them from `.agents/skills/`. Keep one copy and link the other location to it, so both tools load the same files: + +```sh +mkdir -p .agents/skills +ln -s ../../.claude/skills/uts-to- .agents/skills/uts-to- +git add .agents/skills/uts-to- +``` + +Both tools follow directory symlinks. On Windows, Git needs `core.symlinks=true` (and Developer Mode); if that isn't available to your contributors, commit a copy instead, and add a CI check that the two directories are identical. Everything in the skill refers to its own files relative to the skill directory ([3.3](#33-frontmatter-and-arguments)), so either location works. + +Keep two documents with separate roles: the test library's README describes what exists (helpers, seams, layout); `SKILL.md` describes how to author tests. Refer to the README by its full repo path, so it can't be confused with the spec repo's `uts/README.md`. The infra README MUST include a **Known gaps** section: every helper-spec member or contract point, and every corpus construct, that the UTS infra doesn't implement, by tier, in a form the skill's preflight can match against the selected specs (construct or helper name → status and workaround). Write it as a table whose first column is the exact pseudocode token as it appears in `pseudo` fences (for example `respond_with_timeout`, `PING_MESSAGE`, `AWAIT_ALL`), so the preflight matches mechanically: for each row, grep the selected specs' `pseudo` fences for the token; any hit is a conflict. The corpus scanner (section 6) MAY accept spec files as well as a module directory for this. It SHOULD also cover: the per-tier layout (2.2: which tier directory holds what, and the infra-tests tree with its smoke-test and self-test files), the SDK hooks used, each helper-spec symbol → infra symbol, the wait helpers and their default timeouts, the threading and time model, how to run each tier and its smoke tests and self-tests (2.7), and one index of every module's fixture helpers (even those in a separate test-support module). + +**The notes are a map, not an authority.** Where the notes, the spec's IDL and the SDK disagree about the SDK's API, the SDK source is the ground truth: fix the notes. + +**Decide `objects` from evidence.** Whether the skill covers `uts/objects` is decided from the SDK's source, not assumed. If the SDK exposes the LiveObjects public API (the `RealtimeChannel#object` accessor, RTL27, and the public types of `objects-features.md`, or an earlier revision's names), the skill gets full `objects` notes, the mapping entry and the objects infra pieces (2.2, 2.7), even if the implementation is incomplete. Objects runs then default to translate-only (section 4, step E); once evaluated, a failing test is an SDK deviation, gated, or a skipped stub where the API is missing ([7.1](#71-three-acceptable-end-states)). Otherwise the module gets a placeholder notes file, or an entry marked not ready as a whole (one module-level `unready` reason, with no tier paths; 3.2), and the reason is recorded. The [procedure](../skills/uts-to-lang-skill-creator/references/liveobjects-support.md#12-liveobjects-objects-support) gathers the evidence with a script and asks the owner every run. + +### 3.2 The mapping file + +The mapping holds **source → destination**: for each UTS module, one target directory per tier, plus only what the resolver can't derive. Each tier value is **one path**, relative to the repo or to a declared root (Swift's `testRoot`), never machine-absolute. Derive everything else (namespace, package, build target, the test project, each tier's infra sources) in the resolver, never by hand. + +```json +{ + "_description": "Maps each UTS module to its target test directory per tier, for scripts/resolve_uts.py. Update it when a module is added or its tests move.", + "infra": { + "root": "", + "readme": "/README.md", + "testsRoot": "", + "testCommand": "" + }, + "packages": { + "realtime": { "unit": "/unit/realtime", "integration": "/integration/standard/realtime", "proxy": "/integration/proxy/realtime", "notes": "references/realtime-mapping.md" }, + "objects": { "unready": "out of scope: the SDK has no LiveObjects API" } + } +} +``` + +- **Make every derived namespace unique**: keep a module segment after the tier (`unit/realtime`, never bare `unit`), or use a module-specific prefix (ably-java's objects tests use `io.ably.lib.liveobjects.uts.unit`). +- **Sub-directories are mirrored (MUST).** A spec at `uts////.md` goes to `//`, each segment in the language's directory convention: PascalCase in C#; a lowercase identifier in Kotlin, Python and Go; as the repo names directories in Swift. Where the language derives a namespace or package from the path, the segments join it. The mapping keeps one path per tier; the resolver derives the rest ([8.1](#81-resolver-resolve_utspy-must)). A language where each directory is its own compilation unit (Go) MAY flatten instead, recorded as a decision. +- **`notes`** is relative to the skill directory, not the repo root. +- **`infra`** (MUST): exactly four keys: `root`, `readme`, `testsRoot` and `testCommand`. `testsRoot` is the tree of infra smoke tests and self-tests (2.2, 2.7), which can't be derived reliably across languages (it may be a separate test source set). `testCommand` is the one complete command, run from the repo root, that runs the common self-tests plus one tier's smoke tests and self-tests, with `{tier}` (`unit`, `integration` or `proxy`) for the tier (e.g. `dotnet test tests/Uts --filter "uts-infra=common|uts-infra={tier}"`, `pytest test/uts/infra_tests -m "common or {tier}"`) ([2.7](#27-infra-smoke-tests-and-self-tests-must)). The step F preflight runs it mechanically and stops on red. Paths are repo-relative. The resolver reports the UTS infra per tier, with paths validated (still repo-relative), `{tier}` expanded and the tier's helper sources derived ([8.1](#81-resolver-resolve_utspy-must)), so `SKILL.md` never hard-codes infra paths. A mapping from an older skill may name this entry `harness`, its former key: read it as a deprecated alias of `infra`, with a warning (8.1). +- **Derived, never listed.** The test project or build target, the namespace or package root, and each tier's helper sources, which follow the tier layout of [2.2](#22-a-shared-test-library-implementing-the-helper-specs-must) (the root's own files and `common/`, plus each tier directory on the path to the tier; never the infra-tests tree). A UTS infra not yet organised by tier falls back to the whole root until it is restructured. A value the build can't give (for example a C# root namespace that no project file declares) MAY be stored, as a top-level key. +- **Language-specific derivations live in the resolver.** Swift has no package, so the directory is the full story. Kotlin derives `package` from the path after `src/test/kotlin/` and the Gradle module from the first segment. A C# resolver might map `tests/IO.Ably.Tests.Uts/Unit/Realtime` to namespace `IO.Ably.Tests.Uts.Unit.Realtime` (PascalCase segments) and to the nearest ancestor `.csproj`; a Python resolver derives a dotted package path. +- **Hand-maintained entries.** `--create` scaffolds only the repo's default layout. A module whose tests live elsewhere (ably-java's `objects`, in `:liveobjects`) is hand-maintained: the skill MUST say which entries are, and MUST NOT offer `--create` to "fix" them. +- **Readiness is data.** A tier the skill can't offer yet is recorded as not ready, with the reason, so step C refuses it ([4](#4-the-workflow-the-skill-must-implement)). A module out of scope as a whole (`objects` when the SDK has no LiveObjects API, [3.1](#31-layout)) keeps an entry so step B never offers `--create`, with one module-level reason and no tier paths, as `objects` above: never dummy paths, and never `unsupported`, which records a missing capability (below). You MAY store other per-module defaults as data, e.g. `"evaluate": true`. The list of white-box specs lives in the notes ([3.4](#34-recommended-outlines-should) item 8), not here. +- **A module the SDK can't support** (section 2, "Scope follows the SDK's capabilities") keeps its entry with an `"unsupported": ""` marker (any tests in it the SDK can still run are listed in the entry, under `"extraTests"`), so the resolver refuses it with that reason instead of offering to create paths for it. Tests that need a client the SDK lacks are listed per module under `"capabilityInapplicable"`, as in [7.4](#74-language-inapplicable-inputs) case (d). These three scope markers stay in the mapping. +- **Nothing else.** Every other key needs a reason the resolver can't derive it. File lists, per-tier commands, the test project and paths the resolver can find don't belong here. + +### 3.3 Frontmatter and arguments + +Write the skill so the same directory loads in Claude Code and in OpenAI Codex. Both implement the open [Agent Skills](https://agentskills.io/specification) format. Put only the portable fields in `SKILL.md`: + +```yaml +--- +name: uts-to- +description: "Translates Ably UTS (Universal Test Suite) pseudocode specs into runnable LANG tests in REPO. Use when asked to translate, port, derive, generate or re-sync tests from a UTS spec or module (uts/rest, uts/realtime, uts/objects), to evaluate UTS-derived tests against the SDK, or to check UTS coverage. Takes a UTS module directory, validates it, resolves the target test directories, asks for a tier (unit, integration, proxy) and specs, then derives one LANG test per spec Test ID. Not for creating the skill or its UTS infra (use uts-to-lang-skill-creator), or for writing UTS specs." +license: "" +compatibility: "Claude Code or Codex. Needs git, python3 and a local clone of ably/specification (cloned or updated on approval); run on an Opus-class model." +allowed-tools: Bash Read Edit Write +metadata: + version: "1.0.0" + short-description: "Translate UTS specs into LANG tests" +--- +``` + +- **`name`** MUST be `uts-to-`, equal to the skill's directory name: lowercase letters, digits and hyphens, at most 64 characters. (`uts-to-kotlin` omits `name` and relies on the directory name; both tools accept that, but the Agent Skills format requires the field.) +- **`description`** MUST be at most 1,024 characters and contain no `<` or `>`. Write `LANG` and `REPO`, not ``: skill packaging validators (for example skill-creator's `quick_validate.py`) reject angle brackets. It SHOULD be "pushy" and written in the third person: front-load the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to LANG", "re-sync UTS tests") and end with a "Not for …" boundary. Put the usage line in the body, not in the description. Both existing skills' descriptions are procedural, which suits explicit invocation but triggers poorly otherwise. +- **`metadata`** values MUST be strings (quote `"1.0.0"` and `"false"`). Codex shows `metadata.short-description` in its skill list. +- **`allowed-tools`** is space-separated. Claude Code pre-approves the listed tools while the skill runs; Codex ignores the field. Both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). +- **Claude Code-only fields** (`argument-hint`, `model`, `disable-model-invocation`, `when_to_use`, `context`, …) MAY be added only if the team accepts that they aren't portable: Codex ignores them, and they make claude.ai upload and standard packaging fail. Put Codex-only settings (display name, implicit-invocation policy) in an optional `agents/openai.yaml` in the skill directory. +- **Argument (optional):** one of these forms ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first) says which clone each one reads): + - **none:** the shared spec clone. The skill says so in one line, clones it (one question) or syncs it, says what it reads, then asks which module; + - **a spec clone's root** (``): that clone; one that isn't the shared clone is used as it is, with the not-synced warning (the shared clone is synced as usual); then the skill asks which module; + - **a UTS module directory**, directly under `uts/` (`/uts/realtime`): no module question; + - **a bare module name** (`rest`, `realtime`, `objects`), resolved against the shared spec clone (SHOULD): no module question. With the shared clone as the default, this is the normal invocation. + + Only the shared clone is synced; a path into another clone is used as it is, with a warning. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. Don't depend on argument substitution (`$ARGUMENTS` works only in Claude Code): say in words where the argument comes from ("the path or module name given after `/uts-to-LANG` in Claude Code or `$uts-to-LANG` in Codex, or named in the user's message, if any"). **The skill MUST never guess a module.** Where no module is given, it asks which one: one option per module directory under the clone's `uts/`, a module the skill doesn't offer shown with its reason but not selectable (an `unready` module, or an `unsupported` one without `extraTests`; a module with `extraTests` is offered for just those tests), no recommendation, and a way to stop. An argument that is none of these forms gets the usage line, then a stop. The usage line leads with the no-argument form: `/uts-to-LANG [clone path | module path | module name] [--resync]`, "With no argument, uses the shared spec clone and asks which module." +- **Paths to the skill's own files** MUST be relative to the skill directory (the directory containing `SKILL.md`), for example "run `scripts/resolve_uts.py` from this skill's directory". Never write `.claude/skills/uts-to-/scripts/…`: Codex loads the skill from `.agents/skills/`, and `${CLAUDE_SKILL_DIR}` is Claude Code-only. The scripts follow the same rule: they locate the mapping file and the notes relative to their own file (for example `Path(__file__).parent.parent`), never relative to the working directory. +- **Model (SHOULD):** no portable frontmatter field pins a model. State in the opening lines of `SKILL.md` (and in `compatibility`) that the skill should run on an Opus-class model ([Model tier](#model-tier)). Claude Code's `model` field is a non-portable option (see above). +- **Placeholders, not personal paths,** in examples and usage text: `/uts/objects` in the body (angle brackets are fine in the body, but not in `description`). + +### 3.4 Recommended outlines (SHOULD) + +The outlines are SHOULD, except the Infra reference section (MUST, below). Use these shapes (both existing skills converged on them independently of language): + +``` +SKILL.md + Required reading (local clone, 9.4) + Infra reference: per tier, from the resolver's infra output: infra root and README path, + helper sources to read (derived by the tier layout), smoke-test and self-test locations + (the infra-tests tree) and run commands, the generated-test run command, the README's Known gaps + Phase 1, selection: usage guard; step 0; module choice; A resolve; B mapping; C tier; D specs; + E translate vs evaluate; F infra preflight + Phase 2, per spec: 1 read spec + notes; 2 output path; 3 read UTS infra + reference test; + 4 generate (construct table, client construction, mocks, timers, assertions, naming, + file template per tier); 5 compile; 6 run + diagnose (evaluate only); 7 audit + review + Integration tiers: direct-sandbox wiring; proxy wiring (session, connect through proxy, auth, + rule builders, event log); file templates + Final report (section 11) +``` + +`SKILL.md` MUST have the **Infra reference** section. It points at the infra sources, README and infra tests, as the resolver reports them (never a hard-coded file list); it doesn't copy helper signatures inline (inlined copies drift). A construct that relies on a listed known gap is a stop-and-ask or a Mock Infrastructure Limitation, never a substitution (5.2). + +A module notes file SHOULD cover: + +1. **Source of truth and runtime status:** the spec's IDL that the notes apply, including any typed-SDK variant (for objects, the proposed `RTTS` points that both existing skills' notes follow, which aren't in `objects-features.md` yet), and whether the module is implemented (whether evaluate mode is possible). +2. **Layers:** spec names that denote several things (e.g. `LiveMap` as a creation value type, a public view and an internal node). +3. **Entry point and setup:** plugin registration, channel modes. +4. **Async model:** how `AWAIT`, deferred futures and errors render. +5. **Type mapping:** polymorphic spec objects → typed views (which conversions throw and which degrade to null); write values → union/wrapper types; enums and string tags → constants; numeric types. +6. **Mutations, subscriptions and events.** +7. **Messages and errors:** codes, nested `error.cause`, timestamp units. +8. **Internal access:** the ladder ([5.9](#59-internal-access-white-box-unit-specs)), the list of white-box specs (kept only here, not in the mapping), the placement boundary. +9. **Helper-spec coverage table:** every helper-spec symbol → native file and symbol, with each fixture helper's scope (2.2), sanctioned shape differences ("use the pattern; don't add the shorthand") and sanctioned stand-ins ([7.2](#72-infra-stand-ins-are-not-deviations)). +10. **Overrides of the generic flow:** reading list, naming, deviations location, expected audit shortfalls. +11. **Integration helpers.** +12. **A worked example** naming each mechanical rewrite, and a symbol index. +13. **Shape-deviation vocabulary** (`S-1…S-n`), if any. + +A generated test file: + +``` +header: "Derived from uts/ at ably/specification@" (plus "(locally modified; blob )" + when 9.1 applies), tier, disclosures + (infra stand-ins, deviations); integration/proxy: the corresponding unit spec +imports; suite/class (serialised if hooks are process-global) +per test: UTS: tag → test attribute → name with spec point → + // Setup … // Test Steps … // Assertions … (spec comments verbatim) → teardown via scope/hook +file-local helpers: before the first test or in a separate file (the audit attributes anything + after the last tag to the last test; see the audit contract, 8.2) +``` + +--- + +## 4. The workflow the skill must implement + +The workflow has two phases. Implement both. + +### Phase 1: selection (steps 0 and A–F) + +| Step | What happens | Interaction | +|---|---|---| +| **0. Required reading** | Sync the shared spec clone, or warn about another clone the argument names ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)), then read `writing-derived-tests.md` and the "Pseudocode Conventions" section of `uts/README.md` once per run (plus, for the integration and proxy tiers, the "Timeout Strategy", "Avoiding Flaky Tests" and both "Protocol Variants" sections of `integration-testing.md`, not the whole file; plus `proxy.md` for the proxy tier; in evaluate mode, the features specs as needed), from that same clone. Read `writing-test-specs.md` whenever the construct table defers a meaning to it. Don't rely on memory or inlined summaries | — | +| **Choose the module** (before A) | Only when the argument named no module (none, or a clone root; 3.3): list the module directories under the clone's `uts/` (the resolver's module-listing mode, 8.1) and ask which one. A module the skill doesn't offer (an `unready` module, or an `unsupported` one without `extraTests`) is shown with its reason but isn't selectable; an `unsupported` module with `extraTests` is offered for just those tests. No recommendation; never guess | Ask | +| **A. Resolve the module** | Run `resolve_uts.py ""`. If `ok` is false, relay `message` and stop. Treat the output as the single source of truth: never recompute a path or name by hand | Stop on error | +| **B. Confirm or create the mapping** | If mapped, show each tier's target dir verbatim and ask the user to confirm. If unmapped, ask for the target base name (default: the source module name; suggest a rename only when the SDK uses different terminology) and run `resolve_uts.py --create `. If the user says an entry is wrong, re-run `--create ` to overwrite it (keeping its `notes`), unless it is hand-maintained (3.2), in which case edit it by hand | Ask | +| **C. Choose the tier** | Offer only tiers that are present. Where the mapping records a tier, or the whole module, as not ready (per-module data, 3.2), refuse it and name what blocks it. The tier fixes the target dir, the spec list and the translation flow (mocked / sandbox / proxy). Don't re-detect it per spec | Ask | +| **D. Choose specs** | All specs in the tier, or a subset | Ask | +| **E. Translate-only, or translate and evaluate?** | Translate-only: generate, compile, and run the step 7 audit and review, but don't run the tests (the SDK feature may not exist yet). Translate and evaluate: also run and diagnose. If unsure whether the implementation exists, ask rather than guess | Ask | +| **F. Infra preflight** | Compile the UTS infra, run its tests and check its Known gaps (below) | Stop on red | + +Step F runs before anything is generated, for the chosen tier and specs, in both modes: + +- Compile (or collect) the UTS infra and the test target, run the tier's smoke tests and self-tests with the tier's command from the resolver's `infra` output (the mapping's `testCommand`, `{tier}` expanded) ([2.7](#27-infra-smoke-tests-and-self-tests-must)), and check the infra README's Known gaps against the selected specs. Integration and proxy tiers need network (and the proxy binary or platform) for this. +- If anything is red, stop and report it as a UTS infra or environment problem, never as an SDK deviation or a spec error; don't generate tests while the infra tests are red. +- If a selected spec needs a known gap, stop and ask: extend the UTS infra (an infra change, section 5), deselect the spec, or proceed recording a Mock Infrastructure Limitation for the affected lines. Never substitute. +- Exception, for the red check: an SDK-blocked objects smoke test (2.7). When the SDK-blocked items listed in Known gaps (under their expected-failure marker) are the only failures, translate-only runs proceed, and the preflight output lists it. +- Record the result in the final report. + +If the module's notes file exists but is only a placeholder, the skill MUST tell the user the module's mapping hasn't been authored and treat the module as not yet translatable, rather than guess a mapping. + +### Phase 2: per spec (steps 1–7) + +**Batching (SHOULD):** run steps 1–4 for every selected spec, then step 5 once for the module, then steps 6–7 per file. + +1. **Read the spec and the notes.** Read the module notes first (once per run); where they override the generic flow (reading list, internal-access ladder, naming, deviations location), the notes win. Then identify per spec: test cases and IDs, protocol (WS or HTTP), timer use, `## Protocol Variants`, and any delegating, fixture-driven or ID-less structure. +2. **Determine the output path.** Use the resolver's per-spec `targetDir`, `testFile` and class name. The spec's sub-directories below the tier are mirrored under the tier's target directory ([3.2](#32-the-mapping-file)). If a suitable suite already exists, add to it rather than create a duplicate. +3. **Read the UTS infra for exact signatures.** Read the infra README, then all helper sources for the tier (both from the resolver's `infra` output; the sources are derived from the tier layout, 2.2), before generating code. The source wins where the README disagrees; list each disagreement in the final report. Guessed signatures are the most common compile failure. `SKILL.md` SHOULD also name one reviewed, spec-derived test file per tier to read first (ably-cocoa: `ChannelHistoryTests.swift` for direct sandbox, `AuthReauthTests.swift` for proxy; ably-java names `ChannelHistoryTest` (realtime) and `ObjectsLifecycleTest` (objects), both direct sandbox, and no proxy reference). Until one exists, it says `none yet`. After a tier's first reviewed translation (audit clean and, in evaluate mode, one of the three end states, 7.1), the skill proposes that file in its final report and, if the user agrees, replaces the `none yet` line with it; nothing else in the skill changes. Never name a smoke test. +4. **Generate** using the rules in [section 5](#5-translation-rules) and the table in [section 6](#6-pseudocode-construct-catalogue). A construct the skill can't map is a stop-and-ask condition (section 6). +5. **Compile once per module.** Fix and recompile until clean. +6. **Run (evaluate mode only).** Run each class by filter, then diagnose per [section 7](#7-evaluation-and-deviations). +7. **Scripted review (both modes).** Run `audit_translation.py ` using the resolver's paths, then walk the review checklist ([8.3](#83-review-checklist-must-after-the-audit)). Any fix means re-audit, recompile and, in evaluate mode, re-run. + +--- + +## 5. Translation rules + +Every rule below is a MUST unless marked otherwise. + +**When a generated test is wrong, fix the cause, then regenerate.** The cause is a rule in `SKILL.md`, a mapping in the module notes, or an infra helper. A fix made only in the generated file is silently reverted by the next regeneration. + +**Changing the UTS infra during a run** (extending a mock, adding a message template or a helper), by the skill or the user, is an infra change, not a translation step. Make it only within the infra scope the user approved. Add or extend a self-test for the new capability, update the infra README (and its Known gaps), and re-run the tier's smoke tests and self-tests ([2.7](#27-infra-smoke-tests-and-self-tests-must)) before continuing; stop if they are red. Then regenerate the affected tests (fix the cause, then regenerate). The final report lists infra changes separately (section 11). + +### 5.1 Traceability + +- **Exactly one `UTS:` tag per test**, equal to the spec's `**Test ID**`, copied verbatim: `// UTS: ` immediately above the test function (`writing-test-specs.md` "Placement in derived tests"). Where `//` isn't a comment, use the line-comment marker (`# UTS: ` in Python or Ruby); cross-SDK tooling should grep for `UTS: ` without the marker. Never hand-build the prefix: proxy files use `/proxy/…` IDs although they live under `integration/proxy/`, and objects IDs can join spec points (`objects/unit/RTO5c9-RTO20/…`). +- **Never invent a tag.** When one test covers several spec points (e.g. `## Test 26: RTN22/RTC8a -- Server-initiated re-authentication` in `realtime/integration/proxy/auth_reauth.md`), put the extra points in the test name, not in a second tag. Invented tags show up as audit orphans. +- **Test names include the spec point** (`writing-derived-tests.md` Phase 1), as a valid identifier the runner collects: + - Swift `test_RTN16g_`; Kotlin backtick names `` `RTN16g - ` ``; + - C# `RTN16g_`, optionally with a `DisplayName`; + - pytest: file `test_.py`, class `Test`, function `test_rtn16g_` (default collection requires these prefixes); + - a module's notes may change how the name is rendered, but never drop the spec point. +- **A standard file header** naming the source spec path, its tier, the spec-repo commit translated from ([9.1](#91-record-what-you-translated-from-must)), and any disclosures (infra stand-ins, deviations). Integration and proxy headers also name the corresponding unit spec where the spec gives one (`proxy.md` convention 1). See the file outline in [3.4](#34-recommended-outlines-should). +- **Keep the spec's test order** and its section headings as comments (`Setup`, `Test Steps`, `Assertions`, `Test Steps and Assertions`, `Teardown`/`Cleanup`). + +### 5.2 Fidelity + +- **Copy every pseudocode comment verbatim** to the matching step. Don't paraphrase. +- **Keep spec variable names**, adapted to the language's case convention. Name new variables concretely (`attachFrames`, not `fields` or `result2`). +- **Every `ASSERT` and `AWAIT` is either translated at the same place or annotated, never dropped.** If one has no equivalent in the language or SDK, keep the spec line as a comment and add a note explaining why no assertion is emitted. An unannotated drop is a bug. +- **Wait conditions and timeouts must match the spec.** A `poll_until(auth_callback_count > n)` rendered as a poll on a different predicate is a silent semantic change. Any deliberate difference carries a `// NOTE:` (ambiguity, `writing-derived-tests.md` "Flag ambiguity") or `// DEVIATION:` comment. +- **Never substitute a similar mock outcome for one the UTS infra lacks.** If a spec uses a capability the UTS infra doesn't have (e.g. a WS-level `respond_with_timeout()`), extend the mock (preferred, usually small) or record a Mock Infrastructure Limitation. Don't silently use `respond_with_refused()` instead. A designed-in stand-in ([7.2](#72-infra-stand-ins-are-not-deviations)) is not a limitation. +- **Set only the options the spec sets.** The client factory may pre-seed a dummy key; set `key` only when the spec does. Force the protocol format the mocks understand (`writing-derived-tests.md` "Required options vary by SDK"). +- **Helpers return ready-to-assert values** (typed, non-optional unless the spec expects absence), so test bodies stay free of casts and parsing. Helpers used by one suite live before its first test or in a sibling file (8.2); helpers shared across a module's suites live in the module helpers. +- **Use SDK API values, not wire values**, when asserting on decoded objects, and convert units where the SDK's API differs from the wire (e.g. ms on the wire, seconds in the API). +- **Translate spec-local functions** (e.g. `function token_auth_callback(api_key)` in `rest/integration/proxy/rest_fallback.md`) as file-local helpers. + +### 5.3 Structural variants + +| Spec shape | Rule | +|---|---| +| **One ID, several H3 sub-cases** (e.g. `realtime/unit/client/realtime_client.md`, `### RTC1a_1`, `### RTC1a_2`) | One test carrying the one tag; translate every sub-case inside it, in order, or as a parameterised test that keeps the single tag | +| **Group heading with no ID, H3 tests with their own IDs** (e.g. `rest/unit/auth/revoke_tokens.md`) | The H2 is a grouper only. One test per H3 ID. MAY mirror the group as a comment or nested suite | +| **`### Test Cases` table driving `FOR EACH test_case`** | `test_cases` is the table. Render as a table-driven or parameterised test under the one tag. Rows that are language-inapplicable are omitted with a comment ([7.4](#74-language-inapplicable-inputs)) | +| **Delegating test or spec**: a whole file (`realtime/unit/channels/channel_get_message.md`) or a single test inside a mixed spec (RTAN3a in `channel_annotations.md`, a section with no Test ID; RTL10a in `channel_history.md`) that says "the tests in `uts/rest/unit/…` should be used to verify … on a `RealtimeChannel`" | The skill MUST define one policy and apply it everywhere. Recommended: one test with the delegating tag (where the section has no Test ID, as RTAN3a, follow the no-Test-ID row: no invented ID, a `// NOTE:` naming the heading) that runs the referenced REST cases against the realtime client through a shared helper. If that isn't feasible yet, mark the test pending (not commented out) with a `// NOTE:`. The UTS docs don't yet cover delegation | +| **Tests with no Test ID** (`rest/unit/encoding/msgpack_interop.md`) | Translate them; don't invent an ID. Add a `// NOTE:` naming the source heading, and review by hand (the audit can't verify them). *Divergence:* `uts-to-swift` expects a `// UTS:` tag here, which the audit then reports as an orphan | +| **`## Protocol Variants` section (body `json, msgpack`)** | Parameterise over the protocol (`useBinaryProtocol: PROTOCOL == "msgpack"`). The tag and the name stay singular. Specs without the section are JSON only. Proxy tests are always JSON | +| **Spec filed under another module's tree** (e.g. REST faults in `realtime/integration/proxy/rest_faults.md`) | Translate it with the module directory it sits in; the mapping decides the target, not the spec's subject | +| **Inline SDK hints in spec prose** (Dart, ably-js or ably-java names) | Hints for interpretation, not prescriptions. Don't copy them | + +### 5.4 Setup and teardown + +- **Close every client even when the test fails.** `CLOSE_CLIENT(client)` (added to the specs so internal timers stop) must land in `finally`, a teardown hook, or a scoped-resource helper, not only at the end of the body. +- **Restore mocks and cancel SDK timers in teardown**, not at the end of the test (`writing-derived-tests.md` "Cleanup with afterEach"). +- **Never hand-roll teardown** where the UTS infra provides a scope. ably-cocoa uses `with…` scopes because Swift Testing has no async teardown. +- `BEFORE ALL TESTS` / `AFTER ALL TESTS` sandbox provisioning maps to suite-level fixtures where the runner supports them (ably-java `@BeforeAll` with `@TestInstance(PER_CLASS)`; xUnit `IClassFixture`/`IAsyncLifetime`; NUnit `[OneTimeSetUp]`; pytest class- or module-scoped fixtures). Provision once per run wherever the runner supports suite-level fixtures. If it doesn't, per-test provisioning is allowed only as an infra stand-in disclosed in the file header ([7.2](#72-infra-stand-ins-are-not-deviations)); ably-cocoa does this. *Divergence from `writing-test-specs.md`* ("Sandbox App Management": create apps once per test run): per-test provisioning is slower and creates more apps, but it doesn't change what the test asserts. +- **Stop on a failed wait.** If a wait times out, record the failure and stop driving the client in the wrong state; teardown must still run. +- Close only from states where your SDK's `close()` can reach CLOSED; don't await CLOSED from the others (in ably-cocoa: FAILED and INITIALIZED). + +### 5.5 Integration-tier hygiene + +- **Unique channel names** in every integration test: a descriptive part plus a random part (`writing-test-specs.md` "Unique Channel Names"). +- **Each test creates and closes its own proxy session.** +- **No fixed sleeps.** A spec `WAIT` at integration tier is wall-clock; keep it only when the spec has it, with a comment. + +### 5.6 Time and waiting + +- **Unit tier: no real timers** (`writing-derived-tests.md` "No real timers in unit tests"). Use the fake clock for time-dependent behaviour and `process_pending_events()` (a zero-delay yield or internal-queue drain) for async delivery. The only allowed real timer is a safety timeout. +- **Integration and proxy tiers: every timeout is wall-clock** (`writing-derived-tests.md` "Integration timeouts are wall-clock"). In virtual-time runners (kotlinx `runTest`), wrap every real-network wait, including bare future awaits, in a real-clock helper (ably-java `withRealTimeout`). +- **Poll deadlines must not freeze under fake time**: preferably restructure the test so the wall clock needn't be stubbed; otherwise read a monotonic clock (`writing-derived-tests.md` "No real timers in unit tests"). +- **Every wait is bounded.** Use the spec's timeout where it states one. Otherwise, per `integration-testing.md` "Timeout Strategy": pass an explicit timeout on every integration `poll_until` call (10–30 s), and let state waits use helper defaults of 10–15 s. (`poll_until`'s reference interval is 500 ms and its default timeout 10 s; ably-cocoa's integration helpers default to 15 s.) +- **Bound the suite too, with headroom** (`integration-testing.md` "Timeout Strategy"). On the integration and proxy tiers, set a suite- or class-level timeout: about 120 s by default, 30–60 s for one to three fast tests. A proxy test that sets a low `realtimeRequestTimeout` needs at least `realtimeRequestTimeout` + 12 s of that headroom. +- **Operation timeouts are shorter than the suite's** (same section), so a failure says *what* timed out (an HTTP request, a state wait, sandbox provisioning), not just that the suite ran out of time. +- **Prefer state waits.** Map `AWAIT_STATE` to the state-wait helper; use a generic condition poll only when no state applies or the spec itself polls. +- **Keep the spec's `LOOP up to N times: ADVANCE_TIME … BREAK`** as a loop; don't compute exact jumps. +- **Advance fake time, then flush** the async runtime before asserting. + +### 5.7 Waits must catch events + +- **`AWAIT_STATE` must be event-latched and race-free**: subscribe first, then check the current state, and complete exactly once (whichever of the listener and the immediate check wins). Checking before subscribing leaves a window in which the transition is missed. ably-java's `awaitState` does this, with an atomic single-winner resume. *Divergence from `writing-test-specs.md`* ("State Transitions" lists check, then wait): the observable semantics are the same, but subscribe first, for the reason above. The helper also removes its listener on success, timeout and cancellation; leaked listeners keep firing into later tests. +- **A state wait fails fast on FAILED.** If the connection enters FAILED while waiting for another state, fail immediately with the error reason instead of running out the timeout (ably-cocoa's integration `awaitState`). Don't treat a state that is already FAILED at the start of the wait as a failure when the test has just requested a transition out of it (`connect()`, or `authorize()` from FAILED per RTC8c); latch the next transition instead. +- **Transient states use record-and-verify** (`writing-test-specs.md` "Verifying Transient States"): record all state changes, let the cycle finish, then assert `CONTAINS_IN_ORDER`. Even a correct latch misses a state entered and left before the call. Don't insert an intermediate `AWAIT_STATE disconnected`. +- **Polls return the settled value.** For a value-assigned `x = poll_until(…)`, the helper returns the value and later assertions use it. Don't re-read afterwards: an eventually-consistent store can return fewer items on the second read. +- **`poll_until` aborts on an error; `poll_until_success` keeps polling and rethrows the last error on timeout.** Implement them as two helpers; don't emulate `poll_until_success` with a reader that swallows errors to `null`, because the final error is lost. +- **Await the observable effect** of one step before triggering the next (for example, await the seeded state before subscribing, and flush pending events before injecting the next stimulus). The pseudocode assumes synchronous mock delivery; real SDKs don't. + +### 5.8 Assertions + +- **No "accept either behaviour" assertions** (`writing-derived-tests.md` "Avoid the accommodate-both pattern"). Assert the spec or, as a documented deviation, the SDK's actual behaviour. Never both. +- `FAILS WITH error` asserts the `ErrorInfo` fields (`code`, `statusCode`, `message`), not exception type names. +- **`IS `**: a native type check in statically typed languages; in weakly typed ones (Python, Ruby, JavaScript), `isinstance` or a check that the expected members exist (`writing-test-specs.md` "Type Assertions"). Where the static type already guarantees it, see [7.4](#74-language-inapplicable-inputs) case (b). +- Use a hard assertion (stop the test) where later lines depend on the value, and a soft one elsewhere. +- Normalise numeric types where the language compares `Int`, `Long` and `Double` as unequal. + +### 5.9 Internal access (white-box unit specs) + +**Public-API specs use only the public API.** Even where the test target can see internals, unit specs that exercise the public API use only the public API plus infra helpers; so do fixture helpers such as REST provisioning. Only the module's listed white-box specs, and documented deviations, may reach internals. List the white-box specs in the module notes ([3.4](#34-recommended-outlines-should) item 8). + +For white-box specs, use this ladder, documented in the skill or in the module notes: + +1. Use an existing internal exposure. **Check first; never duplicate.** +2. Otherwise add a minimal, test-only exposure in the SDK's sanctioned way (ably-cocoa core: a private header registered in both modulemaps, used via `import Ably.Private`; ably-cocoa objects: a `testsOnly_` accessor in the shared test-support module; C#: `InternalsVisibleTo`). Accessors stay dumb; no logic. +3. If more than a visibility raise is needed, **stop and escalate** to a maintainer. +4. If the symbol is unreachable, keep the spec line as a comment and record a Mock Infrastructure Limitation. + +When a white-box spec asserts on the value an internal call returns, assert on that return value, not on a side channel such as emitted events. Integration and proxy specs use only the public API (`writing-derived-tests.md`, case 3). + +--- + +## 6. Pseudocode construct catalogue + +Fill in the last column for your language; the skill MUST map every construct its specs use. Each row's "meaning" comes from the UTS docs where one exists; rows marked *(undocumented)* are used in the corpus but not defined in the docs, and the meaning is inferred from usage. The Swift and Kotlin cells name only methods and patterns that exist in each skill, its notes, its UTS infra or its generated tests; "—" means that SDK has no rule for the construct, so follow the meaning column. The cells illustrate; they aren't templates. Write your own column from your UTS infra and SDK ([reference implementations](#reference-implementations-last-resort), rule 2). + +**This catalogue is not exhaustive**: the corpus keeps growing and has no grammar. The skill MUST treat any uppercase keyword or snake_case helper it can't map as a stop condition: tell the user, propose a rendering, and add a row to its own table rather than improvising per test. Re-scan the corpus for new keywords on every re-sync ([9.2](#92-a-re-sync-mode-should)). The skill SHOULD ship a corpus scanner (`scripts/scan_constructs.py`, [3.1](#31-layout)) that lists every uppercase keyword and `snake_case(` call in a module's `pseudo` fences, outside comments and string literals, with counts, so the re-scan is mechanical; compare its output with the construct table. + +### 6.1 Async, waiting and state + +| Construct | Meaning | Swift (ably-cocoa) | Kotlin (ably-java) | Your language | +|---|---|---|---|---| +| `AWAIT expr` | Await an async call | `try await op`, or a continuation-bridged helper for callback APIs | `.await()` on a future, or `suspendCancellableCoroutine` around a callback | | +| `f = op()` … later `AWAIT f` (incl. `AWAIT f FAILS WITH error`) | Deferred future: start an operation, act, then await it | `let t = Task { try await op() }` … `try await t.value` (objects module notes) | Hold the `CompletableFuture` (``val getFuture = channel.`object`.get()``) … `getFuture.await()` (generated tests, e.g. `RealtimeObjectTest`) | | +| `AWAIT_STATE x.state == ConnectionState.connected` (also `ChannelState.attached`, bare `CONNECTED`) | Proceed if already true, otherwise wait for the state event; fail on timeout (`writing-test-specs.md` "State Transitions") | `awaitConnectionState(client, .connected)` (unit); `await awaitState(client, .connected)` (integration) | `awaitState(client, ConnectionState.connected)` | | +| `… WITH timeout: N seconds` (`Ns`, `N second`) | Bound on the wait; wall-clock at integration tier | `timeout:` parameter on helpers; no per-call timeout on native `await` (note it in a comment) | `awaitState(client, state, 10.seconds)`; `withRealTimeout` for integration futures | | +| `AWAIT UNTIL `, `WAIT_FOR ` *(undocumented)* | Wait until the condition holds | — | — | | +| `AWAIT_ERROR future` *(undocumented)* | Await a future expected to fail; bind its error | — | — | | +| `AWAIT_ALL futures` *(undocumented)* | Await all futures | — | — | | +| `x.once("connected")` awaited | JS-flavoured one-shot wait; treat as `AWAIT_STATE` | — | — | | +| `poll_until(cond, interval, timeout)`; also the `POLL_UNTIL(timeout:, interval:):` block and `AWAIT pollUntil(CONDITION: …, timeout:)` *(undocumented spellings)* | Re-evaluate until truthy; return the value; an error aborts (`writing-test-specs.md`) | `poll("desc") { … }` (unit, Bool); `pollUntil("desc") { … }` (integration, value-returning) | `pollUntil(10.seconds, 500.milliseconds) { … }` | | +| `poll_until_success(condition: FUNCTION() => …)`; bare form `poll_until_success(, timeout: 15s)` | Errors mean "keep polling"; on timeout rethrow the last error (`uts/README.md`) | — (ports used `pollUntil` with a reader that maps errors to `nil`) | — | | +| `process_pending_events()` | Let queued async work finish; no real delay (`uts/README.md`) | — (objects ports drain the collector queue) | Objects module notes: `` (channel.`object` as DefaultRealtimeObject).asyncFuture { }.await() `` | | +| `WAIT 500ms` | Real delay; discouraged, present at integration tier | `Task.sleep` with a comment (generated tests) | — | | +| `RETURN NEVER_RESOLVING_FUTURE` *(undocumented)* | A callback that never completes | — | — | | +| Quiescence / control-listener (`assert_unchanged_after_quiescence`) | Objects negative assertion (`standard_test_pool.md`) | Objects module notes | Inline control listener in generated tests (e.g. `InstanceTest.kt`); no notes rule | | + +### 6.2 Time + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `enable_fake_timers()` | Install the fake clock before constructing the client | `enableFakeTimers()` before `makeRealtime`/`makeRest` | `FakeClock()` + `enableFakeTimers(fakeClock)` | | +| `ADVANCE_TIME(ms)` | Advance fake time and fire due timers | `advanceTime(byMilliseconds:)` | `fakeClock.advance(…)`, then `yield()` | | +| `LOOP up to N times: ADVANCE_TIME … IF … BREAK` | Time-advancement loop, then a final `AWAIT_STATE` | Keep the loop | Keep the loop | | +| `TestClock()`, `WITH_CLOCK(test_clock):`, `test_clock.advance(…)`, `test_clock.now()` *(undocumented)* | A DI clock scoped to a block | — | — | | +| `SCHEDULE_AFTER(100ms):` inside a mock handler *(undocumented)* | Run the block after fake time advances by the delay | — | — | | +| `NOW()` / `now()`, `now_millis`, `time_now`, `current_time`, `current_fake_time` | Current time (fake, where installed) | — | — | | + +### 6.3 Assertions and matchers + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `ASSERT a == b` | Equality (deep where appropriate) | `#expect(a == b)` | `assertEquals(b, a)` (expected first) | | +| `IS null` / `IS NOT null` (also `IS NULL`, `IS NOT NULL`, `IS undefined`) | Language-appropriate absent value (`uts/README.md`) | `#expect(x == nil)` / `try #require(x)` | `assertNull` / `assertNotNull` | | +| `IS null OR … IS NOT SET` *(undocumented)* | Field absent or null | — | — | | +| `IS ` | Native type check; member check in weakly typed languages ([5.8](#58-assertions)) | `#expect(x is T)` | `assertIs(x)` | | +| `IS SAME AS` / `IS NOT SAME AS` *(undocumented)* | Reference identity | — | — | | +| `IN` / `NOT IN` | Membership or key presence | `#expect(map["k"] != nil)` | `assertContains(map, "key")` | | +| `x HAS member` *(undocumented)* | Member exists | — | — | | +| `CONTAINS` (optionally `(case insensitive)`) | Substring or element | — | — | | +| `STARTS WITH` / `STARTS_WITH`, `ENDS WITH` / `ENDS_WITH` | Prefix / suffix (both spellings occur) | — | — | | +| `matches pattern "regex"`, `MATCHES /regex/` | Regex match | `range(of:options: .regularExpression)` | `assertTrue(x.matches(Regex(…)))` | | +| `CONTAINS_IN_ORDER [a, b, c]` | In-order subsequence: the items appear in this order; other items may be interleaved (`writing-test-specs.md`) | See the first note in 6.7 | See the first note in 6.7 | | +| `ASSERT ALL x IN xs: pred`, `ASSERT ALL x == y FOR x IN xs`, `ASSERT ANY …` *(undocumented)* | Universal / existential check | — | — | | +| `ARE all unique` *(undocumented)* | No duplicates | — | — | | +| `IS EMPTY` / `IS NOT EMPTY` (also lowercase `IS empty` / `IS NOT empty`) | Empty / non-empty | — | — | | +| `IS NOT complete` *(undocumented)* | A deferred future hasn't settled yet (`objects/unit/realtime_object.md`) | — | — | | +| `IS valid`, `IS valid base64url string` *(undocumented)* | Well-formed value of the named kind (`rest/unit/fallback.md`, `objects/unit/object_id.md`) | — | — | | +| `.length == N`, `>=` | Size comparison | `#expect(list.count == N)` | `assertEquals(N, list.size)` | | +| `FAIL("msg")` | Unconditional failure | `Issue.record("msg")` (as in the skill's fail-fast pattern) | `fail("msg")` (kotlin.test; as in the skill's fail-fast pattern) | | + +### 6.4 Errors + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `AWAIT op() FAILS WITH error` / sync `op() FAILS WITH error` | Bind the failure's `ErrorInfo`; assert its fields | Capture the error, then `#expect(error.code == …)` | `assertFailsWith { … }` then `error.errorInfo.code` | | +| `error.cause.code` | Nested cause of an `ErrorInfo` (e.g. RTO23c1 in `objects/unit/realtime_object.md`: 92008 caused by 90000; RTO20e1 requires the same cause) | Objects module notes: check what the implementation populates; else assert the top level and flag a deviation | Objects module notes: read the Java exception cause | | +| `… THROWS error`, `THROWS AblyException WITH:` *(undocumented)* | Same as `FAILS WITH` | — | — | | +| `EXPECT THROW creating Realtime(…)` *(undocumented)* | Construction must fail | — | — | | +| `THROW ErrorInfo(…)` / `THROW AblyException(…)` / `THROW Error(…)` inside a test-supplied callback *(undocumented)* | Raise an error from the callback (e.g. an `authCallback` that fails), so the SDK sees a callback failure | — | — | | +| `TRY: … CATCH` | Discouraged in specs but present; translate literally, but assert error fields | — | — | | + +### 6.5 Control flow, data and parameterisation + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `IF / ELSE IF / ELSE`, `IF … THEN`, `… END`, `WHILE`, `PASS` | Ordinary control flow; `END` closes a block; `PASS` is a no-op | native | native | | +| `AND`, `OR`, `NOT` | Boolean operators | native | native | | +| `LOG "…"` *(undocumented)* | Informational output: render as a test log line or comment, never an assertion | — | — | | +| `FOR EACH test_case IN test_cases:` + `### Test Cases` table | Table-driven test; `test_cases` is the table | — | — | | +| `FOR x IN [..]`, `FOR i IN 0..n-1`, `FOR [k, v] IN x.entries()` | Loops | native | native | | +| `FIND x IN xs WHERE …`, `FILTER xs WHERE …` *(undocumented)*, `.filter(…)`, `.any(…)`, `.map(…)` | First match / filtered list / predicates | native | native | | +| `FUNCTION(params): … RETURN`, `(x) => {…}`, spec-local `function name(…)` | Local function / lambda / file-local helper | native | native | | +| `LISTEN`, `SET()` | Event subscription; set literal | — | — | | +| `"${…}"`, `+` on strings | Interpolation / concatenation | native | native | | +| `PROTOCOL` | Protocol-variant variable | `@Test(arguments: [false, true])` | `@ParameterizedTest @ValueSource(booleans = [false, true])` | | +| `#` and `//` comments | Spec comments: copy verbatim | `//` | `//` | | + +### 6.6 Mocks, fixtures and UTS infra + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `MockHttpClient(onConnectionAttempt:, onRequest:)` + `install_mock()`; `req.respond_with(…)`, `respond_with_delay(ms, status, body)` | `mock_http.md` | Handler-pattern mock, installed before client creation | `install(mock)` | | +| `mock_http.queue_response(…)` and the other `queue_*` calls; `mock_http.captured_requests` *(legacy; deprecated in `writing-test-specs.md` "Common Mistakes to Avoid")* | Pre-queued responses; mock-owned request log ([2.2](#22-a-shared-test-library-implementing-the-helper-specs-must)) | — | — | | +| `mock_http.reset()`, `mock_ws.reset()` | Clear all mock state (`mock_http.md`, `mock_websocket.md`) | — | — | | +| `MockWebSocket(onConnectionAttempt:, onMessageFromClient:, onTextDataFrame:, onBinaryDataFrame:)` | `mock_websocket.md` | Handler pattern, `onConnectionAttempt` only | `onConnectionAttempt`, `onMessageFromClient`, `onTextDataFrame`, `onBinaryDataFrame` callbacks (preferred over the await style) | | +| `on_connect(respond_with:)`, `on_message(action:, respond_with:)`, `on_client_message(…)`, `await_client_message(action:)` *(undocumented alternative API)* | Shorthand handlers | — | — | | +| `conn.respond_with_success/_refused/_timeout/_dns_error/_error`, `respond_with_connected()` | Connection outcomes | `respondWithSuccess`, `respondWithRefused`; no WS-level timeout or DNS error (a known gap: extend the mock or record a limitation) | `respondWithRefused()`, `respondWithTimeout()`, `respondWithDnsError()` | | +| `CONNECTED_MESSAGE`, `CLOSED_MESSAGE`, `DISCONNECTED_MESSAGE`, `HEARTBEAT_MESSAGE`, `ERROR_MESSAGE(code, message)`, `PING_MESSAGE(id)` | Message templates (`mock_websocket.md`) | `.connectedMessage`, `.connected(…)`, `.attached`, `.error`, `.ack`, `.closed()` factories (`connectionStateTtl` in seconds, not wire ms); no `PING_MESSAGE` factory | `CONNECTED_MESSAGE`; `ErrorInfo(message, statusCode, code)` argument order | | +| `mock_ws.active_connection.send_to_client(…)`, `send_to_client_and_close`, `simulate_disconnect`, `close_from_server()`, `send_to_client_raw(json)`, `send_ping_frame` | Server-side actions | `sendToClient`, `sendToClientAndClose`, `simulateDisconnect`; no ping-frame method | `sendToClient`, `sendToClientAndClose`, `simulateDisconnect`; no ping-frame method | | +| await API: `await_connection_attempt()`, `await_request()`, `await_next_message_from_client()`, `await_client_close()` | Await style; set up the next await before responding to the current one | — (not implemented; handler style + attempt counter + `sentMessages` instead) | `awaitConnectionAttempt()`, `awaitRequest()`, `awaitNextMessageFromClient()`, `awaitClientClose()` | | +| `mock_ws.events.filter(…)` with event types `CONNECTION_ATTEMPT`, `CONNECTION_SUCCESS`, `CONNECTION_FAILURE`, `MESSAGE_FROM_CLIENT`, `MESSAGE_TO_CLIENT`, `CLIENT_CLOSE`, `SERVER_DISCONNECT`, `PING_FRAME` (`mock_websocket.md`); `mock_ws.connect_attempts` *(no helper spec; used in a `writing-test-specs.md` example)*, `mock_ws.last_connect_url` *(undocumented)* | Mock event log; connection attempts and the last connect URL | `ws.sentMessages` (client → server frames) | `events: List` | | +| `mock_ws.onConnectionAttempt = …` | Reassign a handler mid-test | — | — | | +| `create_mock_websocket()`, `create_realtime_client(…)` *(undocumented aliases)* | Same as `MockWebSocket()` / `Realtime(…)` | — | — | | +| `MockNetworkListener()`, `simulate_network_lost()` / `simulate_network_available()` *(no helper spec)* | Network-change mock ([2.2](#22-a-shared-test-library-implementing-the-helper-specs-must)) | — | — | | +| `CLOSE_CLIENT(client)` *(undocumented)* | Close the client so its internal timers stop | `closeClient(client)`; integration scopes own it | `client.close()`; in `finally` for proxy tests | | +| `captured_requests.append` / `.push`, `request.url.path` / `request.path`, `.url.query_params` / `.query_params` / `.queryParams` | Request capture (spellings vary) | `Captured` | `CopyOnWriteArrayList` | | +| `parse_json` / `JSON_PARSE` / `JSON_DECODE` / `json_decode`, `json_encode`, `MSGPACK_DECODE` / `msgpack_decode` / `msgpack_deserialize`, `msgpack_encode`, `base64_decode` / `base64_encode`, `load_json`, `load_fixtures`, `toJson()` / `fromJson()`, `encode_uri_component()` / `decode_uri_component`, `base64url_encode` / `base64url_decode`, `encode_utf8` / `utf8_encode` / `utf8_decode`, `parse_query_string`, `parse_form_urlencoded`, `extract_between`, `byte_array([…])`, `to_json`, `msgpack_serialize`, `as_list`, `DateTime(…)` | Data helpers; `toJson`/`fromJson` map to the SDK's idiomatic names (not a deviation) | `[String: Any]` | — | | +| `random_id()`, `random_string`, `unique_channel_name()`, `base64(random_bytes(6))` | Random / unique names | UUID channel names | `UUID.randomUUID()` | | +| `COUNT_UNIQUE(…)`, `EXTRACT_FALLBACK_ID(…)` *(undocumented)* | Spec-level helpers (`fallback_hosts_test.md`) | — | — | | +| `generate_jwt(…)` / `generateJWT(…)`, `extract_key_name(api_key)`, `extract_key_secret(key)`, `get_key_parts`, `request_token_from_sandbox(api_key, params)` *(the last two undocumented)* | Auth helpers | Locally signed `TokenRequest` instead of JWT, with a comment | Locally signed `TokenRequest` | | +| `BEFORE ALL TESTS`, `AFTER ALL TESTS`, `BEFORE EACH TEST`, `AFTER EACH TEST`, `AFTER TEST:` | Fixtures | `with…` scopes, per test | `@BeforeAll` / `@AfterAll` | | +| `POST https://…/apps WITH body from …`, `DELETE … WITH Authorization: Basic …` | Sandbox provisioning | `SandboxApp` | `SandboxApp.create()` / `delete()` | | +| `create_proxy_session(endpoint:, rules:)` (or `rules:` only; see 6.7), `add_rules`, `trigger_action`, `get_log()` / `session.getLog()`, `proxy_port` | `proxy.md` | `withProxySession(rules:)`; typed `ProxyEvent` log | `ProxySession.create(…)`, `finally { session.close() }`; typed log | | +| `"__PASSTHROUGH__"` as a field value in a `replace` action's message *(undocumented)* | A placeholder string, not a uts-proxy feature: uts-proxy (as of v0.3.0) sends a `replace` message verbatim, so the SDK receives the literal value (used in `realtime/integration/proxy/connection_resume.md` Test 22, whose assertions don't depend on it). Treat it as an ordinary string; don't implement a substitution | — | — | | +| `MockVCDiffEncoder()`, `MockVCDiffDecoder`, `FailingMockVCDiffDecoder` | `mock_vcdiff.md` | — | — | | +| `setup_synced_channel`, `setup_synced_channel_no_ack`, `STANDARD_POOL_OBJECTS`, `build_*`, serial helpers, `provision_objects_via_rest` | `standard_test_pool.md` | Unit tier seeds the pool directly (sanctioned stand-in); helpers in the test-support target | `setupSyncedChannel("test")`, `build*` in module `Helpers.kt` | | +| White-box access: `applyOperation`, `channel.object.objectsPool`, `processChannelState`, direct construction, `get_backoff_coefficient(n)`, `get_jitter_coefficient()`, `encode_recovery_key(…)`, `CLEAR channel._lastPayload.messageId` | Internal access ([5.9](#59-internal-access-white-box-unit-specs)). `CLEAR …` in `delta_decoding_test.md` is a private-field write inside an *integration* spec | Core: `import Ably.Private` + private headers; objects: `@testable import` + `testsOnly_` accessors | `internal` within `:liveobjects` | | +| `logLevel: LOG_INFO`, `logHandler: (level, message) => …` (with `level == WARN` / `info` / `debug`), `captured_log_messages.filter(…)`, `captured_logs`, `log_messages` | Log capture; log levels | `CapturingLog`, `log.contains(level:message:)` | — | | +| `MESSAGE_ACTION_INT`, `HAS_OBJECTS`, `HAS_PRESENCE`, protocol actions | Wire constants | — | — | | +| `DEFAULT_REST_HOST`, `SANDBOX_ENDPOINT` *(undefined in the docs)*, spec-local constants such as `DEFAULT_CONNECTION_STATE_TTL = 5000` | Named constants: the SDK's default, the UTS infra's sandbox endpoint, or a value the spec defines | — | — | | +| Reserved words (`channel.object`) | Escape idiomatically; not a deviation | — | `` channel.`object` `` | | + +### 6.7 Notes on the catalogue + +- **`CONTAINS_IN_ORDER` is an in-order subsequence, not a contiguous prefix or an exact filtered list.** `[connecting, disconnected, connecting, connected]` satisfies `CONTAINS_IN_ORDER [disconnected, connecting, connected]`. Kotlin's iterator-`next()` mapping asserts a contiguous prefix. Swift's `filter { [a, b, c].contains($0) } == [a, b, c]` fails when a listed state occurs more than once; its alternative ("walk an index as the spec does") is correct but unspecified. Ship an `assertContainsInOrder` helper in the UTS infra and map to it. (ably-java's `ObjectsFaultsTest.kt` has a correct file-local one; the gap is that it isn't in the UTS infra.) +- **Enum spellings vary**: `ConnectionState.connected`, `ChannelState.attached` and bare `CONNECTED` mean the same. +- **`writing-derived-tests.md` writes `AWAIT_STATE(connection, "connected")`**; the corpus form is `AWAIT_STATE client.connection.state == ConnectionState.connected`. Support both. +- **Proxy session:** `writing-test-specs.md` shows `create_proxy_session(target: TargetConfig(…))`; `proxy.md` and 41 of the 43 corpus calls take `endpoint:`. The other two (`realtime/integration/proxy/presence_reentry.md`) pass only `rules:`, so the session helper must default the endpoint to the sandbox. No corpus call uses `target:`. Follow `proxy.md`. +- **Proxy msgpack:** `proxy.md` gives "SDK doesn't implement msgpack" as the JSON reason; the real reason is that the proxy only supports text frames (`integration-testing.md`). + +--- + +## 7. Evaluation and deviations + +Follow `writing-derived-tests.md` Phase 2 and "Recording deviations" exactly. The skill adds only the language rendering and the stopping rules. + +### 7.1 Three acceptable end states + +Every evaluated test ends in exactly one of these. Never an unexplained red. + +| Outcome | Test state | Rendering the skill must specify | +|---|---|---| +| **Pass** | Green | — | +| **SDK deviation** (`writing-derived-tests.md` 2c) | Green: env-gated skip on `RUN_DEVIATIONS`, or adapted assertion (preferred when the divergence is permanent or intentional), with a `// DEVIATION: see deviations.md` comment | A runtime skip idiom (below) and the reproduction command (`RUN_DEVIATIONS=1 `) | +| **UTS spec error** (`writing-derived-tests.md` 2a) | Red, fails fast, with a message pointing at `deviations.md`. The one acceptable red | The fail idiom (Swift `Issue.record("UTS spec error — fix the spec first; see deviations.md")`, Kotlin `fail(…)`) | + +A translation error (2b) is not an outcome: fix the cause ([section 5](#5-translation-rules)) and regenerate. + +Runtime skip idioms (the gate must read the environment in the test process; a pytest `skipif` decorator does, a compile-time attribute doesn't): + +- Swift Testing: `@Test(.enabled(if: ProcessInfo.processInfo.environment["RUN_DEVIATIONS"] != nil))` +- Kotlin: `if (System.getenv("RUN_DEVIATIONS") == null) return@runTest` +- pytest: `@pytest.mark.skipif(not os.environ.get("RUN_DEVIATIONS"), reason=…)` +- C#: xUnit v2 `[SkippableFact]` + `Skip.If(…)` (Xunit.SkippableFact), xUnit v3 `Assert.Skip(…)`, NUnit `Assert.Ignore(…)` + +**SDK lacks the API entirely**, so the spec-correct test can't compile. First decide, per `writing-derived-tests.md` "Check the SDK's API surface", whether the test is not applicable to this SDK ([7.4](#74-language-inapplicable-inputs) case (c)) or the absence is itself a deviation. If it is a deviation: register the test, keep the uncompilable spec lines as comments with a `// NOTE:`, and mark it skipped (not commented out; `writing-derived-tests.md` "Test-first considerations"). In evaluate mode, record it under Failing Tests with the test impact "skipped stub". *Divergence from `writing-derived-tests.md`*, whose Failing Tests are env-gated: a test that can't compile can't be env-enabled, so it stays a skipped stub until the API exists. In translate-only mode, list it in the final report ([section 11](#11-final-report-format)). + +### 7.2 Infra stand-ins are not deviations + +Using fake timers where the spec uses real ones, a queue-ordering workaround, per-test sandbox provisioning, or seeding state directly instead of driving the spec's mock transport changes *how the test drives the SDK*, not what the SDK does. Explain it in a code comment or the file header. Don't record it in `deviations.md`. This extends `writing-derived-tests.md` case 1 (idiomatic translation) to the UTS infra; the docs don't state it for infra stand-ins explicitly. + +### 7.3 `deviations.md` + +- Use the format in `writing-derived-tests.md` "Recording deviations"; **don't define a home-grown format in the skill**. That means: + - all four headings, in order, with `*(none)*` for an empty one: UTS Spec Errors, Failing Tests, Adapted Tests, Mock Infrastructure Limitations; + - the entry fields: spec point, what the spec says, what the SDK does, root cause, test impact, status (for SDK deviations) and resolution (once resolved); + - where internal-shape differences recur, an `S-1…S-n` vocabulary. +- The skill specifies only where the file lives (one per owning test module) and that new entries go into their category's section, never appended at the end. +- **When it is written.** Per `writing-derived-tests.md` "Test-first considerations", the file is created during evaluation. In translate-only mode, list mock-capability gaps (Mock Infrastructure Limitations) and missing APIs in the final report ([section 11](#11-final-report-format)) instead, and record them on the first evaluate run. + +### 7.4 Language-inapplicable inputs + +Per `uts/README.md`, an input that can't be constructed in your language makes that test or table row **not applicable**: note it in the derived test file, not in `deviations.md`, and don't count it as a coverage gap. Three cases, plus a fourth that isn't about the language: + +| Case | Example | Rule | +|---|---|---| +| **(a) Inexpressible input** | A wrong-typed argument a static type system rejects (`increment("10")`); a non-string key in JavaScript | Not applicable. Omit the test or row with a comment at the omission | +| **(b) Assertion guaranteed by the static type** *(extends the README)* | `ASSERT keys IS Array` on a method whose declared return type is a list | Keep the spec line as a comment noting the compile-time guarantee: an annotated omission, not a deviation | +| **(c) Missing API** | The SDK has no such method | Not one of the README's cases: decide not-applicable vs deviation per `writing-derived-tests.md` "Check the SDK's API surface" ([7.1](#71-three-acceptable-end-states)) | +| **(d) Capability-inapplicable** | The test constructs a client the SDK doesn't have: a REST test that enters presence through a `Realtime` client, run against a REST-only SDK | Not applicable, and not a deviation. Note it in the test file, or list it under the mapping's `capabilityInapplicable` so it is never selected ([3.2](#32-the-mapping-file)). Not language-inapplicable either: it becomes applicable when the SDK gains the client | + +- Not every type mismatch is inapplicable. Calling `increment` on a map is still expressible by casting to the counter view and asserting the throw; translate it. +- Dynamic languages rarely have case (a): translate wrong-type rows as runtime assertions. +- *Divergence:* both existing skills' objects module notes record compile-time-unrepresentable inputs (case a) as deviations, and ably-java's first audit run also filed case (b) type assertions as deviations. New skills follow the README. + +### 7.5 Stopping rules for evaluate mode (SHOULD) + +"Fix until green" needs a bound. The skill should: + +- Stop and ask the user after a fixed number of fix attempts per test (pick one, e.g. three). +- Stop and ask when a failure looks like a sandbox or network problem rather than SDK behaviour (passes alone but fails in the suite; provisioning errors). On the integration and proxy tiers, tests that pass alone but fail together point to sandbox rate limiting or connection exhaustion: raise the suite timeout, don't add retries (`integration-testing.md` "Avoiding Flaky Tests"). +- Stop and report each suspected UTS spec error, with the features-spec quote and a draft spec fix, rather than only recording it. Several upstream spec fixes came out of evaluation runs. +- Report undocumented helper contracts the port had to invent (e.g. the element type of `mock_ws.connect_attempts`, which no helper spec declares) as suspected spec gaps. +- Never fix a test by weakening an assertion, re-reading after a poll, or accepting both behaviours. + +--- + +## 8. Deterministic tooling + +Path validation, mapping, spec discovery, naming and faithfulness checking are mechanical. **Scripts do them identically every run; a model eyeballing two files does them inconsistently.** In ably-java the audit's first use found three undocumented omissions in already-reviewed tests. Data that maintainers update (such as the mapping) belongs in a JSON file, not in script code; start each file with one `_description` key (what the file is and when to update it), and add a `_comment` key only where an entry would otherwise puzzle a maintainer; the scripts ignore keys that start with `_`. + +### 8.1 Resolver (`resolve_uts.py`) (MUST) + +| Requirement | Detail | +|---|---| +| Validate the module | Expand a leading `~`. For a module directory: the parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/`. A bare module name (3.3) is resolved against the shared spec clone (9.4) first. Also accept a spec clone's root (a directory with `uts/`) and no argument (the shared clone): neither names a module, so the resolver answers with the module listing below | +| Module listing | A mode (for example `--list-modules []`, or the same answer for a clone root) printing `{ok: true, specRoot, specRepo?, modules: [{name, mapped, offerable, reason?}]}`: every directory under the clone's `uts/` that has `unit/` or `integration/`. `offerable` is false for an `unready` module and for an `unsupported` one without `extraTests`, with the reason from that marker; an `unsupported` module with `extraTests` is offerable, for just those tests. The module question (4, before step A) reads only this | +| Errors | One JSON object with `ok: false`, a code and a `message`. Example codes (ably-cocoa): `NOT_A_UTS_MODULE_PATH`, `DIR_NOT_FOUND`, `NO_TIER_DIRS`, `MAPPING_NOT_FOUND`, `BAD_MAPPING`, `BAD_TARGET_NAME` | +| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, infra, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, subDir, targetDir, testFile, ?}]}}}`. Per spec: `file` and `subDir` (`""` at the tier base) are relative to the tier's source directory; `targetDir` is the tier's target directory plus the mirrored `subDir` (3.2), and `testFile` the test file's path in it (named `.`, or in the runner's form, e.g. pytest `test_.py` with class `Test`), both repo-relative; the namespace or package per spec wherever the language derives it from the path. The tier-level `targetDir` stays as the base, used by `--create`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty, dirtyFiles, branch, kind, onOriginMain}`, MAY) gives the spec clone's state (`dirtyFiles`: the modified or untracked files under `uts/` and `specifications/`, 9.1; `kind`: the shared clone or the one the argument names, which isn't synced unless it is the shared clone; `onOriginMain`: whether the SHA is on `origin/main`, 9.4) for the file headers and the report (9.1, 9.4). `infra` (`{root, readme, testsRoot, tiers: {: {sources, tests}}}`) is derived from the mapping's infra entry, with paths validated (repo-relative): per tier, `tests` is `testCommand` with `{tier}` expanded, and `sources` follows the tier layout (2.2: the root's own files and `common/`, plus each tier directory on the path to the tier; never the infra-tests tree), or the whole root where the UTS infra isn't organised by tier yet; the Infra reference and the preflight read it (3.2, 4 step F). Downstream steps read only these fields | +| Older mapping shapes | Read them with a warning in the output (for example a `warnings` list), never an error: the entry under its former key `harness` (a deprecated alias of `infra`, in any shape), per-tier `tiers..sources` (used as listed) and `tiers..tests` (that tier's command), and a string `tests` (read as `testsRoot`). Keys the mapping no longer needs (a white-box list, a test project, dummy tier paths beside a module-level `unready`) are ignored; the next Upgrade/Fix removes them (9.5) | +| Tier detection by path | `unit/**`; `integration/**` excluding `integration/proxy/**`; `integration/proxy/**`. Every tier can have sub-directories (e.g. `realtime/integration/channels/`). Use the path only | +| Exclusions | `helpers/` (implement in the UTS infra, never translate), `README.md`, `PLAN.md`, `*_SUMMARY.md`. Match them relative to the tier base, so an ancestor directory in the checkout path can't trip them. Also skip any non-spec notes left inside a tier directory | +| Deterministic naming | Strip a trailing `_test`, convert to the language's convention, add the suffix or prefix the runner needs (`objects_lifecycle_test.md` → Swift `ObjectsLifecycleTests`, Kotlin `ObjectsLifecycleTest`, pytest `test_objects_lifecycle.py` with class `TestObjectsLifecycle`). Each mirrored sub-directory segment is converted to the language's directory convention (3.2), and must also be a valid identifier in the language (no hyphens, not a keyword) | +| Collision detection | Fail loudly if two specs map to one test file, or to one class name in one namespace. In a language with one namespace per test target (Swift), that means one class name anywhere in the target. Warn when a sub-directory segment becomes a namespace segment that shadows a name the tests use (C#: `Encoding` hides `System.Text.Encoding`); the same check applies to the UTS infra's tier namespaces (2.2). Generated names can also collide with native test classes in another target (ably-cocoa `ObjectsPoolTests`); detect this, or always run generated suites with a target-qualified filter (`UTS.`). Runners that import test files by basename (pytest's default import mode) also collide when two tiers or modules produce the same file name (`rest/integration/auth.md` and `realtime/integration/auth.md` → `test_auth.py`): make the target directories packages (`__init__.py`) or use `--import-mode=importlib`, and check file-name uniqueness across the whole mapping | +| `--create` | Validate the existing mapping before writing, so a corrupt file is never written back. Add or replace the named module's entry, and preserve every other entry and the replaced entry's `notes`. Write UTF-8 with LF line endings explicitly. On a bad name, ask again | +| Notes | Resolve `notes` against the skill directory. Return an error, not a silent `null`, when the declared notes file is missing | + +### 8.2 Audit (`audit_translation.py`) (MUST) + +| Check | Why | +|---|---| +| **Test-ID coverage**: spec `**Test ID**` set vs test-file `UTS:` set → `missing`, `orphan`, `duplicate`; a distinct non-zero exit when any is non-empty | A missing ID is a missing test; an orphan is a stale, renamed or invented tag; a duplicate tag hides a method from the ledger | +| **Per-test ledger**: every spec code line, grouped by section, tagged `assert` / `await` / `step` | Setup, operations and assertions are all enumerated | +| **Count assertions and waits separately** | Summed counts let surplus waits mask dropped assertions (ably-cocoa showed this by deleting RTN16f's two assertions) | +| **Ignore commented-out assertions** on the test side | A commented `// assertEquals(…)` isn't an assertion | +| **Never crash** | Always emit one parseable JSON object. Both references use exit 0 for clean, 2 for ID problems and 64 for "couldn't run"; any distinct codes will do | +| **Report what it can't see** | A spec with no `**Test ID**` lines (fixture-driven specs) must be reported as "not verifiable", not as zero tests | + +**Parsing contract.** Implement exactly this, or say where yours differs: + +- **Spec side.** A test starts at each `` **Test ID**: `` `` line (the backticks are part of the marker) and ends at the next one. It SHOULD also end at the next heading at or above the test's own level, so trailing notes or appendix sections aren't attributed to the last test. +- **Headings.** Section headings are any `#`–`####` line outside a fence; inside a fence, `#` starts a pseudocode comment. Combined headings such as `### Test Steps and Assertions` are sections too. +- **Fences.** Read ` ```pseudo ` fences. You MAY also read untagged fences (four in spec files today, all in file preambles before the first test, plus one each in `objects/PLAN.md` and the `standard_test_pool.md` helper, which aren't translated; so this changes nothing yet; the Swift audit reads `pseudo` only). Skip ` ```json ` and other payload fixtures. +- **Classification.** Skip blank and comment lines. Tag `ASSERT` / `ASSERT_*` as `assert`, and also any line containing, outside a trailing comment, `FAILS WITH`, `THROWS`, `EXPECT THROW` or `AWAIT_ERROR`, whether or not it is awaited: the test side counts their native rendering (an expected-failure assertion) as an assertion call, so the spec side must too, or a dropped failure assertion goes unreported and the surplus can mask another dropped `ASSERT`. These failure forms take precedence over the await keywords. Tag `AWAIT` / `AWAIT_STATE` / `AWAIT_ALL` / `AWAIT UNTIL` and the poll forms `poll_until` / `poll_until_success` / `POLL_UNTIL` / `WAIT_FOR` as `await` (whether or not the line starts with `AWAIT`), so spec polls and the test's poll helpers are counted alike; everything else as `step`. Match the longest keyword first, on both sides (`poll_until_success` before `poll_until`; `pollUntil` before `poll`). +- **Test side.** A test's block runs from its `UTS:` tag to the next tag (or end of file). Helpers placed after the last test are attributed to it, and assertions inside shared helpers are invisible to their callers: keep shared helpers in a separate file or before the first test, and annotate call sites that hide spec assertions. + +Only the test-side regexes change per language (tag marker, assertion calls, wait calls, comment syntax); the spec side is the same for every SDK. + +**Output shape (SHOULD).** The guide fixes the checks, not the field names; a shape that covers them: `{ok, spec, test, idCoverage: {specCount, testCount, missing, orphan, duplicate}, perTest: [{id, title, sections, specAsserts, specAwaits, testAssertCount, testWaitCount, assertionShortfall, awaitShortfall}], summary: {testsWithAssertionShortfall, testsWithAwaitShortfall}, notVerifiable?}`, or `{ok: false, error}`. + +SHOULD, beyond the existing audits: + +- **Report the await shortfall separately, as a soft signal** ([8.3](#83-review-checklist-must-after-the-audit)). +- **Condition matching:** compare each `AWAIT` / `poll_until` predicate and timeout with the test's, at least as a side-by-side listing. A count can't catch a changed predicate. +- **Verbatim comment check:** flag spec comments that don't appear verbatim in the test. +- **Stale or renamed IDs:** pair each missing ID with an orphan that shares the descriptive name ([9.3](#93-renamed-merged-and-added-ids)). +- **Structural validation of the spec:** every test heading has an ID or is a recognised grouper or sub-case; the ID category matches the path; `## Test Type` is present; helper references point at files that exist. The spec repo has no automated UTS validation, so the skill is the only check. + +**Test the tooling itself (SHOULD).** Mutation-test the audit: delete an assertion, or duplicate a tag, in a known-good test and confirm the audit flags it. Run it over the whole spec corpus after every script change and confirm zero crashes and no unexpected ledger changes. ably-cocoa did both for each audit change. + +### 8.3 Review checklist (MUST, after the audit) + +The review is static, so it runs in both modes. + +- **Coverage:** `missing`, `orphan` and `duplicate` are empty, or each is explained. These, and a positive assertion shortfall, are the hard checks. +- **Line by line:** every positive assertion shortfall is accounted for by an annotated omission. A negative shortfall (more native assertions than spec `assert` lines, e.g. from type or number normalisation) is fine. A multi-line spec construct (a mock definition, a `ClientOptions(...)` block) appears as several `step` lines; reconcile them as one group. +- **Await shortfall is a soft signal.** The wait counter sees only the UTS infra's wait helpers. Continuation-bridged helpers, native `await` on an async SDK API, and fixture builders legitimately replace them, so a positive await shortfall is expected on natively-async modules. Account for each spec `AWAIT` rather than treating the count as a gate. +- **Setup fidelity:** client options, mock responses, timer use and channel-operation order match the spec. +- **Wait fidelity:** every wait condition and timeout matches, or carries a `NOTE`/`DEVIATION`. +- **Deviation honesty** (evaluate mode): every gated or adapted test has a `deviations.md` entry, and no infra stand-in is recorded as a deviation. + +--- + +## 9. Keeping in sync with spec changes + +UTS has no version number or changelog, and **Test IDs are not stable**: they are renamed when spec points are replaced, and spec and UTS changes land in the same commit. A translation that was correct last month can be stale today. + +### 9.1 Record what you translated from (MUST) + +Stamp each generated file's header with the spec-repo commit SHA it was derived from: + +``` +// Derived from uts/realtime/unit/connection/connection_recovery_test.md +// at ably/specification@ +``` + +A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. If the SHA isn't on `origin/main` (a branch run), list it in the final report: it may stop resolving once that branch is deleted. + +**A SHA identifies only committed content.** If a translated spec file differs from the recorded SHA (`git status --porcelain -- ` is non-empty: staged, unstaged or untracked), stop and ask. If the user proceeds, add `(locally modified; blob >)` to that file's header, list the file in the final report, and have re-sync compare the file with that blob rather than with the SHA. Local changes elsewhere in the clone (for example untracked notes outside the tier directories) only need recording, as 9.4 says. + +**Record the model.** The final report records the model (name and ID) that generated each file ([section 11](#11-final-report-format)). A `// Generated with ` header line MAY be added too, but prefer the report: the line changes, adding diff noise, whenever a different model regenerates the file. + +### 9.2 A re-sync mode (SHOULD) + +Add a mode (for example `/uts-to- --resync` in Claude Code, `$uts-to- --resync` in Codex) that: + +1. Runs the audit over every mapped spec in the module, not only the ones selected. +2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as new; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. If a recorded SHA isn't in the clone, fetch it (with approval, and only in the shared clone; the user fetches it into any other clone); otherwise review each spec stamped with it in full, as changed (base unknown). Then re-scan the corpus for constructs the skill can't map (section 6). +3. Classifies each spec as new, changed (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), unchanged or removed. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](../skills/uts-to-lang-skill-creator/references/upgrade-diff-driven.md#10-upgradefix-diff-driven)). +4. Regenerates the affected tests, preserving only DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. +5. Updates the SHA in each regenerated file's header, and its blob note (9.1): re-stamped if the spec is still locally modified, removed if not (and in the manifest, if kept). +6. Reports the classification ([section 11](#11-final-report-format)). + +Regenerate the whole file when most tests changed; otherwise regenerate only the affected tests. Say which in the report. A re-sync is also needed when the shared UTS infra or module helpers change shape (ably-cocoa reset and regenerated its objects ports after aligning them with `standard_test_pool.md`). After any infra change, re-run every affected tier's smoke tests and self-tests ([2.7](#27-infra-smoke-tests-and-self-tests-must)) first, then regenerate the tests whose rendering depends on the changed helper. + +### 9.3 Renamed, merged and added IDs + +Treat a removed ID plus an added ID as a **possible rename**, keyed on the descriptive name and position, and re-check the assertions: a renamed point can invert its semantics (`RTL15b1/serial-cleared-suspended-1` became `RTL15b2/serial-retained-suspended-1`). + +Examples of the pattern: + +- **Rename not followed.** Both SDKs carried `realtime/unit/RTN16g2/recovery-key-null-inactive-0` after the spec renamed it to `realtime/unit/RTN16g3/recovery-key-null-inactive-0`: ably-cocoa `Test/UTS/unit/realtime/ConnectionRecoveryTests.swift` and ably-java `lib/src/test/kotlin/io/ably/lib/uts/unit/realtime/ConnectionRecoveryTest.kt`. The audit reports one missing and one orphan, but nothing prompted a re-run. +- **Additions not followed.** In ably-java `RealtimeObjectTest.kt`, two `RTO27` tests remained where the spec has a single `RTO27/channel-state-data-lifecycle-0` (the only `RTO27` Test ID the spec has ever had), and `RTO20d4/mixed-null-serials-applies-non-null-0` had no test. + +### 9.4 One pinned local clone, brought up to date first + +Pick one source for docs and specs, and pin it: the local spec clone. By default it is the shared clone that the skill creator also uses (`ABLY_SPEC_CLONE`, else `~/.ably/specification`); a clone root or a module directory given as the argument names its own clone, which is used instead, so an explicit path into any clone keeps working. The sync below runs only when the clone in use is the shared clone, however it was found: a path into any other clone is never synced or fetched, whatever `ABLY_SPEC_SYNC` says, and the skill shows a static warning instead ("Using the spec clone you named (``). It isn't synced; make sure its UTS specs are up to date."); keeping that clone current is the user's responsibility. + +**Bring the shared clone up to date first (SHOULD).** Before reading anything, with the user's approval or their standing `ABLY_SPEC_SYNC=auto` (`off` means no network; the user sets the variable, never the skill), fetch `origin main` and fast-forward `main` with `git merge --ff-only --no-overwrite-ignore`, only if `main` is checked out, has no tracked changes, no merge, rebase or other git operation is in progress, and it is strictly behind `origin/main`. Never pull, reset, stash, rebase or switch branches: on another branch, a dirty tree, or an ahead or diverged `main`, report it and ask (use it as it is, fetch only, or stop). Offline, declined or failed: use the clone as it is, and say how far behind it was at its last fetch. A skill SHOULD do this with a copy of the skill creator's `scripts/ensure_spec_clone.py` ([procedure 13.9](../skills/uts-to-lang-skill-creator/references/orient.md#139-sync-the-spec-clone)). The fast-forward happens before the run, so the run is still pinned: check HEAD again before stamping headers and before the final report, and if it moved, stop and ask. + +**With no argument** ([3.3](#33-frontmatter-and-arguments)), the skill uses the shared clone and says so in one line ("No spec path given: using the shared spec clone at ``."). If the clone doesn't exist, it offers to clone it: one question, even with `ABLY_SPEC_SYNC=auto`, because the first clone needs the network and about 100 MB. If it exists, it syncs it as above. Then it says what it reads: "latest `main` @ ``" only after a successful sync (fast-forwarded, or already level with `origin/main`); otherwise the clone's actual state (another branch, a dirty tree, ahead or diverged, or not synced, with how far behind it was at its last fetch). Then it asks which module ([section 4](#4-the-workflow-the-skill-must-implement)). A clone root as the argument that isn't the shared clone is never synced: it gets the static warning above, then the same module question. The shared clone's root given as the argument is synced as above. + +Then read everything from that clone, at the pinned SHA: + +- the UTS docs (`/uts/docs/writing-derived-tests.md`, `proxy.md`, and for the integration and proxy tiers the "Timeout Strategy", "Avoiding Flaky Tests" and both "Protocol Variants" sections of `integration-testing.md`); +- in evaluate mode, the features specs the decision tree needs (`/specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. + +Record the clone's HEAD SHA, its branch, whether it is dirty and which files are, and the sync's outcome (or "not synced"); a translated spec that is locally modified is handled as in 9.1. + +- *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. +- Fetching docs or specs file by file (for example `gh api …/contents/…` or raw URLs) mixes revisions and records none: don't. + +### 9.5 Upgrading an existing skill (SHOULD) + +Spec changes aren't the only drift. A skill written by hand, by an earlier version of the procedure, or before a change to this guide, can drift from the guide itself. Before you change such a skill: + +1. **Audit** the skill, its UTS infra and its UTS-derived tests against [section 13](#13-checklist-for-a-new-uts-to-lang-skill), and list each item that is missing or doesn't conform, with the evidence. +2. **Let the owner choose** for each item: add it, skip it, or defer it. A skipped or deferred MUST item leaves the skill not conforming, and the final report says so. The infra smoke tests and self-tests (2.7) of every tier the skill keeps offering are the exception: a tier without them is refused until they exist. +3. **Upgrade in place.** Keep what conforms: rules, names, notes, hand-maintained mapping entries and infra pieces. Close the chosen items one at a time, and compare against the results from before the change (resolver output, audit results, infra tests, and compile status): a difference no chosen item explains is a regression. The existing UTS-derived tests are the skill's output, not a constraint on it: read them as evidence (`deviations.md` entries, DEVIATION gates, adapted assertions, and failing or hand-patched tests show where the skill or UTS infra fell short) and carry those lessons into the skill. A derived test that a change makes stale is a candidate for regeneration, not a regression. +4. **Regenerate the skill from scratch** only if the owner chooses to. Change existing UTS-derived tests, stale ones included, only by regenerating them through the skill (re-sync, 9.2) where the owner chooses, never by hand-editing them. + +The skill creator's [Upgrade/Fix, full gap audit](../skills/uts-to-lang-skill-creator/references/upgrade-existing-skill.md#11-upgradefix-full-gap-audit) does this. The skill creator starts every run by detecting the repo's skills, their origin, UTS infra and UTS-tagged tests, and recommends Create or Upgrade/Fix. A repo's own skill and UTS infra are its inputs, not [reference implementations](#reference-implementations-last-resort), even in ably-cocoa and ably-java. + +--- + +## 10. Verification and CI + +Escalate in this order: + +1. **Compile the tests** for the module (`swift build --build-tests`; `./gradlew ::compileTestKotlin`; `dotnet build`). For every runner, check that every tagged test is collected (or executed): the number of distinct collected test functions, counting a parameterised or protocol-variant test once, equals the number of `UTS:` tags: a misnamed pytest class, or a JVM test annotated for an engine the task doesn't run (JUnit 4 vs JUnit Platform), is silently skipped. For pytest use `pytest --collect-only` (plus `mypy` if the repo uses it); for .NET, `dotnet test --list-tests`; for Gradle, compare the task's test-report count. In dynamic languages a missing SDK API surfaces only at run time: run a type checker (`mypy`/`pyright`) over the generated tests where the SDK ships type hints, or report missing APIs as unverified in translate-only mode. MUST, in both modes. +2. **Lint and format** with the repo's gates (EditorConfig, checkstyle, eslint, ruff…). MUST. +3. **Compile as strictly as CI does.** If CI treats warnings as errors, so must the skill. (ably-cocoa's LiveObjects CI failed on a redundant typed-throws cast the local build accepted.) In TypeScript, the test runner may strip types without checking them (`writing-derived-tests.md` "Build pipeline and CI checks"). +4. **Run the generated classes by filter** (evaluate mode): per class, then per tier. Use target-qualified filters where names can collide. +5. **Run the full tier** only when asked; integration tiers are slow and hit the sandbox. +6. **CI:** each suite has exactly one CI home ([2.5](#25-build-and-ci-wiring-must)). Don't gate integration tests behind an env var in the generated code; select them by filter or task. +7. **Commits (SHOULD).** By default the skill doesn't commit: it lists the files it changed (tests, `deviations.md`, mapping, notes) for the user to review. It also lists any project files the build doesn't discover automatically, which the user must update by hand (for example an Xcode project). It doesn't suggest committing tests whose header carries a spec SHA that isn't on any remote (9.1): re-sync them from a pushed SHA first. + +The skill's scripts MUST pin UTF-8 and LF when they write tracked files. + +--- + +## 11. Final report format + +The skill MUST end each run with a report like this: + +``` +UTS translation: / @ ably/specification@ (clone clean|dirty: ) (mode: translate-only | evaluate | resync) +Model: ; sub-agents: + +| Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | +|---|---|---|---|---|---|---|---| +| connection_recovery_test.md | ConnectionRecoveryTests. | 6/6 | clean | 1/1 | ok | 5 pass, 1 gated | Failing Tests: RTN16f | + +Skipped specs (and why): … +Locally modified specs translated (9.1): — blob +Re-sync classification (9.2): new …; changed () …; unchanged …; removed …; changed but not regenerated: — +Not-applicable omissions (7.4): — +Missing APIs found in translate-only mode: — +Mock-capability gaps found in translate-only mode: — +Infra stand-ins disclosed in file headers: … +Constructs the skill couldn't map (stopped and asked): … +Infra preflight (step F): compile ok | smoke n/n | self-tests n/n | known-gap check ok (or the stop reason) +Infra changes made during this run, and the self-tests added or extended: … +Suspected UTS spec errors or gaps (need an upstream fix): — , draft fix: … +Mapping or notes changes: … +Reference test proposed (step 3): → , recorded if the user agrees | none (the tier has one, or no file passed review) +Changed files for review: … +Next steps: … +``` + +--- + +## 12. Lessons learned + +From the git histories of both existing skills. Lessons that became rules live in sections 2–10; these are the rest. + +| Lesson | Where it came from | +|---|---| +| **Lessons must reach `SKILL.md`**, not only UTS infra comments or commit messages | ably-java: several flake fixes stayed in KDoc | +| **Don't cite line numbers in skill text**; they drift. Cite symbols or headings | ably-cocoa | +| **Compile the examples in the skill**, or take them from real, compiling files in your own repo | ably-cocoa: an example helper doesn't compile | +| **Keep generic rules separate from per-language renderings**: write every rule language-neutrally first, then give its rendering for your SDK | Both: the Swift skill began as a port of the Kotlin one, and the generic parts had to be separated afterwards | + +--- + +## 13. Checklist for a new `uts-to-` skill + +The skill and its UTS infra are one deliverable: the skill isn't done until every MUST item below, infra items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. A MUST item that can't be met yet (for example, the repo owner declines the CI change that would run the infra tests) is also marked ✗ with the reason, and the skill doesn't conform to this guide until it is met. An item, or part of one, that needs a capability the SDK lacks (section 2, "Scope follows the SDK's capabilities") is marked n/a with that reason; n/a is not ✗. + +Each mark SHOULD carry its evidence (a command and its result, a file and line, or a record), and someone other than the skill's author SHOULD verify the filled checklist against that evidence; the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#9-phase-7-final-report-and-handover) does both. An item whose behaviour only a translation can show (for example verbatim comments, sub-case policies or `deviations.md` placement) MAY be marked "✓ static; behaviour at first translation": the rule is in the skill, checked without generating a test, and the first real translation run shows the behaviour. + +**UTS infra** + +- [ ] SDK test hooks: WebSocket factory, HTTP client, clock (covering blocking waits and the async runtime's timers), reachability/network monitor, randomness; injected at construction; existing seams first; each new hook minimal, approved and needed by a spec +- [ ] Shared test library: `mock_http.md` (+ legacy `queue_*` and `mock_http.captured_requests`), `mock_websocket.md` (async close, CONNECTED ordering, raw frames, alternative API, undocumented members), `mock_vcdiff.md`, `standard_test_pool.md`, `MockNetworkListener` +- [ ] Wait helpers: race-free event-latched `AWAIT_STATE` failing fast on FAILED, value-returning `poll_until`, `poll_until_success`, `process_pending_events()`, wall-clock timeout wrapper, deadlines safe under fake time +- [ ] Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures +- [ ] UTS infra designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; UTS infra organised by tier (MUST; names SHOULD): proxy under integration, placement by use, names following folders; fixture-helper scope documented +- [ ] Infra README with a Known gaps section (MUST) +- [ ] Infra smoke tests per tier and helper self-tests (2.7): permanent, in CI, ungated, untagged, in a tree by tier, outside generated-test directories (MUST) +- [ ] `SandboxApp` (retries idempotent reads only); `ably-common` submodule +- [ ] `ProxyManager` with a pinned `uts-proxy` version for every developer OS, verified download, health check and port handling; `ProxySession`; string `match.action`; auth through the proxy decided and commented +- [ ] One CI home per suite (MUST); per-tier jobs (SHOULD) +- [ ] Concurrency, time, runner-parallelism and internal-visibility models documented + +**Skill files** + +- [ ] `SKILL.md` frontmatter portable across Claude Code and Codex: `name` equal to the directory; a pushy third-person description of at most 1,024 characters with no `<` or `>` and a "Not for" boundary; string-valued `metadata`; Claude Code-only fields only if accepted as non-portable ([3.3](#33-frontmatter-and-arguments)) +- [ ] `SKILL.md`: a usage guard that doesn't rely on `$ARGUMENTS` and never guesses a module (an optional argument; a module question when none is named); the skill's own files referred to relative to the skill directory, in `SKILL.md` and in the scripts; placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline, including the Infra reference +- [ ] Installed for both tools from one source: `.claude/skills/` and `.agents/skills/`, by in-repo symlink or a CI-checked copy ([3.1](#31-layout)) (SHOULD) +- [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked, `infra` entry (`root`, `readme`, `testsRoot`, `testCommand`); nothing the resolver can derive +- [ ] `resolve_uts.py`: validation (including a clone root) and a module listing, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, mirrored sub-directories, collision detection, validate-then-write `--create` preserving entries, `infra` output derived from the mapping, older mapping shapes read with a warning +- [ ] `audit_translation.py`: the parsing contract, ID coverage, duplicates, separate assert/await counts, ignores commented assertions, never crashes, flags unverifiable specs; itself mutation-tested +- [ ] Per-module notes for every module whose API diverges from the pseudocode; `objects` decided from evidence of the SDK's LiveObjects public API: full notes if it exposes the API, even if incomplete; a placeholder, or an entry marked not ready as a whole, otherwise ([3.1](#31-layout)) +- [ ] Scripts run on every developer platform (path handling, UTF-8, LF) + +**Workflow** + +- [ ] Required reading, including features specs, from the same clone as the specs, the shared clone synced first per [9.4](#94-one-pinned-local-clone-brought-up-to-date-first) (SHOULD) +- [ ] Steps 0 and A–F: the four user questions (mapping, tier, specs, translate vs evaluate) and the infra preflight, which stops on red +- [ ] Steps 1–7 with batching; a reference test per tier; review in both modes +- [ ] Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) +- [ ] Re-sync mode; fix the cause, then regenerate +- [ ] Mid-run infra changes re-run the smoke tests and self-tests and are reported separately + +**Translation rules in `SKILL.md`** + +- [ ] One `UTS: ` per test, verbatim; no invented tags; spec point in a collectable test name +- [ ] File header with source spec, tier, spec SHA, disclosures and (integration/proxy) the corresponding unit spec +- [ ] Verbatim comments, spec variable names, section headings, spec order +- [ ] Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome +- [ ] Policies for sub-cases, groupers, test-case tables, delegating tests and ID-less tests +- [ ] Protocol variants parameterised; tag singular; proxy JSON only +- [ ] Clients closed in teardown; mocks restored; stop on a failed wait +- [ ] Unique channel names; own proxy session per test +- [ ] Unit: fake time and flushes, no real timers. Integration: wall-clock everything +- [ ] Record-and-verify for transient states; polls return the settled value +- [ ] No accommodate-both assertions; `IS ` per typing discipline +- [ ] Public-API specs stay public; internal-access ladder with an escalation stop +- [ ] Construct table ([section 6](#6-pseudocode-construct-catalogue)) filled in for every row + +**Evaluation** + +- [ ] Three end states with the language's runtime skip and fail-fast idioms; `RUN_DEVIATIONS` reproduction command +- [ ] `deviations.md` location per owning module; format deferred to `writing-derived-tests.md`; written at the right time +- [ ] Infra stand-ins disclosed in comments, not in `deviations.md` +- [ ] Language-inapplicable inputs (cases a–c) noted in the test file +- [ ] Stopping rules and spec-error escalation + +**Verification** + +- [ ] Compile (or collect), lint, CI-strict compile, filtered run +- [ ] No commits by default; changed files listed +- [ ] Final report format +- [ ] The skill's own examples compile + +**Creation process** + +- [ ] Skill and UTS infra created on an Opus-class model (MUST); the skill states the model for its runs, or pins it where the tool supports that (SHOULD), and records it in its final report ([Model tier](#model-tier)) +- [ ] Reads of the [reference implementations](#reference-implementations-last-resort), if any, made only as a last resort, recorded, and reported as guide gaps +- [ ] An existing skill, its UTS infra and its UTS-derived tests audited against this checklist before being changed; gaps listed for the owner to choose; working parts preserved ([9.5](#95-upgrading-an-existing-skill-should)) (SHOULD) + +--- + +## Appendix: Existing skills (background and patterns to avoid) + +The existing skills, `uts-to-swift` and `uts-to-kotlin`, are named so that the examples and lessons above can be traced, and as a last-resort reference for creating a skill and its UTS infra ([Reference implementations (last resort)](#reference-implementations-last-resort)). Ably is renaming its SDK repositories from `ably-` to `ably-pubsub-` (`ably-java` is now [`ably/ably-pubsub-java`](https://github.com/ably/ably-pubsub-java)); this guide keeps the short names, and old clones' remotes still work through GitHub's redirect. + +| Skill | Repo and paths | Lessons this guide drew from it | +|---|---|---| +| `uts-to-swift` | ably-cocoa, `.claude/skills/uts-to-swift/`; tests under `Test/UTS/` | Strict compile-time concurrency checking (Swift 6); a callback-based core plus a natively async plugin; scoped-resource teardown without async `tearDown`; separate assertion and wait counts in the audit; "an infra stand-in is not a deviation"; the internal-access ladder; module notes overriding the generic flow | +| `uts-to-kotlin` | ably-java, `.claude/skills/uts-to-kotlin/`; tests in the `lib/` (built by the `:java` Gradle module) and `liveobjects/` (`:liveobjects`) test source sets, UTS infra in the `:uts` Gradle module (path `uts/`) | Deriving a package and build module from the target path; callback- and await-style mocks; race-free `awaitState`; virtual-time pitfalls and real-clock timeouts; idempotent-only sandbox retries; the `S-n` shape-deviation vocabulary | + +Both contain a `SKILL.md`, a mapping file, a resolver and an audit script, and an objects module notes file. + +### Reference implementations (last resort) + +This guide and the UTS docs are meant to be complete. If they don't answer a question while you create or maintain a skill and its UTS infra, you MAY consult the existing skills and their UTS infra, read-only: + +| Repo | Skill | UTS infra | Infra smoke tests | Generated tests | +|---|---|---|---|---| +| [ably-java](https://github.com/ably/ably-pubsub-java) | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-pubsub-java/tree/main/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-pubsub-java/tree/main/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | +| [ably-cocoa](https://github.com/ably/ably-cocoa) | [`.claude/skills/uts-to-swift`](https://github.com/ably/ably-cocoa/tree/main/.claude/skills/uts-to-swift) | The `UTS` test target, [`Test/UTS`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS) (helpers in `Test/UTS/infra`; README `Test/UTS/README.md`), plus the shared test-support targets it depends on: [`Test/AblyTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyTesting) and [`Test/AblyLiveObjectsTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyLiveObjectsTesting) | None: they were retired, a [pattern to avoid](#patterns-to-avoid) | [`Test/UTS/unit`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/unit), [`Test/UTS/integration`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/integration) | + +Rules: + +1. **The authority doesn't change.** Consult the references only after this guide, the UTS docs and the helper specs fail to answer the question. On any conflict, this guide (and, for semantics, the UTS docs, per [How it relates to the other UTS docs](#how-it-relates-to-the-other-uts-docs)) wins; record the conflict. +2. **Learn the pattern; never copy.** Don't copy, port or translate code, scripts, prose, tables or examples verbatim. Take how a problem was structured or solved, and write your own version in your language's idioms, on your SDK's hooks and UTS infra. +3. **Check against [Patterns to avoid](#patterns-to-avoid)** before adopting anything. Both skills and their UTS infra have documented defects, often in exactly the files you would consult. +4. **Record each consultation**: the repo, revision and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. +5. **Creation only.** This applies to creating or maintaining a skill and its UTS infra. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). + +Reading GitHub is network access: the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#reference-implementations-last-resort) gates it, or offers local clones instead. + +### Patterns to avoid + +Observed in the existing skills and their UTS infra when this guide was written, as is every statement about them in this guide; some may since have been fixed. Each is a pattern a new skill shouldn't copy ([Reference implementations](#reference-implementations-last-resort), rule 3). + +**In both skills** + +- **Fetching the manual from `main`, and telling the model to "fetch" the features spec with no pinned source,** while reading specs from a local clone ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). +- **Two sources of tier truth**: path-based in the resolver, content-based in the `SKILL.md` integration section. +- **A `CONTAINS_IN_ORDER` mapping that isn't a subsequence check** ([6.7](#67-notes-on-the-catalogue)). +- **No re-sync mode, no final report, no lint step, no collision detection**, and a silent `null` when a declared notes file is missing. +- **Claude Code-only packaging** ([3.3](#33-frontmatter-and-arguments)): script paths hard-coded as `python3 .claude/skills//scripts/…` and reliance on `$ARGUMENTS`, both of which break under Codex; angle-bracket placeholders (`<…>`) in `description`, which skill packaging validators reject. + +**In other skills** + +- **Fetching docs and specs file by file from the default branch** (`gh api …/contents/…`, raw URLs), unpinned: a run can mix revisions, the headers can record none, and it needs `gh` auth and a POSIX `base64` ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). + +**In the UTS infra** + +- **Smoke tests env-gated and never run in CI, then retired** (ably-cocoa) once spec-derived tests covered their scenarios ([2.7](#27-infra-smoke-tests-and-self-tests-must)). +- **Scenario-only smoke tests** (ably-java) that stay green while a helper breaks its helper-spec contract. +- **A mock `close()` that notifies synchronously** (ably-java), although `mock_websocket.md` requires `onClose` to be called asynchronously. +- **A declared mock event type that is never emitted** (ably-java), so an assertion that filters the event log for it passes on an empty list. + +**Seen in `uts-to-swift`** + +- **Example code that doesn't compile**: the callback-bridging helper in "Assertions (Swift Testing)" has a `guard` body that doesn't exit and doesn't resume the continuation. (The real helper in `TimeTests.swift` is correct.) +- **Describing features that don't exist**: a per-tier class-name suffix the resolver doesn't implement, and a suite that doesn't exist. +- **Contradictory timeout guidance**: "pass an explicit `timeout:` only when the spec states one" alongside "generous timeouts (10–30s)" examples. +- **Pointing to rules in the wrong doc**: it says the value-assigned `poll_until` rule lives in `writing-derived-tests.md`; the reference definition is in `writing-test-specs.md`. +- **Deviations-file rules that contradict the manual**: it says to add the *UTS Spec Errors* heading "on first use if absent"; the manual requires all four headings always present. +- **A UTS infra narrower than the helper spec**: the mock WebSocket lacks the await style, `onMessageFromClient` and raw-frame handlers, ping frames, a PING template, and WS-level `respond_with_timeout` / `respond_with_dns_error`. The infra README documents the first two, the skill documents only the last, and ping is documented nowhere. +- **A sample-based state wait** at the unit tier (`awaitConnectionState`). +- **Test names without the spec point**: the objects module notes override the naming rule so that objects unit tests have bare descriptive names, relying on the `// UTS:` tag alone; `writing-derived-tests.md` requires the spec point in the name. +- **Retrying a non-idempotent provisioning request**: `SandboxApp.create()` retries `POST /apps`. `integration-testing.md` says sandbox apps auto-expire, which justifies best-effort deletion, not retrying creation: a lost response still leaves duplicate apps until they expire. Retry only idempotent requests (2.3). +- **Writing `deviations.md` before evaluation**: it records Mock Infrastructure Limitations at generation time, including in translate-only mode; the manual creates the file during evaluation (7.3). +- **Citing the wrong spec point**: it attributes the fallback-disabling behaviour of explicit `realtimeHost`/`restHost` to REC2c2; it is REC2c6 (REC2c2 covers an `endpoint` hostname). +- **Citing spec points where they don't exist**: the objects module notes place `RTTS1–RTTS10` in `objects-features.md`, which doesn't contain them; they are proposed spec points not yet merged. +- **No rules for** `process_pending_events()`, `poll_until_success`, `CLOSE_CLIENT`, `WAIT` or `BEFORE ALL`; stale statements in the objects module notes and in `Test/UTS/README.md`, which the skill tells the model to read first. + +**Seen in `uts-to-kotlin`** + +- **An audit weaker than its sibling**: it counts `awaitState` / `awaitChannelState` / `pollUntil` as assertions (so surplus waits can mask dropped assertions), counts commented-out assertions, silently collapses duplicate `@UTS` tags, and reads every fenced block rather than only pseudocode. Build the audit from the 8.2 contract instead. +- **A non-documented tag marker**: `/** @UTS */` instead of `// UTS: `. +- **A resolver that can crash**: a malformed `uts-package-mapping.json` raises a Python traceback instead of a structured `ok: false` error. +- **Cleanup at the end of the body**: the unit template closes the client there, not in `finally` or `@AfterEach`. +- **Hard-won rules left out of `SKILL.md`**: `withRealTimeout` and record-and-verify exist in the UTS infra but not in the skill, so some integration tests still await network futures unbounded. +- **Ambiguous or stale references**: a "Proxy integration tests" section that doesn't exist; "`uts/README.md`" meaning either repo's README. +- **Invented tags and silent predicate changes in generated tests**: one proxy and one direct-sandbox test added extra `@UTS` tags for secondary spec points, and one `pollUntil` predicate differs from the spec's with no comment. diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md new file mode 100644 index 000000000..031fde5ba --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -0,0 +1,283 @@ +--- +name: uts-to-lang-skill-creator +description: "Builds or upgrades a uts-to-LANG translator skill, with the UTS infra it runs on, for one Ably Pub/Sub SDK repository on this skill's whitelist (C#, Python, Go, JavaScript, Ruby, ...). Use it whenever asked to create a uts-to-* skill; to audit, fix, upgrade or refresh one after the guide, UTS docs, corpus or SDK changed; to add LiveObjects (uts/objects) support; or to set up UTS test infrastructure (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, infra smoke tests and self-tests). First checks the repo is on the whitelist, then detects what it has (uts-to-* skills and their origin, UTS infra, UTS-tagged tests, REST and Realtime capabilities and scope, LiveObjects API) and recommends Create or Upgrade/Fix, then runs a procedure with user stop points. Needs a local (synced) ably/specification clone. Not for Chat, Spaces, AI Transport, CLI or other non-Pub/Sub repos, for translating specs once a uts-to-* skill exists (use that skill), or for writing UTS specs." +license: Apache-2.0 +compatibility: "Claude Code or Codex, on an Opus-class model. Needs shell and file access, git, Python 3.8+, a local clone of ably/specification (cloned or updated on approval), and the target SDK's build toolchain." +allowed-tools: Read Grep Glob +metadata: + team: "engineering" + version: "1.0.0" + tags: "testing, uts, translator-skill, skill-creator, uts-infra, ably-specification" + marketplace: "false" + short-description: "Detect, then build or upgrade a uts-to-LANG skill and its UTS infra" +--- + +# UTS-to-Lang Skill Creator + +**After a context compaction:** this file may be in your context only in part (Claude Code keeps only its start; Codex may keep none of it). Before your next action, read this whole `SKILL.md` again, then the Design record's Run status if the records exist ([13.7](references/orient.md#137-provenance-d-30)), then the reference of the phase you are in. Never act at a stop point from memory. + +Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create or upgrade a `uts-to-` translator skill, and the UTS test infrastructure (the **UTS infra**) it runs on, for one Ably Pub/Sub SDK repository, starting with [Step 0: Orient](#step-0-orient). [Writing UTS Spec Translator Skills](../../docs/writing-uts-spec-translator-skills.md) (hereafter **the guide**) says what a good translator skill is and why; this skill says how to build one: in what order, with what checks, and where to stop and ask the user. Where the guide holds the reason or the detailed requirement, this skill links to it instead of repeating it; read the linked section. + +The UTS infra and the skill are one deliverable (guide [section 2](../../docs/writing-uts-spec-translator-skills.md#2-the-uts-infrastructure)): the skill isn't complete until the UTS infra's per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-infra-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier it supports. The UTS infra reuses, wraps, extends or builds on the repo's existing tests ([4.2](references/phase-2-design-infra.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)), so the phases run in a fixed order. + +"This skill" is this `SKILL.md` with its `references/` and `assets/templates/`. Section numbers are stable across the files: section n+2 holds Phase n and Orient is section 13; section 2 comes before section 1 so that it survives context compaction. MUST, SHOULD and MAY cite the guide's requirement levels; this skill's own instructions are imperatives. + +## Path convention + +- **Files of this skill** (`references/`, `assets/templates/`, `scripts/`) are relative to ``, the directory that contains this `SKILL.md`, wherever it is installed. Run scripts as `python3 /scripts/.py`. +- **Files of the spec repo** (the guide, the UTS docs, the helper specs, the corpus) are linked relative to this skill's place in a spec clone, `uts/skills/uts-to-lang-skill-creator/`, so the links work on GitHub. A link that climbs out of the skill directory (`../../…` from `SKILL.md`, `../../../…` from `references/`) is a spec-repo path; every other link is a file of this skill. At run time, read a spec-repo path in the clone Orient pinned ([2.6](#26-pin-the-spec-clone)), never against the directory you loaded this skill from and never from GitHub: `…/docs/writing-uts-spec-translator-skills.md#31-layout` means section 3.1 of `/uts/docs/writing-uts-spec-translator-skills.md`. +- **Inputs** come from the user's words: the text after `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex), or their message. Don't rely on argument substitution. + +## Step 0: Orient + +Do this first, in every run, before any phase. `orient.py` is read-only: no build, no test, no network, no file written. Run `python3 /scripts/orient.py [--spec-clone ]` ([13.1](references/orient.md#131-run-it)), then settle its stops in this order; [references/orient.md](references/orient.md) (section 13) has the rules: + +1. **STOP-16**, if the repo isn't plainly eligible ([13.2](references/orient.md#132-repository-eligibility-stop-16)). On a reject, show the script's message exactly and end the run; there is no override. +2. **The spec-clone sync** ([13.9](references/orient.md#139-sync-the-spec-clone)): Step 0's only network use, for the shared clone only, after STOP-6 (not asked if `ABLY_SPEC_SYNC` is `auto` or `off`); rerun `orient.py` if HEAD moved. +3. **STOP-1**, if needed: the model tier (2.8) and anything in `stop1`, such as the spec clone (2.6). +4. **STOP-13**, always: the State summary and the recommended mode ([13.4](references/orient.md#134-the-state-summary-and-stop-13)). Record D-27, then route per [13.5](references/orient.md#135-routing). +5. **STOP-17**, only while open: the UTS infrastructure plan and side, in a follow-up if it depends on the mode ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)). Record D-31. + +## 2. Ground rules + +### 2.1 Your inputs + +You work from four primary sources: + +1. this skill; +2. the guide; +3. the local spec clone: the UTS docs (`uts/README.md`, `uts/docs/`), the helper specs, the UTS corpus (`uts/rest`, `uts/realtime`, `uts/objects`) and, for API and spec-error questions, the features specs under `specifications/`; +4. the target SDK repo. + +and, only as a last resort, the reference implementations. + +**Reference implementations (last resort):** consult the existing `uts-to-swift` and `uts-to-kotlin` skills and their UTS infra only under [Reference implementations (last resort)](#reference-implementations-last-resort) below. Those rules govern other repos: the target repo's own `uts-to-*` skill, UTS infra and UTS-derived tests are input 4; read them in full and change them in place ([11.1](references/upgrade-existing-skill.md#111-when-this-mode-applies)). + +### 2.2 What you may change + +| Action | Allowed? | +|---|---| +| Create and edit files under the skill directory, its `.agents/skills` link or copy (D-01), and the working-records directory | Yes (an existing skill's files: see below) | +| Create new test-only infra code (shared test library, module helpers, smoke tests, UTS test-project wiring) | Only after the user approves the infra design at [STOP-3](#23-stop-points), and only within the scope they approved | +| Change existing test-support code that the SDK's own tests use (shared fakes, fixtures, base classes) | Only with explicit approval for each change, and never in a way that changes behaviour for the existing tests. Prefer wrapping or extending to editing | +| Change SDK production code, including adding test hooks or raising visibility | **Only a minimal hook or visibility raise, approved per change** ([STOP-4](#23-stop-points)), after existing seams and the UTS infra can't reach it ([4.6](references/phase-2-design-infra.md#46-step-2f-propose-the-sdk-test-hooks)) | +| Change CI workflows, build scripts that CI runs, or dependency manifests (including adding a test dependency) | **Only with explicit approval** ([STOP-5](#23-stop-points)). Otherwise write the change as a suggestion in the Final report | +| Edit project or solution files needed to register new test files or projects (`.sln`, `.csproj`, `.projitems`, an Xcode project) | Only with explicit approval ([STOP-5](#23-stop-points)), recorded in the Design record (D-20). If the user declines, give them the exact entries to add, wait until they confirm, and list them in the Final report | +| Stage, commit, push, branch, or open a PR | **No, unless the user asks** | +| Edit anything in the spec clone | No, except the shared clone's sync (13.9). Report spec errors and doc gaps in the Final report with a draft fix | +| Change, fix or delete existing tests (native or UTS-derived) | No, unless the user asks. Report their state and failures as findings | +| Edit, replace or delete files of an existing `uts-to-*` skill or UTS infra that this run didn't create | Only for the items the user chose at STOP-15 (D-29) or the infra rows approved at STOP-3; the objects files that the STOP-14 decision (D-28) changes; when regenerating, replacements the user approves ([11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)); the provenance stamp in its frontmatter (D-30). Never delete without asking | +| Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | +| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies, reference-implementation reads under [2.1](#21-your-inputs)) | No | + +### 2.3 Stop points + +At a stop point, ask as [asking.md](references/asking.md) says (one question per decision, in plain words), then wait for the answer. Record the question and the answer in the decision log ([design-record template](assets/templates/design-record.md)). Orient's stops are recorded once the records exist; a STOP-16 reject ends the run. + +| ID | Phase | When | What you present | +|---|---|---|---| +| **STOP-1** | 0 (Orient) | A required input is missing, the session's model isn't Opus-class ([2.8](#28-model-tier-and-sub-agents)), or no usable spec clone is found (offer to clone it, 13.9), or `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | +| **STOP-2** | 1 (step 1d) | The Repo profile is written | The profile, for confirmation or correction | +| **STOP-3** | 2 (end) | The infra design is written | The gap table, the design and three options: (a) build the missing UTS infra now, (b) plan it only, (c) scope the skill to the tiers possible today | +| **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](references/phase-2-design-infra.md#46-step-2f-propose-the-sdk-test-hooks)) | +| **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | +| **STOP-6** | 0 onwards | Before any network access (the shared clone's sync, 13.9, unless `ABLY_SPEC_SYNC` is `auto` or `off`; network tests, proxy releases, a reference implementation on GitHub or fetching its clone, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads and the sync, whether the approval covers the whole run | +| **STOP-7** | 3 (end) | The UTS infra is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | +| **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | +| **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | +| **STOP-10** | 3, 6 | A Phase 3 build row hits its attempt bound ([5.2](references/phase-3-build-infra.md#52-step-3b-build-in-order-with-acceptance-checks)), or a Phase 6 check does ([8.4](references/phase-6-validate-skill.md#84-fix-the-cause)) | A summary: the failure, what you tried, your diagnosis, the options, and a draft spec fix where relevant | +| **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | +| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections searched; the references read (repo, revision, path) and what they showed; your proposed reading | +| **STOP-13** | 0 (Orient), every run | Always ([13.4](references/orient.md#134-the-state-summary-and-stop-13)) | The State summary, the state class (S0–S4, re-entry) and the options for it, with the recommended one: Create; Upgrade/Fix (diff-driven or full gap audit); regenerate from scratch; resume or restart; for S4, pick a skill, create a new skill or repair the installs first; stop | +| **STOP-14** | 1 (step 1e), every run | Always, in every mode ([12.2](references/liveobjects-support.md#122-stop-14-recommend-then-ask)) | The `detect_liveobjects.py` evidence (names per category, with files and counts), its recommendation and yours; options: full objects support (translate-only if incomplete), placeholder, none now | +| **STOP-15** | 1 (step 1f), Upgrade/Fix | The gap audit, or the list of detected changes (diff-driven), is written ([11.4](references/upgrade-existing-skill.md#114-step-u3-stop-15-choose-per-item), [section 10](references/upgrade-diff-driven.md#10-upgradefix-diff-driven)) | Every non-conforming item or detected change, with the proposed change and the options add, skip or defer (a skipped or deferred MUST row leaves the skill not conforming); or regenerate from scratch instead | +| **STOP-16** | 0 (Orient), first stop, before the spec-clone sync | `check_repo_eligibility.py` rejects the repo, or can't decide ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | Reject: the script's exact message (not on the whitelist; or, for `NO_CLIENT_DEFINITION`, no client defined in this checkout); the run ends, with no override. Ask: the remotes found and the question (confirm a fork, name the GitHub repository, check the path) | +| **STOP-17** | 0 (Orient), after STOP-13 | Open: the profile needs confirming, or the side isn't settled ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31) lists the triggers) | Up to two questions. Is the UTS infrastructure plan right (per tier the infra pieces, smoke tests and self-tests; what's left out and why; the profile's evidence if needed)? Yes, change with evidence (Orient reruns), or stop. How does the UTS infra create clients (the side, provisional until STOP-3)? | + +### 2.4 Never guess + +- **Spec content:** quote the spec, a helper spec or a UTS doc from the clone, with its path and heading. Don't rely on memory of the Ably specification. +- **API signatures:** read the SDK source and the infra source before writing any call. A signature you haven't read is a guess. +- **Commands:** run every build, test, lint and script command you record, and record its exit status. A command you haven't run is marked "unverified" in the working records. +- **Facts about the repo:** every Repo profile entry has evidence (a file path, a symbol or a command and its output). "Unknown" is allowed only with the reason and a question for the user. +- **Undocumented corpus constructs:** the guide marks some constructs *(undocumented)*; their meaning is inferred from usage. Read each use in the corpus before choosing a rendering, and say in the catalogue that the meaning is inferred. +- **Reference-implementation reads:** what another SDK's skill or UTS infra does is evidence of one SDK's choice, not of the spec or of this SDK. Confirm it against the guide or the UTS docs before relying on it. Where they are silent, record the choice as yours in the decision log, citing the read. + +### 2.5 Record every decision + +Every choice you make that isn't forced by the guide or the UTS docs goes into the decision log in the Design record, with the options, the choice, who made it (you or the user) and the reason. That includes infra choices in Phases 2 and 3 (reuse vs build, hook shapes, placement), not only the skill decisions in Phase 4. If a later phase changes an earlier decision, add a new log entry that supersedes the old one; don't edit history. + +### 2.6 Pin the spec clone + +Step 0 syncs the shared clone (no other clone is synced), then Orient pins it: [13.9](references/orient.md#139-sync-the-spec-clone) gives the precedence, the update rule and the outcomes. Record what the sync and `spec_clone_info.py` report: the SHA, the branch, the sync result, the dirty files under `uts/` and `specifications/`, `guideLastChange` and `skillVersion`. Read every spec-repo file at the paths it reports (`paths`), from this clone, for the whole run. A spec you will translate that is modified or untracked is handled as in guide [9.1](../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must). Never fetch, pull or check out outside 13.9. If HEAD moves during the run, stop and ask whether to restart from the new SHA. Untracked notes in the clone (such as `uts/UTS_*.md`) aren't specs. + +### 2.7 Working style + +- Do Step 0 (Orient) before anything else, then Phase 1; don't write infra code before the user approves the infra design (STOP-3), or skill files before the UTS infra is verified (STOP-7). +- Escalate verification in the repo's own order (compile, then build tests, then a filtered run, then wider runs), as recorded in the Repo profile. +- Keep the working records up to date as you go, including the Design record's Run status (13.7); they are also your memory if the session is interrupted. Orient finds them and offers to resume (13.6); then read them first. +- Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. + +### 2.8 Model tier and sub-agents + +- **Model:** run this procedure on the most capable model tier available: Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name (guide [Model tier](../../docs/writing-uts-spec-translator-skills.md#model-tier): MUST, with the reasons). If your environment states your model, record it; otherwise ask the user. If it isn't Opus-class, say so in Orient (STOP-1); if the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. +- **Sub-agents** you spawn for review or validation ([8.3](references/phase-6-validate-skill.md#83-infra-gate-and-selection-dry-run) dry runs, 8.5, the checklist verifier) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, the [8.2](references/phase-6-validate-skill.md#82-test-the-audit-itself) corpus sweep), if at all, and record which steps used one. Parallel work: [parallel-work.md](references/parallel-work.md). +- **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. **State** the generated skill's model tier in its opening lines and `compatibility`; pin it only as D-26 decides. + +## Run modes + +Orient recommends the mode, and the user decides at STOP-13 (D-27). The same phases, ground rules and stop points serve both modes: + +| Mode | When | Read | +|---|---|---| +| **Create** | No `uts-to-*` skill yet (S0); or none, but UTS-tagged tests or infra code to adopt (S1, 2.2) | The phases below | +| **Upgrade/Fix** | A `uts-to-*` skill exists. **Diff-driven** for one this procedure built (S2): follow what changed since its recorded run. **Full gap audit** for one of unknown origin (S3), or by choice: audit the skill, its UTS infra and its UTS-derived tests against the guide, choose per item, upgrade in place. **Regenerate from scratch** only by the user's choice ([11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)) | [Section 10](references/upgrade-diff-driven.md#10-upgradefix-diff-driven) (diff-driven), [section 11](references/upgrade-existing-skill.md#11-upgradefix-full-gap-audit) | + +An interrupted run is resumed or restarted from its records (13.6). In every mode, Phase 1 also asks whether to add LiveObjects (`objects`) support (STOP-14), recommending an answer from `detect_liveobjects.py`'s evidence ([section 12](references/liveobjects-support.md#12-liveobjects-objects-support)). + +## Phases at a glance + +Before starting a phase, read its reference file in full: this table is a map, not enough to act on. In Upgrade/Fix, full gap audit, read [11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)'s row for each phase first (diff-driven: section 10): the phases are scoped to the chosen items, and acceptance items tied to a skipped or deferred gap-audit row are reported ✗ under D-29, not worked. When regenerating from scratch, read 11.5's last paragraph instead. On resuming an interrupted run, read the working records first, then the reference of the phase you are in. + +| Phase | Goal | Read first | Writes | Ends at | +|---|---|---|---|---| +| 0. Orient | Eligibility, capabilities and scope, state class, mode | [Step 0](#step-0-orient); [orient.md](references/orient.md) | nothing | STOP-16 (if not eligible), STOP-1 (if needed), STOP-13, STOP-17 (if open) | +| 1. Understand the repo | A confirmed Repo profile: how the SDK is built, tested and mocked today; the LiveObjects decision; in Upgrade/Fix, the gap audit or the change list | [phase-1](references/phase-1-understand-repo.md) (and sections 10, 11, 12) | `repo-profile.md`; in Upgrade/Fix `skill-gap-audit.md` | STOP-2, STOP-14 (STOP-15 in Upgrade/Fix) | +| 2. Design the UTS infra | Reuse, wrap, extend or build each capability; hooks; placement; tier feasibility | [phase-2](references/phase-2-design-infra.md) | `uts-infra-design.md` | STOP-3 | +| 3. Build and verify the UTS infra | The approved UTS infra, with green per-tier smoke tests and self-tests, wired into CI | [phase-3](references/phase-3-build-infra.md) | infra code and README; build log | STOP-7 | +| 4. Skill design decisions | D-01 to D-26 and D-32 decided; D-27 to D-31 confirmed | [phase-4](references/phase-4-design-record.md) | `design-record.md` | STOP-8 | +| 5. Generate the skill files | Mapping, resolver, audit, construct catalogue, module notes, `SKILL.md` | [phase-5](references/phase-5-generate-skill.md) | the `uts-to-` skill | (STOP-9 if constructs are unmapped) | +| 6. Validate the skill | Resolver, audit mutations, infra gate, dry runs, other paths, lint, examples, the acceptance check | [phase-6](references/phase-6-validate-skill.md) | results (no tests) | (STOP-10 at a bound) | +| 7. Final report and handover | Report; the acceptance checklist completed, checked, verified independently; the completion gate | [section 9](#9-phase-7-final-report-and-handover) | `final-report.md`; `acceptance-checklist.md` with evidence | — | + +## 1. Purpose and how to use this procedure + +### 1.1 Who reads this + +- **The executor** is an LLM agent working inside the target SDK repository, with shell and file access and the ability to ask the user questions. Every instruction is addressed to it ("you"). +- **The user** is an SDK engineer who owns the target repository. They answer at the stop points and approve every change outside the skill directory. +- **A reviewer** may read the working records (Repo profile, Infra design, Design record, Final report) instead of the whole conversation, or this skill to see what the agent will do and where it will ask. + +### 1.2 Invocation + +The user starts a run in the target SDK repo with one sentence, for example: + +``` +/uts-to-lang-skill-creator realtime unit first (Claude Code; $uts-to-lang-skill-creator in Codex) +Audit our uts-to-python skill against the guide and upgrade it; add LiveObjects support. +Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to create a uts-to- skill for this repo. (skill not installed) +``` + +`` is a local clone of `ably/specification`: by default the shared clone ([13.9](references/orient.md#139-sync-the-spec-clone)), so naming one is optional; a clone the user names isn't synced unless it is the shared clone, and otherwise Orient warns them to keep it up to date. Take any path the user names (`--spec-clone `, or a path in their words), and any goals ("objects unit tier first", "rest and realtime only"), from the user's words, and record the goals as inputs. Installation is described in [`uts/README.md`](../../README.md#installing-the-skill-creator). + +### 1.3 Required inputs + +| Input | Example | If missing | +|---|---|---| +| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo: Orient checks that it is an eligible Ably Pub/Sub SDK repository with SDK sources and tests (STOP-16) | +| Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | +| Local spec clone path | `~/.ably/specification` (the directory that contains `uts/` and `specifications/`) | Optional. Located by `scripts/spec_clone_info.py` (precedence: argument, this skill's clone, `ABLY_SPEC_CLONE`, `~/.ably/specification`), or cloned there on approval ([13.9](references/orient.md#139-sync-the-spec-clone)). Only the shared clone is synced. Never fetch the spec or the UTS docs file by file from GitHub | +| User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | +| Model tier | The session's model | See [2.8](#28-model-tier-and-sub-agents) (STOP-1 if it isn't Opus-class) | +| Local clones of ably-cocoa and ably-java (now `ably-pubsub-java`) (optional) | Clones of either repo, read at their current HEAD (ideally a recent `main`) | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after STOP-6 | + +### 1.4 Outputs + +1. **The UTS infra** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-infra-smoke-tests-and-self-tests-must); permanent and run in CI), an infra README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at STOP-3. +2. **The skill** (in Upgrade/Fix, the existing skill, upgraded in place), at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-/` for Codex, unless D-01 says otherwise: `SKILL.md`, `uts-package-mapping.json` (minimal: source → destination plus only what the resolver can't derive, guide [3.2](../../docs/writing-uts-spec-translator-skills.md#32-the-mapping-file)), `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md`, `scripts/scan_constructs.py` if D-19 ships it, and `scripts/ensure_spec_clone.py` per D-32 (guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). +3. **No spec-derived tests:** the first come from the first real run of the generated skill. Phase 6 validates the skill without generating any. +4. **Working records** (*procedure recommendation*): Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; D-24, confirmed in Phase 4). Copy each template from `assets/templates/` there under the same file name, then fill it in: + - [repo-profile](assets/templates/repo-profile.md) and [design-record](assets/templates/design-record.md) (which holds the decision log), at the start of Phase 1; + - [uts-infra-design](assets/templates/uts-infra-design.md) (the gap table, the infra design and the build log), at the start of Phase 2; + - [acceptance-checklist](assets/templates/acceptance-checklist.md), at the start of Phase 2 (in Upgrade/Fix, pre-filled from the gap audit's section B): update the rows a phase produces or verifies when that phase ends, with their evidence; + - [final-report](assets/templates/final-report.md), in Phase 7, with the completed acceptance checklist in its "Acceptance checklist" section; + - in Upgrade/Fix, [skill-gap-audit](assets/templates/skill-gap-audit.md), at U1 (diff-driven: before STOP-15, for section H). + +The templates contain no links, so they stay valid in the SDK repo. Write the records from STOP-13 on, at the location Orient's routing gives (13.5), using the language inferred so far for ``; if STOP-2 changes the language or D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. A later agent reads them to maintain or regenerate the skill ([section 10](references/upgrade-diff-driven.md#10-upgradefix-diff-driven)), so keep them accurate as you go, not only at the end. + +### 1.5 Intent + +The guide's intent ([introduction](../../docs/writing-uts-spec-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the UTS infra and regenerate. Never hand-fix a test the skill generates; fix the skill, notes or UTS infra. + +### 1.6 Which document wins + +| Question | Authority | +|---|---| +| What a faithful translation is; how a failure is diagnosed; the `deviations.md` format | [`writing-derived-tests.md`](../../docs/writing-derived-tests.md) | +| What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../../docs/writing-test-specs.md) | +| Integration and proxy tiers | [`integration-testing.md`](../../docs/integration-testing.md), [`proxy.md`](../../docs/proxy.md) | +| Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | +| What the UTS infra and the skill must contain, and why | the guide ([How it relates to the other UTS docs](../../docs/writing-uts-spec-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | +| How and in what order you build them | this skill | +| The SDK's actual API | the SDK source (guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout), "The notes are a map, not an authority") | +| Questions none of the above answer, as a last resort only | The reference implementations, under [Reference implementations (last resort)](#reference-implementations-last-resort): never an authority | + +If two documents disagree, follow the higher one in the table (and STOP-11 if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. A disagreement between a reference implementation and the guide or the UTS docs is never a STOP-11: follow the guide, and record the disagreement with the read ([2.1](#21-your-inputs)). + +**Procedure recommendations.** Where this skill adds detail the guide doesn't have (for example: the working-records files and their location, the shape of the per-tier readiness data in the mapping (an `unready` object per module), a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *procedure recommendation*: this skill's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). + +## 9. Phase 7: final report and handover + +1. Write `final-report.md` from the [final-report template](assets/templates/final-report.md), and refresh the skill's provenance stamp (D-30, 13.7). +2. Complete the [acceptance checklist](assets/templates/acceptance-checklist.md) in the working records, tracked since Phase 2: it maps each item of the guide's [section 13 checklist](../../docs/writing-uts-spec-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill) to where this procedure produces and verifies it. Give each row its evidence (a command and its result, file:line, or a record ID) and a status: ✓; "✓ static; behaviour at first translation" (the rule's file:line, for a row only translating can show); n/a with the reason ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)); or ✗ with why it isn't met (scoped out, planned, skipped or deferred at STOP-15). A ✓ without evidence counts as open. Then run the automated check (without `--final`, [8.9](references/phase-6-validate-skill.md#89-check-the-acceptance-checklist)) and fix what it finds. +3. **Verify independently.** A fresh sub-agent that built nothing in this run (Opus-class, 2.8) verifies the checklist, read-only. It reads guide §13, the checklist with its evidence and the files; re-runs the cheap checks (the resolver, `check_acceptance.py`, lint); checks the evidence of the ✓ rows (every MUST row, a sample of the rest); and reviews the construct catalogue and the translation rules in the skill's `SKILL.md` and notes against guide §5 and §6, row by row: each rendering names a real UTS infra or SDK symbol, follows the flagged traps (guide 6.7) and doesn't substitute a similar mock outcome (guide 5.2). It returns *agree* or *dispute* per row, with reasons. Fix each dispute, or show it to the user; never override one silently. Record the verifier's model, and the result on the checklist's "Independent verification" line. Without sub-agents, give the user a prompt to run this step in a new session (this step's text, the skill's path and the checklist's path), or record that the verification wasn't independent. +4. **The completion gate.** Record the verification, then run `check_acceptance.py --final` and copy its summary into the report. Set the Design record's Run status to finished only if that gate passes: the checklist has no open row, the automated check has no unexplained fail, the verification has no unresolved dispute, and the infra tests are green, and in CI, at every tier the skill supports (or the report says which tiers aren't and why). Rows marked "✓ static; behaviour at first translation" don't hold the gate open; the report lists them under "Verified at the first real translation". +5. Show the user the report, the list of changed and new files (UTS infra, infra tests, hooks, skill), and the suggested CI changes. **Don't commit** unless asked. +6. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the UTS infra and the skill in separate commits. In Upgrade/Fix, offer one commit per closed gap-audit or change item (from the Design record's changelog), infra rows first. + +## Reference implementations (last resort) + +This section expands the last resort of [2.1](#21-your-inputs). The guide covers what the existing `uts-to-swift` and `uts-to-kotlin` skills and their UTS infra teach, so you shouldn't need them. You MAY consult them, read-only, under the guide's [Reference implementations (last resort)](../../docs/writing-uts-spec-translator-skills.md#reference-implementations-last-resort) rules 1 to 5 (the guide wins, learn the pattern and never copy, check against Patterns to avoid, record each consultation, creation only); that section also lists the repos and paths. This skill adds: + +- **When:** only after this skill, the guide, the UTS docs and the helper specs fail to answer the question. The guide and the UTS docs always win ([1.6](#16-which-document-wins)). +- **Access:** read local clones the user named ([1.3](#13-required-inputs)) at their current HEAD (read the files, or `git -C show HEAD:`); never check out, modify, pull or fetch them on the user's behalf (fetching or pulling is network access). Otherwise read GitHub after STOP-6; the user may approve reference reads once for the whole run, and you record that approval. +- **Recording:** record each read in the Design record ("Reference-implementation reads": repo, revision, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap. + +These rules govern other repos. The target repo's own `uts-to-*` skill, UTS infra and UTS-derived tests are input 4, even in ably-cocoa or ably-java: read them in full and change them in place ([11.1](references/upgrade-existing-skill.md#111-when-this-mode-applies)). A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or their only answer departs from the guide, STOP-12. The generated skill never consults them (D-22). + +## Reference index + +Each phase's reference (sections 3 to 8) is linked from [Phases at a glance](#phases-at-a-glance), and each template from [1.4](#14-outputs). Also read: + +| File | Read it when | Sections | +|---|---|---| +| [orient.md](references/orient.md) | Step 0, every run | 13 | +| [upgrade-diff-driven.md](references/upgrade-diff-driven.md) | Upgrade/Fix, diff-driven (S2) | 10 | +| [upgrade-existing-skill.md](references/upgrade-existing-skill.md) | Upgrade/Fix, full gap audit, or regenerating from scratch | 11 | +| [liveobjects-support.md](references/liveobjects-support.md) | Phase 1 step 1e (every run); writing the objects notes | 12 | +| [phase-4-design-record.md](references/phase-4-design-record.md) | Mapping a guide change to decisions (Upgrade/Fix, diff-driven) | 6 | +| [phase-5-generate-skill.md](references/phase-5-generate-skill.md) | Listing the corpus infra symbols in Phase 2 (7.4) | 7 | +| [asking.md](references/asking.md) | Before asking any stop point: the question format, Orient's ready-made questions, and the AskUserQuestion tool in Claude Code | — | +| [parallel-work.md](references/parallel-work.md) | Before handing work to a sub-agent: which work may run in parallel, and the rules | — | +| [glossary.md](references/glossary.md) | A term is unclear | — | +| [failure-modes.md](references/failure-modes.md) | An infra check, a Phase 6 check or a command fails in a way you don't recognise | — | + +## Scripts + +All but `ensure_spec_clone.py` are read-only Python 3, with no network access; they print to stdout and run from anywhere as `python3 /scripts/.py`. `--help` prints a script's docstring (purpose, usage, output and exit codes) and exits 0; a usage error prints the usage to stderr and exits 2. Each row's linked section has the rules. + +| Script | Use | +|---|---| +| `orient.py [--spec-clone P] [--skill-dir P] [--records P] [--capability-override rest=L,realtime=L] [--side S] [--include-submodules] [--full]` | Step 0: runs `spec_clone_info.py`, the eligibility check, the inspector and both detectors, classifies the repo, recommends a mode, derives the UTS infrastructure plan, and words the STOP-13 and STOP-17 questions ([13.1](references/orient.md#131-run-it)) | +| `spec_clone_info.py [SPEC_CLONE \| --spec-clone P]` | Locate, validate and pin the spec clone; `warnings` is a list, possibly empty ([13.1](references/orient.md#131-run-it)) | +| `ensure_spec_clone.py [--spec-clone P] [--status \| --fetch \| --update \| --bootstrap]` | Step 0's sync of the shared clone (no other clone): the only script that uses the network or changes the spec clone, and only on approval ([13.9](references/orient.md#139-sync-the-spec-clone)); copied into the generated skill (D-32) | +| `check_repo_eligibility.py [--spec-clone P]` | Is the repository on the whitelist? ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | +| `detect_capabilities.py [--spec-clone P] [--include-submodules] [--capability-override rest=full,realtime=absent]` | Capability profile, door sides, suggested scope ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | +| `spec_names.py [--spec-clone P]` | The detectors' names, from the spec clone's IDL, and the capability-inapplicable tests ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | +| `survey_repo.py [ …]` | First pass for the discovery checklist ([3.2](references/phase-1-understand-repo.md#32-step-1b-answer-the-discovery-checklist)) | +| `scan_constructs.py /uts/` | The corpus's constructs and infra calls ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); copied into the generated skill (D-19) | +| `inspect_existing_skill.py [] [--records ]` | Inventory of an existing `uts-to-*` skill, its installs, UTS infra, UTS-derived tests and records; regex hints, not a review ([11.2](references/upgrade-existing-skill.md#112-step-u1-understand-the-existing-skill)) | +| `check_acceptance.py [--records ] [--final] [--markdown]` | The automated acceptance check on a generated skill: the inspector's checks, the working records' acceptance checklist (rows, order, statuses, evidence, reasons) and the records it needs; `--final` adds the completion gate's conditions ([8.9](references/phase-6-validate-skill.md#89-check-the-acceptance-checklist), [9](#9-phase-7-final-report-and-handover)) | +| `detect_liveobjects.py [--spec-clone P] [--include-submodules]` | Evidence for the LiveObjects decision at STOP-14 ([12.1](references/liveobjects-support.md#121-gather-the-evidence-step-1b-p-15)) | +| `check_layout.py [SKILL_MD]` | For maintainers, not a run: do the stop table and the ground rules still end within the compaction window? ([Maintaining this skill](#maintaining-this-skill)) | + +### Maintaining this skill + +What the scripts know about Ably's repositories and SDK names that the spec can't give is data: update the data file, not the code. + +- [`assets/eligibility.json`](assets/eligibility.json) is the whitelist: adding a new SDK repository means adding its name to `repositories`. Update it too when Ably renames or retires a repository (the canonical and planned names), and for the capability overrides and the definition gate's threshold. +- [`assets/capability-names.json`](assets/capability-names.json): update it when an SDK adds a client alias or a door factory, or the spec renames a LiveObjects name its IDL no longer shows. + +Each file starts with one `_description` (what the file is and when to update it); add a `_comment` key only where an entry would otherwise puzzle a maintainer. The scripts ignore `_` keys, check the shape, and report a malformed file as `DATA_FILE_ERROR`, naming the file and the key. Client, channel, connection, presence and LiveObjects names come from the spec clone at run time (13.8) and need no edit. After an edit, rerun Orient on a repository the change affects. + +After a context compaction, Claude Code keeps only the start of this `SKILL.md`: about 5,000 tokens, which it counts as about 20,000 characters of the text after the frontmatter, including a line with the install path. Codex may keep none of it. So Step 0, the stop table and the ground rules (section 2) come first, and everything else follows section 2. After editing anything before the end of section 2, run `python3 /scripts/check_layout.py`: it measures, in characters after the frontmatter, where the stop table and section 2 end, and fails above 16,000 and 18,500. Move material out of the way rather than raising the limits. diff --git a/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json b/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json new file mode 100644 index 000000000..36d739846 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json @@ -0,0 +1,212 @@ +{ + "_description": "Names the detectors need but can't read from the spec's IDL: SDK client aliases, door factories, earlier-revision and prose-only LiveObjects names, and a minimal fallback. Update this file, not the scripts.", + "aliases": { + "rest": [ + "PubSubHttpClient", + "DefaultPubSubHttpClient" + ], + "realtime": [ + "PubSubRealtimeClient", + "DefaultPubSubRealtimeClient", + "PubSubClient" + ] + }, + "doorFactories": { + "_comment": "A door's capability: its return type, else its name (Http or Rest: REST). Its side: its enclosing type (sides.*.types), else a path segment (sides.*.paths), else core.", + "names": [ + "createClient", + "createHttpClient", + "createRestClient", + "createRealtimeClient", + "create_client", + "create_http_client", + "create_rest_client", + "create_realtime_client", + "CreateClient", + "CreateHttpClient", + "CreateRealtimeClient", + "NewClient", + "NewHTTPClient", + "NewRealtimeClient", + "clientBuilder", + "httpClientBuilder", + "realtimeClientBuilder", + "createClientWithOptions", + "createClientWithKey", + "createClientWithToken" + ], + "sides": { + "device": { + "types": [ + "PubSubDevice", + "AblyPubSubDevice" + ], + "paths": [ + "device", + "pubsub-device" + ] + }, + "server": { + "types": [ + "PubSubServer", + "AblyPubSubServer" + ], + "paths": [ + "server", + "pubsub-server" + ] + } + } + }, + "_comment": "legacyPublicApi: public names from earlier objects-features.md revisions. legacyNowInternal: public then, internal now; a public declaration counts as legacy. internalFromProse: internal names the spec gives only in prose, with no IDL entry.", + "legacyPublicApi": [ + "RealtimeObjects", + "getRoot", + "createMap", + "createCounter", + "LiveObjectSubscription", + "LiveMapValueType", + "LiveCounterValueType" + ], + "legacyNowInternal": [ + "LiveObject", + "LiveObjectUpdate", + "LiveMapUpdate", + "LiveCounterUpdate" + ], + "internalFromProse": [ + "ObjectsPool" + ], + "fallback": { + "capability": { + "clients": { + "rest": [ + "Rest", + "RestClient" + ], + "realtime": [ + "Realtime", + "RealtimeClient" + ] + }, + "roles": { + "restClient": { + "names": [ + "RestClient" + ], + "members": { + "request": "RSC19", + "stats": "RSC6a", + "time": "RSC16" + }, + "from": [ + "fallback" + ] + }, + "realtimeClient": { + "names": [ + "RealtimeClient" + ], + "members": { + "connect": "RTC15", + "close": "RTC16" + }, + "from": [ + "fallback" + ] + }, + "restChannel": { + "names": [ + "RestChannel" + ], + "members": { + "publish": "RSL1", + "history": "RSL2" + }, + "from": [ + "fallback" + ] + }, + "channels": { + "names": [ + "RealtimeChannel", + "ChannelStateChange", + "ChannelState", + "ChannelEvent" + ], + "members": { + "attach": "RTL4", + "detach": "RTL5", + "whenState": "RTL25" + }, + "from": [ + "fallback" + ] + }, + "presence": { + "names": [ + "RealtimePresence" + ], + "members": { + "enterClient": "RTP4", + "history": "RTP12" + }, + "from": [ + "fallback" + ] + }, + "connection": { + "names": [ + "Connection", + "ConnectionStateChange", + "ConnectionState", + "ConnectionEvent" + ], + "members": { + "createRecoveryKey": "RTN16g", + "whenState": "RTN26" + }, + "from": [ + "fallback" + ] + } + } + }, + "liveobjects": { + "publicApi": [ + "RealtimeObject", + "PathObject", + "LiveMap", + "LiveCounter", + "compactJson" + ], + "internalApi": [ + "InternalLiveMap", + "InternalLiveCounter", + "LiveObject", + "ObjectsPool" + ], + "shared": [ + "ObjectMessage", + "ObjectOperation" + ], + "sharedConstants": [ + "MAP_SET", + "COUNTER_INC" + ], + "wireOnly": [ + "ObjectState", + "ObjectData" + ], + "wireConstants": [ + "OBJECT_SYNC" + ], + "accessorTypes": [ + "RealtimeObject" + ], + "from": [ + "fallback" + ] + } + } +} diff --git a/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json b/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json new file mode 100644 index 000000000..360222abe --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json @@ -0,0 +1,40 @@ +{ + "_description": "The whitelist of Ably Pub/Sub SDK repositories the skill creator accepts, read by scripts/check_repo_eligibility.py (section 13.2); every other repository is rejected. To support a new SDK repository, add its name here.", + "owner": "ably", + "repositories": [ + "ably-cocoa", "ably-ios", "ably-pubsub-cocoa", + "ably-go", "ably-pubsub-go", + "ably-pubsub-dart", "ably-dart", + "ably-pubsub-dotnet", "ably-dotnet", + "ably-pubsub-flutter", "ably-flutter", + "ably-pubsub-java", "ably-java", + "ably-pubsub-js", "ably-js", + "ably-pubsub-php", "ably-php", + "ably-pubsub-python", "ably-python", + "ably-pubsub-ruby", "ably-ruby", + "ably-ruby-rest", + "ably-rust" + ], + "capabilityOverride": { + "_comment": "ably-ruby-rest is a REST-only gem wrapping ably-ruby as a submodule: it defines no client itself, so the override supplies the profile.", + "ably-ruby-rest": { "rest": "full", "realtime": "absent" } + }, + "canonical": { + "ably-ios": "ably-cocoa", + "ably-dart": "ably-pubsub-dart", + "ably-dotnet": "ably-pubsub-dotnet", + "ably-flutter": "ably-pubsub-flutter", + "ably-java": "ably-pubsub-java", + "ably-js": "ably-pubsub-js", + "ably-php": "ably-pubsub-php", + "ably-python": "ably-pubsub-python", + "ably-ruby": "ably-pubsub-ruby" + }, + "plannedRename": { + "ably-cocoa": "ably-pubsub-cocoa", + "ably-go": "ably-pubsub-go" + }, + "definitionGate": { + "minSourceFilesToReject": 20 + } +} diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md new file mode 100644 index 000000000..e7ca803fd --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md @@ -0,0 +1,64 @@ + + +- Automated check (check_acceptance.py): ; open rows, unexplained fails | not run yet +- Independent verification: , ; agree , disputed , unresolved | not independent: | not run yet + +| Guide §13 item | Produced in | Verified in | Evidence | Status | +|---|---|---|---|---| +| **UTS infra** | | | | | +| SDK test hooks (5), clock covering blocking waits and async-runtime timers, injected at construction; existing seams first; each new hook minimal, approved and needed by a spec | 4.6; 5.2 rows 3–4 | 5.2 acceptance; G-01…G-05; 4.6 seams and alternatives recorded | | | +| Shared test library per helper spec (incl. legacy `queue_*` and `mock_http.captured_requests`; `mock_websocket.md` async close, CONNECTED ordering, raw frames, alternative API and undocumented members; vcdiff; test pool; `MockNetworkListener`) | 4.3; 5.2 rows 5–6, 8, 11 | smoke tests; G-06…G-10 | | | +| Wait helpers (latched, fail-fast `AWAIT_STATE`; value-returning `poll_until`; `poll_until_success`; `process_pending_events`; wall-clock wrapper; fake-time-safe deadlines) | 4.4; 5.2 row 7 | helper tests; G-11 | | | +| Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | | +| UTS infra designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; organised by tier (MUST; names SHOULD): proxy under integration, placement by use, names following folders; fixture-helper scope documented | 4.2; 4.7 (Placement table); 5.2 row 1; 5.3; notes item 9 | G-13; 5.2 layout check; inspector T-tier-layout | | | +| Infra README with a Known gaps section (MUST) | 5.3; 5.2 row 12 | G-14; 8.5 path 9 | | | +| Infra smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, in a tree by tier, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.3 infra gate; 8.5 path 8 (✗ "SDK-blocked" while an objects smoke test is SDK-blocked, 12.5) | | | +| `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | | +| `ProxyManager` (pinned, every OS, verified download, health check, port handling); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | | +| One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | | +| Concurrency, time, parallelism, visibility models documented | Phase 1 (P-09 to P-14) | STOP-2; G-18 | | | +| **Skill files** | | | | | +| `SKILL.md` frontmatter portable across Claude Code and Codex: `name` equal to the directory; pushy third-person description of at most 1,024 characters, no `<` or `>`, a "Not for" boundary; string-valued `metadata`; Claude Code-only fields only if accepted as non-portable | 7.6; D-22; D-26 | 7.6 acceptance; 8.6 | | | +| `SKILL.md`: usage guard that doesn't rely on `$ARGUMENTS` and never guesses a module (optional argument; a module question when none is named); own files referred to relative to the skill directory, in `SKILL.md` and the scripts; placeholder paths; 3.4 outline incl. the Infra reference | 7.6 | 7.6 acceptance; 8.5 paths 1 and 10; 8.9 (inspector) | | | +| Installed for both tools from one source (`.claude/skills/` and `.agents/skills/`, by in-repo symlink or a CI-checked copy) (SHOULD) | D-01; 7.1 | 8.3 dry run in a new session, where the skill is listed (a sub-agent reading it by path doesn't show this); 8.5 path 10 | | | +| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked, `infra` entry (`root`, `readme`, `testsRoot`, `testCommand`); nothing the resolver can derive | 7.1; D-03 | 7.1 acceptance; 8.1 | | | +| `resolve_uts.py`: validation (incl. a clone root) and a module listing, errors, output contract, path tiers, relative exclusions, collectable names, mirrored sub-directories, collisions, validate-then-write `--create` preserving other entries, `infra` output derived from the mapping, older mapping shapes read with a warning | 7.2; D-04; D-05 | 8.1 | | | +| `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | | +| Module notes for every diverging module; `objects` decided from evidence of the LiveObjects public API (full notes if exposed, even if incomplete; a placeholder, or an entry marked not ready as a whole, otherwise) | 7.5; D-23; 12.1–12.3; D-28 (decided at STOP-14) | 7.5 acceptance; 8.5 paths 3–4 | | | +| Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | | +| **Workflow** | | | | | +| Required reading incl. features specs, from the same clone, synced first (SHOULD; D-32) | 7.6 (step 0) | 8.3 dry run (what step 0 read, recorded); 8.5 path 11 | | | +| Steps 0 and A–F: the four questions and the infra preflight (stop on red) | 7.6 | 8.3 dry run (green preflight); 8.5 paths 2–5, 8–9 | | | +| Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 7.6 text; 8.9 (inspector `B-reference-tests`); verifier; at first translation: steps 1–7 run, and the reference test (`none yet`, 8.8) is proposed | | | +| Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 7.4 acceptance (scanner over every covered module, every hit accounted for); 8.5 path 6; at first translation: the stop | | | +| Re-sync mode; fix the cause, then regenerate | 7.6; D-19 | 8.5 path 6 (classification); 7.6 text; at first translation (the first real re-sync): regeneration | | | +| Mid-run infra changes re-run the smoke tests and self-tests and are reported separately | 7.6 (Infra reference) | 7.6 text; verifier; at first translation | | | +| **Translation rules in `SKILL.md`** | | | | | +| One verbatim `UTS: ` per test; no invented tags; spec point in a collectable name | 7.6; D-04, D-06 | 8.2 (the audit enforces it on every file) | | | +| File header: source spec, tier, spec SHA, disclosures, corresponding unit spec | 7.6; D-07 | 8.7 (the per-tier template compiles with its header); 8.5 path 6 (header read) | | | +| Verbatim comments, spec variable names, section headings, spec order | 7.6 | 7.6 text; verifier (guide §5); at first translation | | | +| Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome | 7.6; 7.4 | 8.2 (shortfall mutations); 7.4 (no substitution rows) | | | +| Policies for sub-cases, groupers, test-case tables, delegating tests, ID-less tests | 7.6; D-13 | 7.6 text; D-13; verifier (guide §5); at first translation | | | +| Protocol variants parameterised; tag singular; proxy JSON only | 7.6; D-16 | 8.7 (template and catalogue row compile); verifier | | | +| Clients closed in teardown; mocks restored; stop on a failed wait | 7.6; D-15 | infra self-tests (teardown, waits); 8.7 renderings; verifier; at first translation: the translation side | | | +| Unique channel names; own proxy session per test | 7.6 | infra self-tests (proxy session per test); 8.7 renderings; verifier; at first translation: the translation side | | | +| Unit: fake time and flushes, no real timers. Integration: wall-clock | 7.6; 7.4; D-17 | infra self-tests (fake clock); 8.7 renderings; verifier; at first translation: the translation side | | | +| Record-and-verify for transient states; polls return the settled value | 7.6; 7.4 | infra self-tests (waits, polls); 8.7 renderings; verifier; at first translation: the translation side | | | +| No accommodate-both assertions; `IS ` per typing discipline | 7.6 | 7.6 text; verifier (guide §5); at first translation | | | +| Public-API specs stay public; internal-access ladder with escalation stop | 7.6; 7.5 item 8; D-12 | 7.6 text; 7.5 item 8; D-12; verifier (guide §5); at first translation | | | +| Construct table filled for every row | 7.4 | 7.4 acceptance; 8.7 (every `mapped` rendering compiles); verifier (guide §6) | | | +| **Evaluation** | | | | | +| Three end states; runtime skip and fail-fast idioms; `RUN_DEVIATIONS` command | 7.6; D-09, D-10 | 8.7 (the skip and fail-fast idioms compile); 7.6 text; verifier; at first translation | | | +| `deviations.md` location; format deferred; written at the right time | 7.6; D-08 | 7.6 text; D-08; verifier; at first translation | | | +| Infra stand-ins in comments, not `deviations.md` | 7.4; 7.6 | 7.4 `stand-in` rows; 7.6 text; verifier; at first translation | | | +| Language-inapplicable inputs (cases a–c) noted in the test file | 7.6; D-11 | 7.6 text; D-11; verifier; at first translation | | | +| Stopping rules and spec-error escalation | 7.6; D-21 | 7.6 text; D-21; verifier; at first translation | | | +| **Verification** | | | | | +| Compile (or collect), lint, CI-strict compile, filtered run | 7.6 | 8.6; 8.7; 8.5 path 7 (the filter syntax on an infra class); at first translation: the generated-class run | | | +| No commits by default; changed files listed | 7.6; D-20 | Phase 7 | | | +| Final report format | 7.6 | 7.6 text; verifier (guide §11); at first translation | | | +| The skill's own examples compile | 7.5 item 12; 7.6 | 8.7 | | | +| **Creation process** | | | | | +| Skill and UTS infra created on an Opus-class model (MUST); model stated for runs, or pinned where the tool supports that (SHOULD), and recorded in the skill's final report | 2.8; D-26; 7.6 (frontmatter, final report) | Design record "Inputs at generation time" (Model line, with the dry-run and verifier sub-agents' models); Final report header | | | +| Reference-implementation reads, if any, last resort only, recorded and reported as guide gaps | 2.1; design-record template | Final report "Reference-implementation reads" | | | +| An existing skill, its UTS infra and its UTS-derived tests audited against this checklist before being changed; gaps listed for the owner to choose; working parts preserved (SHOULD) | 13 (Orient); 10; 11.2–11.5; D-27; D-29; skill-gap-audit template | STOP-15; 11.6; Final report "Existing assets and upgrade summary" (n/a in create mode) | | | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md new file mode 100644 index 000000000..67f7543cd --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md @@ -0,0 +1,75 @@ +# Design record: uts-to- + +## Inputs at generation time +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; guide/skill: ; uts-to-lang-skill-creator version +- Repo: +- Repo profile confirmed: ; infra option at STOP-3: (a | b | c); UTS infra confirmed: +- Infra README: +- Model: (Opus-class: yes | no, departure in decision log #); sub-agents: , or none +- Eligibility: via (accept | fork-confirmed) | user-named: () +- Capabilities (D-31): rest ; realtime (); liveobjects ; side: (); scope: ([ + REST-only tests under realtime]); unsupported: (); capability-inapplicable: tests (listed in the notes); names: +- State class (Orient): S0 | S1 | S2 | S3 | S4 (+ re-entry) — ; run mode (D-27): create | upgrade: diff-driven | upgrade: full audit | upgrade: regenerate (resumed | restarted ); target skill: (origin …), or none +- Run status: in progress (Phase , ) | finished () + - After a context compaction: read the skill creator's whole SKILL.md again, then this line, then the current phase's reference, before your next action. + - Finished only at the completion gate: the acceptance checklist has no open row, check_acceptance.py --final has no unexplained fail, the independent verification has no unresolved dispute, and the infra tests are green in CI at every supported tier (or the Final report says why not). +- LiveObjects evidence: detect_liveobjects.py → , ; decision at STOP-14 (D-28) + +## Decisions +| ID | Decision | Choice | Guide-default? | Decided by | Rationale | +|---|---|---|---|---|---| +| D-01 | Skill name and location | | yes / no | user / agent / existing skill | | +| D-02 | Script language and runtime | | | | | +| D-03 | Mapping: module × tier → target dir (hand-maintained entries marked) | (table below) | | | | +| D-04 | Naming and collection rules (with three worked examples) | | | | | +| D-05 | Collision handling | | | | | +| D-06 | Tag syntax | | | | | +| D-07 | File header format | | | | | +| D-08 | deviations.md location(s) | | | | | +| D-09 | Runtime skip idiom; RUN_DEVIATIONS command | | | | | +| D-10 | Fail-fast idiom | | | | | +| D-11 | Pending idiom for missing APIs | | | | | +| D-12 | Internal-access ladder; white-box spec list location | | | | | +| D-13 | Delegation policy | | | | | +| D-14 | Parallelism policy | | | | | +| D-15 | Teardown; suite-level fixtures | | | | | +| D-16 | Protocol-variant rendering | | | | | +| D-17 | Default timeouts and poll interval | | | | | +| D-18 | Auth through the proxy | | | | | +| D-19 | Re-sync flag; manifest; corpus scanner shipped? | | | | | +| D-20 | Commit policy; project/solution-file edits (approved / left to the user) | | | | | +| D-21 | Fix-attempt bound | | | | | +| D-22 | allowed-tools; remote access | | | | | +| D-23 | Notes files per module (full / placeholder; objects per D-28) | | | | | +| D-24 | Working-records location | | | | | +| D-25 | Retired (no pilots): not used | — | — | — | — | +| D-26 | Model tier and pinning for skill runs | | | | | +| D-27 | Run mode and state class (STOP-13) | | | | | +| D-28 | LiveObjects support: full / placeholder / none; translate-only / evaluate; SDK-blocked tiers and evidence, if any (STOP-14, 12.5) | | | | | +| D-29 | Gap-audit choices: add / skip / defer per GA- or CH- item; preserve list (STOP-15; n/a in create mode) | | | | | +| D-30 | Provenance stamp (metadata.generated-by, metadata.records) and Run status | | | | | +| D-31 | Capability profile, skill scope and side: the UTS infrastructure plan (STOP-13 or STOP-17; side re-confirmed at STOP-3) | | | | | +| D-32 | Spec-clone sync in the generated skill (ensure_spec_clone.py copied; ABLY_SPEC_SYNC) | | | | | + +### D-03 mapping +| Module | unit | integration | proxy | notes | readiness and scope markers (unready, unsupported, extraTests, capabilityInapplicable, translateOnly) | hand-maintained? | +|---|---|---|---|---|---|---| + +- infra: root ; readme ; testsRoot ; testCommand +- Values the build can't give (each with why): +- Derived by the resolver, not stored: test project, namespace root, each tier's helper sources (tier layout), per-spec targetDir and testFile (sub-directories mirrored) + +## Phase timings +| Phase | Started (date, time) | Ended (date, time) | Stop waits (STOP-n: asked → answered) | Sub-agents (role → model; task; files written) | +|---|---|---|---|---| + +## Decision log +| # | Date and time | Phase | Question | Options | Answer | By | Supersedes | +|---|---|---|---|---|---|---|---| + +## Reference-implementation reads (last resort, 2.1) +| # | Date | Question | Guide/UTS sections searched | Repo @ revision : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | +|---|---|---|---|---|---|---|---| + +## Changelog (fixes made to the skill, notes or UTS infra) +| # | Date | Found by | Cause (skill / notes / catalogue / script / UTS infra) | Fix | Closes (GA-nn / CH-nn / G-nn, if any) | Tests regenerated | +|---|---|---|---|---|---|---| diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md new file mode 100644 index 000000000..5b4135cde --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md @@ -0,0 +1,88 @@ +# Final report: uts-to- for + +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; repo ; date +- Spec SHA not on origin/main (a branch run): +- Model: (Opus-class: yes | no, and why); sub-agents: , or none +- Scope: ; out of scope and why +- Conforms to the guide: yes | no (MUST rows ✗: …) +- Eligibility: via (accept | fork-confirmed) | user-named: () +- Capabilities (D-31): rest …; realtime …; side: ; scope: ; unsupported: … (capability absent); capability-inapplicable: tests; partial skill: yes, extend by Upgrade/Fix when appears | no +- State class: S0 | S1 | S2 | S3 | S4 (+ re-entry); run mode: create | upgrade: diff-driven | upgrade: full audit | upgrade: regenerate (resumed | restarted); LiveObjects: full | placeholder | none (translate-only | evaluate), from detector recommendation + +## Files created or changed +| Path | New / changed | Kind (hook, UTS infra, smoke test, skill, script, notes, CI suggestion) | Item (GA-nn / CH-nn / G-nn, Upgrade/Fix) | Approved at | +|---|---|---|---|---| +Hand-maintained project files the user must update: … + +## Existing assets and upgrade summary +Create (S1): the UTS-derived tests and infra code found (P-20), whether they match D-03, D-06 and D-07, and the follow-ups proposed. +Upgrade/Fix: items added, skipped (with the reason) and deferred (with the plan), by GA-/CH-/G- ID, from skill-gap-audit.md; infra rows decided at STOP-3; preserve list kept; baseline comparison (resolver, audit, infra tests): differences and the item that explains each; derived tests made stale, and whether they were regenerated; what the old skill had that the new one lacks (regenerate only). + +## Repo profile +Link to repo-profile.md; corrections made at STOP-2. + +## UTS infra +Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test and self-test results per tier, and their CI jobs; the infra gate and dry-run preflight results per tier; infra changes made during Phase 6 (with the self-tests added); the README's Known gaps; the plan for any remaining rows. Link to uts-infra-design.md and the infra README. + +## SDK production changes +| Production file | Change (lines) | Hook | Approved at (STOP-4, decision #) | +|---|---|---|---| +The production diffstat from the build log, matched against each approved proposal's file list. "None" if the run changed no SDK source. + +## Design decisions +Link to design-record.md; decisions that depart from the guide's defaults, and why. + +## Validation results +- Resolver: rest / realtime / objects → ok, spec counts; negative cases → codes +- Audit self-test: each mutation → result; corpus sweep → N specs, 0 crashes, unverifiable specs listed +- Lint: …; CI-strict compile: …; examples, per-tier templates and catalogue renderings verified: N of N (how; scratch files deleted) +- Infra gate per tier: +- Other paths (8.5): each path → result, or not run and why +- Automated check (check_acceptance.py --final): +- Independent verification: ; agree , disputed (each dispute and how it was resolved) | not independent: +- Fixes made to the skill, notes or UTS infra during Phase 6 (from the changelog): … + +## Selection dry runs +| Module × tier | Model (session or sub-agent) | Questions asked | Step 0 read | Preflight | +|---|---|---|---|---| +Reference tests named in SKILL.md: none yet (the skill proposes one after a tier's first reviewed translation). + +## Verified at the first real translation +The acceptance-checklist rows marked "✓ static; behaviour at first translation", each with its static evidence. These checks show the skill conforms to the guide; they can't show its translations come out right. That shows at the first real translation run, and the fix-the-cause, regenerate and re-sync cycle handles it there. + +## SDK-blocked objects tiers +: — ; translate-only until the smoke test passes; acceptance row "Infra smoke tests …" ✗ (SDK-blocked). "None" otherwise. + +## Known limitations +Tiers not ready; `not available` catalogue rows; infra capabilities missing; audit SHOULD items not implemented; platforms the scripts weren't run on. + +## Deviations and spec issues found +- SDK deviations (recorded in deviations.md): … +- Suspected UTS spec errors or gaps: — ; draft fix: … +- Spec-repo doc issues (contradictions between guide and UTS docs; guide gaps revealed by reference-implementation reads): … + +## Native test findings (not fixed) +The SDK's own tests are out of scope: they are reported, never fixed, skipped or rewritten. +- Failing or flaky native tests: — +- SDK bugs found while building: — +"None" otherwise. + +## Reference-implementation reads +Each read from the Design record (question, repo @ revision : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. + +## Suggested CI changes + (not applied unless approved at STOP-5) + +## Next steps +Run the skill on one module per tier first (no spec-derived test exists yet), and review the result against "Verified at the first real translation"; then which modules and tiers to translate next, in order, and why; infra work remaining; deferred gap-audit items with their plan. + +## Run timings +| Phase | Start | End | Duration | Waiting at stop points | Sub-agents (role → model) | +|---|---|---|---|---|---| +From the Design record's Phase timings. Totals: the run, the time waiting for answers, and the time the sub-agents saved, if any. + +## Upgrade/Fix runs (dated) +One entry per diff-driven run (section 10): date; the changes detected (CH- items); the artifacts updated; the checks re-run; the acceptance-checklist rows re-marked. "None" otherwise. + +## Acceptance checklist +(the working records' acceptance-checklist.md, completed: its two result lines and its table, with the Evidence column) diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md new file mode 100644 index 000000000..a42744a6d --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md @@ -0,0 +1,66 @@ +# Repo profile: + +## State summary (Orient, STOP-13) +(paste orient.py's stateSummaryText (the text form of the State summary shown at STOP-13), completed with the model, the capability profile confirmed at STOP-13 or STOP-17 (D-31), and the option chosen at STOP-13) + +- Repo: , HEAD (clean | dirty: ) +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; guide/skill last changed in ; skill version +- Date: ; confirmed by the user on (STOP-2) +- User goals: + +## Build, test and CI +| # | Item | Answer | Evidence (path, symbol or command → result) | Verified? | +|---|---|---|---|---| +| P-01 | Language(s) and version(s) | | | | +| P-02 | Build and test commands (build, build tests, all, class, single test, filter, target-qualified filter) | | | ran / unverified: | +| P-03 | Test framework, runner, assertions, parameterisation, fixtures, teardown, static and runtime skip, fail idiom, collection rules | | | | +| P-04 | Test layout, naming, targets, shared test-support code | | | | +| P-05 | CI workflows, jobs → suites, matrix, network, submodules, secrets | | | | +| P-06 | Lint/format gates, static analysis, type checkers, warnings-as-errors | | | | +| P-07 | Developer and CI platforms; script runtime availability | | | | +| P-08 | Repo conventions; hand-maintained project files; synced manifests | | | | + +## SDK structure for testing +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-09 | Async model; runner async support; virtual time | | | | +| P-10 | Threading, callback queues, internal-queue drain, compile-time concurrency checks | | | | +| P-11 | Every time source (timers, timed waits, delayed callbacks, clock reads) | | | | +| P-12 | Runner parallelism and how to serialise | | | | +| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global, reachable from the SDK language: yes / no, an infra gap) | | | | +| P-14 | Internal-access mechanism and current use | | | | +| P-15 | API surface vs pseudocode, per module (rest / realtime / objects); objects: detect_liveobjects.py summary | | | | + +## Existing native test support +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-16 | Existing mocks, fakes, test doubles (capabilities, style, global?, users) | | | | +| P-17 | Existing test helpers (waits, polls, capture, log capture, assertions, factories) | | | | +| P-18 | Existing sandbox/integration support; fault-injection tooling | | | | +| P-19 | ably-common submodule (path, initialised, fixture files) | | | | +| P-20 | Existing UTS assets and their state (inspect_existing_skill.py: skills and installs, infra entry, tags, header SHAs, deviations.md) | | | | +| P-21 | Sandbox access from developer machines and CI | | | | + +## How the existing tests drive the SDK (step 1c) +| Kind | Test read (path, test name) | Pattern: create client / install double / wait / assert / clean up | +|---|---|---| +| REST request | | | +| Realtime connection state | | | +| Timer-driven behaviour | | | +| Negative assertion | | | +| Internal state | | | +| Sandbox integration | | | +| Plugin module | | | + +- Doubles that already meet helper-spec semantics: … +- Doubles that differ (and how): … +- Race-free waits vs sampling or sleeping waits: … +- Known flakiness and workarounds: … + +## API divergences per module +- rest: … +- realtime: … +- objects: … (public API exposed? implemented? plugin? typed? IDL followed?); detector recommendation: …; decision at STOP-14: … + +## Unknowns and questions for the user +- … diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md new file mode 100644 index 000000000..3dedf881b --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md @@ -0,0 +1,59 @@ +# Gap audit: uts-to- in + +- Existing skill: (installs: <.claude/skills/…, .agents/skills/… (symlink | copy)>); frontmatter version +- Repo (clean | dirty: ) +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)> +- State class (Orient): S2 | S3 | S4; mode at STOP-13: upgrade: diff-driven | upgrade: full audit | upgrade: regenerate — chosen by on +- inspect_existing_skill.py run: ; its checks are regex hints, every row below is confirmed by reading +- Capabilities: recorded D-31 vs Orient now ; delta: + +## A. Inventory +| Part | What exists (path, symbol, command → result) | Baseline result (resolver, audit, infra tests, compile) | +|---|---|---| +| SKILL.md (lines, headings, frontmatter fields) | | | +| uts-package-mapping.json (modules, tiers, notes, infra entry (or the former key `harness`) and its shape: `{root, readme, testsRoot, testCommand}` or older per-tier lists, unready data, `unsupported` / `capabilityInapplicable` / `extraTests` markers; keys the resolver could derive, e.g. file lists, a test project, a white-box list, dummy tier paths) | | | +| scripts/ (resolver, audit, scanner, others) | | | +| references/ (notes per module; placeholders) | | | +| UTS infra (root, README, Known gaps, smoke tests, self-tests, CI jobs) | | | +| UTS-derived tests (see D) | | | +| Working records | | | + +## B. Conformance +One row per guide §13 item outside the UTS infra group, in §13 order (one per acceptance-checklist row). A row (or part of one) needing a capability the SDK lacks is "n/a — capability absent: (D-31)"; one that was n/a when recorded and whose capability now exists is "missing (newly applicable)". Each row and each section-D follow-up gets an ID (GA-01, GA-02, …) used by the decision, the changelog and the Final report. Status: conforms / partial / missing / non-conforming / n/a (reason) / unverified (checked in ). Choice at STOP-15: add / skip / defer (a skipped or deferred MUST row leaves the skill not conforming). The objects half of the module-notes row comes from D-28; Creation-process rows for a skill of unknown origin: "origin unknown; this run meets it for the artifacts it changes". + +| # | Acceptance-checklist item | Level | Status | Evidence | Proposed change | Step | Size | Approval needed | Choice | Re-confirmed (STOP-8 or after Phase 6) | +|---|---|---|---|---|---|---|---|---|---|---| +| GA-01 | | MUST / SHOULD | | | | | | | | | + +## C. UTS infra +Decided per row at STOP-3, not here; G-19/G-20 for offered tiers can't be skipped or deferred. + +| Row | Capability | Quick status | Evidence | Note for Phase 2 | +|---|---|---|---|---| +| G-01 … G-20 | (from the uts-infra-design template) | | | | + +G-13 restructure by tier (when the UTS infra isn't organised by tier; MUST): move list, each file from → to with its namespace or package change and any split by tier (e.g. a file mixing unit factories and every-tier waits); size; infra tests run green before and after; moving with no logic change keeps the preserve list's behaviour (not a rewrite); UTS-derived tests that reference the old namespace become regenerate candidates. + +| From | To | Namespace or package change | Split? | Notes | +|---|---|---|---|---| + +## D. UTS-derived tests +| # | Module × tier (or "outside mapped tiers") | Files | Tags | Tag style | Header SHAs (vs clone HEAD) | No SHA / links main | deviations.md and other deviation records | Compile | Audit | Follow-up proposed | Choice | +|---|---|---|---|---|---|---|---|---|---|---|---| + +Lessons carried into the skill (the tests are evidence, not a constraint): → ; "none" otherwise. + +## E. LiveObjects (decided at STOP-14) +detect_liveobjects.py → , ; decision D-28: full | placeholder | none, translate-only | evaluate; SDK-blocked tiers (12.5): … ; objects items implied: … + +## F. Preserve list +| What is kept as is | Where | Why it conforms | +|---|---|---| + +## G. Decision at STOP-15 +Chosen by on : add …; skip …; defer … (GA- and CH- items); or regenerate from scratch instead. Recorded as D-29. + +## H. Changes since the recorded run (diff-driven Upgrade/Fix, section 10) +| # | What changed (guide / UTS docs / helper specs / corpus / UTS infra / SDK) | Evidence (diff paths) | Artifacts to update | Tests to regenerate | Choice | +|---|---|---|---|---|---| +| CH-01 | | | | | add / skip / defer | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md new file mode 100644 index 000000000..0f4ca50bc --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md @@ -0,0 +1,78 @@ +# Infra design: @ + +- Run mode (D-27): …; upgrade: rows seeded from skill-gap-audit.md section C on , each re-checked per 4.2 +- LiveObjects (D-28): full | placeholder | none +- UTS infrastructure plan (STOP-17 or STOP-13): ; side: (; provisional; re-confirmed at STOP-3: ); differences from the outline: +- Capabilities (D-31): rest …; realtime …; side: …; hooks reachable from the SDK language (P-13): yes | no; rows and tiers needing an absent capability are "n/a — capability absent: (D-31)", not gaps (13.8) + +## Gap table +| # | Capability | Guide | Status | Evidence | Approach (reuse / wrap / extend / build / n/a) | Builds on | Blocks (module/tier) | Production change? | Size | Acceptance check | +|---|---|---|---|---|---|---|---|---|---|---| +| G-01 | WebSocket transport factory hook | 2.1 | present / partial / missing / n/a | | | | realtime/unit, objects/unit | yes / no | S/M/L | | +| G-02 | HTTP client hook | 2.1 | | | | | rest/unit | | | | +| G-03 | Clock hook (timers, timed waits, async-runtime timers) | 2.1 | | | | | | | | | +| G-04 | Network-monitor hook | 2.1 | | | | | | | | | +| G-05 | Randomness hooks (jitter, host shuffle) | 2.1 | | | | | | | | | +| G-06 | MockHttpClient (+ legacy queue_*) | 2.2, mock_http.md | | | | | | | | | +| G-07 | MockWebSocket (+ alternative API, undocumented members, raw frames, ping) | 2.2, mock_websocket.md | | | | | | | | | +| G-08 | MockVCDiff encoder/decoders | 2.2, mock_vcdiff.md | | | | | | | | | +| G-09 | MockNetworkListener | 2.2 | | | | | | | | | +| G-10 | standard_test_pool.md implementation; plugin options builder (only if D-28 = full; else n/a (D-28)) | 2.2, standard_test_pool.md | | | | | objects/unit | | | | +| G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | +| G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | +| G-13 | Placement and the tier layout (MUST: root, unit/, integration/, integration/proxy/; infra tests in their own tier tree; names follow folders); shared / port-only / module helpers; fixture-helper scope documented in the module notes. Upgrade/Fix, not tier-shaped: restructure by tier, with the move list (Placement) | 2.2 | | | | | | | | | +| G-14 | Infra README with Known gaps; mapping `infra` entry (root, readme, testsRoot, testCommand with {tier}; sources derived by the resolver) | 3.1, 3.2 | | | | | | | | | +| G-15 | SandboxApp; ably-common | 2.3 | | | | | */integration | | | | +| G-16 | ProxyManager (pinned, every OS); ProxySession; rule builders; proxy auth | 2.4 | | | | | */proxy | | | | +| G-17 | UTS test target; per-tier selection and filters; one CI home per suite | 2.5 | | | | | | | | | +| G-18 | Concurrency, time, parallelism and visibility models known | 2.6 | | | | | | | | | +| G-19 | Per-tier smoke tests (permanent, in CI, outside generated dirs) | 2.7 | | | | | | | | | +| G-20 | Helper self-tests (common, unit, fake clock, REST, objects, sandbox, proxy) | 2.7 | | | | | | | | | + +## Helper-spec symbol maps (one table per helper spec) +### mock_http.md +| Spec symbol (declared or corpus-only) | Native type.member | Approach | Builds on | Notes | +|---|---|---|---|---| +### mock_websocket.md +### mock_vcdiff.md +### standard_test_pool.md +### MockNetworkListener (no helper spec) + +## Wait and assertion helpers +| Helper | Native symbol | Approach | Guarantees checked by | +|---|---|---|---| + +## Sandbox and proxy +- Sandbox provisioning: …; client targeting (endpoint or hosts): …; suite fixture mapping: … +- Proxy: version …; binaries per OS …; local override …; auth through the proxy …; platform gating … + +## Hook proposals (each needs STOP-4 approval) +| Hook | Existing seams checked (file:line, usable?) | Infra-side alternatives considered (why each falls short) | Specs that need it | API | Default behaviour | Visibility | Files (expected lines) | Installed how | Per-client? | Time sources covered (clock hook) | Approved on | +|---|---|---|---|---|---|---|---|---|---|---|---| + +## Placement +- Infra root: (name SHOULD be infra/); tier directories: , unit/, integration/, integration/proxy/ (MUST); namespaces or packages: +- Infra-tests tree (mapping testsRoot): /tests/, excluded from source derivation> +- Shared test-support: ; port-only UTS infra: ; module helpers: +- Infra tests per tier: + +| Infra file (existing or planned) | Tiers that use it | Directory | Namespace or package | Moved from (Upgrade/Fix) | +|---|---|---|---|---| + +## Tier feasibility +| Module | unit | integration | proxy | +|---|---|---|---| +| rest | ready / after G-.. / not possible: / n/a / n/a — capability absent: (D-31) | | | +| realtime | | | | +| objects | per D-28: full → as above; placeholder → not possible: placeholder notes; none → n/a (D-28) | | | + +## Decision at STOP-3 +(a) build now: | (b) plan only | (c) scope to ready tiers — chosen by on ; design changes requested: … +Upgrade/Fix: per row build / plan (defer) / skip: …; G-19/G-20 for every tier the skill offers can't be skipped or deferred + +## Build log (Phase 3) / plan (option b) +| Order | Row | Work | Acceptance check | Result | Date | +|---|---|---|---|---|---| + +## Confirmation at STOP-7 +Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier matrix: …; confirmed on diff --git a/uts/skills/uts-to-lang-skill-creator/references/asking.md b/uts/skills/uts-to-lang-skill-creator/references/asking.md new file mode 100644 index 000000000..88ffb39e2 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/asking.md @@ -0,0 +1,132 @@ +# Asking at a stop point + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). + +Every stop point (SKILL.md [2.3](../SKILL.md#23-stop-points)) is a question the user has to find, understand and answer. Ask each one the same way, so it stands out in a long run and every answer is unambiguous. This page is the format; the STOP table and the section each stop links to say what goes in it. + +## The question + +One decision per question, and one numbered option list per question. A second decision is a second question, never a lettered sub-question or a second list under the same heading. + +1. **The heading:** the question in bold, numbered in the order you ask it, with the stop ID at the end in italics for the decision log: `` **Question 1: How should this run work on `.claude/skills/uts-to-csharp`?** *(STOP-13)* ``. Number from 1 in each message. +2. **Why:** one short sentence, in plain words. The only exception: when the run assumes something settled without asking (a client setup detected from the UTS code, or taken from the last run's records), add a second plain sentence that names it. +3. **Evidence,** only what the decision needs: a few short bullets in plain words. Long lists stay in the records or the JSON. +4. **The options,** numbered, one line each: `1.