From 6cfeb3fd861ec6a212f7bd7c3b345a870ebbd6d2 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 12:37:05 +0530 Subject: [PATCH 01/33] UTS: add guide to writing uts-to- translator skills Adds a guide for SDK engineers on what a per-language Claude Code skill that translates UTS specs into native tests must contain: harness prerequisites, skill anatomy, the two-phase workflow, translation rules, a pseudocode construct catalogue, evaluation and deviations, resolver and audit tooling, re-sync with spec changes, verification, report format, lessons learned and a checklist. --- .../writing-translator-skills.md | 893 ++++++++++++++++++ 1 file changed, 893 insertions(+) create mode 100644 uts/docs/translator-skills/writing-translator-skills.md diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md new file mode 100644 index 000000000..4237fcb66 --- /dev/null +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -0,0 +1,893 @@ +# Writing UTS Translator Skills + +This guide explains how to build a per-language Claude Code skill — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You don't need to read these skills; everything required is in this guide and the UTS docs. + +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [Generating a UTS Translator Skill](generating-a-translator-skill.md); this guide remains the authority on what the skill must contain. + +**Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. + +- **Easy to use.** One command (`/uts-to- `) and four questions; the skill does the rest. +- **Context-complete.** Every run gathers the full context: the spec, the translation manual, the sources and README of the UTS test infrastructure (the **harness**), the module notes and the resolver output. A good test doesn't depend on anyone's memory. +- **Tests are reproducible artifacts.** The UTS spec plus the skill is the source of truth; the generated test is an output. When a translation is wrong or the spec changes, regenerate it ([section 9](#9-keeping-in-sync-with-spec-changes)) rather than hand-maintain it. +- **Fix the cause, then regenerate.** Fix the skill, the module notes or the harness, never only the generated file. Only DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders and `deviations.md` entries survive regeneration. + +**How to read this guide.** Requirements use three levels: + +- **MUST**: required for a skill that conforms to this guide. Most MUSTs protect translation fidelity (without them a skill produces tests that silently check less than the spec); the rest protect repeatability and reviewability. +- **SHOULD**: strongly recommended. Both existing skills learned these the hard way. +- **MAY**: optional, useful in some SDKs. + +Facts about the existing skills and SDKs are **as of 2026-10-07**: spec commit `12540dcf`; ably-cocoa at `b5074d9b` and ably-java at `0ff24017` (the checkouts reviewed). Remote `main`, fetched the same day (ably-cocoa `2072e0cd`, ably-java `8b7d3f5f`), has no changes to either skill or UTS harness since those commits. They illustrate patterns, and some may since have been fixed. + +## Contents + +1. [Purpose and scope](#1-purpose-and-scope) +2. [Prerequisites in the SDK repo](#2-prerequisites-in-the-sdk-repo) +3. [Skill anatomy](#3-skill-anatomy) +4. [The workflow the skill must implement](#4-the-workflow-the-skill-must-implement) +5. [Translation rules](#5-translation-rules) +6. [Pseudocode construct catalogue](#6-pseudocode-construct-catalogue) +7. [Evaluation and deviations](#7-evaluation-and-deviations) +8. [Deterministic tooling](#8-deterministic-tooling) +9. [Keeping in sync with spec changes](#9-keeping-in-sync-with-spec-changes) +10. [Verification and CI](#10-verification-and-ci) +11. [Final report format](#11-final-report-format) +12. [Lessons learned](#12-lessons-learned) +13. [Checklist for a new `uts-to-` skill](#13-checklist-for-a-new-uts-to-lang-skill) +- [Appendix: Existing skills (background and patterns to avoid)](#appendix-existing-skills-background-and-patterns-to-avoid) + +--- + +## 1. Purpose and scope + +### What a translator skill is + +A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that Claude Code follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. + +### How it relates to the other UTS docs + +| Document | Owns | +|---|---| +| [`uts/README.md`](../../README.md) | Pseudocode conventions: identifier naming, absent values, property access, enum values, `poll_until_success`, `process_pending_events()`, language-inapplicable inputs | +| [`writing-test-specs.md`](../writing-test-specs.md) | The pseudocode reference: Test IDs, mock patterns, `AWAIT_STATE`, `poll_until`, record-and-verify, error pattern, unique channel names | +| [`writing-derived-tests.md`](../writing-derived-tests.md) | **Translation and evaluation semantics.** Faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | +| [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | +| **This guide** | How to package all of the above into a repeatable, scripted Claude Code skill for one SDK | + +`writing-derived-tests.md` is the **source of truth for semantics**. This guide never redefines what a faithful translation is, or how a deviation is classified and recorded; it says how a skill enforces those rules. If the two ever disagree, `writing-derived-tests.md` wins. + +### What the skill owns, and what it defers + +| The skill owns (harness mechanics) | The skill defers (semantics) | +|---|---| +| Resolving where each test file goes and what it is called | What counts as faithful translation → `writing-derived-tests.md` Phase 1 | +| The pseudocode → SDK/test-framework mapping table | The three causes of a failure and the decision tree → `writing-derived-tests.md` Phase 2 | +| Per-module API notes (ably-js-shaped pseudocode → your SDK's API) | The `deviations.md` sections and entry fields → `writing-derived-tests.md` "Recording deviations" | +| The language rendering of each test pattern (skip idiom, fail-fast idiom) | `poll_until` / `poll_until_success` reference definitions → `writing-test-specs.md` | +| Build, run, lint and audit commands | Pseudocode meaning → `uts/README.md`, `writing-test-specs.md` | + +A skill **SHOULD** keep only the language rendering of a rule and link to the source for its meaning. Inline copies of semantic rules drift. Both existing skills replaced their inline `deviations.md` format with a pointer to the manual for this reason, although they still inline the decision tree and test patterns. + +--- + +## 2. Prerequisites in the SDK repo + +A skill cannot make translation mechanical if the harness it translates onto does not exist. Build these **before** writing the skill. In both existing SDKs this took several iterations, and the harness co-evolved with the first real translation run. + +### 2.1 SDK test hooks (MUST) + +The unit tier needs injection points in production code: + +| Hook | Purpose | Examples | +|---|---|---| +| WebSocket transport factory | Install `MockWebSocket` | ably-java `DebugOptions.webSocketEngineFactory`; ably-cocoa `testOptions.transportFactory` | +| HTTP client/executor | Install `MockHttpClient` | ably-java `DebugOptions.httpEngine`; ably-cocoa `testOptions.httpExecutor` | +| Clock / timer source | `enable_fake_timers()`, `ADVANCE_TIME` | ably-java `DebugOptions.clock` (a `Clock` abstraction retrofitted across `Auth`, `Presence`, `Hosts`, `WebSocketTransport`); ably-cocoa `testOptions.timeProvider` | +| Reachability / network monitor | Keep the unit tier hermetic; drive `MockNetworkListener` (`realtime/unit/connection/network_change_test.md`) | ably-cocoa `testOptions.reachabilityClass` | +| Randomness | Jitter (`get_jitter_coefficient()`, `backoff_jitter_test.md`), fallback-host shuffle (RTN17j, `fallback_hosts_test.md`) | ably-cocoa `testOptions.jitterCoefficientGenerator`, `testOptions.shuffleArray` | + +Mocks and the fake clock are injected **at client construction**, so the harness must install them before the client is created. + +**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when `FakeClock.waitOn` did a real timed wait, commit `c4502c67`.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. + +If a hook doesn't exist, add it to the SDK first. Without these hooks, the unit tier is impossible. + +### 2.2 A shared test library implementing the helper specs (MUST) + +Implement each helper spec **once**, as a native library that every derived test calls. Derived tests never hand-roll mock setup, protocol messages or serials. + +| Helper spec | Must implement | Watch for | +|---|---|---| +| `rest/unit/helpers/mock_http.md` | `MockHttpClient`, `PendingConnection`, `PendingRequest`, `install_mock` / `uninstall_mock`, `reset()`, connection-level vs request-level failures | The injection mechanism is implementation-specific. The corpus still uses a legacy API that the helper spec doesn't define and `writing-test-specs.md` "Common Mistakes to Avoid" deprecates: the queue calls (`queue_response`, `queue_responses`, `queue_response_for_host`, `queue_response_for_url`, `queue_timeout`, `queue_delayed_response`; in `rest/unit/fallback.md`, `request.md`, `rest_client.md`) and the mock-owned request log `mock_http.captured_requests` (in `fallback.md`, `request_endpoint.md`, `request.md`, `rest_client.md`): implement them, or map them onto `onRequest` handlers and a local capture list in the skill | +| `realtime/unit/helpers/mock_websocket.md` | `MockWebSocket`, handler and await styles, raw-frame hooks (`onTextDataFrame` / `onBinaryDataFrame`), `respond_with_*`, `send_to_client`, `send_to_client_and_close`, `simulate_disconnect`, ping frames, templates (`CONNECTED_MESSAGE`, `CLOSED_MESSAGE`, `DISCONNECTED_MESSAGE`, `HEARTBEAT_MESSAGE`, `ERROR_MESSAGE(code, message)`, `PING_MESSAGE(id)`) | See the notes below the table | +| `realtime/unit/helpers/mock_vcdiff.md` | `MockVCDiffEncoder`, `MockVCDiffDecoder`, `FailingMockVCDiffDecoder` | Deterministic, not a real codec | +| `objects/helpers/standard_test_pool.md` | The standard pool, `STANDARD_POOL_OBJECTS`, `setup_synced_channel` (and the no-ack variant), every `build_*` builder, serial helpers (`ack_serial`, `remote_serial`, `below_ack_serial`, `SITE_CODE`, `POOL_SERIAL`), quiescence helper, `provision_objects_via_rest` | Serials compare as strings, so hand-written literals sort wrongly. REST provisioning uses the V2 format with the singular `/object` path, and any REST client used must be closed | +| *(no helper spec)* | `MockNetworkListener` with `simulate_network_lost()` / `simulate_network_available()`, installed via `install_mock` | Used by `realtime/unit/connection/network_change_test.md`; needs the network-monitor hook (2.1) | + +`mock_websocket.md` notes: + +- `close()` **must** call `onClose` **asynchronously**; `respond_with_success` **must** complete the connection **first**, then deliver CONNECTED asynchronously. +- `mock_ws.active_connection` is used in the helper spec's own examples but isn't declared in its interface. `connect_attempts`, `last_connect_url` and `create_mock_websocket()` aren't declared in any helper spec (`connect_attempts` is used in the corpus and in a `writing-test-specs.md` example; the other two only in the corpus). Support them anyway. +- The corpus also uses an alternative API: `on_connect(respond_with:)`, `on_message(action:, respond_with:)`, `on_client_message(…)`, `await_client_message(action:)`, `respond_with_connected()`, `close_from_server()`, and `send_to_client_raw(json)`. The last sends an untyped frame (`forwards_compatibility_test.md`), which even a typed SDK's harness must be able to produce. +- State, in the harness README and the skill, whether handler and await styles can be mixed for one event type, and on which thread handlers run. (ably-java's can't be mixed: a set handler consumes the event, so the matching `await_*` never returns; handlers run synchronously on the SDK's thread, so they must not block or await.) + +The library also needs: + +- **Wait helpers:** an event-latched `AWAIT_STATE`, value-returning `poll_until`, `poll_until_success`, `process_pending_events()`, and (for virtual-time runners) a wall-clock timeout wrapper. See [section 5](#5-translation-rules). +- **Thread-safe capture** for lists appended from SDK callbacks (ably-java `CopyOnWriteArrayList`; ably-cocoa `Captured`). +- **A log sink** for "an error is logged" assertions. +- **An in-order subsequence assertion** for `CONTAINS_IN_ORDER`. +- **Caller-attributed failures:** helpers that time out or receive an unexpected error report the failure at the calling test line (Swift `sourceLocation:`, pytest `__tracebackhide__`, C# caller-info attributes), not inside the helper. +- **Harness smoke tests (SHOULD)**, one per tier, as the acceptance gate while the harness is being built. They are structural examples only: not spec-derived, no `UTS:` tag, and the skill must never copy their shape. They MAY be retired once spec-derived tests cover each tier: ably-java keeps them in `:uts`; ably-cocoa removed its own (commit `58fa0f5e`). + +**Placement.** Separate **shared** test-support code (used by both the SDK's native tests and the UTS ports, importing no test framework) from **port-only** harness code (doubles and fixtures used only by UTS ports); keep port-only code with the ports. Module-specific helpers (for example the `standard_test_pool.md` implementation) live next to that module's suites; the shared library stays module-agnostic. For a plugin-backed module such as LiveObjects, provide a client-options builder in the module helpers that registers the plugin, so tests never wire it by hand. + +**Document each fixture helper's scope** in the module notes: which client messages it auto-answers, which spec branches it doesn't wire (a DETACH reply, fake timers), the local pattern tests use for those gaps, and any option tweaks the gap needs. (ably-java: the fake-clock variant of `setupSyncedChannel` needs a large `maxIdleInterval`, or virtual-time advances trip the idle timer.) + +**Reuse before you build.** If the SDK's native tests already have transport doubles, a fake clock or sandbox helpers, assess whether to reuse, wrap or extend them against the helper specs' semantics before writing new ones; never change their behaviour for the native tests that use them. + +**Build the harness to look like the pseudocode.** Callback-style mocks with the spec's handler names, and a client factory that mirrors `Realtime(options: …)`, turn translation into transliteration. ably-java's first generation run replaced a "wrap every callback in a coroutine" rule with callback-style mocks for exactly this reason. + +### 2.3 Sandbox provisioning and fixtures (MUST for integration tiers) + +- A `SandboxApp` helper that creates an app from `ably-common/test-resources/test-app-setup.json` against `sandbox.realtime.ably-nonprod.net` and deletes it best-effort. **Retry only idempotent requests** (fetching the app setup); never retry `POST /apps`, which can create duplicate apps when the response is lost. Check the HTTP status before parsing the body. +- `ably-common` as a submodule. It also provides `msgpack_test_fixtures.json` (`rest/unit/encoding/msgpack_interop.md`) and `encoding.json` (`load_fixtures`, `rest/unit/encoding/message_encoding.md`). +- A way to point the client at the sandbox. The docs use `endpoint: "nonprod:sandbox"`; an SDK without `endpoint` sets `realtimeHost`/`restHost` instead. Explicit hosts disable fallbacks: REC2c6 for `realtimeHost`/`restHost`, REC2c2 for an `endpoint` hostname. + +### 2.4 The proxy (MUST for the proxy tier) + +- A `ProxyManager` that downloads, caches and starts [`ably/uts-proxy`](https://github.com/ably/uts-proxy). Pin the version, and check that the pinned release publishes a binary for every developer OS (its GitHub release assets). An environment override pointing at a locally built binary (`UTS_PROXY_LOCAL_PATH` in both existing harnesses; ably-java also accepts `-Duts.proxy.localPath`) is useful; keep it a configuration override, not a test gate. +- A native `ProxySession` implementing [`proxy.md`](../proxy.md): `create_proxy_session`, `add_rules`, `trigger_action`, `get_log`, `close`, plus rule-builder helpers and a typed event log. +- **The `action` inside a frame `match` (`ws_frame_to_client` / `ws_frame_to_server`) must be a JSON string**: a name (`"ATTACHED"`) or a numeric string (`"11"`), never a number (the proxy rejects a number with HTTP 400). uts-proxy v0.3.0 resolves names only up to `AUTH` (17), so use numeric strings for `OBJECT` (`"19"`), `OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`); an unresolvable name silently never matches. This applies only to `match.action`: a rule's own `action` is an object (`{"type": "suppress"}`), and the `action` inside an injected message stays a number. Rule builders should take an integer and stringify it, as both existing harnesses do. +- Proxy tests always use JSON: the proxy supports text frames only (`integration-testing.md`, "Protocol Variants"). +- **Auth through the proxy.** The proxy hop is plain HTTP/WS (`tls: false`), and RSA1 and RSC18 say Basic Auth over HTTP without TLS results in an error (`writing-test-specs.md` "RSC18 only applies to Basic auth configurations": checked at construction). `proxy.md`, `integration-testing.md` and `writing-test-specs.md` nonetheless show `key: api_key` with `tls: false`. In practice, ably-cocoa authenticates proxied clients with a locally signed `TokenRequest`; ably-java does the same where the spec observes auth (`AuthReauthTest`), but its objects proxy suite (`ObjectsFaultsTest`) uses `key` over the plain hop, which ably-java permits. Token auth is the option valid under RSA1 for every SDK; whichever you use, say why in a comment. +- Platform gating where the proxy can't run (ably-cocoa wraps proxy files in `#if os(macOS)`). + +### 2.5 Build and CI wiring (MUST) + +- Test tasks or filters that select each tier (unit, direct integration, proxy), each with a single-class/single-test filter. +- **Every suite runs in exactly one CI job per platform it targets, and nothing runs green without executing** (MUST). Splitting by tier (unit in the fast check job; integration and proxy in a networked job) is recommended (SHOULD): ably-java does this, while ably-cocoa runs all UTS tiers from one job definition, one job per platform (an iOS / tvOS / macOS matrix; proxy suites compile only on macOS). +- Integration jobs need network access, the `ably-common` submodule and the proxy binary. + +### 2.6 Understand the SDK's concurrency model (MUST) + +Most flakiness in both existing SDKs came from concurrency, not from translation. Before writing the skill, know: + +- Which thread or queue SDK callbacks fire on, and whether listeners can run concurrently with the test body. +- How to drain the SDK's internal work queue (this becomes `process_pending_events()`). One zero-delay yield (e.g. one `await asyncio.sleep(0)`) may not drain chained callbacks; loop until the queue is empty if you can observe it. *Divergence from `writing-derived-tests.md`* ("No real timers in unit tests"), which says one event-loop yield suffices for a negative assertion: that holds for a single JavaScript macrotask queue, but not where the SDK chains work across queues, threads or loop iterations (both existing SDKs needed explicit queue drains). Use one yield only where you have shown it drains everything. +- Whether the test runner virtualises time (kotlinx `runTest` does; this breaks real-network timeouts), and, for async runners, the plugin mode and event-loop scope that suite-level fixtures need (e.g. pytest-asyncio). +- **Whether the runner runs tests in parallel** (xUnit does by default; pytest-xdist and NUnit `[Parallelizable]` when enabled; Swift Testing unless `.serialized`). If any injected mock, fake clock or log sink is process-global, serialise the UTS suites or make the hooks per-client, and say which in the skill. ably-cocoa marks its core realtime/rest and integration suites `@Suite(.serialized)`; its objects unit suites, which inject nothing process-global, are plain parallel `struct`s. +- Whether the language enforces concurrency safety at compile time (Swift 6 `Sendable` drove ably-cocoa's capture types). +- Where white-box unit specs can see internals (Swift `@testable import`, Objective-C private headers, C# `InternalsVisibleTo`, Kotlin `internal` within the owning module). This decides **where tests live**. + +--- + +## 3. Skill anatomy + +### 3.1 Layout + +``` +.claude/skills/uts-to-/ +├── SKILL.md # the procedure +├── uts-package-mapping.json # spec module → target dir/package per tier, + notes pointer +├── scripts/ +│ ├── resolve_uts.py # deterministic selection +│ ├── audit_translation.py # deterministic faithfulness audit +│ └── scan_constructs.py # corpus construct scanner (SHOULD, section 6) +└── references/ + └── -mapping.md # module notes (e.g. objects-mapping.md) +``` + +| File | Purpose | Level | +|---|---|---| +| `SKILL.md` | The two-phase procedure, construct table, a file template per tier, evaluation patterns, tier wiring. Keep it **module-generic**: the core realtime/rest mapping may live here; plugin or typed-module specifics go in notes | MUST | +| `uts-package-mapping.json` | Maps each source module (`rest`, `realtime`, `objects`) and tier (`unit`, `integration`, `proxy`) to one target directory, plus an optional `notes` path | MUST | +| `scripts/resolve_uts.py` | Validates the module dir, reads the mapping, lists specs and derives file/class/package names; prints one JSON object | MUST | +| `scripts/audit_translation.py` | Compares one spec file with its generated test file; prints one JSON object | MUST | +| `scripts/scan_constructs.py` | Lists the uppercase keywords and snake_case calls in a module's `pseudo` fences, so unmapped constructs are found mechanically ([section 6](#6-pseudocode-construct-catalogue)) | SHOULD | +| `references/-mapping.md` | Maps ably-js-shaped pseudocode to your SDK's API for one module; may override parts of the generic flow | MUST wherever the SDK surface diverges from the pseudocode (in practice `objects` for every typed SDK) | + +Both existing skills use Python for scripts. Any language works, but the scripts must run on every developer platform (see [8.1](#81-resolver-resolve_utspy-must) and [section 10](#10-verification-and-ci)). + +Keep two documents with separate roles: the test library's README describes **what exists** (helpers, seams, layout); `SKILL.md` describes **how to author** tests. Refer to the README by its full repo path, so it can't be confused with the spec repo's `uts/README.md`. The harness README SHOULD cover: the SDK hooks used, each helper-spec symbol → harness symbol, the wait helpers and their default timeouts, the threading and time model, how to run each tier, and known gaps against the helper specs. + +**The notes are a map, not an authority.** Where the notes, the spec's IDL and the SDK disagree about the SDK's API, the SDK source is the ground truth: fix the notes. + +### 3.2 The mapping file + +Each tier value is **one path**, relative to the repo or to a declared root (Swift's `testRoot`), never machine-absolute. Derive everything else (namespace, package, build target) from it in the resolver, never by hand. + +```json +{ + "_comment": "Maps each UTS module to its target test dir per tier. Used by scripts/resolve_uts.py.", + "packages": { + "realtime": { "unit": "/unit/realtime", "integration": "/integration/standard/realtime", "proxy": "/integration/proxy/realtime" }, + "objects": { "unit": "...", "integration": "...", "proxy": "...", "notes": "references/objects-mapping.md" } + } +} +``` + +- **Make every derived namespace unique**: keep a module segment after the tier (`unit/realtime`, never bare `unit`), or use a module-specific prefix (ably-java's objects tests use `io.ably.lib.liveobjects.uts.unit`). +- **`notes`** is relative to the skill directory, not the repo root. +- **Language-specific derivations live in the resolver.** Swift has no package, so the directory is the full story. Kotlin derives `package` from the path after `src/test/kotlin/` and the Gradle module from the first segment. A C# resolver might map `tests/IO.Ably.Tests.Uts/Unit/Realtime` to namespace `IO.Ably.Tests.Uts.Unit.Realtime` (PascalCase segments) and to the nearest ancestor `.csproj`; a Python resolver derives a dotted package path. +- **Hand-maintained entries.** `--create` scaffolds only the repo's default layout. A module whose tests live elsewhere (ably-java's `objects`, in `:liveobjects`) is hand-maintained: the skill MUST say which entries are, and must not offer `--create` to "fix" them. +- You MAY store per-module defaults as data, e.g. `"evaluate": true` or a list of white-box specs, instead of prose in the notes. + +### 3.3 Frontmatter and arguments + +```yaml +--- +name: uts-to- +description: "Translate Ably UTS (Universal Test Suite) pseudocode specs into runnable tests in . Use when asked to translate, port, derive, generate or re-sync tests from a UTS spec or module (uts/rest, uts/realtime, uts/objects), to evaluate UTS-derived tests against the SDK, or to check UTS coverage. Takes a UTS module directory, validates it, resolves target test dirs, lets you pick a tier (unit/integration/proxy) and specs, then derives one test per spec Test ID. Usage: /uts-to- " +argument-hint: +allowed-tools: Bash, Read, Edit, Write +--- +``` + +- **`name`** SHOULD be `uts-to-`. (`uts-to-kotlin` omits `name` and relies on the directory name; that works but is less explicit.) +- **`description`** SHOULD be "pushy": list the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to ", "re-sync UTS tests"). Both existing skills' descriptions are procedural, which suits explicit `/` invocation but triggers poorly otherwise. +- **Argument:** a single UTS **module** directory, directly under `uts/`. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. If the argument is empty, the skill MUST print the usage line and stop. +- **`allowed-tools`:** both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-local-clone-vs-fetching-main)). +- **Placeholders, not personal paths,** in examples and usage text: `/uts/objects`. + +### 3.4 Recommended outlines (SHOULD) + +Use these shapes (both existing skills converged on them independently of language): + +``` +SKILL.md + Required reading (local clone, 9.4) + Phase 1, selection: usage guard; step 0; A resolve; B mapping; C tier; D specs; E translate vs evaluate + Phase 2, per spec: 1 read spec + notes; 2 output path; 3 read harness + reference test; + 4 generate (construct table, client construction, mocks, timers, assertions, naming, + file template per tier); 5 compile; 6 run + diagnose (evaluate only); 7 audit + review + Integration tiers: direct-sandbox wiring; proxy wiring (session, connect through proxy, auth, + rule builders, event log); file templates + Final report (section 11) +``` + +A module notes file should cover: + +1. **Source of truth and runtime status:** the spec's IDL that the notes apply, including any typed-SDK variant (for objects, the proposed `RTTS` points (RTTS1–RTTS11 at the branch tip, `c5cc6b8a`) that both existing skills' notes follow; as of `12540dcf` they are on the unmerged `feature/liveobjects-cross-sdk-types-spec` branch, not in `objects-features.md`), and whether the module is implemented (whether evaluate mode is possible). +2. **Layers:** spec names that denote several things (e.g. `LiveMap` as a creation value type, a public view and an internal node). +3. **Entry point and setup:** plugin registration, channel modes. +4. **Async model:** how `AWAIT`, deferred futures and errors render. +5. **Type mapping:** polymorphic spec objects → typed views (which conversions throw and which degrade to null); write values → union/wrapper types; enums and string tags → constants; numeric types. +6. **Mutations, subscriptions and events.** +7. **Messages and errors:** codes, nested `error.cause`, timestamp units. +8. **Internal access:** the ladder ([5.9](#59-internal-access-white-box-unit-specs)), the list of white-box specs, the placement boundary. +9. **Helper-spec coverage table:** every helper-spec symbol → native file and symbol, with each fixture helper's scope (2.2), sanctioned shape differences ("use the pattern; don't add the shorthand") and sanctioned stand-ins ([7.2](#72-harness-stand-ins-are-not-deviations)). +10. **Overrides of the generic flow:** reading list, naming, deviations location, expected audit shortfalls. +11. **Integration helpers.** +12. **A worked example** naming each mechanical rewrite, and a **symbol index**. +13. **Shape-deviation vocabulary** (`S-1…S-n`), if any. + +A generated test file: + +``` +header: "Derived from uts/ at ably/specification@", tier, disclosures + (harness stand-ins, deviations); integration/proxy: the corresponding unit spec +imports; suite/class (serialised if hooks are process-global) +per test: UTS: tag → test attribute → name with spec point → + // Setup … // Test Steps … // Assertions … (spec comments verbatim) → teardown via scope/hook +file-local helpers: before the first test or in a separate file (the audit attributes anything + after the last tag to the last test; see the audit contract, 8.2) +``` + +--- + +## 4. The workflow the skill must implement + +The workflow has two phases. Implement both. + +### Phase 1: selection (steps 0 and A–E) + +| Step | What happens | Interaction | +|---|---|---| +| **0. Required reading** | Read `writing-derived-tests.md` and the "Pseudocode Conventions" section of `uts/README.md` once per run (plus `proxy.md` for the proxy tier; in evaluate mode, the features specs as needed), from the same spec clone as the module ([9.4](#94-local-clone-vs-fetching-main)). Read `writing-test-specs.md` whenever the construct table defers a meaning to it. Don't rely on memory or inlined summaries | — | +| **A. Resolve the module** | Run `resolve_uts.py ""`. If `ok` is false, relay `message` and stop. Treat the output as the **single source of truth**: never recompute a path or name by hand | Stop on error | +| **B. Confirm or create the mapping** | If mapped, show each tier's target dir verbatim and ask the user to confirm. If unmapped, ask for the target base name (default: the source module name; suggest a rename only when the SDK uses different terminology) and run `resolve_uts.py --create `. If the user says an entry is wrong, re-run `--create ` to overwrite it (keeping its `notes`), unless it is hand-maintained (3.2), in which case edit it by hand | **Ask** | +| **C. Choose the tier** | Offer only tiers that are present. Where the mapping records a tier as not ready (per-module data, 3.2), refuse it and name what blocks it. The tier fixes the target dir, the spec list **and** the translation flow (mocked / sandbox / proxy). Don't re-detect it per spec | **Ask** | +| **D. Choose specs** | All specs in the tier, or a subset | **Ask** | +| **E. Translate-only, or translate and evaluate?** | Translate-only: generate, compile, and run the step 7 audit and review, but don't run the tests (the SDK feature may not exist yet). Translate and evaluate: also run and diagnose. If unsure whether the implementation exists, ask rather than guess | **Ask** | + +If the module's notes file exists but is only a placeholder, the skill MUST tell the user the module's mapping hasn't been authored and treat the module as not yet translatable, rather than guess a mapping. + +### Phase 2: per spec (steps 1–7) + +**Batching (SHOULD):** run steps 1–4 for every selected spec, then step 5 once for the module, then steps 6–7 per file. + +1. **Read the spec and the notes.** Read the module notes first (once per run); where they override the generic flow (reading list, internal-access ladder, naming, deviations location), the notes win. Then identify per spec: test cases and IDs, protocol (WS or HTTP), timer use, `## Protocol Variants`, and any delegating, fixture-driven or ID-less structure. +2. **Determine the output path.** Use the resolver's `targetDir` and class name. Flatten spec sub-directories. If a suitable suite already exists, add to it rather than create a duplicate. +3. **Read the harness for exact signatures.** Read the test library's README, then **all** helper sources for the tier, before generating code. Guessed signatures are the most common compile failure. `SKILL.md` SHOULD also name one reviewed, spec-derived test file per tier to read first (ably-cocoa: `ChannelHistoryTests.swift` for direct sandbox, `AuthReauthTests.swift` for proxy; ably-java: `ChannelHistoryTest`, `ObjectsLifecycleTest`). Never point at smoke tests. +4. **Generate** using the rules in [section 5](#5-translation-rules) and the table in [section 6](#6-pseudocode-construct-catalogue). A construct the skill can't map is a **stop-and-ask** condition (section 6). +5. **Compile once per module.** Fix and recompile until clean. +6. **Run (evaluate mode only).** Run each class by filter, then diagnose per [section 7](#7-evaluation-and-deviations). +7. **Scripted review (both modes).** Run `audit_translation.py ` using the resolver's paths, then walk the review checklist ([8.3](#83-review-checklist-must-after-the-audit)). Any fix means re-audit, recompile and, in evaluate mode, re-run. + +--- + +## 5. Translation rules + +Every rule below is a MUST unless marked otherwise. + +**When a generated test is wrong, fix the cause, then regenerate.** The cause is a rule in `SKILL.md`, a mapping in the module notes, or a harness helper. A fix made only in the generated file is silently reverted by the next regeneration. The only per-test state that survives regeneration is listed in [9.2](#92-a-re-sync-mode-should). + +### 5.1 Traceability + +- **Exactly one `UTS:` tag per test**, equal to the spec's `**Test ID**`, copied **verbatim**: `// UTS: ` immediately above the test function (`writing-test-specs.md` "Placement in derived tests"). Where `//` isn't a comment, use the line-comment marker (`# UTS: ` in Python or Ruby); cross-SDK tooling should grep for `UTS: ` without the marker. Never hand-build the prefix: proxy files use `/proxy/…` IDs although they live under `integration/proxy/`, and objects IDs can join spec points (`objects/unit/RTO5c9-RTO20/…`). + - *Divergence:* `uts-to-kotlin` uses a KDoc `/** @UTS */` marker. It works inside ably-java, but a cross-SDK grep for `UTS: ` misses it. New skills should use the documented form. +- **Never invent a tag.** When one test covers several spec points (e.g. `## Test 26: RTN22/RTC8a -- Server-initiated re-authentication` in `realtime/integration/proxy/auth_reauth.md`), put the extra points in the test name, not in a second tag. Invented tags show up as audit orphans. +- **Test names include the spec point** (`writing-derived-tests.md` Phase 1), as a valid identifier the runner collects: + - Swift `test_RTN16g_`; Kotlin backtick names `` `RTN16g - ` ``; + - C# `RTN16g_`, optionally with a `DisplayName`; + - pytest: file `test_.py`, class `Test`, function `test_rtn16g_` (default collection requires these prefixes); + - a module's notes may change how the name is rendered, but the name always includes the spec point (`writing-derived-tests.md` Phase 1 says it *must*). Bare descriptive names that rely on the tag alone are a pattern to avoid. +- **A standard file header** naming the source spec path, its tier, the spec-repo commit translated from ([9.1](#91-record-what-you-translated-from-must)), and any disclosures (harness stand-ins, deviations). Integration and proxy headers also name the corresponding unit spec where the spec gives one (`proxy.md` convention 1). See the file outline in [3.4](#34-recommended-outlines-should). +- **Keep the spec's test order** and its section headings as comments (`Setup`, `Test Steps`, `Assertions`, `Test Steps and Assertions`, `Teardown`/`Cleanup`). + +### 5.2 Fidelity + +- **Copy every pseudocode comment verbatim** to the matching step. Don't paraphrase. +- **Keep spec variable names**, adapted to the language's case convention. Name new variables concretely (`attachFrames`, not `fields` or `result2`). +- **Every `ASSERT` and `AWAIT` is either translated at the same place or annotated, never dropped.** If one has no equivalent in the language or SDK, keep the spec line as a comment and add a note explaining why no assertion is emitted. An unannotated drop is a bug. +- **Wait conditions and timeouts must match the spec.** A `poll_until(auth_callback_count > n)` rendered as a poll on a different predicate is a silent semantic change. Any deliberate difference carries a `// NOTE:` (ambiguity, `writing-derived-tests.md` "Flag ambiguity") or `// DEVIATION:` comment. +- **Never substitute a similar mock outcome for one the harness lacks.** If a spec uses a capability the harness doesn't have (e.g. a WS-level `respond_with_timeout()`), extend the mock (preferred, usually small) or record a Mock Infrastructure Limitation. Don't silently use `respond_with_refused()` instead. A designed-in stand-in ([7.2](#72-harness-stand-ins-are-not-deviations)) is not a limitation. +- **Set only the options the spec sets.** The client factory may pre-seed a dummy key; set `key` only when the spec does. Force the protocol format the mocks understand (`writing-derived-tests.md` "Required options vary by SDK"). +- **Helpers return ready-to-assert values** (typed, non-optional unless the spec expects absence), so test bodies stay free of casts and parsing. Helpers used by one suite live before its first test or in a sibling file (8.2: anything after the last tag is attributed to the last test); helpers shared across a module's suites live in the module helpers. +- **Use SDK API values, not wire values**, when asserting on decoded objects, and convert units where the SDK's API differs from the wire (e.g. ms on the wire, seconds in the API). +- **Translate spec-local functions** (e.g. `function token_auth_callback(api_key)` in `rest/integration/proxy/rest_fallback.md`) as file-local helpers. + +### 5.3 Structural variants + +| Spec shape | Rule | +|---|---| +| **One ID, several H3 sub-cases** (e.g. `realtime/unit/client/realtime_client.md`, `### RTC1a_1`, `### RTC1a_2`) | One test carrying the one tag; translate every sub-case inside it, in order, or as a parameterised test that keeps the single tag | +| **Group heading with no ID, H3 tests with their own IDs** (e.g. `rest/unit/auth/revoke_tokens.md`) | The H2 is a grouper only. One test per H3 ID. MAY mirror the group as a comment or nested suite | +| **`### Test Cases` table driving `FOR EACH test_case`** | `test_cases` is the table. Render as a table-driven or parameterised test under the one tag. Rows that are language-inapplicable are omitted with a comment ([7.4](#74-language-inapplicable-inputs)) | +| **Delegating test or spec**: a whole file (`realtime/unit/channels/channel_get_message.md`) or a single test inside a mixed spec (RTAN3a in `channel_annotations.md`, a section with no Test ID; RTL10a in `channel_history.md`) that says "the tests in `uts/rest/unit/…` should be used to verify … on a `RealtimeChannel`" | The skill MUST define one policy and apply it everywhere. Recommended: one test with the delegating tag (where the section has no Test ID, as RTAN3a, follow the no-Test-ID row: no invented ID, a `// NOTE:` naming the heading) that runs the referenced REST cases against the realtime client through a shared helper. If that isn't feasible yet, mark the test pending (not commented out) with a `// NOTE:`. **This is new policy proposed by this guide**: the UTS docs don't yet cover delegation, and neither existing skill has implemented it | +| **Tests with no Test ID** (`rest/unit/encoding/msgpack_interop.md`) | Translate them; don't invent an ID. Add a `// NOTE:` naming the source heading, and review by hand (the audit can't verify them). *Divergence:* `uts-to-swift` expects a `// UTS:` tag here, which the audit then reports as an orphan | +| **`## Protocol Variants` section (body `json, msgpack`)** | Parameterise over the protocol (`useBinaryProtocol: PROTOCOL == "msgpack"`). The tag and the name stay singular. Specs without the section are JSON only. Proxy tests are always JSON | +| **Spec filed under another module's tree** (e.g. REST faults in `realtime/integration/proxy/rest_faults.md`) | Translate it with the module directory it sits in; the mapping decides the target, not the spec's subject | +| **Inline SDK hints in spec prose** (Dart, ably-js or ably-java names) | Hints for interpretation, not prescriptions. Don't copy them | + +### 5.4 Setup and teardown + +- **Close every client even when the test fails.** `CLOSE_CLIENT(client)` (added to the specs so internal timers stop) must land in `finally`, a teardown hook, or a scoped-resource helper, not only at the end of the body. +- **Restore mocks and cancel SDK timers in teardown**, not at the end of the test (`writing-derived-tests.md` "Cleanup with afterEach"). +- **Never hand-roll teardown** where the harness provides a scope. ably-cocoa uses `with…` scopes because Swift Testing has no async teardown. +- `BEFORE ALL TESTS` / `AFTER ALL TESTS` sandbox provisioning maps to suite-level fixtures where the runner supports them (ably-java `@BeforeAll` with `@TestInstance(PER_CLASS)`; xUnit `IClassFixture`/`IAsyncLifetime`; NUnit `[OneTimeSetUp]`; pytest class- or module-scoped fixtures). Provision once per run wherever the runner supports suite-level fixtures. If it doesn't, per-test provisioning is an acceptable harness adaptation; say so in a comment (ably-cocoa does this). *Divergence from `writing-test-specs.md`* ("Sandbox App Management": create apps once per test run): per-test provisioning is slower and creates more apps, but it doesn't change what the test asserts, so it is allowed only as a harness stand-in disclosed in the file header ([7.2](#72-harness-stand-ins-are-not-deviations)). +- **Stop on a failed wait.** If a wait times out, record the failure and stop driving the client in the wrong state; teardown must still run. +- Close only from states where your SDK's `close()` can reach CLOSED; don't await CLOSED from the others (in ably-cocoa: FAILED and INITIALIZED). + +### 5.5 Integration-tier hygiene + +- **Unique channel names** in every integration test: a descriptive part plus a random part (`writing-test-specs.md` "Unique Channel Names"). +- **Each test creates and closes its own proxy session.** +- **No fixed sleeps.** A spec `WAIT` at integration tier is wall-clock; keep it only when the spec has it, with a comment. + +### 5.6 Time and waiting + +- **Unit tier: no real timers** (`writing-derived-tests.md` "No real timers in unit tests"). Use the fake clock for time-dependent behaviour and `process_pending_events()` (a zero-delay yield or internal-queue drain) for async delivery. The only allowed real timer is a safety timeout. +- **Integration and proxy tiers: every timeout is wall-clock** (`writing-derived-tests.md` "Integration timeouts are wall-clock"). In virtual-time runners (kotlinx `runTest`), wrap every real-network wait, including bare future awaits, in a real-clock helper (ably-java `withRealTimeout`). +- **Poll deadlines must not freeze under fake time**: preferably restructure the test so the wall clock needn't be stubbed; otherwise read a monotonic clock (`writing-derived-tests.md` "No real timers in unit tests"). +- **Every wait is bounded.** Use the spec's timeout where it states one. Otherwise, per `integration-testing.md` "Timeout Strategy": pass an explicit timeout on every integration `poll_until` call (10–30 s), and let state waits use helper defaults of 10–15 s. (`poll_until`'s reference interval is 500 ms and its default timeout 10 s; ably-cocoa's integration helpers default to 15 s.) +- **Prefer state waits.** Map `AWAIT_STATE` to the state-wait helper; use a generic condition poll only when no state applies or the spec itself polls. +- **Keep the spec's `LOOP up to N times: ADVANCE_TIME … BREAK`** as a loop; don't compute exact jumps. +- **Advance fake time, then flush** the async runtime before asserting. + +### 5.7 Waits must catch events + +- **`AWAIT_STATE` must be event-latched and race-free**: subscribe first, then check the current state, and complete exactly once (whichever of the listener and the immediate check wins). Checking before subscribing leaves a window in which the transition is missed. ably-java's `awaitState` does this, with an atomic single-winner resume. *Divergence from `writing-test-specs.md`* ("State Transitions" lists check, then wait): the observable semantics are the same, but a check-then-subscribe implementation misses a transition that happens between the two calls, so subscribe first. The helper also removes its listener on success, timeout and cancellation; leaked listeners keep firing into later tests. +- **A state wait fails fast on FAILED.** If the connection enters FAILED while waiting for another state, fail immediately with the error reason instead of running out the timeout (ably-cocoa's integration `awaitState`). Don't treat a state that is already FAILED at the start of the wait as a failure when the test has just requested a transition out of it (`connect()`, or `authorize()` from FAILED per RTC8c); latch the next transition instead. +- **Transient states use record-and-verify** (`writing-test-specs.md` "Verifying Transient States"): record all state changes, let the cycle finish, then assert `CONTAINS_IN_ORDER`. Even a correct latch misses a state entered and left before the call. Don't insert an intermediate `AWAIT_STATE disconnected`. +- **Polls return the settled value.** For a value-assigned `x = poll_until(…)`, the helper returns the value and later assertions use it. Don't re-read afterwards: an eventually-consistent store can return fewer items on the second read. +- **`poll_until` aborts on an error; `poll_until_success` keeps polling and rethrows the last error on timeout.** Implement them as two helpers; don't emulate `poll_until_success` with a reader that swallows errors to `null`, because the final error is lost. +- **Await the observable effect** of one step before triggering the next. The pseudocode assumes synchronous mock delivery; real SDKs don't. + +### 5.8 Assertions + +- **No "accept either behaviour" assertions** (`writing-derived-tests.md` "Avoid the accommodate-both pattern"). Assert the spec or, as a documented deviation, the SDK's actual behaviour. Never both. +- `FAILS WITH error` asserts the `ErrorInfo` fields (`code`, `statusCode`, `message`), not exception type names. +- **`IS `**: a native type check in statically typed languages; in weakly typed ones (Python, Ruby, JavaScript), `isinstance` or a check that the expected members exist (`writing-test-specs.md` "Type Assertions"). Where the static type already guarantees it, see [7.4](#74-language-inapplicable-inputs) case (b). +- Use a hard assertion (stop the test) where later lines depend on the value, and a soft one elsewhere. +- Normalise numeric types where the language compares `Int`, `Long` and `Double` as unequal. + +### 5.9 Internal access (white-box unit specs) + +**Public-API specs use only the public API.** Even where the test target can see internals, unit specs that exercise the public API use only the public API plus harness helpers; so do fixture helpers such as REST provisioning. Only the module's listed white-box specs, and documented deviations, may reach internals. List the white-box specs in the notes or the mapping file. + +For white-box specs, use this ladder, documented in the skill or in the module notes: + +1. Use an existing internal exposure. **Check first; never duplicate.** +2. Otherwise add a minimal, test-only exposure in the SDK's sanctioned way (ably-cocoa core: a private header registered in both modulemaps, used via `import Ably.Private`; ably-cocoa objects: a `testsOnly_` accessor in the shared test-support module; C#: `InternalsVisibleTo`). Accessors stay dumb; no logic. +3. If more than a visibility raise is needed, **stop and escalate** to a maintainer. +4. If the symbol is unreachable, keep the spec line as a comment and record a Mock Infrastructure Limitation. + +When a white-box spec asserts on the value an internal call returns, assert on that return value, not on a side channel such as emitted events. Integration and proxy specs use only the public API (`writing-derived-tests.md`, case 3). + +--- + +## 6. Pseudocode construct catalogue + +Fill in the last column for your language; the skill MUST map every construct its specs use. Each row's "meaning" comes from the UTS docs where one exists; rows marked *(undocumented)* are used in the corpus but not defined in the docs, and the meaning is inferred from usage. The Swift and Kotlin cells name only methods and patterns that exist in each skill, its notes, its harness or its generated tests; "—" means that SDK has no rule for the construct, so follow the meaning column. + +**This catalogue is not exhaustive**: the corpus keeps growing and has no grammar. The skill MUST treat any uppercase keyword or snake_case helper it can't map as a **stop condition**: tell the user, propose a rendering, and add a row to its own table rather than improvising per test. Re-scan the corpus for new keywords on every re-sync ([9.2](#92-a-re-sync-mode-should)). The skill SHOULD ship a **corpus scanner** (`scripts/scan_constructs.py`, [3.1](#31-layout)) that lists every uppercase keyword and `snake_case(` call in a module's `pseudo` fences, outside comments and string literals, with counts, so the re-scan is mechanical; compare its output with the construct table. + +### 6.1 Async, waiting and state + +| Construct | Meaning | Swift (ably-cocoa) | Kotlin (ably-java) | Your language | +|---|---|---|---|---| +| `AWAIT expr` | Await an async call | `try await op`, or a continuation-bridged helper for callback APIs | `.await()` on a future, or `suspendCancellableCoroutine` around a callback | | +| `f = op()` … later `AWAIT f` (incl. `AWAIT f FAILS WITH error`) | Deferred future: start an operation, act, then await it | `let t = Task { try await op() }` … `try await t.value` (objects module notes) | Hold the `CompletableFuture` (``val getFuture = channel.`object`.get()``) … `getFuture.await()` (generated tests, e.g. `RealtimeObjectTest`) | | +| `AWAIT_STATE x.state == ConnectionState.connected` (also `ChannelState.attached`, bare `CONNECTED`) | Proceed if already true, otherwise wait for the state event; fail on timeout (`writing-test-specs.md` "State Transitions") | `awaitConnectionState(client, .connected)` (unit); `await awaitState(client, .connected)` (integration) | `awaitState(client, ConnectionState.connected)` | | +| `… WITH timeout: N seconds` (`Ns`, `N second`) | Bound on the wait; wall-clock at integration tier | `timeout:` parameter on helpers; no per-call timeout on native `await` (note it in a comment) | `awaitState(client, state, 10.seconds)`; `withRealTimeout` for integration futures | | +| `AWAIT UNTIL `, `WAIT_FOR ` *(undocumented)* | Wait until the condition holds | — | — | | +| `AWAIT_ERROR future` *(undocumented)* | Await a future expected to fail; bind its error | — | — | | +| `AWAIT_ALL futures` *(undocumented)* | Await all futures | — | — | | +| `x.once("connected")` awaited | JS-flavoured one-shot wait; treat as `AWAIT_STATE` | — | — | | +| `poll_until(cond, interval, timeout)`; also the `POLL_UNTIL(timeout:, interval:):` block and `AWAIT pollUntil(CONDITION: …, timeout:)` *(undocumented spellings)* | Re-evaluate until truthy; return the value; an error aborts (`writing-test-specs.md`) | `poll("desc") { … }` (unit, Bool); `pollUntil("desc") { … }` (integration, value-returning) | `pollUntil(10.seconds, 500.milliseconds) { … }` | | +| `poll_until_success(condition: FUNCTION() => …)`; bare form `poll_until_success(, timeout: 15s)` | Errors mean "keep polling"; on timeout rethrow the last error (`uts/README.md`) | — (ports used `pollUntil` with a reader that maps errors to `nil`) | — | | +| `process_pending_events()` | Let queued async work finish; no real delay (`uts/README.md`) | — (objects ports drain the collector queue) | Objects module notes: `` (channel.`object` as DefaultRealtimeObject).asyncFuture { }.await() `` | | +| `WAIT 500ms` | Real delay; discouraged, present at integration tier | `Task.sleep` with a comment (generated tests) | — | | +| `RETURN NEVER_RESOLVING_FUTURE` *(undocumented)* | A callback that never completes | — | — | | +| Quiescence / control-listener (`assert_unchanged_after_quiescence`) | Objects negative assertion (`standard_test_pool.md`) | Objects module notes | Inline control listener in generated tests (e.g. `InstanceTest.kt`); no notes rule | | + +### 6.2 Time + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `enable_fake_timers()` | Install the fake clock **before** constructing the client | `enableFakeTimers()` before `makeRealtime`/`makeRest` | `FakeClock()` + `enableFakeTimers(fakeClock)` | | +| `ADVANCE_TIME(ms)` | Advance fake time and fire due timers | `advanceTime(byMilliseconds:)` | `fakeClock.advance(…)`, then `yield()` | | +| `LOOP up to N times: ADVANCE_TIME … IF … BREAK` | Time-advancement loop, then a final `AWAIT_STATE` | Keep the loop | Keep the loop | | +| `TestClock()`, `WITH_CLOCK(test_clock):`, `test_clock.advance(…)`, `test_clock.now()` *(undocumented)* | A DI clock scoped to a block | — | — | | +| `SCHEDULE_AFTER(100ms):` inside a mock handler *(undocumented)* | Run the block after fake time advances by the delay | — | — | | +| `NOW()` / `now()`, `now_millis`, `time_now`, `current_time`, `current_fake_time` | Current time (fake, where installed) | — | — | | + +### 6.3 Assertions and matchers + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `ASSERT a == b` | Equality (deep where appropriate) | `#expect(a == b)` | `assertEquals(b, a)` (expected first) | | +| `IS null` / `IS NOT null` (also `IS NULL`, `IS NOT NULL`, `IS undefined`) | Language-appropriate absent value (`uts/README.md`) | `#expect(x == nil)` / `try #require(x)` | `assertNull` / `assertNotNull` | | +| `IS null OR … IS NOT SET` *(undocumented)* | Field absent or null | — | — | | +| `IS ` | Native type check; member check in weakly typed languages ([5.8](#58-assertions)) | `#expect(x is T)` | `assertIs(x)` | | +| `IS SAME AS` / `IS NOT SAME AS` *(undocumented)* | Reference identity | — | — | | +| `IN` / `NOT IN` | Membership or key presence | `#expect(map["k"] != nil)` | `assertContains(map, "key")` | | +| `x HAS member` *(undocumented)* | Member exists | — | — | | +| `CONTAINS` (optionally `(case insensitive)`) | Substring or element | — | — | | +| `STARTS WITH` / `STARTS_WITH`, `ENDS WITH` / `ENDS_WITH` | Prefix / suffix (both spellings occur) | — | — | | +| `matches pattern "regex"`, `MATCHES /regex/` | Regex match | `range(of:options: .regularExpression)` | `assertTrue(x.matches(Regex(…)))` | | +| `CONTAINS_IN_ORDER [a, b, c]` | **In-order subsequence**: the items appear in this order; other items may be interleaved (`writing-test-specs.md`) | See the first note in 6.7 | See the first note in 6.7 | | +| `ASSERT ALL x IN xs: pred`, `ASSERT ALL x == y FOR x IN xs`, `ASSERT ANY …` *(undocumented)* | Universal / existential check | — | — | | +| `ARE all unique` *(undocumented)* | No duplicates | — | — | | +| `IS EMPTY` / `IS NOT EMPTY` (also lowercase `IS empty` / `IS NOT empty`) | Empty / non-empty | — | — | | +| `IS NOT complete` *(undocumented)* | A deferred future hasn't settled yet (`objects/unit/realtime_object.md`) | — | — | | +| `IS valid`, `IS valid base64url string` *(undocumented)* | Well-formed value of the named kind (`rest/unit/fallback.md`, `objects/unit/object_id.md`) | — | — | | +| `.length == N`, `>=` | Size comparison | `#expect(list.count == N)` | `assertEquals(N, list.size)` | | +| `FAIL("msg")` | Unconditional failure | `Issue.record("msg")` (as in the skill's fail-fast pattern) | `fail("msg")` (kotlin.test; as in the skill's fail-fast pattern) | | + +### 6.4 Errors + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `AWAIT op() FAILS WITH error` / sync `op() FAILS WITH error` | Bind the failure's `ErrorInfo`; assert its fields | Capture the error, then `#expect(error.code == …)` | `assertFailsWith { … }` then `error.errorInfo.code` | | +| `error.cause.code` | Nested cause of an `ErrorInfo` (e.g. RTO23c1 in `objects/unit/realtime_object.md`: 92008 caused by 90000; RTO20e1 requires the same cause) | Objects module notes: check what the implementation populates; else assert the top level and flag a deviation | Objects module notes: read the Java exception cause | | +| `… THROWS error`, `THROWS AblyException WITH:` *(undocumented)* | Same as `FAILS WITH` | — | — | | +| `EXPECT THROW creating Realtime(…)` *(undocumented)* | Construction must fail | — | — | | +| `THROW ErrorInfo(…)` / `THROW AblyException(…)` / `THROW Error(…)` inside a test-supplied callback *(undocumented)* | Raise an error from the callback (e.g. an `authCallback` that fails), so the SDK sees a callback failure | — | — | | +| `TRY: … CATCH` | Discouraged in specs but present; translate literally, but assert error fields | — | — | | + +### 6.5 Control flow, data and parameterisation + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `IF / ELSE IF / ELSE`, `IF … THEN`, `… END`, `WHILE`, `PASS` | Ordinary control flow; `END` closes a block; `PASS` is a no-op | native | native | | +| `AND`, `OR`, `NOT` | Boolean operators | native | native | | +| `LOG "…"` *(undocumented)* | Informational output: render as a test log line or comment, never an assertion | — | — | | +| `FOR EACH test_case IN test_cases:` + `### Test Cases` table | Table-driven test; `test_cases` is the table | — | — | | +| `FOR x IN [..]`, `FOR i IN 0..n-1`, `FOR [k, v] IN x.entries()` | Loops | native | native | | +| `FIND x IN xs WHERE …`, `FILTER xs WHERE …` *(undocumented)*, `.filter(…)`, `.any(…)`, `.map(…)` | First match / filtered list / predicates | native | native | | +| `FUNCTION(params): … RETURN`, `(x) => {…}`, spec-local `function name(…)` | Local function / lambda / file-local helper | native | native | | +| `LISTEN`, `SET()` | Event subscription; set literal | — | — | | +| `"${…}"`, `+` on strings | Interpolation / concatenation | native | native | | +| `PROTOCOL` | Protocol-variant variable | `@Test(arguments: [false, true])` | `@ParameterizedTest @ValueSource(booleans = [false, true])` | | +| `#` and `//` comments | Spec comments: copy verbatim | `//` | `//` | | + +### 6.6 Mocks, fixtures and harness + +| Construct | Meaning | Swift | Kotlin | Your language | +|---|---|---|---|---| +| `MockHttpClient(onConnectionAttempt:, onRequest:)` + `install_mock()`; `req.respond_with(…)`, `respond_with_delay(ms, status, body)` | `mock_http.md` | Handler-pattern mock, installed before client creation | `install(mock)` | | +| `mock_http.queue_response(…)` and the other `queue_*` calls; `mock_http.captured_requests` *(legacy; deprecated in `writing-test-specs.md` "Common Mistakes to Avoid")* | Pre-queued responses; mock-owned request log ([2.2](#22-a-shared-test-library-implementing-the-helper-specs-must)) | — | — | | +| `mock_http.reset()`, `mock_ws.reset()` | Clear all mock state (`mock_http.md`, `mock_websocket.md`) | — | — | | +| `MockWebSocket(onConnectionAttempt:, onMessageFromClient:, onTextDataFrame:, onBinaryDataFrame:)` | `mock_websocket.md` | Handler pattern, `onConnectionAttempt` only | `onConnectionAttempt`, `onMessageFromClient`, `onTextDataFrame`, `onBinaryDataFrame` callbacks (preferred over the await style) | | +| `on_connect(respond_with:)`, `on_message(action:, respond_with:)`, `on_client_message(…)`, `await_client_message(action:)` *(undocumented alternative API)* | Shorthand handlers | — | — | | +| `conn.respond_with_success/_refused/_timeout/_dns_error/_error`, `respond_with_connected()` | Connection outcomes | `respondWithSuccess`, `respondWithRefused`; no WS-level timeout or DNS error (a known gap: extend the mock or record a limitation) | `respondWithRefused()`, `respondWithTimeout()`, `respondWithDnsError()` | | +| `CONNECTED_MESSAGE`, `CLOSED_MESSAGE`, `DISCONNECTED_MESSAGE`, `HEARTBEAT_MESSAGE`, `ERROR_MESSAGE(code, message)`, `PING_MESSAGE(id)` | Message templates (`mock_websocket.md`) | `.connectedMessage`, `.connected(…)`, `.attached`, `.error`, `.ack`, `.closed()` factories (`connectionStateTtl` in seconds, not wire ms); no `PING_MESSAGE` factory | `CONNECTED_MESSAGE`; `ErrorInfo(message, statusCode, code)` argument order | | +| `mock_ws.active_connection.send_to_client(…)`, `send_to_client_and_close`, `simulate_disconnect`, `close_from_server()`, `send_to_client_raw(json)`, `send_ping_frame` | Server-side actions | `sendToClient`, `sendToClientAndClose`, `simulateDisconnect`; no ping-frame method | `sendToClient`, `sendToClientAndClose`, `simulateDisconnect`; no ping-frame method | | +| await API: `await_connection_attempt()`, `await_request()`, `await_next_message_from_client()`, `await_client_close()` | Await style; set up the next await **before** responding to the current one | — (not implemented; handler style + attempt counter + `sentMessages` instead) | `awaitConnectionAttempt()`, `awaitRequest()`, `awaitNextMessageFromClient()`, `awaitClientClose()` | | +| `mock_ws.events.filter(…)` with event types `CONNECTION_ATTEMPT`, `CONNECTION_SUCCESS`, `CONNECTION_FAILURE`, `MESSAGE_FROM_CLIENT`, `MESSAGE_TO_CLIENT`, `CLIENT_CLOSE`, `SERVER_DISCONNECT`, `PING_FRAME` (`mock_websocket.md`); `mock_ws.connect_attempts` *(no helper spec; used in a `writing-test-specs.md` example)*, `mock_ws.last_connect_url` *(undocumented)* | Mock event log; connection attempts and the last connect URL | `ws.sentMessages` (client → server frames) | `events: List` | | +| `mock_ws.onConnectionAttempt = …` | Reassign a handler mid-test | — | — | | +| `create_mock_websocket()`, `create_realtime_client(…)` *(undocumented aliases)* | Same as `MockWebSocket()` / `Realtime(…)` | — | — | | +| `MockNetworkListener()`, `simulate_network_lost()` / `simulate_network_available()` *(no helper spec)* | Network-change mock ([2.2](#22-a-shared-test-library-implementing-the-helper-specs-must)) | — | — | | +| `CLOSE_CLIENT(client)` *(undocumented)* | Close the client so its internal timers stop | `closeClient(client)`; integration scopes own it | `client.close()`; in `finally` for proxy tests | | +| `captured_requests.append` / `.push`, `request.url.path` / `request.path`, `.url.query_params` / `.query_params` / `.queryParams` | Request capture (spellings vary) | `Captured` | `CopyOnWriteArrayList` | | +| `parse_json` / `JSON_PARSE` / `JSON_DECODE` / `json_decode`, `json_encode`, `MSGPACK_DECODE` / `msgpack_decode` / `msgpack_deserialize`, `msgpack_encode`, `base64_decode` / `base64_encode`, `load_json`, `load_fixtures`, `toJson()` / `fromJson()`, `encode_uri_component()` / `decode_uri_component`, `base64url_encode` / `base64url_decode`, `encode_utf8` / `utf8_encode` / `utf8_decode`, `parse_query_string`, `parse_form_urlencoded`, `extract_between`, `byte_array([…])`, `to_json`, `msgpack_serialize`, `as_list`, `DateTime(…)` | Data helpers; `toJson`/`fromJson` map to the SDK's idiomatic names (not a deviation) | `[String: Any]` | — | | +| `random_id()`, `random_string`, `unique_channel_name()`, `base64(random_bytes(6))` | Random / unique names | UUID channel names | `UUID.randomUUID()` | | +| `COUNT_UNIQUE(…)`, `EXTRACT_FALLBACK_ID(…)` *(undocumented)* | Spec-level helpers (`fallback_hosts_test.md`) | — | — | | +| `generate_jwt(…)` / `generateJWT(…)`, `extract_key_name(api_key)`, `extract_key_secret(key)`, `get_key_parts`, `request_token_from_sandbox(api_key, params)` *(the last two undocumented)* | Auth helpers | Locally signed `TokenRequest` instead of JWT, with a comment | Locally signed `TokenRequest` | | +| `BEFORE ALL TESTS`, `AFTER ALL TESTS`, `BEFORE EACH TEST`, `AFTER EACH TEST`, `AFTER TEST:` | Fixtures | `with…` scopes, per test | `@BeforeAll` / `@AfterAll` | | +| `POST https://…/apps WITH body from …`, `DELETE … WITH Authorization: Basic …` | Sandbox provisioning | `SandboxApp` | `SandboxApp.create()` / `delete()` | | +| `create_proxy_session(endpoint:, rules:)` (or `rules:` only; see 6.7), `add_rules`, `trigger_action`, `get_log()` / `session.getLog()`, `proxy_port` | `proxy.md` | `withProxySession(rules:)`; typed `ProxyEvent` log | `ProxySession.create(…)`, `finally { session.close() }`; typed log | | +| `"__PASSTHROUGH__"` as a field value in a `replace` action's message *(undocumented)* | A uts-proxy sentinel that `proxy.md` doesn't define (used in `realtime/integration/proxy/connection_resume.md`); confirm its behaviour against the pinned uts-proxy release before mapping it | — | — | | +| `MockVCDiffEncoder()`, `MockVCDiffDecoder`, `FailingMockVCDiffDecoder` | `mock_vcdiff.md` | — | — | | +| `setup_synced_channel`, `setup_synced_channel_no_ack`, `STANDARD_POOL_OBJECTS`, `build_*`, serial helpers, `provision_objects_via_rest` | `standard_test_pool.md` | Unit tier seeds the pool directly (sanctioned stand-in); helpers in the test-support target | `setupSyncedChannel("test")`, `build*` in module `Helpers.kt` | | +| White-box access: `applyOperation`, `channel.object.objectsPool`, `processChannelState`, direct construction, `get_backoff_coefficient(n)`, `get_jitter_coefficient()`, `encode_recovery_key(…)`, `CLEAR channel._lastPayload.messageId` | Internal access ([5.9](#59-internal-access-white-box-unit-specs)). `CLEAR …` in `delta_decoding_test.md` is a private-field write inside an *integration* spec | Core: `import Ably.Private` + private headers; objects: `@testable import` + `testsOnly_` accessors | `internal` within `:liveobjects` | | +| `logLevel: LOG_INFO`, `logHandler: (level, message) => …` (with `level == WARN` / `info` / `debug`), `captured_log_messages.filter(…)`, `captured_logs`, `log_messages` | Log capture; log levels | `CapturingLog`, `log.contains(level:message:)` | — | | +| `MESSAGE_ACTION_INT`, `HAS_OBJECTS`, `HAS_PRESENCE`, protocol actions | Wire constants | — | — | | +| `DEFAULT_REST_HOST`, `SANDBOX_ENDPOINT` *(undefined in the docs)*, spec-local constants such as `DEFAULT_CONNECTION_STATE_TTL = 5000` | Named constants: the SDK's default, the harness's sandbox endpoint, or a value the spec defines | — | — | | +| Reserved words (`channel.object`) | Escape idiomatically; not a deviation | — | `` channel.`object` `` | | + +### 6.7 Notes on the catalogue + +- **`CONTAINS_IN_ORDER` is an in-order subsequence, not a contiguous prefix or an exact filtered list.** `[connecting, disconnected, connecting, connected]` satisfies `CONTAINS_IN_ORDER [disconnected, connecting, connected]`. Kotlin's iterator-`next()` mapping asserts a contiguous prefix. Swift's `filter { [a, b, c].contains($0) } == [a, b, c]` fails when a listed state occurs more than once; its alternative ("walk an index as the spec does") is correct but unspecified. Ship an `assertContainsInOrder` helper in the harness and map to it. (ably-java's `ObjectsFaultsTest.kt` has a correct file-local one; the gap is that it isn't in the harness.) +- **Enum spellings vary**: `ConnectionState.connected`, `ChannelState.attached` and bare `CONNECTED` mean the same. +- **`writing-derived-tests.md` writes `AWAIT_STATE(connection, "connected")`**; the corpus form is `AWAIT_STATE client.connection.state == ConnectionState.connected`. Support both. +- **Proxy session:** `writing-test-specs.md` shows `create_proxy_session(target: TargetConfig(…))`; `proxy.md` and 41 of the 43 corpus calls take `endpoint:`. The other two (`realtime/integration/proxy/presence_reentry.md`) pass only `rules:`, so the session helper must default the endpoint to the sandbox. No corpus call uses `target:`. Follow `proxy.md`. +- **Proxy msgpack:** `proxy.md` gives "SDK doesn't implement msgpack" as the JSON reason; the real reason is that the proxy only supports text frames (`integration-testing.md`). +- **Dart, ably-js and ably-java residue** in specs (`Uint8List`, `fromMap`, `.once(…)`, `undefined`) is a hint, not a prescription. + +--- + +## 7. Evaluation and deviations + +Follow `writing-derived-tests.md` Phase 2 and "Recording deviations" exactly. The skill adds only the language rendering and the stopping rules. + +### 7.1 Three acceptable end states + +Every evaluated test ends in exactly one of these. Never an unexplained red. + +| Outcome | Test state | Rendering the skill must specify | +|---|---|---| +| **Pass** | Green | — | +| **SDK deviation** (`writing-derived-tests.md` 2c) | Green: **env-gated skip** on `RUN_DEVIATIONS`, or **adapted assertion** (preferred when the divergence is permanent or intentional), with a `// DEVIATION: see deviations.md` comment | A **runtime** skip idiom (below) and the reproduction command (`RUN_DEVIATIONS=1 `) | +| **UTS spec error** (`writing-derived-tests.md` 2a) | **Red, fails fast** with a message pointing at `deviations.md`. The one acceptable red | The fail idiom (Swift `Issue.record("UTS spec error — fix the spec first; see deviations.md")`, Kotlin `fail(…)`) | + +A translation error (2b) is not an outcome: fix the cause ([section 5](#5-translation-rules)) and regenerate. + +Runtime skip idioms (the gate must read the environment in the test process; a pytest `skipif` decorator does, a compile-time attribute doesn't): + +- Swift Testing: `@Test(.enabled(if: ProcessInfo.processInfo.environment["RUN_DEVIATIONS"] != nil))` +- Kotlin: `if (System.getenv("RUN_DEVIATIONS") == null) return@runTest` +- pytest: `@pytest.mark.skipif(not os.environ.get("RUN_DEVIATIONS"), reason=…)` +- C#: xUnit v2 `[SkippableFact]` + `Skip.If(…)` (Xunit.SkippableFact), xUnit v3 `Assert.Skip(…)`, NUnit `Assert.Ignore(…)` + +**SDK lacks the API entirely**, so the spec-correct test can't compile. First decide, per `writing-derived-tests.md` "Check the SDK's API surface", whether the test is not applicable to this SDK ([7.4](#74-language-inapplicable-inputs) case (c)) or the absence is itself a deviation. If it is a deviation: register the test, keep the uncompilable spec lines as comments with a `// NOTE:`, and mark it skipped (not commented out; `writing-derived-tests.md` "Test-first considerations"). In evaluate mode, record it under **Failing Tests** with the test impact "skipped stub" (one of the test-impact values in `writing-derived-tests.md` "Recording deviations"). *Divergence from `writing-derived-tests.md`*, whose Failing Tests are env-gated: a test that can't compile can't be env-enabled, so it stays a skipped stub until the API exists. In translate-only mode, list it in the final report ([section 11](#11-final-report-format)). + +### 7.2 Harness stand-ins are not deviations + +Using fake timers where the spec uses real ones, a queue-ordering workaround, per-test sandbox provisioning, or seeding state directly instead of driving the spec's mock transport changes *how the test drives the SDK*, not what the SDK does. Explain it in a code comment or the file header. Don't record it in `deviations.md`. This extends `writing-derived-tests.md` case 1 (idiomatic translation) to the harness; the docs don't state it for harness stand-ins explicitly. + +### 7.3 `deviations.md` + +- Use the format in `writing-derived-tests.md` "Recording deviations"; **don't define a home-grown format in the skill**. That means: + - all four headings, in order, with `*(none)*` for an empty one: **UTS Spec Errors**, **Failing Tests**, **Adapted Tests**, **Mock Infrastructure Limitations**; + - the entry fields: spec point, what the spec says, what the SDK does, root cause, test impact, status (for SDK deviations) and resolution (once resolved); + - where internal-shape differences recur, an `S-1…S-n` vocabulary. +- The skill specifies only **where** the file lives (one per owning test module) and that new entries go **into their category's section**, never appended at the end. +- **When it is written.** Per `writing-derived-tests.md` "Test-first considerations", the file is created during evaluation. In translate-only mode, list mock-capability gaps (Mock Infrastructure Limitations) and missing APIs in the final report ([section 11](#11-final-report-format)) instead, and record them on the first evaluate run. When the file is created, it has all four headings. + +### 7.4 Language-inapplicable inputs + +Per `uts/README.md`, an input that can't be constructed in your language makes that test or table row **not applicable**: note it **in the derived test file**, not in `deviations.md`, and don't count it as a coverage gap. Three cases: + +| Case | Example | Rule | +|---|---|---| +| **(a) Inexpressible input** | A wrong-typed argument a static type system rejects (`increment("10")`); a non-string key in JavaScript | Not applicable. Omit the test or row with a comment at the omission | +| **(b) Assertion guaranteed by the static type** *(extends the README)* | `ASSERT keys IS Array` on a method whose declared return type is a list | Keep the spec line as a comment noting the compile-time guarantee: an annotated omission, not a deviation | +| **(c) Missing API** | The SDK has no such method | Not one of the README's cases: decide not-applicable vs deviation per `writing-derived-tests.md` "Check the SDK's API surface" ([7.1](#71-three-acceptable-end-states)) | + +- Not every type mismatch is inapplicable. Calling `increment` on a map is still expressible by casting to the counter view and asserting the throw; translate it. +- Dynamic languages rarely have case (a): translate wrong-type rows as runtime assertions. +- *Divergence:* both existing skills' objects module notes record compile-time-unrepresentable inputs (case a) as deviations, and ably-java's first audit run also filed case (b) type assertions as deviations. New skills should follow the README. + +### 7.5 Stopping rules for evaluate mode (SHOULD) + +"Fix until green" needs a bound. The skill should: + +- Stop and ask the user after a fixed number of fix attempts per test (pick one, e.g. three). +- Stop and ask when a failure looks like a sandbox or network problem rather than SDK behaviour (passes alone but fails in the suite; provisioning errors). +- Stop and report each suspected **UTS spec error**, with the features-spec quote and a draft spec fix, rather than only recording it. Several upstream spec fixes came out of evaluation runs. +- Report undocumented helper contracts the port had to invent (e.g. the element type of `mock_ws.connect_attempts`, which no helper spec declares) as suspected spec gaps. +- Never fix a test by weakening an assertion, re-reading after a poll, or accepting both behaviours. + +--- + +## 8. Deterministic tooling + +Path validation, mapping, spec discovery, naming and faithfulness checking are mechanical. **Scripts do them identically every run; a model eyeballing two files does them inconsistently.** In ably-java the audit's first use found three undocumented omissions in already-reviewed tests. + +### 8.1 Resolver (`resolve_uts.py`): MUST + +| Requirement | Detail | +|---|---| +| Validate the module | Expand a leading `~`. The parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/` | +| Errors | One JSON object with `ok: false`, a code and a `message`. Example codes (ably-cocoa): `NOT_A_UTS_MODULE_PATH`, `DIR_NOT_FOUND`, `NO_TIER_DIRS`, `MAPPING_NOT_FOUND`, `BAD_MAPPING`, `BAD_TARGET_NAME` | +| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty}`, MAY) gives the spec clone's state for the file headers and the report (9.1, 9.4). Downstream steps read only these fields | +| Tier detection by path | `unit/**`; `integration/**` excluding `integration/proxy/**`; `integration/proxy/**`. Every tier can have sub-directories (e.g. `realtime/integration/channels/`). Use the path only. (Both existing skills also describe content-based proxy detection in their integration sections; drop it.) | +| Exclusions | `helpers/` (implement as harness, never translate), `README.md`, `PLAN.md`, `*_SUMMARY.md`. Match them **relative to the tier base**, so an ancestor directory in the checkout path can't trip them. *(New, defensive; nothing matches today:)* also skip any non-spec notes left inside a tier directory | +| Deterministic naming | Strip a trailing `_test`, convert to the language's convention, add the suffix or prefix the runner needs (`objects_lifecycle_test.md` → Swift `ObjectsLifecycleTests`, Kotlin `ObjectsLifecycleTest`, pytest `test_objects_lifecycle.py` with class `TestObjectsLifecycle`) | +| Collision detection *(new)* | Fail loudly if flattening sub-directories maps two specs to one class. Generated names can also collide with native test classes in another target (ably-cocoa `ObjectsPoolTests`); detect this, or always run generated suites with a target-qualified filter (`UTS.`). Runners that import test files by basename (pytest's default import mode) also collide when two tiers or modules produce the same file name (`rest/integration/auth.md` and `realtime/integration/auth.md` → `test_auth.py`): make the target directories packages (`__init__.py`) or use `--import-mode=importlib`, and check file-name uniqueness across the whole mapping | +| `--create` | Validate the existing mapping **before** writing, so a corrupt file is never written back. Add or replace the named module's entry, and preserve every other entry and the replaced entry's `notes`. Write UTF-8 with LF line endings explicitly. On a bad name, ask again | +| Notes | Resolve `notes` against the skill directory. *(New:)* return an error, not a silent `null`, when the declared notes file is missing | + +### 8.2 Audit (`audit_translation.py`): MUST + +| Check | Why | +|---|---| +| **Test-ID coverage**: spec `**Test ID**` set vs test-file `UTS:` set → `missing`, `orphan`, `duplicate`; a distinct non-zero exit when any is non-empty | A missing ID is a missing test; an orphan is a stale, renamed or invented tag; a duplicate tag hides a method from the ledger | +| **Per-test ledger**: every spec code line, grouped by section, tagged `assert` / `await` / `step` | Setup, operations and assertions are all enumerated | +| **Count assertions and waits separately** | Summed counts let surplus waits mask dropped assertions (ably-cocoa showed this by deleting RTN16f's two assertions; the ably-java audit still sums them) | +| **Ignore commented-out assertions** on the test side | A commented `// assertEquals(…)` isn't an assertion | +| **Never crash** | Always emit one parseable JSON object. Both references use exit 0 for clean, 2 for ID problems and 64 for "couldn't run"; any distinct codes will do | +| **Report what it can't see** *(new)* | A spec with no `**Test ID**` lines (fixture-driven specs) must be reported as "not verifiable", not as zero tests | + +**Parsing contract.** Implement exactly this, or say where yours differs: + +- **Spec side.** A test starts at each `` **Test ID**: `` `` line (the backticks are part of the marker) and ends at the next one. *(New, SHOULD:)* also end it at the next heading at or above the test's own level, so trailing notes or appendix sections aren't attributed to the last test. +- **Headings.** Section headings are any `#`–`####` line **outside** a fence; inside a fence, `#` starts a pseudocode comment. Combined headings such as `### Test Steps and Assertions` are sections too. +- **Fences.** Read ` ```pseudo ` fences. You MAY also read untagged fences (four in spec files today, all in file preambles before the first test, plus one each in `objects/PLAN.md` and the `standard_test_pool.md` helper, which aren't translated; so this changes nothing yet; the Swift audit reads `pseudo` only). Skip ` ```json ` and other payload fixtures. +- **Classification.** Skip blank and comment lines. Tag `ASSERT` / `ASSERT_*` as `assert`; `AWAIT` / `AWAIT_STATE` / `AWAIT_ERROR` / `AWAIT_ALL` / `AWAIT UNTIL` / `EXPECT` and the poll forms `poll_until` / `poll_until_success` / `POLL_UNTIL` / `WAIT_FOR` as `await` (whether or not the line starts with `AWAIT`), so spec polls and the test's poll helpers are counted alike; everything else as `step`. Match the longest keyword first, on both sides (`poll_until_success` before `poll_until`; `pollUntil` before `poll`). +- **Test side.** A test's block runs from its `UTS:` tag to the next tag (or end of file). Helpers placed after the last test are attributed to it, and assertions inside shared helpers are invisible to their callers: keep shared helpers in a separate file or before the first test, and annotate call sites that hide spec assertions. + +Only the test-side regexes change per language (tag marker, assertion calls, wait calls, comment syntax); the spec side is the same for every SDK. + +**Output shape (SHOULD).** The guide fixes the checks, not the field names; a shape that covers them: `{ok, spec, test, idCoverage: {specCount, testCount, missing, orphan, duplicate}, perTest: [{id, title, sections, specAsserts, specAwaits, testAssertCount, testWaitCount, assertionShortfall, awaitShortfall}], summary: {testsWithAssertionShortfall, testsWithAwaitShortfall}, notVerifiable?}`, or `{ok: false, error}`. + +SHOULD, beyond the existing audits: + +- **Report the await shortfall separately, as a soft signal** ([8.3](#83-review-checklist-must-after-the-audit)). +- **Condition matching:** compare each `AWAIT` / `poll_until` predicate and timeout with the test's, at least as a side-by-side listing. A count can't catch a changed predicate. +- **Verbatim comment check:** flag spec comments that don't appear verbatim in the test. +- **Stale or renamed IDs:** pair each missing ID with an orphan that shares the descriptive name ([9.3](#93-renamed-merged-and-added-ids)). +- **Structural validation of the spec:** every test heading has an ID or is a recognised grouper or sub-case; the ID category matches the path; `## Test Type` is present; helper references point at files that exist. The spec repo has no automated UTS validation, so the skill is the only check. + +**Test the tooling itself (SHOULD).** Mutation-test the audit: delete an assertion, or duplicate a tag, in a known-good test and confirm the audit flags it. Run it over the whole spec corpus after every script change and confirm zero crashes and no unexpected ledger changes. ably-cocoa did both for each audit change. + +### 8.3 Review checklist (MUST, after the audit) + +The review is static, so it runs in both modes. + +- **Coverage:** `missing`, `orphan` and `duplicate` are empty, or each is explained. These, and a positive assertion shortfall, are the hard checks. +- **Line by line:** every positive assertion shortfall is accounted for by an annotated omission. A negative shortfall (more native assertions than spec `ASSERT`s, e.g. from type or number normalisation) is fine. A multi-line spec construct (a mock definition, a `ClientOptions(...)` block) appears as several `step` lines; reconcile them as one group. +- **Await shortfall is a soft signal.** The wait counter sees only the harness's wait helpers. Continuation-bridged helpers, native `await` on an async SDK API, and fixture builders legitimately replace them, so a positive await shortfall is expected on natively-async modules. Account for each spec `AWAIT` rather than treating the count as a gate. +- **Setup fidelity:** client options, mock responses, timer use and channel-operation order match the spec. +- **Wait fidelity:** every wait condition and timeout matches, or carries a `NOTE`/`DEVIATION`. +- **Deviation honesty** (evaluate mode): every gated or adapted test has a `deviations.md` entry, and no harness stand-in is recorded as a deviation. + +--- + +## 9. Keeping in sync with spec changes + +UTS has no version number or changelog, and **Test IDs are not stable**: they are renamed when spec points are replaced, and spec and UTS changes land in the same commit. A translation that was correct last month can be stale today. + +### 9.1 Record what you translated from (MUST) + +Stamp each generated file's header with the spec-repo commit SHA it was derived from: + +``` +// Derived from uts/realtime/unit/connection/connection_recovery_test.md +// at ably/specification@ +``` + +A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. + +### 9.2 A re-sync mode (SHOULD) + +Neither existing skill has one; both have stale suites as a result. Add a mode (for example `/uts-to- --resync`) that: + +1. Runs the audit over **every** mapped spec in the module, not only the ones selected. +2. Runs `git diff ..HEAD -- uts/` to list changed spec files, and re-scans the corpus for constructs the skill can't map (section 6). +3. Classifies each spec as **new**, **changed** (ID set changed, or pseudocode changed), **unchanged** or **removed**. +4. Regenerates the affected tests, **preserving only** DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. +5. Updates the SHA in each regenerated file's header (and in the manifest, if kept). +6. Reports the classification ([section 11](#11-final-report-format)). + +Regenerate the whole file when most tests changed; otherwise regenerate only the affected tests. Say which in the report. A re-sync is also needed when the shared harness or module helpers change shape (ably-cocoa reset and regenerated its objects ports after aligning them with `standard_test_pool.md`). + +### 9.3 Renamed, merged and added IDs + +Treat a removed ID plus an added ID as a **possible rename**, keyed on the descriptive name and position, and **re-check the assertions**: a renamed point can invert its semantics (`RTL15b1/serial-cleared-suspended-1` became `RTL15b2/serial-retained-suspended-1`). + +Examples of the pattern, as of 2026-10-07 (spec `12540dcf`): + +- **Rename not followed.** Both SDKs carried `realtime/unit/RTN16g2/recovery-key-null-inactive-0` after the spec renamed it to `realtime/unit/RTN16g3/recovery-key-null-inactive-0` (spec commit `d0d1c02f`): ably-cocoa `Test/UTS/unit/realtime/ConnectionRecoveryTests.swift` and ably-java `lib/src/test/kotlin/io/ably/lib/uts/unit/realtime/ConnectionRecoveryTest.kt`. The audit reports one missing and one orphan, but nothing prompted a re-run. +- **Additions not followed.** In ably-java `RealtimeObjectTest.kt`, two `RTO27` tests remained where the spec has a single `RTO27/channel-state-data-lifecycle-0` (the only `RTO27` Test ID in the spec's history, added in `65e6dd5e`), and `RTO20d4/mixed-null-serials-applies-non-null-0` (added in `6425db00`) had no test. + +### 9.4 Local clone vs fetching `main` + +Pick **one** source for docs and specs, and pin it. This guide recommends **reading everything from the local spec clone** the module directory came from: + +- the UTS docs (`/../docs/writing-derived-tests.md`, `proxy.md`); +- in evaluate mode, the features specs the decision tree needs (`/../../specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. + +Record the clone's HEAD SHA and whether it is dirty. You MAY warn when the clone is behind `origin/main`. + +- *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. +- *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md`, and the features specs, from GitHub `main` while reading specs from the local clone, so the two can skew. + +--- + +## 10. Verification and CI + +Escalate in this order: + +1. **Compile the tests** for the module (`swift build --build-tests`; `./gradlew ::compileTestKotlin`; `dotnet build`). For every runner, check that the number of collected (or executed) tests equals the number of `UTS:` tags: a misnamed pytest class, or a JVM test annotated for an engine the task doesn't run (JUnit 4 vs JUnit Platform), is silently skipped. For pytest use `pytest --collect-only` (plus `mypy` if the repo uses it); for .NET, `dotnet test --list-tests`; for Gradle, compare the task's test-report count. In dynamic languages a missing SDK API surfaces only at run time: run a type checker (`mypy`/`pyright`) over the generated tests where the SDK ships type hints, or report missing APIs as unverified in translate-only mode. MUST, in both modes. +2. **Lint and format** with the repo's gates (EditorConfig, checkstyle, eslint, ruff…). MUST. Neither existing skill runs lint; both repos needed it. +3. **Compile as strictly as CI does.** If CI treats warnings as errors, so must the skill. (ably-cocoa's LiveObjects CI failed on a redundant typed-throws cast the local build accepted.) In TypeScript, the test runner may strip types without checking them (`writing-derived-tests.md` "Build pipeline and CI checks"). +4. **Run the generated classes by filter** (evaluate mode): per class, then per tier. Use target-qualified filters where names can collide. +5. **Run the full tier** only when asked; integration tiers are slow and hit the sandbox. +6. **CI:** each suite has exactly one CI home ([2.5](#25-build-and-ci-wiring-must)). Don't gate integration tests behind an env var in the generated code; select them by filter or task. +7. **Commits (SHOULD).** By default the skill doesn't commit: it lists the files it changed (tests, `deviations.md`, mapping, notes) for the user to review. It also lists any project files the build doesn't discover automatically, which the user must update by hand (for example an Xcode project). + +The skill's scripts MUST pin UTF-8 and LF when they write tracked files. + +--- + +## 11. Final report format + +Neither existing skill defines one; define it. The skill MUST end each run with a report like this: + +``` +UTS translation: / @ ably/specification@ (clone clean|dirty) (mode: translate-only | evaluate | resync) + +| Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | +|---|---|---|---|---|---|---|---| +| connection_recovery_test.md | ConnectionRecoveryTests. | 6/6 | clean | 1/1 | ok | 5 pass, 1 gated | Failing Tests: RTN16f | + +Skipped specs (and why): … +Not-applicable omissions (7.4): — +Missing APIs found in translate-only mode: — +Mock-capability gaps found in translate-only mode: — +Harness stand-ins disclosed in file headers: … +Constructs the skill couldn't map (stopped and asked): … +Suspected UTS spec errors or gaps (need an upstream fix): — , draft fix: … +Mapping or notes changes: … +Changed files for review: … +Next steps: … +``` + +--- + +## 12. Lessons learned + +From the git histories of both existing skills. Most lessons are already rules in sections 2–10; these are the ones that aren't, or that are easy to forget. + +| Lesson | Where it came from | +|---|---| +| **Build the harness to look like the pseudocode.** Callback mocks and a builder DSL make translation mechanical | ably-java's first generation run | +| **Move everything mechanical into scripts.** Paths, names, mapping and coverage checks drift when the model does them | Both: resolver, then audit | +| **Lessons must reach `SKILL.md`**, not only harness comments or commit messages | ably-java: several flake fixes stayed in KDoc | +| **Defer formats to the manual**; inline copies drift | Both: deviations format | +| **Don't cite line numbers in skill text**; they drift. Cite symbols or headings | ably-cocoa | +| **Compile the examples in the skill**, or take them from real, compiling files in your own repo | ably-cocoa: an example helper doesn't compile | +| **Placement follows visibility**: put white-box tests where internals are visible; share the harness as a library | ably-java tests moved to `:liveobjects` | +| **Evaluation finds spec bugs**; fix them at source | Both: several upstream spec PRs | +| **Concurrency is the main source of flakiness**: thread-safe recording lists, await the seed before subscribing, flush before injecting a stimulus, a fake clock that runs to quiescence | ably-java CI fixes | +| **Keep generic rules separate from per-language renderings**: write every rule language-neutrally first, then give its rendering for your SDK | Both: the Swift skill began as a port of the Kotlin one, and the generic parts had to be separated afterwards | + +--- + +## 13. Checklist for a new `uts-to-` skill + +**Prerequisites** + +- [ ] SDK test hooks: WebSocket factory, HTTP client, clock (covering blocking waits and the async runtime's timers), reachability/network monitor, randomness; injected at construction +- [ ] Shared test library: `mock_http.md` (+ legacy `queue_*` and `mock_http.captured_requests`), `mock_websocket.md` (async close, CONNECTED ordering, raw frames, alternative API, undocumented members), `mock_vcdiff.md`, `standard_test_pool.md`, `MockNetworkListener` +- [ ] Wait helpers: race-free event-latched `AWAIT_STATE` failing fast on FAILED, value-returning `poll_until`, `poll_until_success`, `process_pending_events()`, wall-clock timeout wrapper, deadlines safe under fake time +- [ ] Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures +- [ ] Shared vs port-only placement; fixture-helper scope documented +- [ ] Harness smoke tests per tier while building (SHOULD) +- [ ] `SandboxApp` (retries idempotent reads only); `ably-common` submodule +- [ ] `ProxyManager` with a pinned `uts-proxy` version for every developer OS; `ProxySession`; string `match.action`; auth through the proxy decided and commented +- [ ] One CI home per suite (MUST); per-tier jobs (SHOULD) +- [ ] Concurrency, time, runner-parallelism and internal-visibility models documented + +**Skill files** + +- [ ] `SKILL.md` with `name: uts-to-`, a pushy description with trigger phrases, `argument-hint`, usage guard, placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline +- [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked +- [ ] `resolve_uts.py`: validation, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, collision detection, validate-then-write `--create` preserving entries +- [ ] `audit_translation.py`: the parsing contract, ID coverage, duplicates, separate assert/await counts, ignores commented assertions, never crashes, flags unverifiable specs; itself mutation-tested +- [ ] Per-module notes for every module whose API diverges from the pseudocode (at least `objects`, if the SDK implements it) +- [ ] Scripts run on every developer platform (path handling, UTF-8, LF) + +**Workflow** + +- [ ] Required reading, including features specs, from the same clone as the specs +- [ ] Steps 0 and A–E with the four user questions (mapping, tier, specs, translate vs evaluate) +- [ ] Steps 1–7 with batching; a reference test per tier; review in both modes +- [ ] Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) +- [ ] Re-sync mode; fix the cause, then regenerate + +**Translation rules in `SKILL.md`** + +- [ ] One `UTS: ` per test, verbatim; no invented tags; spec point in a collectable test name +- [ ] File header with source spec, tier, spec SHA, disclosures and (integration/proxy) the corresponding unit spec +- [ ] Verbatim comments, spec variable names, section headings, spec order +- [ ] Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome +- [ ] Policies for sub-cases, groupers, test-case tables, delegating tests (new policy) and ID-less tests +- [ ] Protocol variants parameterised; tag singular; proxy JSON only +- [ ] Clients closed in teardown; mocks restored; stop on a failed wait +- [ ] Unique channel names; own proxy session per test +- [ ] Unit: fake time and flushes, no real timers. Integration: wall-clock everything +- [ ] Record-and-verify for transient states; polls return the settled value +- [ ] No accommodate-both assertions; `IS ` per typing discipline +- [ ] Public-API specs stay public; internal-access ladder with an escalation stop +- [ ] Construct table ([section 6](#6-pseudocode-construct-catalogue)) filled in for every row + +**Evaluation** + +- [ ] Three end states with the language's runtime skip and fail-fast idioms; `RUN_DEVIATIONS` reproduction command +- [ ] `deviations.md` location per owning module; format deferred to `writing-derived-tests.md`; written at the right time +- [ ] Harness stand-ins disclosed in comments, not in `deviations.md` +- [ ] Language-inapplicable inputs (cases a–c) noted in the test file +- [ ] Stopping rules and spec-error escalation + +**Verification** + +- [ ] Compile (or collect), lint, CI-strict compile, filtered run +- [ ] No commits by default; changed files listed +- [ ] Final report format +- [ ] The skill's own examples compile + +--- + +## Appendix: Existing skills (background and patterns to avoid) + +You don't need to read these skills; everything required is in this guide. They are named only so that the examples and lessons above can be traced. + +| Skill | Repo and paths | Lessons this guide drew from it | +|---|---|---| +| `uts-to-swift` | ably-cocoa, `.claude/skills/uts-to-swift/`; tests under `Test/UTS/` | Strict compile-time concurrency checking (Swift 6); a callback-based core plus a natively async plugin; scoped-resource teardown without async `tearDown`; separate assertion and wait counts in the audit; "a harness stand-in is not a deviation"; the internal-access ladder; module notes overriding the generic flow | +| `uts-to-kotlin` | ably-java, `.claude/skills/uts-to-kotlin/`; tests in the `lib/` and `liveobjects/` test source sets, harness in `:uts` | Deriving a package and build module from the target path; callback- and await-style mocks; race-free `awaitState`; virtual-time pitfalls and real-clock timeouts; idempotent-only sandbox retries; the `S-n` shape-deviation vocabulary | + +Both contain a `SKILL.md`, a mapping file, a resolver and an audit script, and an objects module notes file. + +### Patterns to avoid + +Observed in the existing skills as of 2026-10-07 (the reviewed checkouts, ably-cocoa `b5074d9b` and ably-java `0ff24017`; unchanged on remote `main` as fetched that day); some may since be fixed. Each is a pattern a new skill shouldn't copy. + +**In both skills** + +- **Fetching the manual and features specs from `main`** while reading specs from a local clone ([9.4](#94-local-clone-vs-fetching-main)). +- **Two sources of tier truth**: path-based in the resolver, content-based in the `SKILL.md` integration section. +- **A `CONTAINS_IN_ORDER` mapping that isn't a subsequence check** ([6.7](#67-notes-on-the-catalogue)). +- **No re-sync mode, no final report, no lint step, no collision detection**, and a silent `null` when a declared notes file is missing. + +**Seen in `uts-to-swift`** + +- **Example code that doesn't compile**: the callback-bridging helper in "Assertions (Swift Testing)" has a `guard` body that doesn't exit and doesn't resume the continuation. (The real helper in `TimeTests.swift` is correct.) +- **Describing features that don't exist**: a per-tier class-name suffix the resolver doesn't implement, and a suite that doesn't exist. +- **Contradictory timeout guidance**: "pass an explicit `timeout:` only when the spec states one" alongside "generous timeouts (10–30s)" examples. +- **Pointing to rules in the wrong doc**: it says the value-assigned `poll_until` rule lives in `writing-derived-tests.md`; the reference definition is in `writing-test-specs.md`. +- **Deviations-file rules that contradict the manual**: it says to add the *UTS Spec Errors* heading "on first use if absent"; the manual requires all four headings always present. +- **A harness narrower than the helper spec**: the mock WebSocket lacks the await style, `onMessageFromClient` and raw-frame handlers, ping frames, a PING template, and WS-level `respond_with_timeout` / `respond_with_dns_error`. The harness README documents the first two, the skill documents only the last, and ping is documented nowhere. +- **A sample-based state wait** at the unit tier (`awaitConnectionState`). +- **Test names without the spec point**: the objects module notes override the naming rule so that objects unit tests have bare descriptive names, relying on the `// UTS:` tag alone; `writing-derived-tests.md` requires the spec point in the name. +- **Retrying a non-idempotent provisioning request**: `SandboxApp.create()` retries `POST /apps`. `integration-testing.md` says sandbox apps auto-expire, which justifies best-effort deletion, not retrying creation: a lost response still leaves duplicate apps until they expire. Retry only idempotent requests (2.3). +- **Writing `deviations.md` before evaluation**: it records Mock Infrastructure Limitations at generation time, including in translate-only mode; the manual creates the file during evaluation (7.3). +- **Citing the wrong spec point**: it attributes the fallback-disabling behaviour of explicit `realtimeHost`/`restHost` to REC2c2; it is REC2c6 (REC2c2 covers an `endpoint` hostname). +- **Citing spec points where they don't exist**: the objects module notes place `RTTS1–RTTS10` in `objects-features.md`; they exist only on the unmerged `feature/liveobjects-cross-sdk-types-spec` branch. +- **No rules for** `process_pending_events()`, `poll_until_success`, `CLOSE_CLIENT`, `WAIT` or `BEFORE ALL`; stale statements in the objects module notes and in `Test/UTS/README.md`, which the skill tells the model to read first. + +**Seen in `uts-to-kotlin`** + +- **An audit weaker than its sibling**: it counts `awaitState` / `awaitChannelState` / `pollUntil` as assertions (so surplus waits can mask dropped assertions), counts commented-out assertions, silently collapses duplicate `@UTS` tags, and reads every fenced block rather than only pseudocode. Build the audit from the 8.2 contract instead. +- **A non-documented tag marker**: `/** @UTS */` instead of `// UTS: `. +- **A resolver that can crash**: a malformed `uts-package-mapping.json` raises a Python traceback instead of a structured `ok: false` error. +- **Cleanup at the end of the body**: the unit template closes the client there, not in `finally` or `@AfterEach`. +- **Hard-won rules left out of `SKILL.md`**: `withRealTimeout` and record-and-verify exist in the harness but not in the skill, so some integration tests still await network futures unbounded. +- **Ambiguous or stale references**: a "Proxy integration tests" section that doesn't exist; "`uts/README.md`" meaning either repo's README. +- **Invented tags and silent predicate changes in generated tests**: one proxy and one direct-sandbox test added extra `@UTS` tags for secondary spec points, and one `pollUntil` predicate differs from the spec's with no comment. From a69352423ce5ef54a86803e8035c246dee5cdb33 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 12:37:05 +0530 Subject: [PATCH 02/33] UTS: add agent procedure for generating a translator skill Adds a step-by-step procedure an LLM agent follows to create a uts-to- skill for an SDK repo: study the repo's existing code, tests and test support; design, build and verify the UTS harness; then generate the skill and validate it with pilot translations. Links both docs from uts/README.md. --- uts/README.md | 5 + .../generating-a-translator-skill.md | 1138 +++++++++++++++++ 2 files changed, 1143 insertions(+) create mode 100644 uts/docs/translator-skills/generating-a-translator-skill.md diff --git a/uts/README.md b/uts/README.md index 8cc4d9b9d..6ded173b8 100644 --- a/uts/README.md +++ b/uts/README.md @@ -33,6 +33,9 @@ uts/ ├── docs/ # Guides and reference │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests +│ ├── translator-skills/ # Building uts-to- translator skills +│ │ ├── writing-translator-skills.md # What a translator skill must contain +│ │ └── generating-a-translator-skill.md # Agent procedure to build one │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix @@ -138,6 +141,8 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations +- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` Claude Code skill: prerequisites, workflow, construct mapping, audit tooling, re-sync +- **[Generating a UTS Translator Skill](docs/translator-skills/generating-a-translator-skill.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design and build the UTS test infrastructure (harness), generate the skill, validate it with pilot translations - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs diff --git a/uts/docs/translator-skills/generating-a-translator-skill.md b/uts/docs/translator-skills/generating-a-translator-skill.md new file mode 100644 index 000000000..dbcfba260 --- /dev/null +++ b/uts/docs/translator-skills/generating-a-translator-skill.md @@ -0,0 +1,1138 @@ +# Generating a UTS Translator Skill + +This is a step-by-step procedure for an LLM agent (for example Claude Code) to create a `uts-to-` translator skill for one Ably SDK repository. [Writing UTS Translator Skills](writing-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why. This document says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this document links to it, and you must read the linked section. + +The order is fixed, because each step stands on the one before it: + +1. **Understand the repo:** how the SDK is built, tested and mocked today (Phase 1). +2. **Design the UTS test infrastructure (the harness)** for this language from what you found: what to reuse, what to wrap, what to build (Phase 2). +3. **Build and verify the harness** (Phase 3). +4. **Create the `uts-to-` skill on top of it** (Phases 4 and 5). +5. **Validate the skill with pilot translations** (Phase 6), then report (Phase 7). + +A skill written before its harness exists can't make translation mechanical (guide [section 2](writing-translator-skills.md#2-prerequisites-in-the-sdk-repo)), and a harness designed without studying the repo's existing tests duplicates or fights them. A human may skim this document to see what the agent will do and where it will ask for decisions. + +## Contents + +1. [Purpose and how to use this procedure](#1-purpose-and-how-to-use-this-procedure) +2. [Ground rules](#2-ground-rules) +3. [Phase 1: understand the repo](#3-phase-1-understand-the-repo) +4. [Phase 2: design the harness](#4-phase-2-design-the-harness) +5. [Phase 3: build and verify the harness](#5-phase-3-build-and-verify-the-harness) +6. [Phase 4: skill design decisions](#6-phase-4-skill-design-decisions) +7. [Phase 5: generate the skill files](#7-phase-5-generate-the-skill-files) +8. [Phase 6: validate the skill](#8-phase-6-validate-the-skill) +9. [Phase 7: final report and handover](#9-phase-7-final-report-and-handover) +10. [Maintenance](#10-maintenance) +- [Appendix A: Repo profile template](#appendix-a-repo-profile-template) +- [Appendix B: Gap table and harness design template](#appendix-b-gap-table-and-harness-design-template) +- [Appendix C: Design record template](#appendix-c-design-record-template) +- [Appendix D: Final report template](#appendix-d-final-report-template) +- [Appendix E: Acceptance checklist (guide section 13)](#appendix-e-acceptance-checklist-guide-section-13) +- [Appendix F: Glossary](#appendix-f-glossary) +- [Appendix G: Common failure modes and remedies](#appendix-g-common-failure-modes-and-remedies) + +--- + +## 1. Purpose and how to use this procedure + +### 1.1 Who reads this + +- **The executor** is an LLM agent working inside the target SDK repository, with shell access, file read/write access, and the ability to ask the user questions. Every instruction below is addressed to it ("you"). +- **The user** is an SDK engineer who owns the target repository. They answer the questions at the stop points and approve every change outside the skill directory. +- **The lead reviewer** may skim the working records (Repo profile, Harness design, Design record, Final report) instead of the whole conversation. + +### 1.2 Invocation + +The user starts a run with one sentence, for example: + +``` +Follow /uts/docs/translator-skills/generating-a-translator-skill.md to create a uts-to- skill for this repo. +``` + +`` is the user's local clone of `ably/specification`. If the user also states goals ("objects unit tier first", "rest and realtime only"), record them as inputs. + +### 1.3 Required inputs + +| Input | Example | If missing | +|---|---|---| +| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo; confirm it has SDK sources and tests | +| Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | +| Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | **Stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | +| User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | + +### 1.4 Outputs + +1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, per-tier smoke tests, a harness README and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). +2. **The skill**, at `.claude/skills/uts-to-/` unless the Design record says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py` and `references/-mapping.md` (layout: guide [3.1](writing-translator-skills.md#31-layout)). +3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. +4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): + - `repo-profile.md` ([Appendix A](#appendix-a-repo-profile-template)); + - `uts-infra-design.md`: the gap table, the harness design and the build log ([Appendix B](#appendix-b-gap-table-and-harness-design-template)); + - `design-record.md` ([Appendix C](#appendix-c-design-record-template)), which also holds the decision log; + - `final-report.md` ([Appendix D](#appendix-d-final-report-template)), with the acceptance checklist ([Appendix E](#appendix-e-acceptance-checklist-guide-section-13)). + +Write them at the default location from Phase 1 on; if D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. + +The working records are what a later agent reads to maintain or regenerate the skill ([section 10](#10-maintenance)), so keep them accurate as you go, not only at the end. + +### 1.5 Intent + +The guide's intent ([introduction](writing-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the harness and regenerate. Never hand-fix a pilot test and call the skill done. + +### 1.6 Which document wins + +| Question | Authority | +|---|---| +| What a faithful translation is; how a failure is diagnosed; the `deviations.md` format | [`writing-derived-tests.md`](../writing-derived-tests.md) | +| What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../writing-test-specs.md) | +| Integration and proxy tiers | [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | +| Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | +| What the harness and the skill must contain, and why | the guide | +| How and in what order you build them | this document | +| The SDK's actual API | the SDK source (guide [3.1](writing-translator-skills.md#31-layout), "The notes are a map, not an authority") | + +If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. + +**Procedure recommendations.** Where this document adds detail the guide doesn't have (for example: the working-records files and their location, smoke tests as gates, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this document's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). + +--- + +## 2. Ground rules + +### 2.1 Your only inputs + +You work from exactly four sources: + +1. this document; +2. the guide; +3. the local spec clone: the UTS docs (`uts/README.md`, `uts/docs/`), the helper specs, the UTS corpus (`uts/rest`, `uts/realtime`, `uts/objects`) and, for API and spec-error questions, the features specs under `specifications/`; +4. the target SDK repo. + +**Don't look for other SDKs' translator skills or UTS harnesses**, including the two the guide cites ([guide appendix](writing-translator-skills.md#appendix-existing-skills-background-and-patterns-to-avoid)). Don't clone, fetch, browse or search for them. Their lessons are already in the guide. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), treat it as background only: never port, copy, translate or fetch it. Write every hook, helper, script, rule and example yourself, from the guide, the UTS docs and the target repo. A harness and skill built from this procedure must stand on those alone. If you think you need something only another SDK's skill could tell you, that is a gap in the guide: **STOP-12**, and record it in the Final report. + +### 2.2 What you may change + +| Action | Allowed? | +|---|---| +| Create and edit files under the skill directory and the working-records directory | Yes | +| Create new test-only harness code (shared test library, module helpers, smoke tests, UTS test-project wiring) | Only after the user approves the harness design at [STOP-3](#23-stop-points), and only within the scope they approved | +| Change existing test-support code that the SDK's own tests use (shared fakes, fixtures, base classes) | Only with explicit approval for each change, and never in a way that changes behaviour for the existing tests. Prefer wrapping or extending to editing | +| Change SDK production code, including adding test hooks or raising visibility | **Only with explicit approval for each change** ([STOP-4](#23-stop-points)). Present the proposal first ([4.6](#46-step-2f-propose-the-sdk-test-hooks)) | +| Change CI workflows, build scripts that CI runs, or dependency manifests (including adding a test dependency) | **Only with explicit approval** ([STOP-5](#23-stop-points)). Otherwise write the change as a suggestion in the Final report | +| Edit project or solution files needed to register new test files or projects (`.sln`, `.csproj`, `.projitems`, an Xcode project, etc.) | Only with explicit approval ([STOP-5](#23-stop-points)), recorded in the Design record (D-20). If the user declines, give them the exact entries to add, wait until they confirm, and list them in the Final report | +| Stage, commit, push, branch, or open a PR | **No, unless the user asks** | +| Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | +| Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | +| Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | +| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies) | No | + +### 2.3 Stop points + +At a stop point, present what you found and the options, then wait for the user's answer. Record the question and the answer in the decision log ([Appendix C](#appendix-c-design-record-template)). + +| ID | Phase | When | What you present | +|---|---|---|---| +| **STOP-1** | Start | A required input is missing ([1.3](#13-required-inputs)) | The missing input and why it's needed | +| **STOP-2** | 1 (end) | The Repo profile is written | The profile, for confirmation or correction | +| **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | +| **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](#46-step-2f-propose-the-sdk-test-hooks)) | +| **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | +| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases), or running a full suite | The command, the expected duration, what it touches | +| **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests per tier, updated tier feasibility), before any skill file is written | +| **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | +| **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | +| **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](#52-step-3b-build-in-order-with-acceptance-checks), [8.4](#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | +| **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | +| **STOP-12** | Any | You would need another SDK's skill or harness to proceed ([2.1](#21-your-only-inputs)) | The question you can't answer from the guide | + +### 2.4 Never guess + +- **Spec content:** quote the spec, a helper spec or a UTS doc from the clone, with its path and heading. Don't rely on memory of the Ably specification. +- **API signatures:** read the SDK source and the harness source before writing any call. A signature you haven't read is a guess. +- **Commands:** run every build, test, lint and script command you record, and record its exit status. A command you haven't run is marked "unverified" in the working records. +- **Facts about the repo:** every Repo profile entry has evidence (a file path, a symbol or a command and its output). "Unknown" is allowed only with the reason and a question for the user. +- **Undocumented corpus constructs:** the guide marks some constructs *(undocumented)*; their meaning is inferred from usage. Read each use in the corpus before choosing a rendering, and say in the catalogue that the meaning is inferred. + +### 2.5 Record every decision + +Every choice you make that isn't forced by the guide or the UTS docs goes into the decision log in the Design record, with the options, the choice, who made it (you or the user) and the reason. That includes harness choices in Phases 2 and 3 (reuse vs build, hook shapes, placement), not only the skill decisions in Phase 4. If a later phase changes an earlier decision, add a new log entry that supersedes the old one; don't edit history. + +### 2.6 Pin the spec clone + +At the start of the run: + +```sh +git -C rev-parse HEAD # full SHA +git -C status --porcelain -- uts specifications +git -C log -1 --format='%H %cd' -- uts/docs/translator-skills +``` + +Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), and the guide's last-change commit. You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. + +Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. + +### 2.7 Working style + +- Do Phase 1 before anything else, and don't write harness code before the user approves the harness design (STOP-3), or skill files before the harness is verified (STOP-7). +- Escalate verification in the repo's own order (compile, then build tests, then a filtered run, then wider runs), as recorded in the Repo profile. +- Keep the working records up to date as you go; they are also your memory if the session is interrupted. On resuming, read them first. +- Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. + +--- + +## 3. Phase 1: understand the repo + +**Goal:** a written, user-confirmed Repo profile that explains how the SDK is built, tested and mocked today, in enough depth to design the harness from it. This phase is mandatory and comes before anything else: the harness you design in Phase 2 should extend what the repo already has, follow its test style, and plug into its build and CI, and the skill's file locations, idioms and commands all derive from this profile. + +### 3.1 Step 1a: read the repo's own instructions + +Read these first, if they exist, and follow them for the rest of the run: `CLAUDE.md` (root and nested), `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `.editorconfig`, any `docs/` or `Docs/` pages about testing, and any existing `.claude/` directory (settings, skills, commands). Record any instruction that constrains you (for example "don't edit project file X", "use helper Y instead of Z"). + +### 3.2 Step 1b: answer the discovery checklist + +Answer every item. For each, record the answer and its evidence in the Repo profile. The "how to find it" column lists starting points; follow the evidence wherever it leads. + +**Build, test and CI** + +| # | Find | How to find it | +|---|---|---| +| P-01 | **Language(s) and version(s)**: SDK language, test language (they can differ), toolchain version, language mode or edition | Toolchain pins (`global.json`, `.python-version`, `.nvmrc`, `go.mod`, `rust-toolchain*`, `Package.swift` tools version, Gradle `jvmToolchain`); `git ls-files` extension counts | +| P-02 | **Build system and commands**: build the SDK; build the tests; run all tests; run one class; run one test; run by filter; target-qualified filters | Build files (`*.sln`/`*.csproj`, `pyproject.toml`/`tox.ini`/`noxfile.py`, `package.json` scripts, `build.gradle*`, `Package.swift`, `Makefile`, wrapper scripts); CI workflow steps; CONTRIBUTING. **Run** the build-tests and single-class commands and record the exit status. For interpreted languages, "build the tests" means collection (e.g. `pytest --collect-only`) plus the repo's type checker. Record the command that lists collected tests (e.g. `pytest --collect-only -q`, `dotnet test --list-tests`) | +| P-03 | **Test framework(s)**: runner, assertion library, parameterisation, suite-level fixtures (before/after all), teardown hooks, **static and runtime skip idioms**, unconditional-fail idiom, **collection rules** (what names and files the runner picks up) | Test dependencies in the build files; existing tests; the runner's config (`pytest.ini`, `conftest.py`, `xunit.runner.json`, `jest.config.*`). Check whether a runtime skip needs an extra package (guide [7.1](writing-translator-skills.md#71-three-acceptable-end-states)) | +| P-04 | **Test layout and naming**: test projects/targets/source sets, their directories, naming conventions for files, classes and methods, shared test-support projects, and which test targets can see which code | Directory listing of test roots; build files for test targets | +| P-05 | **CI**: every workflow that builds or tests; which job runs which suites; OS and runtime matrix; network access; submodule checkout; secrets; how a new suite or test target would be picked up | `.github/workflows/*`, other CI configs; follow each test step to the command it runs | +| P-06 | **Lint, format and strictness gates**: EditorConfig, formatters, linters, static analysis, type checkers; whether CI treats warnings as errors; whether the local build is laxer than CI | Lint configs, `Makefile` lint targets, CI lint jobs, compiler flags in build files | +| P-07 | **Developer and CI platforms**: which OSes developers and CI use; whether the script runtime the skill will use (e.g. `python3`) is available on all of them; where a proxy binary would have to run | CONTRIBUTING, CI matrix, existing scripts | +| P-08 | **Repo conventions**: licence or file headers, comment style, line endings, commit message rules, files kept in sync (e.g. several dependency manifests), project files not discovered by the build | CLAUDE.md, CONTRIBUTING, `.editorconfig`, `.gitattributes`, recent commits | + +**How the SDK is structured for testing** + +| # | Find | How to find it | +|---|---|---| +| P-09 | **Async model**: callbacks, futures/promises, `async`/`await`, coroutines; whether SDK APIs are callback-based, async, or both; the test runner's async support and its mode; whether the runner **virtualises time** | SDK public API signatures; runner plugins (e.g. pytest-asyncio mode); existing async tests (guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)) | +| P-10 | **Concurrency and threading**: which thread, queue or loop SDK callbacks run on; whether listeners can run concurrently with the test body; how to drain the SDK's internal work queue (this becomes `process_pending_events()`); whether the language checks concurrency safety at compile time | SDK source: dispatch, executor or event-loop code; how existing tests wait for callbacks | +| P-11 | **Time model**: **every time source** the SDK uses: timers, timed blocking waits, scheduled or delayed callbacks, wall-clock and monotonic reads | Grep the SDK source for the language's sleep, timer, wait, delay and now APIs; list each call site's owner (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)) | +| P-12 | **Runner parallelism**: are tests run in parallel by default, and how is it turned off per suite? | Runner docs and config; existing serialisation attributes or collection settings | +| P-13 | **Injection points (test hooks)** for each of: WebSocket transport, HTTP client, clock/timers, reachability/network monitor, randomness (retry jitter, fallback-host shuffle). For each: symbol, how it is set (option, constructor argument, global, subclass), whether it is set at client construction, per-client or process-global, test-only or public | SDK client-options and any debug/test-options types; constructors of the transport, HTTP, timer and network layers; existing tests that replace them | +| P-14 | **Internal-access mechanism**: how tests reach non-public code, and where the repo already uses it | e.g. `InternalsVisibleTo`, `@testable import`, private headers and module maps, same-package placement, `_`-prefixed conventions; grep for existing test-only accessors | +| P-15 | **SDK API surface vs the pseudocode, per module**: client constructors and options, async return types, the error type and how `ErrorInfo` fields (`code`, `statusCode`, `message`, cause) are read, enum renderings, protocol-format option (JSON vs msgpack), `endpoint` vs explicit hosts, the plugin mechanism. For `objects` (LiveObjects): is it implemented, where, as a plugin or built in, typed or dynamic, and which spec IDL it follows | Public API; compare with the pseudocode in two or three specs per module and with `specifications/features.md` / `objects-features.md` in the clone. Produce a short divergence list per module | + +**Existing native test support** + +| # | Find | How to find it | +|---|---|---| +| P-16 | **Existing mocks, fakes and test doubles**: for HTTP, WebSocket/transport, clock/timers, network monitor, randomness, delta decoding, plugins. For each: file and symbol, what it can do (capture requests or frames, inject responses or server messages, simulate refusal, timeout, DNS failure, disconnect), its style (handler/callback, queued responses, await), whether it is process-global, and which native tests use it | Search the test roots for `Mock`, `Fake`, `Stub`, `Test*Transport`, `*Handler`, clock or timer types; follow the hook symbols from P-13 to their test-side implementations | +| P-17 | **Existing test helpers**: state waits, polling helpers, async-to-sync bridges, event capture or recorder types, log capture, custom assertions, client factories, test-options builders | Shared test-support code; base test classes; helpers used by many test files | +| P-18 | **Existing sandbox and integration support**: how native integration tests provision a sandbox app (from `test-app-setup.json`?), point clients at the sandbox, retry, clean up; any proxy or fault-injection tooling | Integration test directories and their fixtures; environment variables they read | +| P-19 | **`ably-common`**: submodule path and name (it may not be called `ably-common`), whether it is initialised, and whether `test-resources/test-app-setup.json`, `msgpack_test_fixtures.json` and `encoding.json` are present | `.gitmodules`; `git submodule status`; list the files | +| P-20 | **Existing UTS assets and their state**: any UTS harness, UTS-derived tests (`UTS:` tags), `deviations.md` files, previous `uts-to-*` skills in this repo. For each asset: does it compile, do its tests pass, which spec SHA do headers name, which helper-spec symbols exist | `grep -rn "UTS:" `; search for `MockWebSocket`, `MockHttpClient`, `SandboxApp`, `uts-proxy`, `deviations.md`, `.claude/skills/uts-*` | +| P-21 | **Sandbox access**: whether tests can reach `sandbox.realtime.ably-nonprod.net` from developer machines and CI | Existing integration tests; CI network settings (guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)). Probing the sandbox, or running existing integration tests for P-20, is network access: **STOP-6** first | + +Useful generic commands (adapt to the platform): + +```sh +git -C rev-parse HEAD; git -C status --porcelain +git -C ls-files | sed 's/.*\.//' | sort | uniq -c | sort -rn | head -20 # extension counts +git -C ls-files | grep -iE '(^|/)(test|tests|spec)s?/' | head -50 # test roots +git -C submodule status +grep -rnE "class (Mock|Fake|Stub)|(Mock|Fake)[A-Z][A-Za-z]*(Transport|Http|Clock|Timer|Socket)" | head -40 +grep -rn "UTS:" | head # previous UTS ports +``` + +### 3.3 Step 1c: study how the existing tests drive the SDK + +The checklist tells you what exists; this step tells you how it is used. Read, in full, at least one existing native test of each kind below (if the repo has it), and write down the pattern each follows: how the client is created, how the double is installed, how the test waits, how it asserts, how it cleans up. + +| Kind | What to learn from it | +|---|---| +| A REST test that checks an outgoing request | How HTTP is intercepted; how requests are captured and responses injected | +| A realtime test that drives connection state | How the transport is replaced; how server messages are injected; how state changes are awaited | +| A test of timer-driven behaviour (retry, timeout, heartbeat) | Whether time is faked or real, and how it is advanced | +| A test that checks something did **not** happen | How the test flushes pending work before a negative assertion | +| A test that reaches internal state | The internal-access mechanism in practice | +| An integration test against the sandbox | Provisioning, client options, timeouts, cleanup | +| A test from each plugin or optional module (e.g. LiveObjects) | How the plugin is registered in tests; its own test helpers | + +Then answer, in the Repo profile: + +- Which existing doubles behave like the helper specs need (see the semantics in [4.3](#43-step-2c-map-each-helper-spec-to-native-code)), and which differ? +- Which existing waits are race-free (subscribe first, then check), and which sample state or sleep? +- Where do the existing tests flake or carry workarounds (retries, sleeps, `@Ignore`/skip with a reason, comments about races)? These mark concurrency or timing problems the harness must avoid. + +### 3.4 Step 1d: write and confirm the Repo profile + +Write `repo-profile.md` from [Appendix A](#appendix-a-repo-profile-template). Record the repo's HEAD SHA and whether it is dirty. Then **STOP-2**: show the user the profile, highlight every "unknown" and every inference, and ask them to confirm or correct it. Apply their corrections before continuing. + +**Done when:** every P-row has an answer with evidence or an explicit "unknown" with a question; the build-tests and single-class commands have been run (or marked unverified with the reason); the existing-test patterns of 3.3 are written down; the user has confirmed the profile. + +--- + +## 4. Phase 2: design the harness + +**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](writing-translator-skills.md#2-prerequisites-in-the-sdk-repo) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. + +Design principles: + +- **Follow guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)**: implement each helper spec once, and build the harness to look like the pseudocode. When an existing double has the right behaviour but a different shape, **wrap it in a pseudocode-shaped facade** rather than teaching the skill to translate into the old shape. +- **Reuse what is correct; don't reuse what is subtly wrong.** An existing double that misses a helper-spec guarantee (for example a synchronous `close()`) will make the UTS ports flaky. Extend or replace it for the UTS ports, leaving the native tests' behaviour unchanged. +- **Hooks** are injected at client construction, per-client where the SDK allows (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)). + +### 4.1 Step 2a: list the required capabilities + +Start the gap table from rows G-01 to G-18 of [Appendix B](#appendix-b-gap-table-and-harness-design-template), which cover every requirement of guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) to [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. + +**Check:** every bullet and table row of guide section 2 maps to a gap-table row. + +### 4.2 Step 2b: assess each capability: reuse, wrap, extend or build + +This is the guide's "Reuse before you build" ([2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)) applied to every row. For each row, record the status (`present`, `partial`, `missing`, `n/a`), the evidence (file and symbol, or what you searched for), and the **approach**: + +| Approach | When | +|---|---| +| **Reuse** | An existing component already meets the requirement's semantics and shape (verified, not assumed) | +| **Wrap** | The semantics are right but the shape differs from the pseudocode; add a facade with the spec's names | +| **Extend** | The component is close but lacks capabilities (e.g. no WS-level timeout, no raw frames); add them without changing existing behaviour | +| **Build** | Nothing suitable exists, or what exists is semantically wrong for UTS | +| **n/a** | The SDK lacks the feature (e.g. `MockVCDiff*` when the SDK has no delta support); confirm against the features spec in the clone and give the reason | + +Also record which tiers and modules each row blocks, whether closing it needs a production-code change, and a size (S/M/L). "Partial" always says what is missing. + +If harness code already exists, **check it, don't trust it**: compile it, run its tests by filter, and compare its behaviour with the semantics in 4.3 and 4.4. + +### 4.3 Step 2c: map each helper spec to native code + +Read each helper spec in full, then design its native implementation. For each, produce a symbol table in `uts-infra-design.md`: every symbol the helper spec declares **and** every harness symbol the corpus uses (find them with the scanner in [7.4](#74-step-5d-fill-the-construct-catalogue); the corpus uses members the helper specs don't declare) → native type and member → approach (reuse, wrap, extend, build) → the existing component it builds on, if any. + +| Helper spec | Design questions to answer | Requirements to check against | +|---|---|---| +| [`mock_http.md`](../../rest/unit/helpers/mock_http.md) | Which HTTP hook does `install_mock` use? Does an existing HTTP double capture method, URL, headers and body? Can it fail at connection level separately from request level? Will the legacy `queue_*` calls be implemented, or mapped onto `onRequest` handlers in the skill? | The helper spec; guide 2.2 `mock_http.md` row | +| [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md) | Which transport hook installs it? Does an existing transport double exist, and is its `close()` asynchronous? How are typed messages and raw untyped frames sent to the client? How are client frames captured? | The helper spec (including "Async Behavior and Event Loop Considerations"); guide 2.2 `mock_websocket.md` row and the notes below it | +| [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md) | Does the SDK support deltas, and how is a decoder registered? | The helper spec; guide 2.2 | +| [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | Which channel and plugin setup does `setup_synced_channel` need? Where do module helpers live (where internals are visible, if white-box specs need them)? How is the plugin registered in a client-options builder? | The helper spec; guide 2.2 `standard_test_pool.md` row and "Document each fixture helper's scope" | +| *(no helper spec)* `MockNetworkListener` | Is there a network-monitor hook? | guide 2.2 last table row; `realtime/unit/connection/network_change_test.md` | + +Where a design question can only be answered by a production change, it becomes a hook proposal (4.6). + +### 4.4 Step 2d: design the wait and assertion helpers + +Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the tests that prove them are in [5.2](#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. + +| Helper | Requirements | +|---|---| +| `AWAIT_STATE` (connection and channel) | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | +| `poll_until`, `poll_until_success` | `writing-test-specs.md`; `uts/README.md`; guide 5.7; `writing-derived-tests.md` "No real timers in unit tests" (deadlines) | +| `process_pending_events()` | `uts/README.md`; guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | +| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock" | +| Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must), [5.6](writing-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | +| Thread-safe capture, log sink, caller-attributed failures | guide 2.2 | +| `assertContainsInOrder` | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | + +### 4.5 Step 2e: design the sandbox and proxy helpers + +- **Sandbox** (guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). +- **Proxy** (guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). + +### 4.6 Step 2f: propose the SDK test hooks + +For every hook row that isn't `present`, write a proposal in `uts-infra-design.md`: + +- the API (name, type, where it is set), modelled on the repo's existing test options if it has any (P-13); +- the default behaviour (unchanged for users); +- its visibility (test-only if the language allows); +- every production file it touches; +- how the harness installs it at client construction, and whether it is per-client; +- for the clock hook, the list of time sources from P-11 it covers, including timed blocking waits and the async runtime's timers. + +These proposals are presented at STOP-3 and need individual approval at **STOP-4** before you implement them. + +### 4.7 Step 2g: plan placement, smoke tests and build order + +- **Placement** (guide 2.2, "Placement"): name the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). +- **Smoke tests**, one per tier you plan to support: structural acceptance tests only, with no `UTS:` tag, never copied by the skill (guide 2.2). The acceptance checks in [5.2](#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. +- **Build order:** follow the order in 5.2, dropping rows that are `present` or out of scope. + +### 4.8 Step 2h: derive tier feasibility + +Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "objects not implemented in this SDK: translate-only at most"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). + +### 4.9 Step 2i: STOP-3, approve the design and choose the scope + +Present `uts-infra-design.md`: the gap table, the helper-spec mappings, the wait-helper design, the sandbox and proxy design, the hook proposals, placement, the build order and the tier matrix. Ask the user to choose: + +- **(a) Build the missing harness now**, all or a named subset. Recommend this option (the skill depends on the harness), but the user chooses. Continue with Phase 3. +- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) step C requires (see the readiness data in [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). +- **(c) Scope the skill to the tiers possible today.** As (b), without the plan. + +Record the answer and any design changes the user asked for. Under (b) and (c), the skill's tier step (C) must offer only ready tiers, and the Final report lists the rest as known limitations. + +--- + +## 5. Phase 3: build and verify the harness + +**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has a passing smoke test. Skip the rows the user didn't choose to build at STOP-3. + +### 5.1 Step 3a: rules while building + +- **Production hooks only after STOP-4 approval**, one proposal at a time, in the smallest form that works. +- **Don't change existing behaviour** of doubles or helpers the native tests use; extend, wrap or add new code instead (see [2.2](#22-what-you-may-change)). +- Follow the repo's test style (P-03, P-04, P-08): naming, file headers, assertion library, formatting. +- Run the existing tests that touch anything you changed, by filter, before and after the change. +- Log every design change you make while building in the decision log; update `uts-infra-design.md`. + +### 5.2 Step 3b: build in order, with acceptance checks + +Each step has an acceptance check; don't start the next step until it passes (or the user accepts a documented exception). **Bound:** if a row's check still fails after three fix attempts, stop (**STOP-10**) with a summary: the failure, what you tried, your diagnosis, and the options (change the design and log it, accept a documented exception, or descope the tiers the row blocks). + +Treating the smoke tests as gates is a *procedure recommendation*; the guide makes smoke tests a SHOULD ([2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)). + +| Order | Build | Acceptance check | +|---|---|---| +| 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal smoke test (one trivial passing test) compiles, is collected and passes under the single-class filter | +| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)); a failing helper reports the caller's line | +| 3 | Approved SDK hooks | Each hook replaces the real implementation for one client without affecting another client created afterwards (or, if process-global, that fact is recorded for decision D-14); the existing tests still pass | +| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)) | +| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test:** a request is captured with method, path, headers and body; connection-level and request-level failures are distinguishable | +| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test:** connect to CONNECTED via `respond_with_success`; `close()` calls `onClose` asynchronously; CONNECTED is delivered after the connection completes; a raw untyped frame can be sent; a client frame is captured | +| 7 | Wait helpers per [4.4](#44-step-2d-design-the-wait-and-assertion-helpers) | Tests of the helpers: a state reached before the wait starts and one reached during it both complete; a FAILED transition fails fast; `poll_until` returns the value; `poll_until_success` on an always-failing condition raises the last error; a poll deadline still expires with the fake clock installed; `process_pending_events()` lets a queued callback run with no real delay | +| 8 | `MockNetworkListener` (if the SDK has a network monitor), `MockVCDiff*` (if the SDK supports deltas) | A smoke test drives each one | +| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test:** provision, connect, close, delete; `POST /apps` is never retried | +| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test:** create a session with one rule, connect through it, read the event log, close the session; a rule matching `ATTACHED` by string action fires | +| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test:** `setup_synced_channel` completes and one pool object can be read; the serial helpers sort as the helper spec requires | +| 12 | Harness README describing what exists | See [5.3](#53-step-3c-write-the-harness-readme) | +| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing (guide [2.5](writing-translator-skills.md#25-build-and-ci-wiring-must)) | + +Then compile all test targets as strictly as CI does (P-06) and run the repo's lint gates over the new code. + +### 5.3 Step 3c: write the harness README + +Write a README next to the harness covering everything guide [3.1](writing-translator-skills.md#31-layout) and [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: + +- take the symbol map from the [4.3](#43-step-2c-map-each-helper-spec-to-native-code) tables as built, and the known gaps from the gap table's `partial` and `missing` rows; +- add the layout (shared, port-only, module helpers, test targets) and how to run each smoke test; +- link each fixture helper's scope to the module notes (item 9), where guide 2.2 puts it; +- record the README's full repo path in the Design record. + +### 5.4 Step 3d: STOP-7, confirm the harness + +Re-run the assessment of [4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test results per tier, the files created or changed, and the updated tier matrix. Only then continue to the skill. + +**Done when:** every row the user chose to build is `present`; each supported tier has a passing smoke test; the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. + +--- + +## 6. Phase 4: skill design decisions + +**Goal:** every choice the skill bakes in is made once, with the user, and written down. + +### 6.1 Step 4a: draft the Design record + +For each decision below, propose an answer derived from the Repo profile, the built harness and the guide, then present them together. Write them into `design-record.md` ([Appendix C](#appendix-c-design-record-template)). + +| ID | Decision | Guide reference | How to propose it | +|---|---|---|---| +| D-01 | Skill name and location | [3.1](writing-translator-skills.md#31-layout), [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/` | +| D-02 | Script language and minimum runtime version | [3.1](writing-translator-skills.md#31-layout), [8](writing-translator-skills.md#8-deterministic-tooling) | Python 3 unless P-07 shows it isn't available on every developer platform | +| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](writing-translator-skills.md#32-the-mapping-file), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](#47-step-2g-plan-placement-smoke-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that | +| D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](writing-translator-skills.md#51-traceability), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | +| D-05 | Name collisions with native tests | [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | +| D-06 | **Tag syntax** | [5.1](writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test function | +| D-07 | File header format | [5.1](writing-translator-skills.md#51-traceability), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@`, disclosures; integration/proxy also the corresponding unit spec | +| D-08 | **`deviations.md` location** per owning test module | [7.3](writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | +| D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | +| D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | +| D-11 | Pending idiom for a missing API (registered, skipped, not commented out) | [7.1](writing-translator-skills.md#71-three-acceptable-end-states), [7.4](writing-translator-skills.md#74-language-inapplicable-inputs) | The framework's static skip or pending marker | +| D-12 | **Internal-access ladder**: the existing exposure to check first, the sanctioned way to add a test-only exposure, who to escalate to; where the white-box spec list lives | [5.9](writing-translator-skills.md#59-internal-access-white-box-unit-specs) | From P-14. List white-box specs in the module notes or the mapping file | +| D-13 | **Delegation policy** for delegating specs and tests | [5.3](writing-translator-skills.md#53-structural-variants) | The guide's recommendation (one test with the delegating tag, running the referenced REST cases through a shared helper; else pending with a `NOTE`). The guide calls this new policy: confirm with the user | +| D-14 | Parallelism: serialise UTS suites, or rely on per-client hooks | [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | From P-12 and the hooks as built | +| D-15 | Teardown mechanism (scope, hook, `finally`) and suite-level fixtures for `BEFORE ALL TESTS` | [5.4](writing-translator-skills.md#54-setup-and-teardown) | From P-03 and the harness | +| D-16 | Protocol-variant rendering (`PROTOCOL`) | [5.3](writing-translator-skills.md#53-structural-variants) | The framework's parameterisation over `json`/`msgpack`; tag stays singular | +| D-17 | Default timeouts and poll interval | [5.6](writing-translator-skills.md#56-time-and-waiting) | As guide 5.6 "Every wait is bounded": the spec's timeout where it states one; otherwise an explicit 10–30 s timeout on every integration `poll_until` and 10–15 s helper defaults for state waits. Record the chosen values | +| D-18 | Auth through the proxy | [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | As designed in [4.5](#45-step-2e-design-the-sandbox-and-proxy-helpers) | +| D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](#74-step-5d-fill-the-construct-catalogue)) | +| D-20 | **Commit policy**; project/solution-file edits | [10](writing-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | +| D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](writing-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | +| D-22 | `allowed-tools` and remote access | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash, Read, Edit, Write`; no `WebFetch`, because everything is read from the local clone | +| D-23 | Which modules get notes files, and which are placeholders | [3.1](writing-translator-skills.md#31-layout), [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | +| D-24 | Location of the working records, and whether they are kept in the repo | [1.4](#14-outputs) of this document | Default `.claude/skills/uts-to-/generation/`, kept for maintenance | +| D-25 | Pilot specs and mode per tier | [8.3](#83-pilot-translate-one-spec-per-available-tier) of this document | One per ready module × tier in scope, and at least one per module with a full notes file, chosen by the criteria there | + +### 6.2 Step 4b: STOP-8, confirm + +Show the Design record. Make clear which decisions follow directly from the guide (the user may still override them, but should know they are departing from it) and which are genuine choices. Record the answers. Don't generate any skill file before this confirmation. + +--- + +## 7. Phase 5: generate the skill files + +Generate the files in this order: each one depends on the previous ones. `SKILL.md` comes last because it references all the others; its outline is fixed by the guide, so you can draft it early, but finish it last. + +### 7.1 Step 5a: create the layout and the mapping file + +1. Create the directory layout from guide [3.1](writing-translator-skills.md#31-layout) (plus the working-records directory, D-24). +2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](writing-translator-skills.md#32-the-mapping-file). +3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. + +**Acceptance:** + +- [ ] Every module the user wants has an entry; every tier value is one path relative to the repo (or a declared root), never machine-absolute. +- [ ] Every derived namespace is unique (a module segment after the tier, or a module-specific prefix). +- [ ] `notes` paths are relative to the skill directory and the files exist (placeholders at this point are fine). +- [ ] Hand-maintained entries are marked (e.g. in `_comment`) and listed in the Design record. +- [ ] Every tier the harness design marks unready has an `unready` entry with the blocking gap-table row. +- [ ] The file is valid JSON, UTF-8, LF line endings. + +### 7.2 Step 5b: write the resolver + +Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. + +Output contract: the fields of guide 8.1, including `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: + +```json +{ + "ok": true, + "sourceModule": "realtime", + "specRepo": { "sha": "", "dirty": false }, + "mapped": true, + "translationNotes": "", + "tiers": { + "unit": { + "present": true, + "sourceDir": "/uts/realtime/unit", + "targetDir": "", + "namespace": "", + "buildTarget": "", + "ready": true, + "specs": [ { "file": "connection/connection_failures_test.md", "className": "ConnectionFailuresTests" } ] + }, + "integration": { "present": true, "...": "..." }, + "proxy": { "present": true, "ready": false, "blockedBy": "G-16: ProxySession not built", "...": "..." } + } +} +``` + +Errors are `{"ok": false, "code": "", "message": ""}`, with a distinct code for each failure guide 8.1 describes (not a UTS module path, directory not found, no tier directories, mapping missing or malformed, bad target name; the guide's example codes are a fine naming), plus one for a declared notes file that is missing (e.g. `NOTES_NOT_FOUND`) and one for a name collision (e.g. `NAME_COLLISION`). + +**Acceptance** (each verified by running the script in [8.1](#81-run-the-resolver-on-every-module)): + +- [ ] Every row of guide 8.1 is implemented: validation, error objects, output contract, path-only tier detection (including specs in tier sub-directories, e.g. `realtime/integration/channels/`), exclusions relative to the tier base, deterministic runner-collectable naming (D-04), collision detection (D-05, including file-name uniqueness where the runner needs it), validate-then-write `--create` writing the D-03 default layout, an error for a missing notes file. +- [ ] `ready`/`blockedBy` match the mapping's `unready` data; `testFile` is present wherever the file name isn't derived from `className`. +- [ ] Never crashes: any exception becomes `ok: false` with a code. + +### 7.3 Step 5c: write the audit + +Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](#21-your-only-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: + +| Test-side pattern | Source | +|---|---| +| Tag marker (`UTS:` after the line-comment marker) | D-06 | +| Assertion calls (every assertion form the catalogue maps `ASSERT …` and `FAILS WITH` to) | [7.4](#74-step-5d-fill-the-construct-catalogue) | +| Wait calls (every harness helper the catalogue maps `AWAIT`, `AWAIT_STATE`, `poll_until`, … to) | [7.4](#74-step-5d-fill-the-construct-catalogue); the harness README | +| Comment syntax (line and block), so commented-out assertions are ignored | P-01 | + +Write these as data at the top of the script, so a catalogue change is a one-line edit. Match the longest keyword first on both sides. + +Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifiable` for a spec with no `**Test ID**` lines (not zero tests); if you depart from it, document your shape in `SKILL.md`. Use one exit code for clean, a distinct one for ID problems (missing, orphan, duplicate) and another for "couldn't run". + +**Acceptance** (verified in [8.2](#82-test-the-audit-itself)): + +- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including its SHOULD end-at-heading rule (requiring that SHOULD is a *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. +- [ ] Where your parsing differs from the contract, `SKILL.md` says so (guide 8.2 "Implement exactly this, or say where yours differs"). +- [ ] SHOULD items you implemented are listed in `SKILL.md`; the ones you didn't are listed in the Final report. + +### 7.4 Step 5d: fill the construct catalogue + +This step produces the construct table that goes into `SKILL.md` (and, for module-specific rows, into the notes). + +1. **Start from every row of guide [section 6](writing-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](writing-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. +2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells; they illustrate other SDKs. +3. **Scan the corpus for constructs the guide doesn't list** (guide section 6 says the catalogue isn't exhaustive). Run the scanner below over each module the skill covers, and add a row for every uppercase keyword or snake_case call you can't account for. +4. **Give every row a status:** + - `mapped`: a direct rendering; + - `mapped (inferred)`: an *(undocumented)* construct, meaning inferred from its uses (list one use); + - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](writing-translator-skills.md#72-harness-stand-ins-are-not-deviations)); + - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation), and the Final report has a plan; + - `n/a`: no spec in the covered modules uses it (say so after scanning). +5. **No blank cells.** Collect every row the skill can't render and has no plan for, from one scan, into a single **STOP-9**. + +Corpus scanner (prints the uppercase keywords and snake_case calls used in `pseudo` fences, outside comments and string literals, with counts; compare its output with the catalogue, and use it in Phase 2 to list the harness symbols the corpus uses): + +```python +# scan_constructs.py /uts/ +import collections, pathlib, re, sys + +root = pathlib.Path(sys.argv[1]) +keywords, calls = collections.Counter(), collections.Counter() +for path in root.rglob("*.md"): + in_pseudo = False + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("```"): + in_pseudo = (line.strip() == "```pseudo") if not in_pseudo else False + continue + if not in_pseudo or line.lstrip().startswith(("#", "//")): + continue + line = re.sub(r'"[^"]*"', '""', line) # ignore string literals + line = re.split(r"\s(?:#|//)\s", line, maxsplit=1)[0] # drop trailing comments + keywords.update(re.findall(r"\b[A-Z][A-Z_]{2,}\b", line)) + calls.update(re.findall(r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\s*\(", line)) +for title, counter in (("KEYWORDS", keywords), ("SNAKE_CASE CALLS", calls)): + print("==", title) + for name, count in sorted(counter.items()): + print(f"{count:6d} {name}") +``` + +The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_SET`, `LWW`); classify those under the guide's "wire constants" and enum rows rather than as new constructs. The scanner sees only `snake_case(` calls; method-style harness calls such as `mock_ws.active_connection.send_to_client(…)` are caught, but camelCase members (`onConnectionAttempt`) are not, so also read the helper specs' interfaces. Ship it as `scripts/scan_constructs.py` in the skill, as guide [section 6](writing-translator-skills.md#6-pseudocode-construct-catalogue) and [3.1](writing-translator-skills.md#31-layout) recommend (SHOULD), because a re-sync re-scans the corpus (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 2); record the choice in the Design record (D-19). + +**Acceptance:** + +- [ ] Every guide section 6 row is present with a rendering and a status; none blank. +- [ ] Every scanner hit in the covered modules is either a catalogue row, a wire constant or enum value, a spec-local name, or internal pseudocode of a helper spec or `objects/PLAN.md` (implemented as harness or ignored, never translated). +- [ ] Every rendering names a symbol that exists in the harness or the SDK (you read it), or is marked `not available` with a plan. +- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide 5.7, 6.7). + +### 7.5 Step 5e: write the module notes + +Write `references/-mapping.md` for every module chosen in D-23. Cover the thirteen items in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should) ("A module notes file should cover"), in that order. For each item, either fill it or write "Not applicable: ". The items that most need evidence: + +- **Item 1 (source of truth and runtime status):** name the IDL the notes apply (for `objects`, check whether the SDK follows `objects-features.md` in the clone or another variant; guide 3.4 item 1 explains the typed-SDK case) and whether the module is implemented, which decides whether evaluate mode is possible. +- **Item 5 (type mapping):** for every polymorphic or union type in the pseudocode, the SDK type and which conversions throw or return null. Read the SDK source for each. +- **Item 8 (internal access):** the ladder from D-12 and the list of white-box specs. To find candidates, look for specs that construct internal classes or call members the public API doesn't have (the guide's white-box row in [6.6](writing-translator-skills.md#66-mocks-fixtures-and-harness) lists examples), then confirm each against the SDK's public surface. +- **Item 9 (helper-spec coverage):** the module's rows of the symbol tables from [4.3](#43-step-2c-map-each-helper-spec-to-native-code), as built, with each fixture helper's scope and every sanctioned stand-in. +- **Item 12 (worked example):** translate one short real test from the module by hand, name each mechanical rewrite, and make sure it compiles (it becomes one of the skill's examples; [8.7](#87-verify-the-skills-own-examples)). + +A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), "If the module's notes file exists but is only a placeholder"). + +**Acceptance:** + +- [ ] All thirteen items present or marked not applicable with a reason. +- [ ] Every SDK and harness symbol named in the notes exists at the recorded repo SHA. +- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide 4, step 1). +- [ ] No line numbers; cite symbols and headings (guide [12](writing-translator-skills.md#12-lessons-learned)). + +### 7.6 Step 5f: write `SKILL.md` + +Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide [1](writing-translator-skills.md#1-purpose-and-scope), "What the skill owns, and what it defers"). + +| Part of `SKILL.md` | Content | Guide | +|---|---|---| +| Frontmatter | `name: uts-to-`; a pushy `description` with trigger phrases; `argument-hint`; `allowed-tools` from D-22 | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | +| Usage guard | Empty argument → print the usage line and stop; placeholder paths only | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | +| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | +| Skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | +| Skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README by its full repo path; for the reference test per tier, write `none yet` until [8.8](#88-name-the-reference-tests) names one | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit) | +| Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](writing-translator-skills.md#5-translation-rules) | +| Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) | +| File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](writing-translator-skills.md#34-recommended-outlines-should) | +| Evaluation | Guide 7 rendered with D-08 to D-11 and D-21 | [7](writing-translator-skills.md#7-evaluation-and-deviations) | +| Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](writing-translator-skills.md#55-integration-tier-hygiene) | +| Verification | Guide 10 with the exact commands from the Repo profile | [10](writing-translator-skills.md#10-verification-and-ci) | +| Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should), [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| Final report | The format | [11](writing-translator-skills.md#11-final-report-format) | +| Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](writing-translator-skills.md#10-verification-and-ci) | + +**Acceptance:** + +- [ ] Every part in the table is present. +- [ ] The skill is module-generic: module-specific rules live in the notes. +- [ ] No inline copy of a semantic rule where a link to its source would do; in particular the `deviations.md` format is referenced, not redefined. +- [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills. +- [ ] Every command in it is one the Repo profile records as run. (Running each through the skill, and compiling every example, are Phase 6 checks: [8.5](#85-exercise-the-skills-other-paths), [8.7](#87-verify-the-skills-own-examples).) +- [ ] Every helper or SDK symbol it names exists. + +--- + +## 8. Phase 6: validate the skill + +**Goal:** evidence that the skill works on this repo, on top of the harness you built. This phase is mandatory. Record every result in the Final report. + +### 8.1 Run the resolver on every module + +Run the resolver on `/uts/rest`, `/uts/realtime` and `/uts/objects`, whether or not they are in scope (an unmapped module must resolve with `mapped: false`, not crash). For each, check: + +- [ ] `ok: true`; tiers `present` match `ls /uts/`. +- [ ] Spec counts match the files on disk minus the exclusions. A cross-check: `find /uts// -name '*.md' -not -path '*/helpers/*'` (add the proxy exclusion for the integration tier). +- [ ] No helper spec, `README.md`, `PLAN.md` or `*_SUMMARY.md` appears in `specs`. +- [ ] Every `className` follows D-04 and is collected by the runner; no collisions (or the collision is reported). +- [ ] `targetDir`, namespace and build target match the mapping and the Design record. +- [ ] The same module given as a `~`-prefixed path, and (on Windows) with backslash separators, resolves identically (guide 8.1 "Validate the module"). + +Then run the negative cases and check each returns `ok: false` with the right code: a path whose parent isn't `uts`; a missing directory; a module with no tier directories (a scratch directory named `uts/` will do); a mapping entry whose notes file is missing (use a scratch copy of the mapping); `--create` with an invalid name; `--create` on a scratch copy of a deliberately corrupt mapping (the file must be left unchanged). + +### 8.2 Test the audit itself + +Use one known-good test file (the pilot output from [8.3](#83-pilot-translate-one-spec-per-available-tier) once it is reviewed; until then, a hand-written file with the correct tags) and make scratch copies for each mutation. If you started with the hand-written file, run the mutations again once the pilot output passes review: + +| Mutation | Expected audit result | +|---|---| +| Delete one assertion | A positive assertion shortfall for that test | +| Comment out one assertion | The same shortfall (commented assertions don't count) | +| Duplicate one test's tag | `duplicate` lists the ID; distinct non-zero exit | +| Add a tag that isn't in the spec | `orphan` lists it | +| Remove one tag | `missing` lists it | +| Add surplus waits to a test with a deleted assertion | The assertion shortfall is still reported (counts aren't summed) | +| Delete the poll call that renders a spec `poll_until` (with no `AWAIT` on the spec line) | An await shortfall for that test (spec polls count as waits) | + +**Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l`` (as of `12540dcf`: rest 571, realtime 554, objects 339). + +### 8.3 Pilot-translate one spec per available tier + +Choose one spec per ready module × tier in scope, and at least one per module with a full notes file (D-25). A good pilot is short, representative of the tier, and uses the harness features most likely to be wrong: + +| Tier | Choose a spec that uses | Example candidates in the corpus | +|---|---|---| +| REST unit | `MockHttpClient`, request capture, `FAILS WITH` | `rest/unit/time.md` | +| Realtime unit | `MockWebSocket`, `enable_fake_timers`, `ADVANCE_TIME`, `AWAIT_STATE` | `realtime/unit/connection/when_state_test.md` | +| Objects unit | `setup_synced_channel`, builders, serial helpers | `objects/unit/internal_live_counter_api.md` (then a mutation spec) | +| Direct integration | `SandboxApp`, `BEFORE ALL TESTS`, `## Protocol Variants` | `rest/integration/publish.md` (has `## Protocol Variants`), or `rest/integration/time_stats.md` then `realtime/integration/channel_history_test.md` | +| Proxy | `create_proxy_session`, rules, the event log | `rest/integration/proxy/rest_fallback.md` or `realtime/integration/proxy/connection_resume.md` | + +The candidates are examples only. Before choosing one, confirm it exists in your clone (`ls /uts/`) and still uses the features listed (grep for them); check it against the module's scope and the SDK's features. Prefer specs with no existing UTS-derived test in the repo (P-20); if the best candidate has one, ask the user before the skill adds to or regenerates it. + +**Run the pilot the way a user would.** + +- **Loading the skill.** A skill created during a session may not be listed until a new session starts. Ask the user to start a new Claude Code session in the repo (or to reload skills, if their client supports it) and confirm `/uts-to-` is listed. A fresh session also tests that the skill is context-complete on its own (guide intent). +- **Answering its questions.** The answers are the D-25 choices, confirmed at STOP-8 (mapping confirmation, tier, specs, translate-only or evaluate). Either the user runs `/uts-to- /uts/` in the new session and answers with them, or, if the skill is listed in a session you control, you invoke it and answer with exactly those choices. If you use a subagent, give it only the invocation and the D-25 answers, none of this run's context. Any answer not covered by D-25 goes to the user. +- **Record each question the skill asked.** More than the four questions, or a question it should have answered from its own files, is a skill defect ([8.4](#84-fix-the-cause-then-regenerate)). + +Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: + +1. Compile (and collect: the collected count equals the number of `UTS:` tags). +2. Audit; walk the review checklist (guide [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit)). +3. In evaluate mode, and after **STOP-6** for network tiers, run the class by filter and diagnose per `writing-derived-tests.md` Phase 2. +4. Review the generated file line by line against the spec yourself, and show it to the user. + +### 8.4 Fix the cause, then regenerate + +For every defect in a pilot test: + +1. Classify it: a rule in `SKILL.md`, a mapping in the notes, a catalogue row, a resolver or audit bug, or a harness gap. (A UTS spec error is not a skill defect: record it, emit the fail-fast test per D-10, and **STOP-10**.) +2. Fix that cause. **Never edit only the generated test.** A harness fix follows the rules of [5.1](#51-step-3a-rules-while-building), re-runs the affected smoke tests and updates the harness README. +3. Delete the generated test and regenerate it with the skill. +4. Re-audit, recompile and, in evaluate mode, re-run. +5. Log the defect and the fix in the Design record's changelog. + +**Bound:** once a pilot spec has been regenerated three times (or the D-21 count) without a clean audit and, in evaluate mode, one of the three end states, stop (**STOP-10**) with a summary of every defect found and fixed so far. Count all regenerations of that spec together, not per defect. + +### 8.5 Exercise the skill's other paths + +Run each path through the skill (in the session from [8.3](#83-pilot-translate-one-spec-per-available-tier)) and record the result: + +1. **Usage guard:** no argument → the usage line, then stop. +2. **Bad module path:** a path whose parent isn't `uts` → the resolver's error relayed, then stop. +3. **Placeholder notes:** a module whose notes file is a placeholder → refused, with the reason. +4. **Unready tier:** a tier the mapping marks unready → not offered at step C, with its `blockedBy`. +5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. +6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification, that the header SHA is updated, and that only the survivors in guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. +7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. + +### 8.6 Lint and CI-strict compile + +Run the repo's lint and format gates (P-06) over every file you created or changed, including the harness, the skill's scripts and Markdown where the repo's EditorConfig covers them. Compile the test targets as strictly as CI does (warnings as errors, static analysis, type checkers). Run the skill's scripts on every developer platform you can, or note which you couldn't. + +### 8.7 Verify the skill's own examples + +Every code example in `SKILL.md` and the notes must compile (guide [12](writing-translator-skills.md#12-lessons-learned)). For each example, either: + +- name the real, compiled file it was copied from (and check it still matches), or +- paste it into a scratch file in the test target, compile, and delete the scratch file. + +Record which examples were verified and how. + +### 8.8 Name the reference tests + +After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), step 3). Never name a smoke test. + +**Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected and every command in `SKILL.md` has been run; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. + +--- + +## 9. Phase 7: final report and handover + +1. Write `final-report.md` from [Appendix D](#appendix-d-final-report-template). +2. Fill the acceptance checklist in [Appendix E](#appendix-e-acceptance-checklist-guide-section-13): tick each item, or explain why it isn't met (scoped out, planned, not applicable). +3. Show the user the report, the list of changed and new files (harness, hooks, skill, pilot tests), and the suggested CI changes. **Don't commit** unless asked. +4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. + +--- + +## 10. Maintenance + +The harness and the skill must stay in step with four things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. + +| What changed | How to detect it | What to update | +|---|---|---| +| **The guide** | `git -C diff ..HEAD -- uts/docs/translator-skills/` | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](#61-step-4a-draft-the-design-record) and [7.6](#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | +| **The UTS docs or helper specs** | `git -C diff ..HEAD -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and README, the catalogue rows and notes item 9; then re-sync the affected tests | +| **The UTS corpus** | The skill's own re-sync mode (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| **The SDK** (API, hooks, test layout, CI) | `git -C diff ..HEAD` over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | + +Rules for every maintenance run: + +- Follow the same ground rules ([section 2](#2-ground-rules)) and stop points. +- Start from the working records; update the Repo profile and the harness design if anything they describe has changed. +- Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)). +- After any harness change, re-run the smoke tests; after any script change, repeat [8.1](#81-run-the-resolver-on-every-module) and [8.2](#82-test-the-audit-itself). +- Record the new spec SHA and repo SHA in the Design record and append a changelog entry. + +--- + +## Appendix A: Repo profile template + +```markdown +# Repo profile: + +- Repo: , HEAD (clean | dirty: ) +- Spec clone: , HEAD (clean | dirty: ); guide last changed in +- Date: ; confirmed by the user on (STOP-2) +- User goals: + +## Build, test and CI +| # | Item | Answer | Evidence (path, symbol or command → result) | Verified? | +|---|---|---|---|---| +| P-01 | Language(s) and version(s) | | | | +| P-02 | Build and test commands (build, build tests, all, class, single test, filter, target-qualified filter) | | | ran / unverified: | +| P-03 | Test framework, runner, assertions, parameterisation, fixtures, teardown, static and runtime skip, fail idiom, collection rules | | | | +| P-04 | Test layout, naming, targets, shared test-support code | | | | +| P-05 | CI workflows, jobs → suites, matrix, network, submodules, secrets | | | | +| P-06 | Lint/format gates, static analysis, type checkers, warnings-as-errors | | | | +| P-07 | Developer and CI platforms; script runtime availability | | | | +| P-08 | Repo conventions; hand-maintained project files; synced manifests | | | | + +## SDK structure for testing +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-09 | Async model; runner async support; virtual time | | | | +| P-10 | Threading, callback queues, internal-queue drain, compile-time concurrency checks | | | | +| P-11 | Every time source (timers, timed waits, delayed callbacks, clock reads) | | | | +| P-12 | Runner parallelism and how to serialise | | | | +| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global) | | | | +| P-14 | Internal-access mechanism and current use | | | | +| P-15 | API surface vs pseudocode, per module (rest / realtime / objects) | | | | + +## Existing native test support +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-16 | Existing mocks, fakes, test doubles (capabilities, style, global?, users) | | | | +| P-17 | Existing test helpers (waits, polls, capture, log capture, assertions, factories) | | | | +| P-18 | Existing sandbox/integration support; fault-injection tooling | | | | +| P-19 | ably-common submodule (path, initialised, fixture files) | | | | +| P-20 | Existing UTS assets and their state | | | | +| P-21 | Sandbox access from developer machines and CI | | | | + +## How the existing tests drive the SDK (step 1c) +| Kind | Test read (path, test name) | Pattern: create client / install double / wait / assert / clean up | +|---|---|---| +| REST request | | | +| Realtime connection state | | | +| Timer-driven behaviour | | | +| Negative assertion | | | +| Internal state | | | +| Sandbox integration | | | +| Plugin module | | | + +- Doubles that already meet helper-spec semantics: … +- Doubles that differ (and how): … +- Race-free waits vs sampling or sleeping waits: … +- Known flakiness and workarounds: … + +## API divergences per module +- rest: … +- realtime: … +- objects: … (implemented? plugin? typed? IDL followed?) + +## Unknowns and questions for the user +- … +``` + +## Appendix B: Gap table and harness design template + +```markdown +# Harness design: @ + +## Gap table +| # | Capability | Guide | Status | Evidence | Approach (reuse / wrap / extend / build / n/a) | Builds on | Blocks (module/tier) | Production change? | Size | Acceptance check | +|---|---|---|---|---|---|---|---|---|---|---| +| G-01 | WebSocket transport factory hook | 2.1 | present / partial / missing / n/a | | | | realtime/unit, objects/unit | yes / no | S/M/L | | +| G-02 | HTTP client hook | 2.1 | | | | | rest/unit | | | | +| G-03 | Clock hook (timers, timed waits, async-runtime timers) | 2.1 | | | | | | | | | +| G-04 | Network-monitor hook | 2.1 | | | | | | | | | +| G-05 | Randomness hooks (jitter, host shuffle) | 2.1 | | | | | | | | | +| G-06 | MockHttpClient (+ legacy queue_*) | 2.2, mock_http.md | | | | | | | | | +| G-07 | MockWebSocket (+ alternative API, undocumented members, raw frames, ping) | 2.2, mock_websocket.md | | | | | | | | | +| G-08 | MockVCDiff encoder/decoders | 2.2, mock_vcdiff.md | | | | | | | | | +| G-09 | MockNetworkListener | 2.2 | | | | | | | | | +| G-10 | standard_test_pool.md implementation; plugin options builder | 2.2, standard_test_pool.md | | | | | | | | | +| G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | +| G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | +| G-13 | Placement (shared / port-only / module helpers); fixture-helper scope documented in the module notes; harness README | 2.2, 3.1 | | | | | | | | | +| G-14 | Harness smoke tests per tier | 2.2 | | | | | | | | | +| G-15 | SandboxApp; ably-common | 2.3 | | | | | */integration | | | | +| G-16 | ProxyManager (pinned, every OS); ProxySession; rule builders; proxy auth | 2.4 | | | | | */proxy | | | | +| G-17 | UTS test target; per-tier selection and filters; one CI home per suite | 2.5 | | | | | | | | | +| G-18 | Concurrency, time, parallelism and visibility models known | 2.6 | | | | | | | | | + +## Helper-spec symbol maps (one table per helper spec) +### mock_http.md +| Spec symbol (declared or corpus-only) | Native type.member | Approach | Builds on | Notes | +|---|---|---|---|---| +### mock_websocket.md +### mock_vcdiff.md +### standard_test_pool.md +### MockNetworkListener (no helper spec) + +## Wait and assertion helpers +| Helper | Native symbol | Approach | Guarantees checked by | +|---|---|---|---| + +## Sandbox and proxy +- Sandbox provisioning: …; client targeting (endpoint or hosts): …; suite fixture mapping: … +- Proxy: version …; binaries per OS …; local override …; auth through the proxy …; platform gating … + +## Hook proposals (each needs STOP-4 approval) +| Hook | API | Default behaviour | Visibility | Files touched | Installed how | Per-client? | Approved on | +|---|---|---|---|---|---|---|---| + +## Placement +- Shared test-support: ; port-only harness: ; module helpers: + +## Tier feasibility +| Module | unit | integration | proxy | +|---|---|---|---| +| rest | ready / after G-.. / not possible: / n/a | | | +| realtime | | | | +| objects | | | | + +## Decision at STOP-3 +(a) build now: | (b) plan only | (c) scope to ready tiers — chosen by on ; design changes requested: … + +## Build log (Phase 3) / plan (option b) +| Order | Row | Work | Acceptance check | Result | Date | +|---|---|---|---|---|---| + +## Confirmation at STOP-7 +Smoke tests per tier: …; updated tier matrix: …; confirmed on +``` + +## Appendix C: Design record template + +```markdown +# Design record: uts-to- + +## Inputs at generation time +- Spec clone: (clean | dirty); guide: +- Repo: +- Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: +- Harness README: + +## Decisions +| ID | Decision | Choice | Guide-default? | Decided by | Rationale | +|---|---|---|---|---|---| +| D-01 | Skill name and location | | yes / no | user / agent | | +| D-02 | Script language and runtime | | | | | +| D-03 | Mapping: module × tier → target dir (hand-maintained entries marked) | (table below) | | | | +| D-04 | Naming and collection rules (with three worked examples) | | | | | +| D-05 | Collision handling | | | | | +| D-06 | Tag syntax | | | | | +| D-07 | File header format | | | | | +| D-08 | deviations.md location(s) | | | | | +| D-09 | Runtime skip idiom; RUN_DEVIATIONS command | | | | | +| D-10 | Fail-fast idiom | | | | | +| D-11 | Pending idiom for missing APIs | | | | | +| D-12 | Internal-access ladder; white-box spec list location | | | | | +| D-13 | Delegation policy | | | | | +| D-14 | Parallelism policy | | | | | +| D-15 | Teardown; suite-level fixtures | | | | | +| D-16 | Protocol-variant rendering | | | | | +| D-17 | Default timeouts and poll interval | | | | | +| D-18 | Auth through the proxy | | | | | +| D-19 | Re-sync flag; manifest; corpus scanner shipped? | | | | | +| D-20 | Commit policy; project/solution-file edits (approved / left to the user) | | | | | +| D-21 | Fix-attempt bound | | | | | +| D-22 | allowed-tools; remote access | | | | | +| D-23 | Notes files per module (full / placeholder) | | | | | +| D-24 | Working-records location | | | | | +| D-25 | Pilot specs and mode per tier | | | | | + +### D-03 mapping +| Module | unit | integration | proxy | notes | hand-maintained? | +|---|---|---|---|---|---| + +## Decision log +| # | Date | Phase | Question | Options | Answer | By | Supersedes | +|---|---|---|---|---|---|---|---| + +## Changelog (fixes made to the skill, notes or harness) +| # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Tests regenerated | +|---|---|---|---|---|---| +``` + +## Appendix D: Final report template + +```markdown +# Final report: uts-to- for + +- Spec clone (clean | dirty); repo ; date +- Scope: ; out of scope and why + +## Files created or changed +| Path | New / changed | Kind (hook, harness, smoke test, skill, script, notes, pilot test, CI suggestion) | Approved at | +|---|---|---|---| +Hand-maintained project files the user must update: … + +## Repo profile +Link to repo-profile.md; corrections made at STOP-2. + +## Harness +Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test results per tier; the plan for any remaining rows. Link to uts-infra-design.md and the harness README. + +## Design decisions +Link to design-record.md; decisions that depart from the guide's defaults, and why. + +## Validation results +- Resolver: rest / realtime / objects → ok, spec counts; negative cases → codes +- Audit self-test: each mutation → result; corpus sweep → N specs, 0 crashes, unverifiable specs listed +- Lint: …; CI-strict compile: …; examples verified: N of N (how) + +## Pilot results +(each pilot run's full section-11 report as the skill printed it: the header line, the table row, and every labelled line, including not-applicable omissions, missing APIs, mock-capability gaps, harness stand-ins and constructs it couldn't map; the table below collects the rows) +| Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | +|---|---|---|---|---|---|---|---| +Fixes made to the skill, notes or harness during the pilot (from the changelog): … +Reference tests named in SKILL.md: + +## Known limitations +Tiers not ready; `not available` catalogue rows; harness capabilities missing; audit SHOULD items not implemented; platforms the scripts weren't run on. + +## Deviations and spec issues found +- SDK deviations (recorded in deviations.md): … +- Suspected UTS spec errors or gaps: — ; draft fix: … +- Spec-repo doc issues (contradictions between guide and UTS docs): … + +## Suggested CI changes + (not applied unless approved at STOP-5) + +## Next steps +Which modules and tiers to translate next, in order, and why; harness work remaining. + +## Acceptance checklist +(Appendix E, filled) +``` + +## Appendix E: Acceptance checklist (guide section 13) + +Fill this in the Final report. Each row is one item of the guide's [checklist](writing-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill), with where this procedure produces or verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). + +| Guide §13 item | Produced in | Verified in | Status | +|---|---|---|---| +| **Prerequisites** | | | | +| SDK test hooks (5), clock covering blocking waits and async-runtime timers, injected at construction | 4.6; 5.2 rows 3–4 | 5.2 acceptance; G-01…G-05 | | +| Shared test library per helper spec (incl. legacy `queue_*` and `mock_http.captured_requests`; `mock_websocket.md` async close, CONNECTED ordering, raw frames, alternative API and undocumented members; vcdiff; test pool; `MockNetworkListener`) | 4.3; 5.2 rows 5–6, 8, 11 | smoke tests; G-06…G-10 | | +| Wait helpers (latched, fail-fast `AWAIT_STATE`; value-returning `poll_until`; `poll_until_success`; `process_pending_events`; wall-clock wrapper; fake-time-safe deadlines) | 4.4; 5.2 row 7 | helper tests; G-11 | | +| Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | +| Shared vs port-only placement; fixture-helper scope documented | 4.7; 5.3; notes item 9 | G-13 | | +| Harness smoke tests per tier (SHOULD) | 4.7; 5.2 rows 5–11 | STOP-7; G-14 | | +| `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | +| `ProxyManager` (pinned, every OS); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | +| One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | +| Concurrency, time, parallelism, visibility models documented | Phase 1 (P-09 to P-14) | STOP-2; G-18 | | +| **Skill files** | | | | +| `SKILL.md`: name, pushy description, `argument-hint`, usage guard, placeholder paths, 3.4 outline | 7.6 | 7.6 acceptance; 8.3 | | +| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked | 7.1 | 7.1 acceptance; 8.1 | | +| `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries | 7.2 | 8.1 | | +| `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | +| Per-module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | +| Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | +| **Workflow** | | | | +| Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | +| Steps 0 and A–E with the four questions | 7.6 | 8.3 pilot; 8.5 | | +| Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 8.3 pilot; 8.8 | | +| Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 8.3 pilot; 8.5 (re-sync) | | +| Re-sync mode; fix the cause, then regenerate | 7.6; D-19 | 8.5 (re-sync); 8.4 (fix and regenerate) | | +| **Translation rules in `SKILL.md`** | | | | +| One verbatim `UTS: ` per test; no invented tags; spec point in a collectable name | 7.6; D-04, D-06 | 8.2, 8.3 | | +| File header: source spec, tier, spec SHA, disclosures, corresponding unit spec | 7.6; D-07 | 8.3 | | +| Verbatim comments, spec variable names, section headings, spec order | 7.6 | 8.3 review | | +| Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome | 7.6; 7.4 | 8.3 audit | | +| Policies for sub-cases, groupers, test-case tables, delegating tests, ID-less tests | 7.6; D-13 | 8.3 (where the pilot has them) | | +| Protocol variants parameterised; tag singular; proxy JSON only | 7.6; D-16 | 8.3 integration pilot | | +| Clients closed in teardown; mocks restored; stop on a failed wait | 7.6; D-15 | 8.3 | | +| Unique channel names; own proxy session per test | 7.6 | 8.3 integration/proxy pilots | | +| Unit: fake time and flushes, no real timers. Integration: wall-clock | 7.6; 7.4; D-17 | 8.3 | | +| Record-and-verify for transient states; polls return the settled value | 7.6; 7.4 | 8.3 | | +| No accommodate-both assertions; `IS ` per typing discipline | 7.6 | 8.3 review | | +| Public-API specs stay public; internal-access ladder with escalation stop | 7.6; 7.5 item 8; D-12 | 8.3 | | +| Construct table filled for every row | 7.4 | 7.4 acceptance | | +| **Evaluation** | | | | +| Three end states; runtime skip and fail-fast idioms; `RUN_DEVIATIONS` command | 7.6; D-09, D-10 | 8.3 (evaluate) | | +| `deviations.md` location; format deferred; written at the right time | 7.6; D-08 | 8.3 (evaluate) | | +| Harness stand-ins in comments, not `deviations.md` | 7.4; 7.6 | 8.3 review | | +| Language-inapplicable inputs (cases a–c) noted in the test file | 7.6; D-11 | 8.3 (where they occur) | | +| Stopping rules and spec-error escalation | 7.6; D-21 | 8.4 | | +| **Verification** | | | | +| Compile (or collect), lint, CI-strict compile, filtered run | 7.6 | 8.3, 8.6 | | +| No commits by default; changed files listed | 7.6; D-20 | Phase 7 | | +| Final report format | 7.6 | 8.3 (the skill's report) | | +| The skill's own examples compile | 7.5 item 12; 7.6 | 8.7 | | + +## Appendix F: Glossary + +| Term | Meaning | +|---|---| +| **UTS** | Universal Test Suite: language-neutral pseudocode test specs in `uts/` of the spec repo | +| **Module** | A top-level corpus directory: `rest`, `realtime`, `objects` | +| **Tier** | `unit` (mocked transports), `integration` (direct sandbox), `proxy` (sandbox through `uts-proxy`), detected by path | +| **Spec** | One Markdown file of UTS tests; **features spec** means the Ably specification under `specifications/` | +| **Test ID / tag** | The `**Test ID**` of a spec test; the `UTS: ` comment above the derived test | +| **Helper spec** | A spec of harness behaviour (`mock_http.md`, `mock_websocket.md`, `mock_vcdiff.md`, `standard_test_pool.md`); implemented once, never translated | +| **Hook** | An injection point in SDK production code that the harness uses (transport, HTTP, clock, network monitor, randomness) | +| **Harness (UTS test infrastructure)** | The native test library implementing the helper specs, wait helpers, sandbox and proxy helpers and fixtures | +| **Reuse / wrap / extend / build** | The four ways to provide a harness capability from what the repo has ([4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)) | +| **Shared / port-only** | Harness code used by native tests and UTS ports alike / used only by UTS ports | +| **Smoke test** | A structural harness acceptance test; not spec-derived, no tag, never a reference test | +| **Reference test** | A reviewed, spec-derived test per tier that the skill reads before generating | +| **Notes** | `references/-mapping.md`: how ably-js-shaped pseudocode maps to this SDK for one module | +| **Resolver** | `resolve_uts.py`: validates a module and derives targets and names; its output is the single source of truth | +| **Audit** | `audit_translation.py`: compares a spec with its test file (IDs, ledger, counts) | +| **Ledger / shortfall** | The per-test list of spec lines by kind / spec count minus test count, for assertions and waits separately | +| **Translate-only / evaluate** | Generate and statically check / also run and diagnose | +| **Deviation** | An SDK behaviour that differs from a correct spec, recorded in `deviations.md` | +| **Stand-in** | A harness-level difference in how a test drives the SDK; disclosed in a comment, not a deviation | +| **White-box spec** | A unit spec that drives internal APIs; listed in the notes | +| **Delegating spec/test** | One that says to run another module's tests against this client | +| **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | +| **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | +| **Phase N / skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Skill Phase 1" (selection, steps 0 and A–E) and "skill Phase 2" (per spec, steps 1–7) are the generated skill's workflow (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | +| **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | + +## Appendix G: Common failure modes and remedies + +All from the guide and the UTS docs; follow the link for the detail. + +| Symptom | Likely cause | Remedy | Source | +|---|---|---|---| +| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the collected count with the tag count | guide [10](writing-translator-skills.md#10-verification-and-ci), [5.1](writing-translator-skills.md#51-traceability) | +| Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | +| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests" | +| Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) | +| A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | +| `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | +| A state wait runs out the timeout after the connection failed | No fail-fast on FAILED | Fail immediately on FAILED, except when the test just requested a transition out of it | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | +| Assertions after a poll see fewer items | The test re-reads after polling an eventually-consistent store | Use the value the poll returned | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | +| `poll_until_success` times out with no useful error | Emulated with a reader that swallows errors | A separate helper that rethrows the last error | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events); `uts/README.md` | +| Negative assertions pass or fail at random | One zero-delay yield doesn't drain chained callbacks | Drain until the internal queue is empty; or the quiescence control-listener | guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must); `standard_test_pool.md` "Negative-assertion quiescence" | +| Tests pass alone, fail together | Process-global hooks with a parallel runner; leaked clients or timers | Serialise or make hooks per-client (D-14); close clients and restore mocks in teardown | guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must), [5.4](writing-translator-skills.md#54-setup-and-teardown) | +| State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | +| An objects test's operation is silently ignored | Hand-written serial literal sorts before the pool serial | Use the serial helpers | `standard_test_pool.md`; guide 2.2 | +| Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | +| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions" | +| Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | +| Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | +| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning" | +| Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | +| Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must) | +| Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | +| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | +| Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](writing-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | +| A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](writing-translator-skills.md#5-translation-rules) | +| Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide [1](writing-translator-skills.md#1-purpose-and-scope), [12](writing-translator-skills.md#12-lessons-learned) | From 8a1bb95e5db6296a1768473e00126fd627ed48db Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 14:01:13 +0530 Subject: [PATCH 03/33] UTS: rename translator skill procedure to uts-to-lang-skill-creator Renames generating-a-translator-skill.md to uts-to-lang-skill-creator.md, matching the naming of skill-creator skills, and updates its title and the links to it from the guide and uts/README.md. --- uts/README.md | 4 ++-- ...ing-a-translator-skill.md => uts-to-lang-skill-creator.md} | 4 ++-- uts/docs/translator-skills/writing-translator-skills.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) rename uts/docs/translator-skills/{generating-a-translator-skill.md => uts-to-lang-skill-creator.md} (99%) diff --git a/uts/README.md b/uts/README.md index 6ded173b8..769f61af2 100644 --- a/uts/README.md +++ b/uts/README.md @@ -35,7 +35,7 @@ uts/ │ ├── writing-derived-tests.md # How to translate specs into SDK tests │ ├── translator-skills/ # Building uts-to- translator skills │ │ ├── writing-translator-skills.md # What a translator skill must contain -│ │ └── generating-a-translator-skill.md # Agent procedure to build one +│ │ └── uts-to-lang-skill-creator.md # Agent procedure to build one │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix @@ -142,7 +142,7 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations - **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` Claude Code skill: prerequisites, workflow, construct mapping, audit tooling, re-sync -- **[Generating a UTS Translator Skill](docs/translator-skills/generating-a-translator-skill.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design and build the UTS test infrastructure (harness), generate the skill, validate it with pilot translations +- **[UTS-to-Lang Skill Creator](docs/translator-skills/uts-to-lang-skill-creator.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design and build the UTS test infrastructure (harness), generate the skill, validate it with pilot translations - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs diff --git a/uts/docs/translator-skills/generating-a-translator-skill.md b/uts/docs/translator-skills/uts-to-lang-skill-creator.md similarity index 99% rename from uts/docs/translator-skills/generating-a-translator-skill.md rename to uts/docs/translator-skills/uts-to-lang-skill-creator.md index dbcfba260..69ddb944e 100644 --- a/uts/docs/translator-skills/generating-a-translator-skill.md +++ b/uts/docs/translator-skills/uts-to-lang-skill-creator.md @@ -1,4 +1,4 @@ -# Generating a UTS Translator Skill +# UTS-to-Lang Skill Creator This is a step-by-step procedure for an LLM agent (for example Claude Code) to create a `uts-to-` translator skill for one Ably SDK repository. [Writing UTS Translator Skills](writing-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why. This document says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this document links to it, and you must read the linked section. @@ -47,7 +47,7 @@ A skill written before its harness exists can't make translation mechanical (gui The user starts a run with one sentence, for example: ``` -Follow /uts/docs/translator-skills/generating-a-translator-skill.md to create a uts-to- skill for this repo. +Follow /uts/docs/translator-skills/uts-to-lang-skill-creator.md to create a uts-to- skill for this repo. ``` `` is the user's local clone of `ably/specification`. If the user also states goals ("objects unit tier first", "rest and realtime only"), record them as inputs. diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index 4237fcb66..93d8a6214 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -2,7 +2,7 @@ This guide explains how to build a per-language Claude Code skill — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You don't need to read these skills; everything required is in this guide and the UTS docs. -If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [Generating a UTS Translator Skill](generating-a-translator-skill.md); this guide remains the authority on what the skill must contain. +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [UTS-to-Lang Skill Creator](uts-to-lang-skill-creator.md); this guide remains the authority on what the skill must contain. **Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. From 5016c00d3d4adf2eb742196ff9dd21e051f0ed8a Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 14:01:13 +0530 Subject: [PATCH 04/33] UTS: make the harness part of translator skill setup Treats the harness (UTS test infrastructure) and the skill as one deliverable: the harness is designed from the repo's existing test support, and permanent, CI-run per-tier smoke tests and helper self-tests prove it before the skill is generated. The skill gets a Harness reference section, the mapping and resolver report the harness, and a preflight step runs the tier's harness tests before translating. Also: - allows reading the existing Swift and Kotlin skills and harnesses, pinned to reviewed commits, as a guarded last resort when the guide and UTS docs don't answer a question; never copied, always recorded - recommends Opus-class models: required for creating the skill and harness, recommended for translation runs, and recorded in reports - tightens procedure ordering, collected-test counting, known-gaps matching and other points found in review, and corrects claims about the existing skills --- uts/README.md | 6 +- .../uts-to-lang-skill-creator.md | 262 ++++++++++------ .../writing-translator-skills.md | 279 +++++++++++++++--- 3 files changed, 409 insertions(+), 138 deletions(-) diff --git a/uts/README.md b/uts/README.md index 769f61af2..a24ba5f6e 100644 --- a/uts/README.md +++ b/uts/README.md @@ -34,7 +34,7 @@ uts/ │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests │ ├── translator-skills/ # Building uts-to- translator skills -│ │ ├── writing-translator-skills.md # What a translator skill must contain +│ │ ├── writing-translator-skills.md # What a translator skill and its harness must contain │ │ └── uts-to-lang-skill-creator.md # Agent procedure to build one │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) @@ -141,8 +141,8 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations -- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` Claude Code skill: prerequisites, workflow, construct mapping, audit tooling, re-sync -- **[UTS-to-Lang Skill Creator](docs/translator-skills/uts-to-lang-skill-creator.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design and build the UTS test infrastructure (harness), generate the skill, validate it with pilot translations +- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` Claude Code skill and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync +- **[UTS-to-Lang Skill Creator](docs/translator-skills/uts-to-lang-skill-creator.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs diff --git a/uts/docs/translator-skills/uts-to-lang-skill-creator.md b/uts/docs/translator-skills/uts-to-lang-skill-creator.md index 69ddb944e..862e57eb7 100644 --- a/uts/docs/translator-skills/uts-to-lang-skill-creator.md +++ b/uts/docs/translator-skills/uts-to-lang-skill-creator.md @@ -10,7 +10,9 @@ The order is fixed, because each step stands on the one before it: 4. **Create the `uts-to-` skill on top of it** (Phases 4 and 5). 5. **Validate the skill with pilot translations** (Phase 6), then report (Phase 7). -A skill written before its harness exists can't make translation mechanical (guide [section 2](writing-translator-skills.md#2-prerequisites-in-the-sdk-repo)), and a harness designed without studying the repo's existing tests duplicates or fights them. A human may skim this document to see what the agent will do and where it will ask for decisions. +The harness and the skill are **one deliverable** (guide [section 2](writing-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). + +A harness designed without studying the repo's existing tests duplicates or fights them. A human may skim this document to see what the agent will do and where it will ask for decisions. ## Contents @@ -38,7 +40,7 @@ A skill written before its harness exists can't make translation mechanical (gui ### 1.1 Who reads this -- **The executor** is an LLM agent working inside the target SDK repository, with shell access, file read/write access, and the ability to ask the user questions. Every instruction below is addressed to it ("you"). +- **The executor** is an LLM agent working inside the target SDK repository, with shell access, file read/write access, and the ability to ask the user questions, running on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). Every instruction below is addressed to it ("you"). - **The user** is an SDK engineer who owns the target repository. They answer the questions at the stop points and approve every change outside the skill directory. - **The lead reviewer** may skim the working records (Repo profile, Harness design, Design record, Final report) instead of the whole conversation. @@ -60,11 +62,13 @@ Follow /uts/docs/translator-skills/uts-to-lang-skill-creator.md to c | Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | | Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | **Stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | | User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | +| Model tier | The session's model, Opus-class ([2.8](#28-model-tier-and-sub-agents)) | Take it from your environment if it states your model; otherwise ask. If the model isn't Opus-class: **STOP-1** | +| Local clones of ably-cocoa and ably-java (optional) | Clones that contain commits `b5074d9b` and `0ff24017` | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | ### 1.4 Outputs -1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, per-tier smoke tests, a harness README and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). -2. **The skill**, at `.claude/skills/uts-to-/` unless the Design record says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py` and `references/-mapping.md` (layout: guide [3.1](writing-translator-skills.md#31-layout)). +1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). +2. **The skill**, at `.claude/skills/uts-to-/` unless the Design record says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](writing-translator-skills.md#31-layout)). 3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. 4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): - `repo-profile.md` ([Appendix A](#appendix-a-repo-profile-template)); @@ -88,28 +92,39 @@ The guide's intent ([introduction](writing-translator-skills.md)) applies to thi | What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../writing-test-specs.md) | | Integration and proxy tiers | [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | | Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | -| What the harness and the skill must contain, and why | the guide | +| What the harness and the skill must contain, and why | the guide ([How it relates to the other UTS docs](writing-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | | How and in what order you build them | this document | | The SDK's actual API | the SDK source (guide [3.1](writing-translator-skills.md#31-layout), "The notes are a map, not an authority") | +| Questions none of the above answer, as a last resort only | The reference implementations, under [2.1](#21-your-inputs): never an authority | -If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. +If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. A disagreement between a reference implementation and the guide or the UTS docs is never a STOP-11: follow the guide, and record the disagreement with the read ([2.1](#21-your-inputs)). -**Procedure recommendations.** Where this document adds detail the guide doesn't have (for example: the working-records files and their location, smoke tests as gates, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this document's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). +**Procedure recommendations.** Where this document adds detail the guide doesn't have (for example: the working-records files and their location, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this document's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). --- ## 2. Ground rules -### 2.1 Your only inputs +### 2.1 Your inputs -You work from exactly four sources: +You work from four primary sources: 1. this document; 2. the guide; 3. the local spec clone: the UTS docs (`uts/README.md`, `uts/docs/`), the helper specs, the UTS corpus (`uts/rest`, `uts/realtime`, `uts/objects`) and, for API and spec-error questions, the features specs under `specifications/`; 4. the target SDK repo. -**Don't look for other SDKs' translator skills or UTS harnesses**, including the two the guide cites ([guide appendix](writing-translator-skills.md#appendix-existing-skills-background-and-patterns-to-avoid)). Don't clone, fetch, browse or search for them. Their lessons are already in the guide. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), treat it as background only: never port, copy, translate or fetch it. Write every hook, helper, script, rule and example yourself, from the guide, the UTS docs and the target repo. A harness and skill built from this procedure must stand on those alone. If you think you need something only another SDK's skill could tell you, that is a gap in the guide: **STOP-12**, and record it in the Final report. +and, only as a last resort, the reference implementations below. + +**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](writing-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: + +1. You have searched this document, the guide, the UTS docs and the helper specs, and they don't answer the question. +2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) with `git -C show :` and `git -C ls-tree -r --name-only -- `, never checking out, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. If a clone lacks the pinned commit, fetching it is network access too. +3. You take a **pattern**, not text: how something was structured or solved. Never copy, port or translate code, scripts, prose, tables or examples verbatim. Write every hook, helper, script, rule and example yourself, in this language's idioms, from the guide, the UTS docs and the target repo. +4. You check what you took against the guide's [Patterns to avoid](writing-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. +5. You record each consultation in the Design record ("Reference-implementation reads": repo, commit, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. + +A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](#83-pilot-translate-one-spec-per-available-tier)). ### 2.2 What you may change @@ -125,7 +140,7 @@ You work from exactly four sources: | Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | | Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | | Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | -| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies) | No | +| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies, reference-implementation reads under [2.1](#21-your-inputs)) | No | ### 2.3 Stop points @@ -133,18 +148,18 @@ At a stop point, present what you found and the options, then wait for the user' | ID | Phase | When | What you present | |---|---|---|---| -| **STOP-1** | Start | A required input is missing ([1.3](#13-required-inputs)) | The missing input and why it's needed | +| **STOP-1** | Start | A required input is missing, or the session's model isn't Opus-class ([1.3](#13-required-inputs), [2.8](#28-model-tier-and-sub-agents)) | The missing input (or the model found) and why it's needed | | **STOP-2** | 1 (end) | The Repo profile is written | The profile, for confirmation or correction | | **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | | **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](#46-step-2f-propose-the-sdk-test-hooks)) | | **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | -| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases), or running a full suite | The command, the expected duration, what it touches | -| **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests per tier, updated tier feasibility), before any skill file is written | +| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases, reading a reference implementation on GitHub or fetching its commit, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | +| **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | | **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | | **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | | **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](#52-step-3b-build-in-order-with-acceptance-checks), [8.4](#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | | **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | -| **STOP-12** | Any | You would need another SDK's skill or harness to proceed ([2.1](#21-your-only-inputs)) | The question you can't answer from the guide | +| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, commit, path) and what they showed; the reading you propose | ### 2.4 Never guess @@ -153,6 +168,7 @@ At a stop point, present what you found and the options, then wait for the user' - **Commands:** run every build, test, lint and script command you record, and record its exit status. A command you haven't run is marked "unverified" in the working records. - **Facts about the repo:** every Repo profile entry has evidence (a file path, a symbol or a command and its output). "Unknown" is allowed only with the reason and a question for the user. - **Undocumented corpus constructs:** the guide marks some constructs *(undocumented)*; their meaning is inferred from usage. Read each use in the corpus before choosing a rendering, and say in the catalogue that the meaning is inferred. +- **Reference-implementation reads:** what another SDK's skill or harness does is evidence of one SDK's choice, not of the spec or of this SDK. Confirm it against the guide or the UTS docs before relying on it. Where they are silent, record the choice as yours in the decision log, citing the read. ### 2.5 Record every decision @@ -179,6 +195,13 @@ Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) a - Keep the working records up to date as you go; they are also your memory if the session is interrupted. On resuming, read them first. - Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. +### 2.8 Model tier and sub-agents + +- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model): MUST.** The guide's [Model tier](writing-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. +- **Sub-agents** you spawn for review, validation or the pilot ([8.3](#83-pilot-translate-one-spec-per-available-tier)) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, running the [8.2](#82-test-the-audit-itself) corpus sweep and collecting its JSON), if at all, and record which steps used one. +- **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. +- **Pin** the generated skill's model where the tooling allows (D-26). + --- ## 3. Phase 1: understand the repo @@ -270,7 +293,7 @@ Write `repo-profile.md` from [Appendix A](#appendix-a-repo-profile-template). Re ## 4. Phase 2: design the harness -**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](writing-translator-skills.md#2-prerequisites-in-the-sdk-repo) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. +**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](writing-translator-skills.md#2-the-harness-uts-test-infrastructure) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. Design principles: @@ -280,7 +303,7 @@ Design principles: ### 4.1 Step 2a: list the required capabilities -Start the gap table from rows G-01 to G-18 of [Appendix B](#appendix-b-gap-table-and-harness-design-template), which cover every requirement of guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) to [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. +Start the gap table from rows G-01 to G-20 of [Appendix B](#appendix-b-gap-table-and-harness-design-template), which cover every requirement of guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) to [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. **Check:** every bullet and table row of guide section 2 maps to a gap-table row. @@ -316,14 +339,14 @@ Where a design question can only be answered by a production change, it becomes ### 4.4 Step 2d: design the wait and assertion helpers -Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the tests that prove them are in [5.2](#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. +Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the self-tests that prove them (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are in [5.2](#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. | Helper | Requirements | |---|---| | `AWAIT_STATE` (connection and channel) | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | | `poll_until`, `poll_until_success` | `writing-test-specs.md`; `uts/README.md`; guide 5.7; `writing-derived-tests.md` "No real timers in unit tests" (deadlines) | | `process_pending_events()` | `uts/README.md`; guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | -| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock" | +| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | | Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must), [5.6](writing-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | | Thread-safe capture, log sink, caller-attributed failures | guide 2.2 | | `assertContainsInOrder` | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | @@ -331,7 +354,7 @@ Design these against the async and concurrency model from P-09 to P-12 and the e ### 4.5 Step 2e: design the sandbox and proxy helpers - **Sandbox** (guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). -- **Proxy** (guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). +- **Proxy** (guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Include every operational requirement in guide 2.4. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). ### 4.6 Step 2f: propose the SDK test hooks @@ -346,31 +369,33 @@ For every hook row that isn't `present`, write a proposal in `uts-infra-design.m These proposals are presented at STOP-3 and need individual approval at **STOP-4** before you implement them. -### 4.7 Step 2g: plan placement, smoke tests and build order +### 4.7 Step 2g: plan placement, harness tests and build order -- **Placement** (guide 2.2, "Placement"): name the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). -- **Smoke tests**, one per tier you plan to support: structural acceptance tests only, with no `UTS:` tag, never copied by the skill (guide 2.2). The acceptance checks in [5.2](#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. +- **Placement** (guide 2.2, "Placement" and "Recommended layout"): name the harness library and its per-tier directories, the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). +- **Harness tests:** the per-tier smoke tests and helper self-tests of guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must), item by item, for every tier you plan to support. They are permanent and run in CI; they carry no `UTS:` tag and are never copied by the skill; they live outside every directory the mapping will give the resolver as a `targetDir`. Name the command that runs each tier's set plus the common self-tests (guide 2.7 "Selectable"); it goes into the mapping's `harness` entry (D-03). Name the CI job that runs it (5.2 row 13). The acceptance checks in [5.2](#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. - **Build order:** follow the order in 5.2, dropping rows that are `present` or out of scope. ### 4.8 Step 2h: derive tier feasibility -Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "objects not implemented in this SDK: translate-only at most"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). +Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "not possible until the SDK has an objects API (placeholder notes; the skill refuses the module)"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). A tier is **ready** only when its G-19 and G-20 rows (smoke tests and self-tests, guide 2.7) are present, green and wired into CI (or the CI change was declined at STOP-5 and is reported as unmet, 5.4), or will be built in Phase 3. ### 4.9 Step 2i: STOP-3, approve the design and choose the scope Present `uts-infra-design.md`: the gap table, the helper-spec mappings, the wait-helper design, the sandbox and proxy design, the hook proposals, placement, the build order and the tier matrix. Ask the user to choose: - **(a) Build the missing harness now**, all or a named subset. Recommend this option (the skill depends on the harness), but the user chooses. Continue with Phase 3. -- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) step C requires (see the readiness data in [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). +- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4, step C](writing-translator-skills.md#phase-1-selection-steps-0-and-af) requires (see the readiness data in [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). - **(c) Scope the skill to the tiers possible today.** As (b), without the plan. +Under (b) and (c), G-19 and G-20 for every tier the skill will offer can't be planned or scoped out; Phase 3 and STOP-7 still run for them. + Record the answer and any design changes the user asked for. Under (b) and (c), the skill's tier step (C) must offer only ready tiers, and the Final report lists the rest as known limitations. --- ## 5. Phase 3: build and verify the harness -**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has a passing smoke test. Skip the rows the user didn't choose to build at STOP-3. +**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has green smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)), wired into CI. This is part of the skill's deliverable, not a preliminary. Skip the rows the user didn't choose to build at STOP-3. ### 5.1 Step 3a: rules while building @@ -384,23 +409,25 @@ Record the answer and any design changes the user asked for. Under (b) and (c), Each step has an acceptance check; don't start the next step until it passes (or the user accepts a documented exception). **Bound:** if a row's check still fails after three fix attempts, stop (**STOP-10**) with a summary: the failure, what you tried, your diagnosis, and the options (change the design and log it, accept a documented exception, or descope the tiers the row blocks). -Treating the smoke tests as gates is a *procedure recommendation*; the guide makes smoke tests a SHOULD ([2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)). +The smoke tests and self-tests in the checks below are the permanent harness tests of guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) (MUST): build them as this phase's gates, and keep them in the repo and in CI afterwards, where the generated skill's preflight runs them. Each row's check is an explicit assertion in a harness test, not a manual observation. + +Rows 4–7 depend on each other: build them in order, but run an acceptance item that needs a component from a later row as soon as that component exists (the fake-clock check needs a transport mock; the REST row's realtime-driven request needs `MockWebSocket`; the realtime row's state waits need row 7), and don't leave row 7 until every row 4–7 item passes. Common items that need an integration component (the integration state-wait helper, integration teardown) run at row 9, once that component exists; they are part of row 9's acceptance. | Order | Build | Acceptance check | |---|---|---| -| 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal smoke test (one trivial passing test) compiles, is collected and passes under the single-class filter | -| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)); a failing helper reports the caller's line | +| 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal harness test (one trivial passing test; delete it once row 2's self-tests exist) compiles, is collected and passes under the single-class filter and under the tier's harness-test command | +| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken, and a repeated state passes (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)); capture lists are safe under concurrent appends; a failing helper reports the caller's line | | 3 | Approved SDK hooks | Each hook replaces the real implementation for one client without affecting another client created afterwards (or, if process-global, that fact is recorded for decision D-14); the existing tests still pass | -| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)) | -| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test:** a request is captured with method, path, headers and body; connection-level and request-level failures are distinguishable | -| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test:** connect to CONNECTED via `respond_with_success`; `close()` calls `onClose` asynchronously; CONNECTED is delivered after the connection completes; a raw untyped frame can be sent; a client frame is captured | -| 7 | Wait helpers per [4.4](#44-step-2d-design-the-wait-and-assertion-helpers) | Tests of the helpers: a state reached before the wait starts and one reached during it both complete; a FAILED transition fails fast; `poll_until` returns the value; `poll_until_success` on an always-failing condition raises the last error; a poll deadline still expires with the fake clock installed; `process_pending_events()` lets a queued callback run with no real delay | +| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it, and nothing waits on the real clock (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)); one advance runs cascaded work to quiescence (zero-delay reschedules and timers created mid-advance) | +| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, REST entry, each as an explicit assertion | +| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, realtime entry, each as an explicit assertion | +| 7 | Wait helpers per [4.4](#44-step-2d-design-the-wait-and-assertion-helpers) | **Helper self-tests:** every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Common and Unit, fake clock entries, each as an explicit assertion; in addition, `AWAIT_STATE` completes for a state reached before or during the wait (the teardown self-test follows your SDK's `close()` behaviour, guide 5.4) | | 8 | `MockNetworkListener` (if the SDK has a network monitor), `MockVCDiff*` (if the SDK supports deltas) | A smoke test drives each one | -| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test:** provision, connect, close, delete; `POST /apps` is never retried | -| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test:** create a session with one rule, connect through it, read the event log, close the session; a rule matching `ATTACHED` by string action fires | -| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test:** `setup_synced_channel` completes and one pool object can be read; the serial helpers sort as the helper spec requires | -| 12 | Harness README describing what exists | See [5.3](#53-step-3c-write-the-harness-readme) | -| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing (guide [2.5](writing-translator-skills.md#25-build-and-ci-wiring-must)) | +| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Direct sandbox entry, each as an explicit assertion | +| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Proxy entry, each as an explicit assertion; clients authenticate as decided in D-18 | +| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, objects entry, each as an explicit assertion. The smoke test lives beside the module helpers (outside every `targetDir`) and is selected by the unit tier's harness command | +| 12 | Harness README describing what exists, with its Known gaps | See [5.3](#53-step-3c-write-the-harness-readme) | +| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing; each tier's smoke tests and self-tests run, ungated, in that tier's job (guide [2.5](writing-translator-skills.md#25-build-and-ci-wiring-must), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)); the repo's existing test commands and jobs (e.g. bare `pytest`, `dotnet test` on the solution) don't also collect the UTS or harness tests (exclude them via `testpaths`/`--ignore`, a solution filter or a separate project), unless that job is deliberately their one CI home | Then compile all test targets as strictly as CI does (P-06) and run the repo's lint gates over the new code. @@ -409,15 +436,17 @@ Then compile all test targets as strictly as CI does (P-06) and run the repo's l Write a README next to the harness covering everything guide [3.1](writing-translator-skills.md#31-layout) and [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: - take the symbol map from the [4.3](#43-step-2c-map-each-helper-spec-to-native-code) tables as built, and the known gaps from the gap table's `partial` and `missing` rows; -- add the layout (shared, port-only, module helpers, test targets) and how to run each smoke test; +- add the layout (shared, port-only, module helpers, test targets) and the complete command that runs each tier's smoke tests and self-tests; +- write the **Known gaps** section (MUST): every helper-spec member, contract point or corpus construct the harness doesn't implement, by tier, as a table whose first column is the exact pseudocode token, matched by grepping the selected specs' `pseudo` fences (guide 3.1), then status and workaround, so the skill's preflight can match it against selected specs; +- index every module's fixture helpers in one place, even those in a separate test-support module; - link each fixture helper's scope to the module notes (item 9), where guide 2.2 puts it; - record the README's full repo path in the Design record. ### 5.4 Step 3d: STOP-7, confirm the harness -Re-run the assessment of [4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test results per tier, the files created or changed, and the updated tier matrix. Only then continue to the skill. +Re-run the assessment of [4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test and self-test results per tier (each run by its tier's filter), their CI wiring, the files created or changed, and the updated tier matrix. Only then continue to the skill. -**Done when:** every row the user chose to build is `present`; each supported tier has a passing smoke test; the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. +**Done when:** every row the user chose to build is `present`; each supported tier has green smoke tests and self-tests covering guide 2.7, wired into CI (or, if the user declined the CI change at STOP-5, listed in the Final report as an unmet requirement; the tier stays ready, because the skill's preflight still runs the harness tests locally on every run, and the harness README states that CI doesn't yet run them); the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. If the user declines network access at STOP-6, the integration and proxy tiers aren't supported: mark them unready (`blockedBy`: harness tests not run) and list them in the Final report. --- @@ -432,11 +461,11 @@ For each decision below, propose an answer derived from the Repo profile, the bu | ID | Decision | Guide reference | How to propose it | |---|---|---|---| | D-01 | Skill name and location | [3.1](writing-translator-skills.md#31-layout), [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/` | -| D-02 | Script language and minimum runtime version | [3.1](writing-translator-skills.md#31-layout), [8](writing-translator-skills.md#8-deterministic-tooling) | Python 3 unless P-07 shows it isn't available on every developer platform | -| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](writing-translator-skills.md#32-the-mapping-file), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](#47-step-2g-plan-placement-smoke-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that | +| D-02 | Script language and minimum runtime version | [3.1](writing-translator-skills.md#31-layout), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [10](writing-translator-skills.md#10-verification-and-ci) | Python 3 unless P-07 shows it isn't available on every developer platform | +| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](writing-translator-skills.md#32-the-mapping-file), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](#47-step-2g-plan-placement-harness-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that. Record the mapping's `harness` entry (root, README, per-tier sources and harness-test commands from 4.7) | | D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](writing-translator-skills.md#51-traceability), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | | D-05 | Name collisions with native tests | [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | -| D-06 | **Tag syntax** | [5.1](writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test function | +| D-06 | **Tag syntax** | [5.1](writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | | D-07 | File header format | [5.1](writing-translator-skills.md#51-traceability), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@`, disclosures; integration/proxy also the corresponding unit spec | | D-08 | **`deviations.md` location** per owning test module | [7.3](writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | | D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | @@ -452,10 +481,11 @@ For each decision below, propose an answer derived from the Repo profile, the bu | D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](#74-step-5d-fill-the-construct-catalogue)) | | D-20 | **Commit policy**; project/solution-file edits | [10](writing-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | | D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](writing-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | -| D-22 | `allowed-tools` and remote access | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash, Read, Edit, Write`; no `WebFetch`, because everything is read from the local clone | -| D-23 | Which modules get notes files, and which are placeholders | [3.1](writing-translator-skills.md#31-layout), [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | +| D-22 | `allowed-tools` and remote access | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash, Read, Edit, Write`; no `WebFetch`, because everything is read from the local clone (the skill never consults the reference implementations, [2.1](#21-your-inputs)) | +| D-23 | Which modules get notes files, and which are placeholders | [3.1](writing-translator-skills.md#31-layout), [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | | D-24 | Location of the working records, and whether they are kept in the repo | [1.4](#14-outputs) of this document | Default `.claude/skills/uts-to-/generation/`, kept for maintenance | | D-25 | Pilot specs and mode per tier | [8.3](#83-pilot-translate-one-spec-per-available-tier) of this document | One per ready module × tier in scope, and at least one per module with a full notes file, chosen by the criteria there | +| D-26 | Model tier for skill runs: pin the model, or state it | [Model tier](writing-translator-skills.md#model-tier), [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | Pin to an Opus-class model if the tooling supports it (verify the mechanism in the current Claude Code documentation; don't assume a frontmatter field); otherwise state it in the opening lines of `SKILL.md`. Either way, the skill's final report records the model | ### 6.2 Step 4b: STOP-8, confirm @@ -471,7 +501,7 @@ Generate the files in this order: each one depends on the previous ones. `SKILL. 1. Create the directory layout from guide [3.1](writing-translator-skills.md#31-layout) (plus the working-records directory, D-24). 2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](writing-translator-skills.md#32-the-mapping-file). -3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. +3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data, and guide [4, step C](writing-translator-skills.md#phase-1-selection-steps-0-and-af) refuses unready tiers), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. **Acceptance:** @@ -480,13 +510,14 @@ Generate the files in this order: each one depends on the previous ones. `SKILL. - [ ] `notes` paths are relative to the skill directory and the files exist (placeholders at this point are fine). - [ ] Hand-maintained entries are marked (e.g. in `_comment`) and listed in the Design record. - [ ] Every tier the harness design marks unready has an `unready` entry with the blocking gap-table row. +- [ ] The mapping has a `harness` entry (root, README, per-tier sources and the complete smoke-test and self-test command), every path in it exists, no harness test lies inside any tier `targetDir`, and no `targetDir` lies inside the harness root (guide [3.2](writing-translator-skills.md#32-the-mapping-file), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)). - [ ] The file is valid JSON, UTF-8, LF line endings. ### 7.2 Step 5b: write the resolver Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. -Output contract: the fields of guide 8.1, including `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: +Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: ```json { @@ -495,6 +526,11 @@ Output contract: the fields of guide 8.1, including `testRoot` when the mapping "specRepo": { "sha": "", "dirty": false }, "mapped": true, "translationNotes": "", + "harness": { + "root": "", + "readme": "", + "tiers": { "unit": { "sources": ["..."], "tests": "" }, "integration": { "...": "..." }, "proxy": { "...": "..." } } + }, "tiers": { "unit": { "present": true, @@ -516,12 +552,12 @@ Errors are `{"ok": false, "code": "", "message": ""}`, with a distin **Acceptance** (each verified by running the script in [8.1](#81-run-the-resolver-on-every-module)): - [ ] Every row of guide 8.1 is implemented: validation, error objects, output contract, path-only tier detection (including specs in tier sub-directories, e.g. `realtime/integration/channels/`), exclusions relative to the tier base, deterministic runner-collectable naming (D-04), collision detection (D-05, including file-name uniqueness where the runner needs it), validate-then-write `--create` writing the D-03 default layout, an error for a missing notes file. -- [ ] `ready`/`blockedBy` match the mapping's `unready` data; `testFile` is present wherever the file name isn't derived from `className`. +- [ ] `harness` matches the mapping, with every path validated (repo-relative) and existing; `ready`/`blockedBy` match the mapping's `unready` data; `testFile` is present wherever the file name isn't derived from `className`. - [ ] Never crashes: any exception becomes `ok: false` with a code. ### 7.3 Step 5c: write the audit -Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](#21-your-only-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: +Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](#21-your-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: | Test-side pattern | Source | |---|---| @@ -536,7 +572,7 @@ Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifi **Acceptance** (verified in [8.2](#82-test-the-audit-itself)): -- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including its SHOULD end-at-heading rule (requiring that SHOULD is a *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. +- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including the end-at-heading rule (a guide SHOULD that this procedure requires: *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. - [ ] Where your parsing differs from the contract, `SKILL.md` says so (guide 8.2 "Implement exactly this, or say where yours differs"). - [ ] SHOULD items you implemented are listed in `SKILL.md`; the ones you didn't are listed in the Final report. @@ -545,13 +581,13 @@ Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifi This step produces the construct table that goes into `SKILL.md` (and, for module-specific rows, into the notes). 1. **Start from every row of guide [section 6](writing-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](writing-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. -2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells; they illustrate other SDKs. +2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells, or anything seen in a reference-implementation read ([2.1](#21-your-inputs)); they illustrate other SDKs. 3. **Scan the corpus for constructs the guide doesn't list** (guide section 6 says the catalogue isn't exhaustive). Run the scanner below over each module the skill covers, and add a row for every uppercase keyword or snake_case call you can't account for. 4. **Give every row a status:** - `mapped`: a direct rendering; - `mapped (inferred)`: an *(undocumented)* construct, meaning inferred from its uses (list one use); - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](writing-translator-skills.md#72-harness-stand-ins-are-not-deviations)); - - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation), and the Final report has a plan; + - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation; never substitute a similar mock outcome, guide [5.2](writing-translator-skills.md#52-fidelity)), and the Final report has a plan; - `n/a`: no spec in the covered modules uses it (say so after scanning). 5. **No blank cells.** Collect every row the skill can't render and has no plan for, from one scan, into a single **STOP-9**. @@ -588,7 +624,7 @@ The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_ - [ ] Every guide section 6 row is present with a rendering and a status; none blank. - [ ] Every scanner hit in the covered modules is either a catalogue row, a wire constant or enum value, a spec-local name, or internal pseudocode of a helper spec or `objects/PLAN.md` (implemented as harness or ignored, never translated). - [ ] Every rendering names a symbol that exists in the harness or the SDK (you read it), or is marked `not available` with a plan. -- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide 5.7, 6.7). +- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide [5.7](writing-translator-skills.md#57-waits-must-catch-events), [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). ### 7.5 Step 5e: write the module notes @@ -600,26 +636,28 @@ Write `references/-mapping.md` for every module chosen in D-23. Cover th - **Item 9 (helper-spec coverage):** the module's rows of the symbol tables from [4.3](#43-step-2c-map-each-helper-spec-to-native-code), as built, with each fixture helper's scope and every sanctioned stand-in. - **Item 12 (worked example):** translate one short real test from the module by hand, name each mechanical rewrite, and make sure it compiles (it becomes one of the skill's examples; [8.7](#87-verify-the-skills-own-examples)). -A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), "If the module's notes file exists but is only a placeholder"). +A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), "If the module's notes file exists but is only a placeholder"). **Acceptance:** - [ ] All thirteen items present or marked not applicable with a reason. - [ ] Every SDK and harness symbol named in the notes exists at the recorded repo SHA. -- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide 4, step 1). +- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide [4, step 1](writing-translator-skills.md#phase-2-per-spec-steps-17)). - [ ] No line numbers; cite symbols and headings (guide [12](writing-translator-skills.md#12-lessons-learned)). ### 7.6 Step 5f: write `SKILL.md` -Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide [1](writing-translator-skills.md#1-purpose-and-scope), "What the skill owns, and what it defers"). +Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide ["What the skill owns, and what it defers"](writing-translator-skills.md#what-the-skill-owns-and-what-it-defers)). | Part of `SKILL.md` | Content | Guide | |---|---|---| -| Frontmatter | `name: uts-to-`; a pushy `description` with trigger phrases; `argument-hint`; `allowed-tools` from D-22 | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | +| Frontmatter | `name: uts-to-`; a pushy `description` with trigger phrases; `argument-hint`; `allowed-tools` from D-22; the model pin from D-26, where the tooling supports one (otherwise the model statement in the opening lines) | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | | Usage guard | Empty argument → print the usage line and stop; placeholder paths only | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | -| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | -| Skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | -| Skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README by its full repo path; for the reference test per tier, write `none yet` until [8.8](#88-name-the-reference-tests) names one | [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit) | +| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | +| Skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | +| Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](writing-translator-skills.md#34-recommended-outlines-should), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README and helper sources from the resolver's `harness` output; for the reference test per tier, write `none yet` until [8.8](#88-name-the-reference-tests) names one | [4](writing-translator-skills.md#phase-2-per-spec-steps-17), [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit) | | Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](writing-translator-skills.md#5-translation-rules) | | Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) | | File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](writing-translator-skills.md#34-recommended-outlines-should) | @@ -627,7 +665,7 @@ Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-ou | Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](writing-translator-skills.md#55-integration-tier-hygiene) | | Verification | Guide 10 with the exact commands from the Repo profile | [10](writing-translator-skills.md#10-verification-and-ci) | | Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should), [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Final report | The format | [11](writing-translator-skills.md#11-final-report-format) | +| Final report | The format, including the model line | [11](writing-translator-skills.md#11-final-report-format) | | Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](writing-translator-skills.md#10-verification-and-ci) | **Acceptance:** @@ -635,7 +673,9 @@ Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-ou - [ ] Every part in the table is present. - [ ] The skill is module-generic: module-specific rules live in the notes. - [ ] No inline copy of a semantic rule where a link to its source would do; in particular the `deviations.md` format is referenced, not redefined. -- [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills. +- [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills (reference-implementation reads during creation are recorded in the working records, never cited in the skill). +- [ ] None of the guide's [Patterns to avoid](writing-translator-skills.md#patterns-to-avoid) appears in the skill, its notes, its scripts or the harness. +- [ ] The Harness reference names only files, filters and commands that exist, and doesn't inline helper signatures. - [ ] Every command in it is one the Repo profile records as run. (Running each through the skill, and compiling every example, are Phase 6 checks: [8.5](#85-exercise-the-skills-other-paths), [8.7](#87-verify-the-skills-own-examples).) - [ ] Every helper or SDK symbol it names exists. @@ -654,7 +694,7 @@ Run the resolver on `/uts/rest`, `/uts/realtime` and `` will do); a mapping entry whose notes file is missing (use a scratch copy of the mapping); `--create` with an invalid name; `--create` on a scratch copy of a deliberately corrupt mapping (the file must be left unchanged). @@ -690,13 +730,15 @@ The candidates are examples only. Before choosing one, confirm it exists in your **Run the pilot the way a user would.** -- **Loading the skill.** A skill created during a session may not be listed until a new session starts. Ask the user to start a new Claude Code session in the repo (or to reload skills, if their client supports it) and confirm `/uts-to-` is listed. A fresh session also tests that the skill is context-complete on its own (guide intent). -- **Answering its questions.** The answers are the D-25 choices, confirmed at STOP-8 (mapping confirmation, tier, specs, translate-only or evaluate). Either the user runs `/uts-to- /uts/` in the new session and answers with them, or, if the skill is listed in a session you control, you invoke it and answer with exactly those choices. If you use a subagent, give it only the invocation and the D-25 answers, none of this run's context. Any answer not covered by D-25 goes to the user. -- **Record each question the skill asked.** More than the four questions, or a question it should have answered from its own files, is a skill defect ([8.4](#84-fix-the-cause-then-regenerate)). +- **Loading the skill.** A skill created during a session may not be listed until a new session starts. Ask the user to start a new Claude Code session in the repo, on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)) (or to reload skills, if their client supports it), and confirm `/uts-to-` is listed. A fresh session also tests that the skill is context-complete on its own (guide intent), so the pilot never consults the reference implementations. +- **Answering its questions.** The answers are D-03 (mapping confirmation) and D-25 (tier, specs, mode), confirmed at STOP-8. Either the user runs `/uts-to- /uts/` in the new session and answers with them, or, if the skill is listed in a session you control, you invoke it and answer with exactly those choices. If you use a subagent, run it on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)), and give it only the invocation and the D-25 answers, none of this run's context. It must not consult the reference implementations or any other SDK's skill, because the pilot tests the skill alone. Record the pilot session's or subagent's model with the pilot results. Any answer not covered by D-25 goes to the user. +- **Record each question the skill asked.** More than the four questions, other than a stop the guide requires (an unmapped module's name at step B, a known gap or red harness at step F, an unmapped construct, the fix-attempt bound), or a question it should have answered from its own files, is a skill defect ([8.4](#84-fix-the-cause-then-regenerate)). + +**A green preflight comes first.** The skill's step F (after **STOP-6** for network tiers) must pass for the pilot's tier before the pilot translates anything; a pilot run on a red harness doesn't count. Record the preflight result with the pilot. If network access is declined, that tier's pilot isn't run and is reported as unvalidated. Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: -1. Compile (and collect: the collected count equals the number of `UTS:` tags). +1. Compile (and collect: every tagged test function is collected; count a parameterised or protocol-variant test once). 2. Audit; walk the review checklist (guide [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit)). 3. In evaluate mode, and after **STOP-6** for network tiers, run the class by filter and diagnose per `writing-derived-tests.md` Phase 2. 4. Review the generated file line by line against the spec yourself, and show it to the user. @@ -706,7 +748,7 @@ Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: For every defect in a pilot test: 1. Classify it: a rule in `SKILL.md`, a mapping in the notes, a catalogue row, a resolver or audit bug, or a harness gap. (A UTS spec error is not a skill defect: record it, emit the fail-fast test per D-10, and **STOP-10**.) -2. Fix that cause. **Never edit only the generated test.** A harness fix follows the rules of [5.1](#51-step-3a-rules-while-building), re-runs the affected smoke tests and updates the harness README. +2. Fix that cause. **Never edit only the generated test.** A harness fix follows the rules of [5.1](#51-step-3a-rules-while-building), adds or extends a self-test that would have caught the defect, re-runs the affected tiers' smoke tests and self-tests, and updates the harness README (and its Known gaps). 3. Delete the generated test and regenerate it with the skill. 4. Re-audit, recompile and, in evaluate mode, re-run. 5. Log the defect and the fix in the Design record's changelog. @@ -724,6 +766,8 @@ Run each path through the skill (in the session from [8.3](#83-pilot-translate-o 5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. 6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification, that the header SHA is updated, and that only the survivors in guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. 7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. +8. **Red harness:** on a scratch copy (or a temporary working-tree edit you revert afterwards), break one harness test (for example, make the WebSocket mock refuse every connection), or run an integration tier with the network unavailable → the skill stops at step F before generating anything, reports a harness or environment problem (not an SDK deviation), and names the failing test. Restore the change and confirm the preflight is green again. +9. **Known gap:** select a spec that uses a construct the harness README lists under Known gaps → the preflight reports the conflict and stops to ask (guide 4 step F), rather than generating a substituted rendering. If Known gaps is empty, use a scratch copy of the README with an entry for a construct the selected spec uses. ### 8.6 Lint and CI-strict compile @@ -740,9 +784,9 @@ Record which examples were verified and how. ### 8.8 Name the reference tests -After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement), step 3). Never name a smoke test. +After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4, step 3](writing-translator-skills.md#phase-2-per-spec-steps-17)). Never name a smoke test. -**Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected and every command in `SKILL.md` has been run; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. +**Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected (including the red-harness stop) and every command in `SKILL.md` has been run; every pilot ran after a green preflight; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. --- @@ -750,28 +794,29 @@ After review, name one spec-derived test file per tier in `SKILL.md` as the file 1. Write `final-report.md` from [Appendix D](#appendix-d-final-report-template). 2. Fill the acceptance checklist in [Appendix E](#appendix-e-acceptance-checklist-guide-section-13): tick each item, or explain why it isn't met (scoped out, planned, not applicable). -3. Show the user the report, the list of changed and new files (harness, hooks, skill, pilot tests), and the suggested CI changes. **Don't commit** unless asked. +3. Show the user the report, the list of changed and new files (harness, harness tests, hooks, skill, pilot tests), and the suggested CI changes. The run is complete only if the harness tests are green, and in CI, at every tier the skill supports; otherwise the report says which tiers aren't and why. **Don't commit** unless asked. 4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. --- ## 10. Maintenance -The harness and the skill must stay in step with four things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. +The harness and the skill must stay in step with five things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. | What changed | How to detect it | What to update | |---|---|---| | **The guide** | `git -C diff ..HEAD -- uts/docs/translator-skills/` | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](#61-step-4a-draft-the-design-record) and [7.6](#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | -| **The UTS docs or helper specs** | `git -C diff ..HEAD -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and README, the catalogue rows and notes item 9; then re-sync the affected tests | +| **The UTS docs or helper specs** | `git -C diff ..HEAD -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | | **The UTS corpus** | The skill's own re-sync mode (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | | **The SDK** (API, hooks, test layout, CI) | `git -C diff ..HEAD` over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | Rules for every maintenance run: -- Follow the same ground rules ([section 2](#2-ground-rules)) and stop points. +- Follow the same ground rules ([section 2](#2-ground-rules)) and stop points, including the model tier ([2.8](#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](#21-your-inputs)). If the guide's "as of" commits have moved since a recorded read, re-check that read's conclusion against the updated Patterns to avoid before relying on it again. - Start from the working records; update the Repo profile and the harness design if anything they describe has changed. - Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)). -- After any harness change, re-run the smoke tests; after any script change, repeat [8.1](#81-run-the-resolver-on-every-module) and [8.2](#82-test-the-audit-itself). +- After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](#81-run-the-resolver-on-every-module) and [8.2](#82-test-the-audit-itself). - Record the new spec SHA and repo SHA in the Design record and append a changelog entry. --- @@ -864,12 +909,14 @@ Rules for every maintenance run: | G-10 | standard_test_pool.md implementation; plugin options builder | 2.2, standard_test_pool.md | | | | | | | | | | G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | | G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | -| G-13 | Placement (shared / port-only / module helpers); fixture-helper scope documented in the module notes; harness README | 2.2, 3.1 | | | | | | | | | -| G-14 | Harness smoke tests per tier | 2.2 | | | | | | | | | +| G-13 | Placement and recommended harness layout (shared / port-only / module helpers); fixture-helper scope documented in the module notes | 2.2 | | | | | | | | | +| G-14 | Harness README with Known gaps; mapping `harness` entry (root, README, per-tier sources and harness-test commands) | 3.1, 3.2 | | | | | | | | | | G-15 | SandboxApp; ably-common | 2.3 | | | | | */integration | | | | | G-16 | ProxyManager (pinned, every OS); ProxySession; rule builders; proxy auth | 2.4 | | | | | */proxy | | | | | G-17 | UTS test target; per-tier selection and filters; one CI home per suite | 2.5 | | | | | | | | | | G-18 | Concurrency, time, parallelism and visibility models known | 2.6 | | | | | | | | | +| G-19 | Per-tier smoke tests (permanent, in CI, outside generated dirs) | 2.7 | | | | | | | | | +| G-20 | Helper self-tests (common, unit, fake clock, REST, objects, sandbox, proxy) | 2.7 | | | | | | | | | ## Helper-spec symbol maps (one table per helper spec) ### mock_http.md @@ -910,7 +957,7 @@ Rules for every maintenance run: |---|---|---|---|---|---| ## Confirmation at STOP-7 -Smoke tests per tier: …; updated tier matrix: …; confirmed on +Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier matrix: …; confirmed on ``` ## Appendix C: Design record template @@ -923,6 +970,7 @@ Smoke tests per tier: …; updated tier matrix: …; confirmed on - Repo: - Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: - Harness README: +- Model: (Opus-class: yes | no, departure in decision log #); sub-agents: , or none ## Decisions | ID | Decision | Choice | Guide-default? | Decided by | Rationale | @@ -952,6 +1000,7 @@ Smoke tests per tier: …; updated tier matrix: …; confirmed on | D-23 | Notes files per module (full / placeholder) | | | | | | D-24 | Working-records location | | | | | | D-25 | Pilot specs and mode per tier | | | | | +| D-26 | Model tier and pinning for skill runs | | | | | ### D-03 mapping | Module | unit | integration | proxy | notes | hand-maintained? | @@ -961,6 +1010,10 @@ Smoke tests per tier: …; updated tier matrix: …; confirmed on | # | Date | Phase | Question | Options | Answer | By | Supersedes | |---|---|---|---|---|---|---|---| +## Reference-implementation reads (last resort, 2.1) +| # | Date | Question | Guide/UTS sections searched | Repo @ commit : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | +|---|---|---|---|---|---|---|---| + ## Changelog (fixes made to the skill, notes or harness) | # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Tests regenerated | |---|---|---|---|---|---| @@ -972,6 +1025,7 @@ Smoke tests per tier: …; updated tier matrix: …; confirmed on # Final report: uts-to- for - Spec clone (clean | dirty); repo ; date +- Model: (Opus-class: yes | no, and why); sub-agents: , or none - Scope: ; out of scope and why ## Files created or changed @@ -983,7 +1037,7 @@ Hand-maintained project files the user must update: … Link to repo-profile.md; corrections made at STOP-2. ## Harness -Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test results per tier; the plan for any remaining rows. Link to uts-infra-design.md and the harness README. +Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test and self-test results per tier, and their CI jobs; preflight results from the pilot runs; harness changes made during Phase 6 (with the self-tests added); the README's Known gaps; the plan for any remaining rows. Link to uts-infra-design.md and the harness README. ## Design decisions Link to design-record.md; decisions that depart from the guide's defaults, and why. @@ -1006,7 +1060,10 @@ Tiers not ready; `not available` catalogue rows; harness capabilities missing; a ## Deviations and spec issues found - SDK deviations (recorded in deviations.md): … - Suspected UTS spec errors or gaps: — ; draft fix: … -- Spec-repo doc issues (contradictions between guide and UTS docs): … +- Spec-repo doc issues (contradictions between guide and UTS docs; guide gaps revealed by reference-implementation reads): … + +## Reference-implementation reads +Each read from the Design record (question, repo @ commit : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. ## Suggested CI changes (not applied unless approved at STOP-5) @@ -1024,30 +1081,32 @@ Fill this in the Final report. Each row is one item of the guide's [checklist](w | Guide §13 item | Produced in | Verified in | Status | |---|---|---|---| -| **Prerequisites** | | | | +| **Harness** | | | | | SDK test hooks (5), clock covering blocking waits and async-runtime timers, injected at construction | 4.6; 5.2 rows 3–4 | 5.2 acceptance; G-01…G-05 | | | Shared test library per helper spec (incl. legacy `queue_*` and `mock_http.captured_requests`; `mock_websocket.md` async close, CONNECTED ordering, raw frames, alternative API and undocumented members; vcdiff; test pool; `MockNetworkListener`) | 4.3; 5.2 rows 5–6, 8, 11 | smoke tests; G-06…G-10 | | | Wait helpers (latched, fail-fast `AWAIT_STATE`; value-returning `poll_until`; `poll_until_success`; `process_pending_events`; wall-clock wrapper; fake-time-safe deadlines) | 4.4; 5.2 row 7 | helper tests; G-11 | | | Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | -| Shared vs port-only placement; fixture-helper scope documented | 4.7; 5.3; notes item 9 | G-13 | | -| Harness smoke tests per tier (SHOULD) | 4.7; 5.2 rows 5–11 | STOP-7; G-14 | | +| Harness designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; recommended harness layout; fixture-helper scope documented | 4.2; 4.7; 5.3; notes item 9 | G-13 | | +| Harness README with a Known gaps section (MUST) | 5.3; 5.2 row 12 | G-14; 8.5 path 9 | | +| Harness smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.5 path 8 | | | `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | -| `ProxyManager` (pinned, every OS); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | +| `ProxyManager` (pinned, every OS, verified download, health check, port handling); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | | One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | | Concurrency, time, parallelism, visibility models documented | Phase 1 (P-09 to P-14) | STOP-2; G-18 | | | **Skill files** | | | | -| `SKILL.md`: name, pushy description, `argument-hint`, usage guard, placeholder paths, 3.4 outline | 7.6 | 7.6 acceptance; 8.3 | | -| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked | 7.1 | 7.1 acceptance; 8.1 | | -| `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries | 7.2 | 8.1 | | +| `SKILL.md`: name, pushy description, `argument-hint`, usage guard, placeholder paths, 3.4 outline incl. the Harness reference | 7.6 | 7.6 acceptance; 8.3 | | +| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked, `harness` entry | 7.1 | 7.1 acceptance; 8.1 | | +| `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries, `harness` output | 7.2 | 8.1 | | | `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | -| Per-module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | +| Module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | | Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | | **Workflow** | | | | | Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | -| Steps 0 and A–E with the four questions | 7.6 | 8.3 pilot; 8.5 | | +| Steps 0 and A–F: the four questions and the harness preflight (stop on red) | 7.6 | 8.3 pilot (green preflight first); 8.5 paths 8–9 | | | Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 8.3 pilot; 8.8 | | | Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 8.3 pilot; 8.5 (re-sync) | | | Re-sync mode; fix the cause, then regenerate | 7.6; D-19 | 8.5 (re-sync); 8.4 (fix and regenerate) | | +| Mid-run harness changes re-run the smoke tests and self-tests and are reported separately | 7.6 (Harness reference) | 8.4 step 2; final report | | | **Translation rules in `SKILL.md`** | | | | | One verbatim `UTS: ` per test; no invented tags; spec point in a collectable name | 7.6; D-04, D-06 | 8.2, 8.3 | | | File header: source spec, tier, spec SHA, disclosures, corresponding unit spec | 7.6; D-07 | 8.3 | | @@ -1073,6 +1132,9 @@ Fill this in the Final report. Each row is one item of the guide's [checklist](w | No commits by default; changed files listed | 7.6; D-20 | Phase 7 | | | Final report format | 7.6 | 8.3 (the skill's report) | | | The skill's own examples compile | 7.5 item 12; 7.6 | 8.7 | | +| **Creation process** | | | | +| Skill and harness created on an Opus-class model (MUST); model pinned or stated for runs (SHOULD) and recorded in the skill's final report | 2.8; D-26; 7.6 (frontmatter, final report) | Design record "Inputs at generation time" (Model line); 8.3 (pilot model recorded); Final report header | | +| Reference-implementation reads, if any, last resort only, recorded and reported as guide gaps | 2.1; Appendix C | Final report "Reference-implementation reads" | | ## Appendix F: Glossary @@ -1088,8 +1150,12 @@ Fill this in the Final report. Each row is one item of the guide's [checklist](w | **Harness (UTS test infrastructure)** | The native test library implementing the helper specs, wait helpers, sandbox and proxy helpers and fixtures | | **Reuse / wrap / extend / build** | The four ways to provide a harness capability from what the repo has ([4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)) | | **Shared / port-only** | Harness code used by native tests and UTS ports alike / used only by UTS ports | -| **Smoke test** | A structural harness acceptance test; not spec-derived, no tag, never a reference test | +| **Smoke test** | A per-tier harness test that drives one tier end to end through the real hooks (guide 2.7); permanent, in CI, not spec-derived, no tag, never a reference test | +| **Self-test** | A harness test that checks one helper obeys its helper-spec contract (guide 2.7); permanent, in CI, no tag | +| **Preflight** | The generated skill's step F: compile, run the tier's smoke tests and self-tests, check Known gaps; stop on red | | **Reference test** | A reviewed, spec-derived test per tier that the skill reads before generating | +| **Reference implementations** | The existing `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java) skills and their harnesses, at the commits the guide pins: a last-resort, read-only reference for patterns, never copied and never an authority ([2.1](#21-your-inputs)). Not a reference test | +| **Opus-class model** | The most capable model tier available (Claude Opus or an equivalent); required for this procedure and its sub-agents ([2.8](#28-model-tier-and-sub-agents)) | | **Notes** | `references/-mapping.md`: how ably-js-shaped pseudocode maps to this SDK for one module | | **Resolver** | `resolve_uts.py`: validates a module and derives targets and names; its output is the single source of truth | | **Audit** | `audit_translation.py`: compares a spec with its test file (IDs, ledger, counts) | @@ -1101,7 +1167,7 @@ Fill this in the Final report. Each row is one item of the guide's [checklist](w | **Delegating spec/test** | One that says to run another module's tests against this client | | **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | | **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | -| **Phase N / skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Skill Phase 1" (selection, steps 0 and A–E) and "skill Phase 2" (per spec, steps 1–7) are the generated skill's workflow (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | +| **Phase N / skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Skill Phase 1" (selection, steps 0 and A–F) and "skill Phase 2" (per spec, steps 1–7) are the generated skill's workflow (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | | **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | ## Appendix G: Common failure modes and remedies @@ -1110,9 +1176,9 @@ All from the guide and the UTS docs; follow the link for the detail. | Symptom | Likely cause | Remedy | Source | |---|---|---|---| -| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the collected count with the tag count | guide [10](writing-translator-skills.md#10-verification-and-ci), [5.1](writing-translator-skills.md#51-traceability) | +| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the distinct collected test functions (a parameterised test counted once) with the tag count | guide [10](writing-translator-skills.md#10-verification-and-ci), [5.1](writing-translator-skills.md#51-traceability) | | Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | -| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests" | +| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | | Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) | | A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | | `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | @@ -1124,15 +1190,15 @@ All from the guide and the UTS docs; follow the link for the detail. | State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | | An objects test's operation is silently ignored | Hand-written serial literal sorts before the pool serial | Use the serial helpers | `standard_test_pool.md`; guide 2.2 | | Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | -| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions" | +| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions"; guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | | Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | | Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | -| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning" | +| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning"; guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | | Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | | Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must) | | Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | -| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement) | +| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | | Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](writing-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | | A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](writing-translator-skills.md#5-translation-rules) | | Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide [1](writing-translator-skills.md#1-purpose-and-scope), [12](writing-translator-skills.md#12-lessons-learned) | +| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](writing-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](writing-translator-skills.md#12-lessons-learned) | diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index 93d8a6214..4a5152521 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -1,11 +1,12 @@ # Writing UTS Translator Skills -This guide explains how to build a per-language Claude Code skill — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You don't need to read these skills; everything required is in this guide and the UTS docs. +This guide explains how to build a per-language Claude Code skill — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). -If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [UTS-to-Lang Skill Creator](uts-to-lang-skill-creator.md); this guide remains the authority on what the skill must contain. +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [UTS-to-Lang Skill Creator](uts-to-lang-skill-creator.md), on an Opus-class model ([Model tier](#model-tier)); this guide remains the authority on what the skill must contain. **Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. +- **Harness and skill are one deliverable.** The skill translates onto a harness built in the same repo from its existing test setup, and it isn't complete until that harness's smoke tests and self-tests are green, in CI, at every tier the skill supports ([section 2](#2-the-harness-uts-test-infrastructure)). - **Easy to use.** One command (`/uts-to- `) and four questions; the skill does the rest. - **Context-complete.** Every run gathers the full context: the spec, the translation manual, the sources and README of the UTS test infrastructure (the **harness**), the module notes and the resolver output. A good test doesn't depend on anyone's memory. - **Tests are reproducible artifacts.** The UTS spec plus the skill is the source of truth; the generated test is an output. When a translation is wrong or the spec changes, regenerate it ([section 9](#9-keeping-in-sync-with-spec-changes)) rather than hand-maintain it. @@ -22,7 +23,7 @@ Facts about the existing skills and SDKs are **as of 2026-10-07**: spec commit ` ## Contents 1. [Purpose and scope](#1-purpose-and-scope) -2. [Prerequisites in the SDK repo](#2-prerequisites-in-the-sdk-repo) +2. [The harness (UTS test infrastructure)](#2-the-harness-uts-test-infrastructure) 3. [Skill anatomy](#3-skill-anatomy) 4. [The workflow the skill must implement](#4-the-workflow-the-skill-must-implement) 5. [Translation rules](#5-translation-rules) @@ -42,7 +43,7 @@ Facts about the existing skills and SDKs are **as of 2026-10-07**: spec commit ` ### What a translator skill is -A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that Claude Code follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. +A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that Claude Code follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a harness in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). ### How it relates to the other UTS docs @@ -53,6 +54,7 @@ A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and no | [`writing-derived-tests.md`](../writing-derived-tests.md) | **Translation and evaluation semantics.** Faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | | [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | | **This guide** | How to package all of the above into a repeatable, scripted Claude Code skill for one SDK | +| Existing skills and harnesses ([Reference implementations](#reference-implementations-last-resort)) | Nothing normative: a last-resort, read-only reference for patterns when creating a skill, below every document above | `writing-derived-tests.md` is the **source of truth for semantics**. This guide never redefines what a faithful translation is, or how a deviation is classified and recorded; it says how a skill enforces those rules. If the two ever disagree, `writing-derived-tests.md` wins. @@ -68,11 +70,25 @@ A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and no A skill **SHOULD** keep only the language rendering of a rule and link to the source for its meaning. Inline copies of semantic rules drift. Both existing skills replaced their inline `deviations.md` format with a pointer to the manual for this reason, although they still inline the decision tree and test patterns. +### Model tier + +Creating a skill and its harness, and running the skill, are long, multi-file reasoning tasks in which a silent omission costs far more than compute. Use **the most capable model tier available (Claude Opus or an equivalent Opus-class model)**. "Opus-class" means that tier, whatever its current version, not a particular release. + +| Task | Level | Why | +|---|---|---| +| Creating the skill and building its harness (the [procedure](uts-to-lang-skill-creator.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | +| Translating and evaluating with the finished skill (`/uts-to-`) | **SHOULD** | The resolver and the audit ([section 8](#8-deterministic-tooling)) make paths, names, ID coverage and assertion counts mechanical, but not the rest. Matching wait predicates and timeouts, copying comments verbatim, setup fidelity, and diagnosing a failure as SDK, spec or translation ([7](#7-evaluation-and-deviations), [8.3](#83-review-checklist-must-after-the-audit)) remain model judgement, and a count can't catch a changed predicate ([8.2](#82-audit-audit_translationpy-must)) | +| Sub-agents spawned during either (review, validation; pilot runs during creation serve creation) | **The level of the task they serve** | A reviewer weaker than the author misses what the author missed. A cheaper model MAY be used only for a clearly mechanical step whose result a script verifies (for example, running the audit over the corpus and collecting its JSON), if at all | + +The deterministic scripts reduce the dependence on the model; they don't remove it. Pin the model where the tooling allows ([3.3](#33-frontmatter-and-arguments)), and record the model used in every run's final report ([section 11](#11-final-report-format)). + --- -## 2. Prerequisites in the SDK repo +## 2. The harness (UTS test infrastructure) + +The harness is part of the deliverable, not an optional prerequisite: a `uts-to-` skill is delivered together with its harness, and it isn't complete until the harness's smoke tests and self-tests ([2.7](#27-harness-smoke-tests-and-self-tests-must)) are green, in CI, at every tier the skill supports. A skill can't make translation mechanical if the harness it translates onto doesn't exist or doesn't keep its contracts. -A skill cannot make translation mechanical if the harness it translates onto does not exist. Build these **before** writing the skill. In both existing SDKs this took several iterations, and the harness co-evolved with the first real translation run. +**Design the harness from the repo's existing test setup.** For every capability in 2.1–2.7, first find what the SDK's native tests already have, then decide to **reuse** it, **wrap** it in a pseudocode-shaped facade, **extend** it, or **build** new (2.2, "Reuse before you build"). Build the harness, and its tests, **before** writing the skill. In both existing SDKs this took several iterations, and the harness co-evolved with the first real translation run. ### 2.1 SDK test hooks (MUST) @@ -88,7 +104,7 @@ The unit tier needs injection points in production code: Mocks and the fake clock are injected **at client construction**, so the harness must install them before the client is created. -**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when `FakeClock.waitOn` did a real timed wait, commit `c4502c67`.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. +**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when a retry ran on wall-clock time through `FakeClock.waitOn`, which does a real timed wait, commit `c4502c67`.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. Two workable shapes: route every SDK delay (`sleep`, `call_later`, timeouts) through the clock hook, so `ADVANCE_TIME` runs them; or run unit tests on a virtual-time event loop whose `time()` the fake clock controls, in which case the wall-clock timeout wrapper must not use the loop's clock (it measures virtual time). If a hook doesn't exist, add it to the SDK first. Without these hooks, the unit tier is impossible. @@ -118,13 +134,33 @@ The library also needs: - **A log sink** for "an error is logged" assertions. - **An in-order subsequence assertion** for `CONTAINS_IN_ORDER`. - **Caller-attributed failures:** helpers that time out or receive an unexpected error report the failure at the calling test line (Swift `sourceLocation:`, pytest `__tracebackhide__`, C# caller-info attributes), not inside the helper. -- **Harness smoke tests (SHOULD)**, one per tier, as the acceptance gate while the harness is being built. They are structural examples only: not spec-derived, no `UTS:` tag, and the skill must never copy their shape. They MAY be retired once spec-derived tests cover each tier: ably-java keeps them in `:uts`; ably-cocoa removed its own (commit `58fa0f5e`). +- **Per-tier smoke tests and helper self-tests (MUST)**, permanent and run in CI: the minimum set, and the rules for where they live and how they run, are in [2.7](#27-harness-smoke-tests-and-self-tests-must). **Placement.** Separate **shared** test-support code (used by both the SDK's native tests and the UTS ports, importing no test framework) from **port-only** harness code (doubles and fixtures used only by UTS ports); keep port-only code with the ports. Module-specific helpers (for example the `standard_test_pool.md` implementation) live next to that module's suites; the shared library stays module-agnostic. For a plugin-backed module such as LiveObjects, provide a client-options builder in the module helpers that registers the plugin, so tests never wire it by hand. +**Recommended layout (SHOULD).** One harness library (a test-support module, target or package) that every target hosting UTS tests consumes as a test dependency, organised by tier, with its smoke tests and self-tests beside it and the generated tests kept separate: + +``` +/ # the harness library; one per SDK + README.md # what exists, how to run each tier's harness tests, Known gaps (3.1) + common/ # wait and poll helpers, capture, log sink, assertContainsInOrder, caller attribution + unit/ # client factory, MockWebSocket, MockHttpClient, fake clock, message templates, optional mocks + integration/ # SandboxApp, suite fixture with guaranteed teardown + proxy/ # ProxyManager, ProxySession, rule builders + tests/ # smoke tests and helper self-tests (2.7), by tier: + common/ unit/ integration/ proxy/ + # the mapping's target directories (3.2), outside / + unit// integration/standard// integration/proxy// + # next to each module's suites, depending on , never the reverse; + # a module's smoke test lives beside its helpers (outside every target dir) + # and is selected by the unit tier's harness command +``` + +Where the SDK already has a test-options type, add the hooks to it, and implement the SDK's own transport and HTTP abstractions rather than patching internals. Keep the harness tests where one per-tier filter selects them (2.5), and where no generated test can collide with or overwrite them. (ably-java's `:uts` Gradle module is one example of this shape: per-tier harness packages, with the smoke tests in the module's own test set.) + **Document each fixture helper's scope** in the module notes: which client messages it auto-answers, which spec branches it doesn't wire (a DETACH reply, fake timers), the local pattern tests use for those gaps, and any option tweaks the gap needs. (ably-java: the fake-clock variant of `setupSyncedChannel` needs a large `maxIdleInterval`, or virtual-time advances trip the idle timer.) -**Reuse before you build.** If the SDK's native tests already have transport doubles, a fake clock or sandbox helpers, assess whether to reuse, wrap or extend them against the helper specs' semantics before writing new ones; never change their behaviour for the native tests that use them. +**Reuse before you build.** If the SDK's native tests already have transport doubles, a fake clock or sandbox helpers, assess whether to reuse, wrap or extend them against the helper specs' semantics before writing new ones; never change their behaviour for the native tests that use them. Existing doubles bound to another test framework, or built for real-network interception rather than the helper specs' in-process semantics, are usually better left alone: build pseudocode-shaped doubles on the same SDK hooks instead, and extract shared plumbing (such as sandbox retries) for both. (One SDK kept its legacy doubles for its native suite and built new framework-free UTS doubles on the same transport and HTTP seams.) **Build the harness to look like the pseudocode.** Callback-style mocks with the spec's handler names, and a client factory that mirrors `Realtime(options: …)`, turn translation into transliteration. ably-java's first generation run replaced a "wrap every callback in a coroutine" rule with callback-style mocks for exactly this reason. @@ -136,7 +172,7 @@ The library also needs: ### 2.4 The proxy (MUST for the proxy tier) -- A `ProxyManager` that downloads, caches and starts [`ably/uts-proxy`](https://github.com/ably/uts-proxy). Pin the version, and check that the pinned release publishes a binary for every developer OS (its GitHub release assets). An environment override pointing at a locally built binary (`UTS_PROXY_LOCAL_PATH` in both existing harnesses; ably-java also accepts `-Duts.proxy.localPath`) is useful; keep it a configuration override, not a test gate. +- A `ProxyManager` that downloads, caches and starts [`ably/uts-proxy`](https://github.com/ably/uts-proxy). Pin the version, and check that the pinned release publishes a binary for every developer OS (its GitHub release assets). An environment override pointing at a locally built binary (`UTS_PROXY_LOCAL_PATH` in both existing harnesses; ably-java also accepts `-Duts.proxy.localPath`) is useful; keep it a configuration override, not a test gate. Operationally, the manager also verifies the downloaded archive (a checksum), serialises the download across concurrent test processes, waits for the proxy's control endpoint to report healthy before returning, reaps the proxy process when the test process exits, and handles the port: proxy suites share one control port, so run them in one process (one fork or worker), or give each worker its own port. - A native `ProxySession` implementing [`proxy.md`](../proxy.md): `create_proxy_session`, `add_rules`, `trigger_action`, `get_log`, `close`, plus rule-builder helpers and a typed event log. - **The `action` inside a frame `match` (`ws_frame_to_client` / `ws_frame_to_server`) must be a JSON string**: a name (`"ATTACHED"`) or a numeric string (`"11"`), never a number (the proxy rejects a number with HTTP 400). uts-proxy v0.3.0 resolves names only up to `AUTH` (17), so use numeric strings for `OBJECT` (`"19"`), `OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`); an unresolvable name silently never matches. This applies only to `match.action`: a rule's own `action` is an object (`{"type": "suppress"}`), and the `action` inside an injected message stays a number. Rule builders should take an integer and stringify it, as both existing harnesses do. - Proxy tests always use JSON: the proxy supports text frames only (`integration-testing.md`, "Protocol Variants"). @@ -145,7 +181,7 @@ The library also needs: ### 2.5 Build and CI wiring (MUST) -- Test tasks or filters that select each tier (unit, direct integration, proxy), each with a single-class/single-test filter. +- Test tasks or filters that select each tier (unit, direct integration, proxy), each with a single-class/single-test filter, and, per tier, a filter and the complete command that runs that tier's harness smoke tests and self-tests ([2.7](#27-harness-smoke-tests-and-self-tests-must)). The harness tests run ungated in their tier's CI job. Check the repo's existing test commands and jobs: a bare `pytest`, `dotnet test` on the solution, or a Gradle `test` that aggregates every module also collects the new harness and UTS tests (including network tiers). Exclude them there (`testpaths`/`--ignore`, a solution filter, a separate project or task), or include them deliberately in that job as their one CI home. - **Every suite runs in exactly one CI job per platform it targets, and nothing runs green without executing** (MUST). Splitting by tier (unit in the fast check job; integration and proxy in a networked job) is recommended (SHOULD): ably-java does this, while ably-cocoa runs all UTS tiers from one job definition, one job per platform (an iOS / tvOS / macOS matrix; proxy suites compile only on macOS). - Integration jobs need network access, the `ably-common` submodule and the proxy binary. @@ -160,6 +196,113 @@ Most flakiness in both existing SDKs came from concurrency, not from translation - Whether the language enforces concurrency safety at compile time (Swift 6 `Sendable` drove ably-cocoa's capture types). - Where white-box unit specs can see internals (Swift `@testable import`, Objective-C private headers, C# `InternalsVisibleTo`, Kotlin `internal` within the owning module). This decides **where tests live**. +### 2.7 Harness smoke tests and self-tests (MUST) + +The harness has two kinds of test of its own, both permanent and both required: + +- **Per-tier smoke tests:** end-to-end wiring of one tier through the real SDK hooks (unit with mocks and the fake clock; direct sandbox; proxy). +- **Helper self-tests:** contract tests that each harness helper obeys its helper spec, or this guide's rule for it. + +**Why both.** Spec-derived tests can't be the harness's acceptance gate. They are regenerated, they may legitimately be red (a UTS spec error), and a failing one has to be triaged as SDK, spec or harness. A smoke-test or self-test failure is unambiguously a harness or environment fault. Scenario-only smoke tests aren't enough: they follow a happy path, so a helper that breaks its contract (a mock `close()` that notifies synchronously, a declared mock event that is never emitted, a state wait that samples) still passes, and the defect surfaces later as a flaky spec-derived test. Both existing SDKs hit this (as of 2026-10-07). One had connect-only smoke tests, env-gated and never run in CI, then removed them; its sampling state wait, error-swallowing polls and teardown hangs were later found only through flaky spec-derived tests. The other's scenario smoke tests, run in CI, caught a fake clock that waited on the real clock, but not its synchronous mock `close()` or a mock event type it declares and never emits. + +**Rules (MUST):** + +- **Permanent.** Don't retire them when spec-derived tests cover a tier. +- **In CI, ungated.** They run, ungated, in whichever CI job runs their tier ([2.5](#25-build-and-ci-wiring-must)); no environment variable may skip them (compile-time platform gating per 2.4 aside). +- **Untagged and structural.** No `UTS:` tag; each is named after the capability it proves; each passes on its own under the single-class filter. +- **Never copied.** The skill never copies their shape and never names them as reference tests. Its Harness reference ([3.4](#34-recommended-outlines-should)) may cite a smoke test as a wiring example only (client construction, suite fixtures, teardown). +- **Separate.** They live with the harness (layout in 2.2), outside every directory the mapping gives the resolver as a target, so generated tests can't collide with or overwrite them. +- **Selectable.** One complete command per tier runs that tier's smoke tests and self-tests (plus the common self-tests). The mapping declares it ([3.2](#32-the-mapping-file)), and the skill's preflight runs it ([4](#4-the-workflow-the-skill-must-implement), step F). +- **Explicit.** Each item below is an explicit assertion. One scenario test may cover several items; an item covered only implicitly doesn't count. Every event type the harness declares is emitted and asserted (or the type is removed). A helper-spec contract point the harness doesn't meet is listed in the harness README's Known gaps ([3.1](#31-layout)), not left untested. + +**Minimum set per tier** (a tier the skill doesn't support needs none of its entries): + +**Common (every tier)** + +- Self-tests: + - `assertContainsInOrder` has subsequence semantics (interleaving passes, a repeated state passes, wrong order fails) + - `poll_until` returns the settled value and aborts on an error + - `poll_until_success` rethrows the last error on timeout + - The wall-clock timeout wrapper bounds a wait, including under a virtual-time runner + - Capture lists are safe under concurrent appends + - A failing helper reports the caller's line + - Every state-wait helper (unit and integration) latches a state entered and left within one scheduling tick and fails fast on FAILED + - Teardown closes clients and cancels timers when the body throws, and, from a state where the SDK's `close()` can't reach CLOSED ([5.4](#54-setup-and-teardown)), doesn't wait for CLOSED + +**Unit, realtime (mock WebSocket and fake clock)** + +- Smoke test: + - A factory-built client with the mock installed reaches CONNECTED via `respond_with_success` and `CONNECTED_MESSAGE`, and a template field (e.g. the connection id) reaches the client + - The connection attempt exposes host, port, TLS and query params + - A client→server frame (ATTACH, with a non-ASCII channel name) is captured and decoded + - A server→client frame (ATTACHED) changes state + - `simulate_disconnect` yields DISCONNECTED, observed by record-and-verify + - A timer-driven retry doesn't happen before `ADVANCE_TIME` and does after it + - The await style drives refused attempts through to SUSPENDED +- Self-tests: + - Mock `close()` calls `onClose` asynchronously + - CONNECTED is delivered after the connection completes + - A raw untyped frame can be sent + - WS-level timeout and DNS-error outcomes exist + - Handler and await styles both work, and the documented mixing rule holds + - Every declared mock event type is emitted, in order + - `process_pending_events()` runs a queued callback with no real delay + +**Unit, fake clock** + +- Smoke test: (driven by the realtime smoke test) +- Self-tests: + - One `ADVANCE_TIME` runs cascaded work to quiescence (zero-delay reschedules, timers created mid-advance) + - The fake clock never waits on the real clock (the SDK's timed blocking waits and the async runtime's timers follow it) + - A poll deadline still expires with the fake clock installed + +**Unit, REST (mock HTTP)** + +- Smoke test: + - A request is captured with method, path, headers and body + - `respond_with` status, body and headers are decoded by the SDK + - A realtime-driven HTTP request (e.g. token auth via `authUrl`) is served by the HTTP mock while the WS mock serves the connection +- Self-tests: + - Connection-level and request-level failures are distinguishable + - Handler and await styles both work + +**Unit, optional mocks** + +- Smoke test: + - `MockNetworkListener` (if the SDK has a network monitor) and `MockVCDiff*` (if it supports deltas) each drive the SDK once + +**Unit, objects (each module with a fixture helper)** + +- Smoke test: + - `setup_synced_channel` (or its sanctioned stand-in) completes and one pool object reads its seeded value +- Self-tests: + - Serial helpers sort as the helper spec requires + +**Direct sandbox** + +- Smoke test: + - An app is provisioned once per suite (or per test where 5.4 allows it, disclosed) and deleted in teardown, and its keys look right + - A client connects to the sandbox (the recorded state sequence ends CONNECTED) + - Attach, subscribe and ack-awaited publish on a unique channel, with messages received in order + - A REST read (history) returns them via `poll_until` + - It runs once per protocol variant + - Every wait is wall-clock and bounded +- Self-tests: + - `SandboxApp` creates and deletes an app, and never retries `POST /apps` + +**Proxy** + +- Smoke test: + - The pinned proxy (or the local override) is fetched, verified, started and healthy + - A pass-through session connects a client with the chosen proxy auth + - The typed log shows `ws_connect` and a server→client CONNECTED frame + - `trigger_action(disconnect)` is followed by a recorded DISCONNECTED, then recovery + - A frame-match rule with a string `match.action` (e.g. `"11"`) fires once, then the client recovers + - Sessions and clients close in teardown, even on failure +- Self-tests: + - Rule builders stringify `match.action` + - A session with `rules:` only defaults its endpoint to the sandbox ([6.7](#67-notes-on-the-catalogue)) + --- ## 3. Skill anatomy @@ -181,7 +324,7 @@ Most flakiness in both existing SDKs came from concurrency, not from translation | File | Purpose | Level | |---|---|---| | `SKILL.md` | The two-phase procedure, construct table, a file template per tier, evaluation patterns, tier wiring. Keep it **module-generic**: the core realtime/rest mapping may live here; plugin or typed-module specifics go in notes | MUST | -| `uts-package-mapping.json` | Maps each source module (`rest`, `realtime`, `objects`) and tier (`unit`, `integration`, `proxy`) to one target directory, plus an optional `notes` path | MUST | +| `uts-package-mapping.json` | Maps each source module (`rest`, `realtime`, `objects`) and tier (`unit`, `integration`, `proxy`) to one target directory, plus an optional `notes` path; declares the harness (root, README, per-tier sources and harness-test command) | MUST | | `scripts/resolve_uts.py` | Validates the module dir, reads the mapping, lists specs and derives file/class/package names; prints one JSON object | MUST | | `scripts/audit_translation.py` | Compares one spec file with its generated test file; prints one JSON object | MUST | | `scripts/scan_constructs.py` | Lists the uppercase keywords and snake_case calls in a module's `pseudo` fences, so unmapped constructs are found mechanically ([section 6](#6-pseudocode-construct-catalogue)) | SHOULD | @@ -189,7 +332,7 @@ Most flakiness in both existing SDKs came from concurrency, not from translation Both existing skills use Python for scripts. Any language works, but the scripts must run on every developer platform (see [8.1](#81-resolver-resolve_utspy-must) and [section 10](#10-verification-and-ci)). -Keep two documents with separate roles: the test library's README describes **what exists** (helpers, seams, layout); `SKILL.md` describes **how to author** tests. Refer to the README by its full repo path, so it can't be confused with the spec repo's `uts/README.md`. The harness README SHOULD cover: the SDK hooks used, each helper-spec symbol → harness symbol, the wait helpers and their default timeouts, the threading and time model, how to run each tier, and known gaps against the helper specs. +Keep two documents with separate roles: the test library's README describes **what exists** (helpers, seams, layout); `SKILL.md` describes **how to author** tests. Refer to the README by its full repo path, so it can't be confused with the spec repo's `uts/README.md`. The harness README MUST include a **Known gaps** section: every helper-spec member or contract point, and every corpus construct, that the harness doesn't implement, by tier, in a form the skill's preflight can match against the selected specs (construct or helper name → status and workaround). Write it as a table whose first column is the exact pseudocode token as it appears in `pseudo` fences (for example `respond_with_timeout`, `PING_MESSAGE`, `AWAIT_ALL`), so the preflight matches mechanically: for each row, grep the selected specs' `pseudo` fences for the token; any hit is a conflict. The corpus scanner (section 6) MAY accept spec files as well as a module directory for this. It SHOULD also cover: the SDK hooks used, each helper-spec symbol → harness symbol, the wait helpers and their default timeouts, the threading and time model, how to run each tier and its smoke tests and self-tests (2.7), and one index of every module's fixture helpers (even those in a separate test-support module). **The notes are a map, not an authority.** Where the notes, the spec's IDL and the SDK disagree about the SDK's API, the SDK source is the ground truth: fix the notes. @@ -200,6 +343,15 @@ Each tier value is **one path**, relative to the repo or to a declared root (Swi ```json { "_comment": "Maps each UTS module to its target test dir per tier. Used by scripts/resolve_uts.py.", + "harness": { + "root": "", + "readme": "/README.md", + "tiers": { + "unit": { "sources": ["/common", "/unit"], "tests": "" }, + "integration": { "sources": ["/common", "/integration"], "tests": "" }, + "proxy": { "sources": ["/common", "/integration", "/proxy"], "tests": "" } + } + }, "packages": { "realtime": { "unit": "/unit/realtime", "integration": "/integration/standard/realtime", "proxy": "/integration/proxy/realtime" }, "objects": { "unit": "...", "integration": "...", "proxy": "...", "notes": "references/objects-mapping.md" } @@ -209,6 +361,7 @@ Each tier value is **one path**, relative to the repo or to a declared root (Swi - **Make every derived namespace unique**: keep a module segment after the tier (`unit/realtime`, never bare `unit`), or use a module-specific prefix (ably-java's objects tests use `io.ably.lib.liveobjects.uts.unit`). - **`notes`** is relative to the skill directory, not the repo root. +- **`harness`** (MUST): the harness root and README, and per tier the helper sources to read and the complete command, run from the repo root, that runs that tier's smoke tests and self-tests (e.g. `dotnet test tests/Uts --filter "Category=UtsHarness.Unit|Category=UtsHarness.Common"`, `pytest test/uts/harness/tests/common test/uts/harness/tests/unit`) ([2.7](#27-harness-smoke-tests-and-self-tests-must)). Paths are repo-relative. The resolver reports it, with paths validated (still repo-relative) ([8.1](#81-resolver-resolve_utspy-must)), so `SKILL.md` never hard-codes harness paths. - **Language-specific derivations live in the resolver.** Swift has no package, so the directory is the full story. Kotlin derives `package` from the path after `src/test/kotlin/` and the Gradle module from the first segment. A C# resolver might map `tests/IO.Ably.Tests.Uts/Unit/Realtime` to namespace `IO.Ably.Tests.Uts.Unit.Realtime` (PascalCase segments) and to the nearest ancestor `.csproj`; a Python resolver derives a dotted package path. - **Hand-maintained entries.** `--create` scaffolds only the repo's default layout. A module whose tests live elsewhere (ably-java's `objects`, in `:liveobjects`) is hand-maintained: the skill MUST say which entries are, and must not offer `--create` to "fix" them. - You MAY store per-module defaults as data, e.g. `"evaluate": true` or a list of white-box specs, instead of prose in the notes. @@ -228,16 +381,21 @@ allowed-tools: Bash, Read, Edit, Write - **`description`** SHOULD be "pushy": list the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to ", "re-sync UTS tests"). Both existing skills' descriptions are procedural, which suits explicit `/` invocation but triggers poorly otherwise. - **Argument:** a single UTS **module** directory, directly under `uts/`. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. If the argument is empty, the skill MUST print the usage line and stop. - **`allowed-tools`:** both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-local-clone-vs-fetching-main)). +- **Model (SHOULD):** if your Claude Code version lets a skill declare the model it runs on, pin the skill to an Opus-class model ([Model tier](#model-tier)). Check the current Claude Code documentation for the mechanism rather than assuming a frontmatter field. If it can't, say in the opening lines of `SKILL.md` that the skill should be run on an Opus-class model. Either way, the skill records the model in its final report ([section 11](#11-final-report-format)). - **Placeholders, not personal paths,** in examples and usage text: `/uts/objects`. ### 3.4 Recommended outlines (SHOULD) -Use these shapes (both existing skills converged on them independently of language): +The outlines are SHOULD; the Harness reference section within the `SKILL.md` outline is MUST (below). Use these shapes (both existing skills converged on them independently of language): ``` SKILL.md Required reading (local clone, 9.4) - Phase 1, selection: usage guard; step 0; A resolve; B mapping; C tier; D specs; E translate vs evaluate + Harness reference: per tier, from the resolver's harness output: harness root and README path, + helper sources to read, smoke-test and self-test locations and run commands, the + generated-test run command, the README's Known gaps + Phase 1, selection: usage guard; step 0; A resolve; B mapping; C tier; D specs; + E translate vs evaluate; F harness preflight Phase 2, per spec: 1 read spec + notes; 2 output path; 3 read harness + reference test; 4 generate (construct table, client construction, mocks, timers, assertions, naming, file template per tier); 5 compile; 6 run + diagnose (evaluate only); 7 audit + review @@ -246,6 +404,8 @@ SKILL.md Final report (section 11) ``` +`SKILL.md` MUST have the **Harness reference** section. It points at the harness sources, README and harness tests; it doesn't copy helper signatures inline (inlined copies drift). A construct that relies on a listed known gap is a stop-and-ask or a Mock Infrastructure Limitation, never a substitution (5.2). A smoke test may be named as a wiring example only, never as a reference test (2.7). + A module notes file should cover: 1. **Source of truth and runtime status:** the spec's IDL that the notes apply, including any typed-SDK variant (for objects, the proposed `RTTS` points (RTTS1–RTTS11 at the branch tip, `c5cc6b8a`) that both existing skills' notes follow; as of `12540dcf` they are on the unmerged `feature/liveobjects-cross-sdk-types-spec` branch, not in `objects-features.md`), and whether the module is implemented (whether evaluate mode is possible). @@ -278,9 +438,9 @@ file-local helpers: before the first test or in a separate file (the audit attri ## 4. The workflow the skill must implement -The workflow has two phases. Implement both. +The workflow has two phases. Implement both. Runs SHOULD use an Opus-class model ([Model tier](#model-tier)); the final report records the model used ([section 11](#11-final-report-format)). -### Phase 1: selection (steps 0 and A–E) +### Phase 1: selection (steps 0 and A–F) | Step | What happens | Interaction | |---|---|---| @@ -290,6 +450,7 @@ The workflow has two phases. Implement both. | **C. Choose the tier** | Offer only tiers that are present. Where the mapping records a tier as not ready (per-module data, 3.2), refuse it and name what blocks it. The tier fixes the target dir, the spec list **and** the translation flow (mocked / sandbox / proxy). Don't re-detect it per spec | **Ask** | | **D. Choose specs** | All specs in the tier, or a subset | **Ask** | | **E. Translate-only, or translate and evaluate?** | Translate-only: generate, compile, and run the step 7 audit and review, but don't run the tests (the SDK feature may not exist yet). Translate and evaluate: also run and diagnose. If unsure whether the implementation exists, ask rather than guess | **Ask** | +| **F. Harness preflight** | Before generating anything, for the chosen tier and specs, in both modes: compile (or collect) the harness and the test target; run the tier's smoke tests and self-tests with the resolver's `harness` command ([2.7](#27-harness-smoke-tests-and-self-tests-must)); check the harness README's Known gaps against the selected specs. Integration and proxy tiers need network (and the proxy binary or platform) for this. If anything is red, **stop** and report it as a harness or environment problem, never as an SDK deviation or a spec error; don't generate tests on a red harness. If a selected spec needs a known gap, stop and ask: extend the harness (a harness change, section 5), deselect the spec, or proceed recording a Mock Infrastructure Limitation for the affected lines. Never substitute. Record the result in the final report | Stop on red | If the module's notes file exists but is only a placeholder, the skill MUST tell the user the module's mapping hasn't been authored and treat the module as not yet translatable, rather than guess a mapping. @@ -299,7 +460,7 @@ If the module's notes file exists but is only a placeholder, the skill MUST tell 1. **Read the spec and the notes.** Read the module notes first (once per run); where they override the generic flow (reading list, internal-access ladder, naming, deviations location), the notes win. Then identify per spec: test cases and IDs, protocol (WS or HTTP), timer use, `## Protocol Variants`, and any delegating, fixture-driven or ID-less structure. 2. **Determine the output path.** Use the resolver's `targetDir` and class name. Flatten spec sub-directories. If a suitable suite already exists, add to it rather than create a duplicate. -3. **Read the harness for exact signatures.** Read the test library's README, then **all** helper sources for the tier, before generating code. Guessed signatures are the most common compile failure. `SKILL.md` SHOULD also name one reviewed, spec-derived test file per tier to read first (ably-cocoa: `ChannelHistoryTests.swift` for direct sandbox, `AuthReauthTests.swift` for proxy; ably-java: `ChannelHistoryTest`, `ObjectsLifecycleTest`). Never point at smoke tests. +3. **Read the harness for exact signatures.** Read the harness README, then **all** helper sources for the tier (both from the resolver's `harness` output), before generating code. The source wins where the README disagrees; list each disagreement in the final report. Guessed signatures are the most common compile failure. `SKILL.md` SHOULD also name one reviewed, spec-derived test file per tier to read first (ably-cocoa: `ChannelHistoryTests.swift` for direct sandbox, `AuthReauthTests.swift` for proxy; ably-java names `ChannelHistoryTest` (realtime) and `ObjectsLifecycleTest` (objects), both direct sandbox, and no proxy reference). Never name a smoke test as the reference test. 4. **Generate** using the rules in [section 5](#5-translation-rules) and the table in [section 6](#6-pseudocode-construct-catalogue). A construct the skill can't map is a **stop-and-ask** condition (section 6). 5. **Compile once per module.** Fix and recompile until clean. 6. **Run (evaluate mode only).** Run each class by filter, then diagnose per [section 7](#7-evaluation-and-deviations). @@ -313,6 +474,8 @@ Every rule below is a MUST unless marked otherwise. **When a generated test is wrong, fix the cause, then regenerate.** The cause is a rule in `SKILL.md`, a mapping in the module notes, or a harness helper. A fix made only in the generated file is silently reverted by the next regeneration. The only per-test state that survives regeneration is listed in [9.2](#92-a-re-sync-mode-should). +**Changing the harness during a run** (extending a mock, adding a message template or a helper), by the skill or the user, is a harness change, not a translation step. Make it only within the harness scope the user approved. Add or extend a self-test for the new capability, update the harness README (and its Known gaps), and re-run the tier's smoke tests and self-tests ([2.7](#27-harness-smoke-tests-and-self-tests-must)) before continuing; stop if they are red. Then regenerate the affected tests (fix the cause, then regenerate). The final report lists harness changes separately (section 11). + ### 5.1 Traceability - **Exactly one `UTS:` tag per test**, equal to the spec's `**Test ID**`, copied **verbatim**: `// UTS: ` immediately above the test function (`writing-test-specs.md` "Placement in derived tests"). Where `//` isn't a comment, use the line-comment marker (`# UTS: ` in Python or Ruby); cross-SDK tooling should grep for `UTS: ` without the marker. Never hand-build the prefix: proxy files use `/proxy/…` IDs although they live under `integration/proxy/`, and objects IDs can join spec points (`objects/unit/RTO5c9-RTO20/…`). @@ -410,7 +573,7 @@ When a white-box spec asserts on the value an internal call returns, assert on t ## 6. Pseudocode construct catalogue -Fill in the last column for your language; the skill MUST map every construct its specs use. Each row's "meaning" comes from the UTS docs where one exists; rows marked *(undocumented)* are used in the corpus but not defined in the docs, and the meaning is inferred from usage. The Swift and Kotlin cells name only methods and patterns that exist in each skill, its notes, its harness or its generated tests; "—" means that SDK has no rule for the construct, so follow the meaning column. +Fill in the last column for your language; the skill MUST map every construct its specs use. Each row's "meaning" comes from the UTS docs where one exists; rows marked *(undocumented)* are used in the corpus but not defined in the docs, and the meaning is inferred from usage. The Swift and Kotlin cells name only methods and patterns that exist in each skill, its notes, its harness or its generated tests; "—" means that SDK has no rule for the construct, so follow the meaning column. The cells illustrate; they aren't templates. Write your own column from your harness and SDK, and don't copy theirs, or anything seen in the [reference implementations](#reference-implementations-last-resort). **This catalogue is not exhaustive**: the corpus keeps growing and has no grammar. The skill MUST treat any uppercase keyword or snake_case helper it can't map as a **stop condition**: tell the user, propose a rendering, and add a row to its own table rather than improvising per test. Re-scan the corpus for new keywords on every re-sync ([9.2](#92-a-re-sync-mode-should)). The skill SHOULD ship a **corpus scanner** (`scripts/scan_constructs.py`, [3.1](#31-layout)) that lists every uppercase keyword and `snake_case(` call in a module's `pseudo` fences, outside comments and string literals, with counts, so the re-scan is mechanical; compare its output with the construct table. @@ -614,7 +777,7 @@ Path validation, mapping, spec discovery, naming and faithfulness checking are m |---|---| | Validate the module | Expand a leading `~`. The parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/` | | Errors | One JSON object with `ok: false`, a code and a `message`. Example codes (ably-cocoa): `NOT_A_UTS_MODULE_PATH`, `DIR_NOT_FOUND`, `NO_TIER_DIRS`, `MAPPING_NOT_FOUND`, `BAD_MAPPING`, `BAD_TARGET_NAME` | -| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty}`, MAY) gives the spec clone's state for the file headers and the report (9.1, 9.4). Downstream steps read only these fields | +| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, harness, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty}`, MAY) gives the spec clone's state for the file headers and the report (9.1, 9.4). `harness` (`{root, readme, tiers: {: {sources, tests}}}`) echoes the mapping's harness entry with paths validated (repo-relative), for the Harness reference and the preflight (3.2, 4 step F). Downstream steps read only these fields | | Tier detection by path | `unit/**`; `integration/**` excluding `integration/proxy/**`; `integration/proxy/**`. Every tier can have sub-directories (e.g. `realtime/integration/channels/`). Use the path only. (Both existing skills also describe content-based proxy detection in their integration sections; drop it.) | | Exclusions | `helpers/` (implement as harness, never translate), `README.md`, `PLAN.md`, `*_SUMMARY.md`. Match them **relative to the tier base**, so an ancestor directory in the checkout path can't trip them. *(New, defensive; nothing matches today:)* also skip any non-spec notes left inside a tier directory | | Deterministic naming | Strip a trailing `_test`, convert to the language's convention, add the suffix or prefix the runner needs (`objects_lifecycle_test.md` → Swift `ObjectsLifecycleTests`, Kotlin `ObjectsLifecycleTest`, pytest `test_objects_lifecycle.py` with class `TestObjectsLifecycle`) | @@ -683,6 +846,8 @@ Stamp each generated file's header with the spec-repo commit SHA it was derived A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. +**Record the model.** The run's final report, which is the run record, records the model (name and ID) that generated or regenerated each file ([section 11](#11-final-report-format); part of that MUST format). A header line such as `// Generated with ` MAY be added as well. It changes whenever a different model regenerates the file, which adds diff noise, so prefer the run record. + ### 9.2 A re-sync mode (SHOULD) Neither existing skill has one; both have stale suites as a result. Add a mode (for example `/uts-to- --resync`) that: @@ -694,7 +859,7 @@ Neither existing skill has one; both have stale suites as a result. Add a mode ( 5. Updates the SHA in each regenerated file's header (and in the manifest, if kept). 6. Reports the classification ([section 11](#11-final-report-format)). -Regenerate the whole file when most tests changed; otherwise regenerate only the affected tests. Say which in the report. A re-sync is also needed when the shared harness or module helpers change shape (ably-cocoa reset and regenerated its objects ports after aligning them with `standard_test_pool.md`). +Regenerate the whole file when most tests changed; otherwise regenerate only the affected tests. Say which in the report. A re-sync is also needed when the shared harness or module helpers change shape (ably-cocoa reset and regenerated its objects ports after aligning them with `standard_test_pool.md`). After any harness change, re-run every affected tier's smoke tests and self-tests ([2.7](#27-harness-smoke-tests-and-self-tests-must)) first, then regenerate the tests whose rendering depends on the changed helper. ### 9.3 Renamed, merged and added IDs @@ -715,7 +880,7 @@ Pick **one** source for docs and specs, and pin it. This guide recommends **read Record the clone's HEAD SHA and whether it is dirty. You MAY warn when the clone is behind `origin/main`. - *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. -- *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md`, and the features specs, from GitHub `main` while reading specs from the local clone, so the two can skew. +- *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md` from GitHub `main`, and tell the model to "fetch" the features spec with no pinned source, while reading specs from the local clone, so the two can skew. --- @@ -723,7 +888,7 @@ Record the clone's HEAD SHA and whether it is dirty. You MAY warn when the clone Escalate in this order: -1. **Compile the tests** for the module (`swift build --build-tests`; `./gradlew ::compileTestKotlin`; `dotnet build`). For every runner, check that the number of collected (or executed) tests equals the number of `UTS:` tags: a misnamed pytest class, or a JVM test annotated for an engine the task doesn't run (JUnit 4 vs JUnit Platform), is silently skipped. For pytest use `pytest --collect-only` (plus `mypy` if the repo uses it); for .NET, `dotnet test --list-tests`; for Gradle, compare the task's test-report count. In dynamic languages a missing SDK API surfaces only at run time: run a type checker (`mypy`/`pyright`) over the generated tests where the SDK ships type hints, or report missing APIs as unverified in translate-only mode. MUST, in both modes. +1. **Compile the tests** for the module (`swift build --build-tests`; `./gradlew ::compileTestKotlin`; `dotnet build`). For every runner, check that every tagged test is collected (or executed): the number of distinct collected test functions, counting a parameterised or protocol-variant test once, equals the number of `UTS:` tags (the runner lists each parameter case separately, so group by function): a misnamed pytest class, or a JVM test annotated for an engine the task doesn't run (JUnit 4 vs JUnit Platform), is silently skipped. For pytest use `pytest --collect-only` (plus `mypy` if the repo uses it); for .NET, `dotnet test --list-tests`; for Gradle, compare the task's test-report count. In dynamic languages a missing SDK API surfaces only at run time: run a type checker (`mypy`/`pyright`) over the generated tests where the SDK ships type hints, or report missing APIs as unverified in translate-only mode. MUST, in both modes. 2. **Lint and format** with the repo's gates (EditorConfig, checkstyle, eslint, ruff…). MUST. Neither existing skill runs lint; both repos needed it. 3. **Compile as strictly as CI does.** If CI treats warnings as errors, so must the skill. (ably-cocoa's LiveObjects CI failed on a redundant typed-throws cast the local build accepted.) In TypeScript, the test runner may strip types without checking them (`writing-derived-tests.md` "Build pipeline and CI checks"). 4. **Run the generated classes by filter** (evaluate mode): per class, then per tier. Use target-qualified filters where names can collide. @@ -741,6 +906,7 @@ Neither existing skill defines one; define it. The skill MUST end each run with ``` UTS translation: / @ ably/specification@ (clone clean|dirty) (mode: translate-only | evaluate | resync) +Model: ; sub-agents: | Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | |---|---|---|---|---|---|---|---| @@ -752,6 +918,8 @@ Missing APIs found in translate-only mode: — Mock-capability gaps found in translate-only mode: — Harness stand-ins disclosed in file headers: … Constructs the skill couldn't map (stopped and asked): … +Harness preflight (step F): compile ok | smoke n/n | self-tests n/n | known-gap check ok (or the stop reason) +Harness changes made during this run, and the self-tests added or extended: … Suspected UTS spec errors or gaps (need an upstream fix): — , draft fix: … Mapping or notes changes: … Changed files for review: … @@ -769,36 +937,40 @@ From the git histories of both existing skills. Most lessons are already rules i | **Build the harness to look like the pseudocode.** Callback mocks and a builder DSL make translation mechanical | ably-java's first generation run | | **Move everything mechanical into scripts.** Paths, names, mapping and coverage checks drift when the model does them | Both: resolver, then audit | | **Lessons must reach `SKILL.md`**, not only harness comments or commit messages | ably-java: several flake fixes stayed in KDoc | -| **Defer formats to the manual**; inline copies drift | Both: deviations format | +| **Defer formats to the manual**; inline copies drift | The existing skills' inline `deviations.md` format, since replaced by a pointer to the manual (section 1) | | **Don't cite line numbers in skill text**; they drift. Cite symbols or headings | ably-cocoa | | **Compile the examples in the skill**, or take them from real, compiling files in your own repo | ably-cocoa: an example helper doesn't compile | | **Placement follows visibility**: put white-box tests where internals are visible; share the harness as a library | ably-java tests moved to `:liveobjects` | -| **Evaluation finds spec bugs**; fix them at source | Both: several upstream spec PRs | +| **Evaluation finds spec bugs**; fix them at source | Evaluation runs of the existing skills | | **Concurrency is the main source of flakiness**: thread-safe recording lists, await the seed before subscribing, flush before injecting a stimulus, a fake clock that runs to quiescence | ably-java CI fixes | +| **Keep the harness smoke tests and self-tests permanently, in CI**; scenario-only smoke tests miss contract violations, and without them harness defects surface as flaky spec-derived tests that must be triaged as SDK, spec or harness | Both: one SDK's env-gated smoke tests never ran in CI and were removed; the other's scenario smoke tests missed two helper contract violations | | **Keep generic rules separate from per-language renderings**: write every rule language-neutrally first, then give its rendering for your SDK | Both: the Swift skill began as a port of the Kotlin one, and the generic parts had to be separated afterwards | --- ## 13. Checklist for a new `uts-to-` skill -**Prerequisites** +The skill and its harness are one deliverable: the skill isn't done until every MUST item below, harness items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. + +**Harness** - [ ] SDK test hooks: WebSocket factory, HTTP client, clock (covering blocking waits and the async runtime's timers), reachability/network monitor, randomness; injected at construction - [ ] Shared test library: `mock_http.md` (+ legacy `queue_*` and `mock_http.captured_requests`), `mock_websocket.md` (async close, CONNECTED ordering, raw frames, alternative API, undocumented members), `mock_vcdiff.md`, `standard_test_pool.md`, `MockNetworkListener` - [ ] Wait helpers: race-free event-latched `AWAIT_STATE` failing fast on FAILED, value-returning `poll_until`, `poll_until_success`, `process_pending_events()`, wall-clock timeout wrapper, deadlines safe under fake time - [ ] Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures -- [ ] Shared vs port-only placement; fixture-helper scope documented -- [ ] Harness smoke tests per tier while building (SHOULD) +- [ ] Harness designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; recommended harness layout; fixture-helper scope documented +- [ ] Harness README with a Known gaps section (MUST) +- [ ] Harness smoke tests per tier and helper self-tests (2.7): permanent, in CI, ungated, untagged, outside generated-test directories (MUST) - [ ] `SandboxApp` (retries idempotent reads only); `ably-common` submodule -- [ ] `ProxyManager` with a pinned `uts-proxy` version for every developer OS; `ProxySession`; string `match.action`; auth through the proxy decided and commented +- [ ] `ProxyManager` with a pinned `uts-proxy` version for every developer OS, verified download, health check and port handling; `ProxySession`; string `match.action`; auth through the proxy decided and commented - [ ] One CI home per suite (MUST); per-tier jobs (SHOULD) - [ ] Concurrency, time, runner-parallelism and internal-visibility models documented **Skill files** -- [ ] `SKILL.md` with `name: uts-to-`, a pushy description with trigger phrases, `argument-hint`, usage guard, placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline -- [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked -- [ ] `resolve_uts.py`: validation, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, collision detection, validate-then-write `--create` preserving entries +- [ ] `SKILL.md` with `name: uts-to-`, a pushy description with trigger phrases, `argument-hint`, usage guard, placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline, including the Harness reference +- [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked, `harness` entry +- [ ] `resolve_uts.py`: validation, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, collision detection, validate-then-write `--create` preserving entries, `harness` output - [ ] `audit_translation.py`: the parsing contract, ID coverage, duplicates, separate assert/await counts, ignores commented assertions, never crashes, flags unverifiable specs; itself mutation-tested - [ ] Per-module notes for every module whose API diverges from the pseudocode (at least `objects`, if the SDK implements it) - [ ] Scripts run on every developer platform (path handling, UTF-8, LF) @@ -806,10 +978,11 @@ From the git histories of both existing skills. Most lessons are already rules i **Workflow** - [ ] Required reading, including features specs, from the same clone as the specs -- [ ] Steps 0 and A–E with the four user questions (mapping, tier, specs, translate vs evaluate) +- [ ] Steps 0 and A–F: the four user questions (mapping, tier, specs, translate vs evaluate) and the harness preflight, which stops on red - [ ] Steps 1–7 with batching; a reference test per tier; review in both modes - [ ] Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) - [ ] Re-sync mode; fix the cause, then regenerate +- [ ] Mid-run harness changes re-run the smoke tests and self-tests and are reported separately **Translation rules in `SKILL.md`** @@ -842,30 +1015,62 @@ From the git histories of both existing skills. Most lessons are already rules i - [ ] Final report format - [ ] The skill's own examples compile +**Creation process** + +- [ ] Skill and harness created on an Opus-class model (MUST); the skill pins the model or states it for its runs (SHOULD), and records it in its final report ([Model tier](#model-tier)) +- [ ] Reads of the [reference implementations](#reference-implementations-last-resort), if any, made only as a last resort, recorded, and reported as guide gaps + --- ## Appendix: Existing skills (background and patterns to avoid) -You don't need to read these skills; everything required is in this guide. They are named only so that the examples and lessons above can be traced. +You shouldn't need to read these skills: this guide and the UTS docs are the authority. They are named so that the examples and lessons above can be traced, and as a last-resort reference for creating a skill and its harness ([Reference implementations (last resort)](#reference-implementations-last-resort)). | Skill | Repo and paths | Lessons this guide drew from it | |---|---|---| | `uts-to-swift` | ably-cocoa, `.claude/skills/uts-to-swift/`; tests under `Test/UTS/` | Strict compile-time concurrency checking (Swift 6); a callback-based core plus a natively async plugin; scoped-resource teardown without async `tearDown`; separate assertion and wait counts in the audit; "a harness stand-in is not a deviation"; the internal-access ladder; module notes overriding the generic flow | -| `uts-to-kotlin` | ably-java, `.claude/skills/uts-to-kotlin/`; tests in the `lib/` and `liveobjects/` test source sets, harness in `:uts` | Deriving a package and build module from the target path; callback- and await-style mocks; race-free `awaitState`; virtual-time pitfalls and real-clock timeouts; idempotent-only sandbox retries; the `S-n` shape-deviation vocabulary | +| `uts-to-kotlin` | ably-java, `.claude/skills/uts-to-kotlin/`; tests in the `lib/` (built by the `:java` Gradle module) and `liveobjects/` (`:liveobjects`) test source sets, harness in the `:uts` Gradle module (path `uts/`) | Deriving a package and build module from the target path; callback- and await-style mocks; race-free `awaitState`; virtual-time pitfalls and real-clock timeouts; idempotent-only sandbox retries; the `S-n` shape-deviation vocabulary | Both contain a `SKILL.md`, a mapping file, a resolver and an audit script, and an objects module notes file. +### Reference implementations (last resort) + +This guide and the UTS docs are meant to be complete. If they don't answer a question while you create or maintain a skill and its harness, you MAY consult the existing skills and harnesses, read-only, at the commits this guide reviewed: + +| Repo @ commit | Skill | Harness | Harness smoke tests | Generated tests | +|---|---|---|---|---| +| [ably-java](https://github.com/ably/ably-java) @ `0ff24017` | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | +| [ably-cocoa](https://github.com/ably/ably-cocoa) @ `b5074d9b` | [`.claude/skills/uts-to-swift`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/.claude/skills/uts-to-swift) | The `UTS` test target, [`Test/UTS`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS) (helpers in `Test/UTS/infra`; README `Test/UTS/README.md`), plus the shared test-support targets it depends on: [`Test/AblyTesting`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/AblyTesting) and [`Test/AblyLiveObjectsTesting`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/AblyLiveObjectsTesting) | None: they were retired, a [pattern to avoid](#patterns-to-avoid) | [`Test/UTS/unit`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS/unit), [`Test/UTS/integration`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS/integration) | + +Rules: + +1. **The authority doesn't change.** Consult the references only after this guide, the UTS docs and the helper specs fail to answer the question. On any conflict, this guide (and, for semantics, the UTS docs, per [How it relates to the other UTS docs](#how-it-relates-to-the-other-uts-docs)) wins; record the conflict. +2. **Learn the pattern; never copy.** Don't copy, port or translate code, scripts, prose, tables or examples verbatim. Take how a problem was structured or solved, and write your own version in your language's idioms, on your SDK's hooks and harness. +3. **Check against [Patterns to avoid](#patterns-to-avoid)** before adopting anything. Both skills and both harnesses have documented defects, often in exactly the files you would consult. +4. **Read the pinned commits** linked above, where Patterns to avoid applies exactly. `main` may since have improved; if you consult `main`, re-check what you take against Patterns to avoid, and record which revision you used. +5. **Record each consultation**: the repo, commit and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. +6. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. + +Reading GitHub is network access: the [procedure](uts-to-lang-skill-creator.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. + ### Patterns to avoid -Observed in the existing skills as of 2026-10-07 (the reviewed checkouts, ably-cocoa `b5074d9b` and ably-java `0ff24017`; unchanged on remote `main` as fetched that day); some may since be fixed. Each is a pattern a new skill shouldn't copy. +Observed in the existing skills and their harnesses as of 2026-10-07 (the reviewed checkouts, ably-cocoa `b5074d9b` and ably-java `0ff24017`; unchanged on remote `main` as fetched that day); some may since be fixed. Each is a pattern a new skill shouldn't copy. Check anything taken from the [reference implementations](#reference-implementations-last-resort) against this list. When the "as of" commits in the introduction move, update the links in that table and this list together. **In both skills** -- **Fetching the manual and features specs from `main`** while reading specs from a local clone ([9.4](#94-local-clone-vs-fetching-main)). +- **Fetching the manual from `main`, and telling the model to "fetch" the features spec with no pinned source,** while reading specs from a local clone ([9.4](#94-local-clone-vs-fetching-main)). - **Two sources of tier truth**: path-based in the resolver, content-based in the `SKILL.md` integration section. - **A `CONTAINS_IN_ORDER` mapping that isn't a subsequence check** ([6.7](#67-notes-on-the-catalogue)). - **No re-sync mode, no final report, no lint step, no collision detection**, and a silent `null` when a declared notes file is missing. +**In the harnesses** + +- **Smoke tests env-gated and never run in CI, then retired** (ably-cocoa) once spec-derived tests covered their scenarios ([2.7](#27-harness-smoke-tests-and-self-tests-must)). +- **Scenario-only smoke tests** (ably-java) that stay green while a helper breaks its helper-spec contract. +- **A mock `close()` that notifies synchronously** (ably-java), although `mock_websocket.md` requires `onClose` to be called asynchronously. +- **A declared mock event type that is never emitted** (ably-java), so an assertion that filters the event log for it passes on an empty list. + **Seen in `uts-to-swift`** - **Example code that doesn't compile**: the callback-bridging helper in "Assertions (Swift Testing)" has a `guard` body that doesn't exit and doesn't resume the continuation. (The real helper in `TimeTests.swift` is correct.) From 0df3556225ccf7af7e614d29eb514980223c2334 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 20:33:21 +0530 Subject: [PATCH 05/33] UTS: address review feedback on translator skill docs - count expected-failure forms (FAILS WITH, THROWS, EXPECT THROW, AWAIT_ERROR) as assertions in the audit contract, so surplus waits can't mask a dropped assertion - report two-letter keywords (IS, IF, IN, OR, AS) in the corpus scanner - treat any change to a spec file as changed during re-sync, and diff against the working tree, including untracked specs - stop and ask before translating a locally modified spec, and record its blob hash when the user proceeds - state that a tier whose harness tests aren't wired into CI keeps the skill non-conforming until CI runs them --- .../uts-to-lang-skill-creator.md | 33 ++++++++++--------- .../writing-translator-skills.md | 25 ++++++++------ 2 files changed, 32 insertions(+), 26 deletions(-) diff --git a/uts/docs/translator-skills/uts-to-lang-skill-creator.md b/uts/docs/translator-skills/uts-to-lang-skill-creator.md index 862e57eb7..504ba99d9 100644 --- a/uts/docs/translator-skills/uts-to-lang-skill-creator.md +++ b/uts/docs/translator-skills/uts-to-lang-skill-creator.md @@ -184,7 +184,7 @@ git -C status --porcelain -- uts specifications git -C log -1 --format='%H %cd' -- uts/docs/translator-skills ``` -Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), and the guide's last-change commit. You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. +Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), and the guide's last-change commit. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. @@ -377,7 +377,7 @@ These proposals are presented at STOP-3 and need individual approval at **STOP-4 ### 4.8 Step 2h: derive tier feasibility -Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "not possible until the SDK has an objects API (placeholder notes; the skill refuses the module)"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). A tier is **ready** only when its G-19 and G-20 rows (smoke tests and self-tests, guide 2.7) are present, green and wired into CI (or the CI change was declined at STOP-5 and is reported as unmet, 5.4), or will be built in Phase 3. +Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "not possible until the SDK has an objects API (placeholder notes; the skill refuses the module)"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). A tier is **ready** only when its G-19 and G-20 rows (smoke tests and self-tests, guide 2.7) are present, green and wired into CI (or the CI change was declined at STOP-5 and is reported as unmet, so the skill doesn't yet conform to guide 2.5/2.7; see [5.4](#54-step-3d-stop-7-confirm-the-harness)), or will be built in Phase 3. ### 4.9 Step 2i: STOP-3, approve the design and choose the scope @@ -446,7 +446,7 @@ Write a README next to the harness covering everything guide [3.1](writing-trans Re-run the assessment of [4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test and self-test results per tier (each run by its tier's filter), their CI wiring, the files created or changed, and the updated tier matrix. Only then continue to the skill. -**Done when:** every row the user chose to build is `present`; each supported tier has green smoke tests and self-tests covering guide 2.7, wired into CI (or, if the user declined the CI change at STOP-5, listed in the Final report as an unmet requirement; the tier stays ready, because the skill's preflight still runs the harness tests locally on every run, and the harness README states that CI doesn't yet run them); the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. If the user declines network access at STOP-6, the integration and proxy tiers aren't supported: mark them unready (`blockedBy`: harness tests not run) and list them in the Final report. +**Done when:** every row the user chose to build is `present`; each supported tier has green smoke tests and self-tests covering guide 2.7, wired into CI (or, if the user declined the CI change at STOP-5, listed in the Final report as an unmet requirement: the skill does not conform to guide 2.5/2.7 until CI runs the harness tests, so mark the Appendix E rows "Harness smoke tests per tier and helper self-tests …" and "One CI home per suite …" ✗ with that reason; the tier stays ready, because the skill's preflight still runs the harness tests locally on every run, and the harness README states that CI doesn't yet run them); the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. If the user declines network access at STOP-6, the integration and proxy tiers aren't supported: mark them unready (`blockedBy`: harness tests not run) and list them in the Final report. --- @@ -466,7 +466,7 @@ For each decision below, propose an answer derived from the Repo profile, the bu | D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](writing-translator-skills.md#51-traceability), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | | D-05 | Name collisions with native tests | [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | | D-06 | **Tag syntax** | [5.1](writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | -| D-07 | File header format | [5.1](writing-translator-skills.md#51-traceability), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@`, disclosures; integration/proxy also the corresponding unit spec | +| D-07 | File header format | [5.1](writing-translator-skills.md#51-traceability), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@` (plus `(locally modified; blob )` for a locally modified spec, guide 9.1), disclosures; integration/proxy also the corresponding unit spec | | D-08 | **`deviations.md` location** per owning test module | [7.3](writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | | D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | | D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | @@ -517,13 +517,13 @@ Generate the files in this order: each one depends on the previous ones. `SKILL. Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. -Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: +Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty, dirtyFiles}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: ```json { "ok": true, "sourceModule": "realtime", - "specRepo": { "sha": "", "dirty": false }, + "specRepo": { "sha": "", "dirty": false, "dirtyFiles": [] }, "mapped": true, "translationNotes": "", "harness": { @@ -562,7 +562,7 @@ Implement `scripts/audit_translation.py` to every row and the parsing contract o | Test-side pattern | Source | |---|---| | Tag marker (`UTS:` after the line-comment marker) | D-06 | -| Assertion calls (every assertion form the catalogue maps `ASSERT …` and `FAILS WITH` to) | [7.4](#74-step-5d-fill-the-construct-catalogue) | +| Assertion calls (every assertion form the catalogue maps `ASSERT …`, `FAILS WITH`, `THROWS`, `EXPECT THROW` and `AWAIT_ERROR` to) | [7.4](#74-step-5d-fill-the-construct-catalogue) | | Wait calls (every harness helper the catalogue maps `AWAIT`, `AWAIT_STATE`, `poll_until`, … to) | [7.4](#74-step-5d-fill-the-construct-catalogue); the harness README | | Comment syntax (line and block), so commented-out assertions are ignored | P-01 | @@ -572,7 +572,7 @@ Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifi **Acceptance** (verified in [8.2](#82-test-the-audit-itself)): -- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including the end-at-heading rule (a guide SHOULD that this procedure requires: *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. +- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including the end-at-heading rule (a guide SHOULD that this procedure requires: *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `FAILS WITH` / `THROWS` / `EXPECT THROW` / `AWAIT_ERROR` lines counted as assertions, awaited or not, and `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. - [ ] Where your parsing differs from the contract, `SKILL.md` says so (guide 8.2 "Implement exactly this, or say where yours differs"). - [ ] SHOULD items you implemented are listed in `SKILL.md`; the ones you didn't are listed in the Final report. @@ -609,7 +609,7 @@ for path in root.rglob("*.md"): continue line = re.sub(r'"[^"]*"', '""', line) # ignore string literals line = re.split(r"\s(?:#|//)\s", line, maxsplit=1)[0] # drop trailing comments - keywords.update(re.findall(r"\b[A-Z][A-Z_]{2,}\b", line)) + keywords.update(re.findall(r"\b[A-Z][A-Z_]+\b", line)) calls.update(re.findall(r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\s*\(", line)) for title, counter in (("KEYWORDS", keywords), ("SNAKE_CASE CALLS", calls)): print("==", title) @@ -653,7 +653,7 @@ Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-ou |---|---|---| | Frontmatter | `name: uts-to-`; a pushy `description` with trigger phrases; `argument-hint`; `allowed-tools` from D-22; the model pin from D-26, where the tooling supports one (otherwise the model statement in the opening lines) | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | | Usage guard | Empty argument → print the usage line and stop; placeholder paths only | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | -| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | +| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state (the resolver's `specRepo`); for each selected spec in `dirtyFiles`, stop and ask, and if the user proceeds stamp its blob (guide 9.1) | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | | Skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | | Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | | Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](writing-translator-skills.md#34-recommended-outlines-should), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | @@ -710,6 +710,7 @@ Use one known-good test file (the pilot output from [8.3](#83-pilot-translate-on | Add a tag that isn't in the spec | `orphan` lists it | | Remove one tag | `missing` lists it | | Add surplus waits to a test with a deleted assertion | The assertion shortfall is still reported (counts aren't summed) | +| Delete the native assertion that renders a spec `FAILS WITH` (awaited or not) | A positive assertion shortfall for that test | | Delete the poll call that renders a spec `poll_until` (with no `AWAIT` on the spec line) | An await shortfall for that test (spec polls count as waits) | **Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l`` (as of `12540dcf`: rest 571, realtime 554, objects 339). @@ -764,7 +765,7 @@ Run each path through the skill (in the session from [8.3](#83-pilot-translate-o 3. **Placeholder notes:** a module whose notes file is a placeholder → refused, with the reason. 4. **Unready tier:** a tier the mapping marks unready → not offered at step C, with its `blockedBy`. 5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. -6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification, that the header SHA is updated, and that only the survivors in guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. +6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification (including a change outside the pseudocode, such as `## Protocol Variants`, reported as **changed**, and, run against a scratch clone of the spec repo (`git clone `; the spec clone itself is never edited, 2.2), an uncommitted edit to that spec reported as **changed**), that the header SHA is updated, and that only the survivors in guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. 7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. 8. **Red harness:** on a scratch copy (or a temporary working-tree edit you revert afterwards), break one harness test (for example, make the WebSocket mock refuse every connection), or run an integration tier with the network unavailable → the skill stops at step F before generating anything, reports a harness or environment problem (not an SDK deviation), and names the failing test. Restore the change and confirm the preflight is green again. 9. **Known gap:** select a spec that uses a construct the harness README lists under Known gaps → the preflight reports the conflict and stops to ask (guide 4 step F), rather than generating a substituted rendering. If Known gaps is empty, use a scratch copy of the README with an entry for a construct the selected spec uses. @@ -805,11 +806,11 @@ The harness and the skill must stay in step with five things that change indepen | What changed | How to detect it | What to update | |---|---|---| -| **The guide** | `git -C diff ..HEAD -- uts/docs/translator-skills/` | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](#61-step-4a-draft-the-design-record) and [7.6](#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | -| **The UTS docs or helper specs** | `git -C diff ..HEAD -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | +| **The guide** | `git -C diff -- uts/docs/translator-skills/` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](#61-step-4a-draft-the-design-record) and [7.6](#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | +| **The UTS docs or helper specs** | `git -C diff -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | | **The UTS corpus** | The skill's own re-sync mode (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | | **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | -| **The SDK** (API, hooks, test layout, CI) | `git -C diff ..HEAD` over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | +| **The SDK** (API, hooks, test layout, CI) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | Rules for every maintenance run: @@ -966,7 +967,7 @@ Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier # Design record: uts-to- ## Inputs at generation time -- Spec clone: (clean | dirty); guide: +- Spec clone: (clean | dirty: ); guide: - Repo: - Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: - Harness README: @@ -1024,7 +1025,7 @@ Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier ```markdown # Final report: uts-to- for -- Spec clone (clean | dirty); repo ; date +- Spec clone (clean | dirty: ); repo ; date - Model: (Opus-class: yes | no, and why); sub-agents: , or none - Scope: ; out of scope and why diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index 4a5152521..71bd54407 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -425,7 +425,8 @@ A module notes file should cover: A generated test file: ``` -header: "Derived from uts/ at ably/specification@", tier, disclosures +header: "Derived from uts/ at ably/specification@" (plus "(locally modified; blob )" + when 9.1 applies), tier, disclosures (harness stand-ins, deviations); integration/proxy: the corresponding unit spec imports; suite/class (serialised if hooks are process-global) per test: UTS: tag → test attribute → name with spec point → @@ -777,7 +778,7 @@ Path validation, mapping, spec discovery, naming and faithfulness checking are m |---|---| | Validate the module | Expand a leading `~`. The parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/` | | Errors | One JSON object with `ok: false`, a code and a `message`. Example codes (ably-cocoa): `NOT_A_UTS_MODULE_PATH`, `DIR_NOT_FOUND`, `NO_TIER_DIRS`, `MAPPING_NOT_FOUND`, `BAD_MAPPING`, `BAD_TARGET_NAME` | -| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, harness, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty}`, MAY) gives the spec clone's state for the file headers and the report (9.1, 9.4). `harness` (`{root, readme, tiers: {: {sources, tests}}}`) echoes the mapping's harness entry with paths validated (repo-relative), for the Harness reference and the preflight (3.2, 4 step F). Downstream steps read only these fields | +| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, harness, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty, dirtyFiles}`, MAY) gives the spec clone's state (`dirtyFiles`: the modified or untracked files under `uts/` and `specifications/`, 9.1) for the file headers and the report (9.1, 9.4). `harness` (`{root, readme, tiers: {: {sources, tests}}}`) echoes the mapping's harness entry with paths validated (repo-relative), for the Harness reference and the preflight (3.2, 4 step F). Downstream steps read only these fields | | Tier detection by path | `unit/**`; `integration/**` excluding `integration/proxy/**`; `integration/proxy/**`. Every tier can have sub-directories (e.g. `realtime/integration/channels/`). Use the path only. (Both existing skills also describe content-based proxy detection in their integration sections; drop it.) | | Exclusions | `helpers/` (implement as harness, never translate), `README.md`, `PLAN.md`, `*_SUMMARY.md`. Match them **relative to the tier base**, so an ancestor directory in the checkout path can't trip them. *(New, defensive; nothing matches today:)* also skip any non-spec notes left inside a tier directory | | Deterministic naming | Strip a trailing `_test`, convert to the language's convention, add the suffix or prefix the runner needs (`objects_lifecycle_test.md` → Swift `ObjectsLifecycleTests`, Kotlin `ObjectsLifecycleTest`, pytest `test_objects_lifecycle.py` with class `TestObjectsLifecycle`) | @@ -801,7 +802,7 @@ Path validation, mapping, spec discovery, naming and faithfulness checking are m - **Spec side.** A test starts at each `` **Test ID**: `` `` line (the backticks are part of the marker) and ends at the next one. *(New, SHOULD:)* also end it at the next heading at or above the test's own level, so trailing notes or appendix sections aren't attributed to the last test. - **Headings.** Section headings are any `#`–`####` line **outside** a fence; inside a fence, `#` starts a pseudocode comment. Combined headings such as `### Test Steps and Assertions` are sections too. - **Fences.** Read ` ```pseudo ` fences. You MAY also read untagged fences (four in spec files today, all in file preambles before the first test, plus one each in `objects/PLAN.md` and the `standard_test_pool.md` helper, which aren't translated; so this changes nothing yet; the Swift audit reads `pseudo` only). Skip ` ```json ` and other payload fixtures. -- **Classification.** Skip blank and comment lines. Tag `ASSERT` / `ASSERT_*` as `assert`; `AWAIT` / `AWAIT_STATE` / `AWAIT_ERROR` / `AWAIT_ALL` / `AWAIT UNTIL` / `EXPECT` and the poll forms `poll_until` / `poll_until_success` / `POLL_UNTIL` / `WAIT_FOR` as `await` (whether or not the line starts with `AWAIT`), so spec polls and the test's poll helpers are counted alike; everything else as `step`. Match the longest keyword first, on both sides (`poll_until_success` before `poll_until`; `pollUntil` before `poll`). +- **Classification.** Skip blank and comment lines. Tag `ASSERT` / `ASSERT_*` as `assert`, and also *(new; both existing audits tag these as `step` or `await`)* any line containing, outside a trailing comment, `FAILS WITH`, `THROWS`, `EXPECT THROW` or `AWAIT_ERROR`, whether or not it is awaited: the test side counts their native rendering (an expected-failure assertion) as an assertion call, so the spec side must too, or a dropped failure assertion goes unreported and the surplus can mask another dropped `ASSERT`. These failure forms take precedence over the await keywords. Tag `AWAIT` / `AWAIT_STATE` / `AWAIT_ALL` / `AWAIT UNTIL` and the poll forms `poll_until` / `poll_until_success` / `POLL_UNTIL` / `WAIT_FOR` as `await` (whether or not the line starts with `AWAIT`), so spec polls and the test's poll helpers are counted alike; everything else as `step`. Match the longest keyword first, on both sides (`poll_until_success` before `poll_until`; `pollUntil` before `poll`). - **Test side.** A test's block runs from its `UTS:` tag to the next tag (or end of file). Helpers placed after the last test are attributed to it, and assertions inside shared helpers are invisible to their callers: keep shared helpers in a separate file or before the first test, and annotate call sites that hide spec assertions. Only the test-side regexes change per language (tag marker, assertion calls, wait calls, comment syntax); the spec side is the same for every SDK. @@ -823,7 +824,7 @@ SHOULD, beyond the existing audits: The review is static, so it runs in both modes. - **Coverage:** `missing`, `orphan` and `duplicate` are empty, or each is explained. These, and a positive assertion shortfall, are the hard checks. -- **Line by line:** every positive assertion shortfall is accounted for by an annotated omission. A negative shortfall (more native assertions than spec `ASSERT`s, e.g. from type or number normalisation) is fine. A multi-line spec construct (a mock definition, a `ClientOptions(...)` block) appears as several `step` lines; reconcile them as one group. +- **Line by line:** every positive assertion shortfall is accounted for by an annotated omission. A negative shortfall (more native assertions than spec `assert` lines, e.g. from type or number normalisation) is fine. A multi-line spec construct (a mock definition, a `ClientOptions(...)` block) appears as several `step` lines; reconcile them as one group. - **Await shortfall is a soft signal.** The wait counter sees only the harness's wait helpers. Continuation-bridged helpers, native `await` on an async SDK API, and fixture builders legitimately replace them, so a positive await shortfall is expected on natively-async modules. Account for each spec `AWAIT` rather than treating the count as a gate. - **Setup fidelity:** client options, mock responses, timer use and channel-operation order match the spec. - **Wait fidelity:** every wait condition and timeout matches, or carries a `NOTE`/`DEVIATION`. @@ -846,6 +847,8 @@ Stamp each generated file's header with the spec-repo commit SHA it was derived A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. +**A SHA identifies only committed content.** If a translated spec file differs from the recorded SHA (`git status --porcelain -- ` is non-empty: staged, unstaged or untracked), stop and ask. If the user proceeds, add `(locally modified; blob >)` to that file's header, list the file in the final report, and have re-sync compare the file with that blob rather than with the SHA. Local changes elsewhere in the clone (for example untracked notes outside the tier directories) only need recording, as 9.4 says. + **Record the model.** The run's final report, which is the run record, records the model (name and ID) that generated or regenerated each file ([section 11](#11-final-report-format); part of that MUST format). A header line such as `// Generated with ` MAY be added as well. It changes whenever a different model regenerates the file, which adds diff noise, so prefer the run record. ### 9.2 A re-sync mode (SHOULD) @@ -853,10 +856,10 @@ A per-module manifest (spec file → SHA → test file) MAY be kept as well. Lin Neither existing skill has one; both have stale suites as a result. Add a mode (for example `/uts-to- --resync`) that: 1. Runs the audit over **every** mapped spec in the module, not only the ones selected. -2. Runs `git diff ..HEAD -- uts/` to list changed spec files, and re-scans the corpus for constructs the skill can't map (section 6). -3. Classifies each spec as **new**, **changed** (ID set changed, or pseudocode changed), **unchanged** or **removed**. +2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as **new**; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. Then re-scan the corpus for constructs the skill can't map (section 6). +3. Classifies each spec as **new**, **changed** (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), **unchanged** or **removed**. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](uts-to-lang-skill-creator.md#10-maintenance)). 4. Regenerates the affected tests, **preserving only** DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. -5. Updates the SHA in each regenerated file's header (and in the manifest, if kept). +5. Updates the SHA in each regenerated file's header, and its blob note (9.1): re-stamped if the spec is still locally modified, removed if not (and in the manifest, if kept). 6. Reports the classification ([section 11](#11-final-report-format)). Regenerate the whole file when most tests changed; otherwise regenerate only the affected tests. Say which in the report. A re-sync is also needed when the shared harness or module helpers change shape (ably-cocoa reset and regenerated its objects ports after aligning them with `standard_test_pool.md`). After any harness change, re-run every affected tier's smoke tests and self-tests ([2.7](#27-harness-smoke-tests-and-self-tests-must)) first, then regenerate the tests whose rendering depends on the changed helper. @@ -877,7 +880,7 @@ Pick **one** source for docs and specs, and pin it. This guide recommends **read - the UTS docs (`/../docs/writing-derived-tests.md`, `proxy.md`); - in evaluate mode, the features specs the decision tree needs (`/../../specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. -Record the clone's HEAD SHA and whether it is dirty. You MAY warn when the clone is behind `origin/main`. +Record the clone's HEAD SHA, whether it is dirty and which files are; a translated spec that is locally modified is handled as in 9.1. You MAY warn when the clone is behind `origin/main`. - *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. - *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md` from GitHub `main`, and tell the model to "fetch" the features spec with no pinned source, while reading specs from the local clone, so the two can skew. @@ -905,7 +908,7 @@ The skill's scripts MUST pin UTF-8 and LF when they write tracked files. Neither existing skill defines one; define it. The skill MUST end each run with a report like this: ``` -UTS translation: / @ ably/specification@ (clone clean|dirty) (mode: translate-only | evaluate | resync) +UTS translation: / @ ably/specification@ (clone clean|dirty: ) (mode: translate-only | evaluate | resync) Model: ; sub-agents: | Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | @@ -913,6 +916,8 @@ Model: ; sub-agents: | connection_recovery_test.md | ConnectionRecoveryTests. | 6/6 | clean | 1/1 | ok | 5 pass, 1 gated | Failing Tests: RTN16f | Skipped specs (and why): … +Locally modified specs translated (9.1): — blob +Re-sync classification (9.2): new …; changed () …; unchanged …; removed …; changed but not regenerated: — Not-applicable omissions (7.4): — Missing APIs found in translate-only mode: — Mock-capability gaps found in translate-only mode: — @@ -950,7 +955,7 @@ From the git histories of both existing skills. Most lessons are already rules i ## 13. Checklist for a new `uts-to-` skill -The skill and its harness are one deliverable: the skill isn't done until every MUST item below, harness items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. +The skill and its harness are one deliverable: the skill isn't done until every MUST item below, harness items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. A MUST item that can't be met yet (for example, the repo owner declines the CI change that would run the harness tests) is also marked ✗ with the reason, and the skill doesn't conform to this guide until it is met. **Harness** From a7fc036c565874a4eb3a035e4c1d63c4567d9dea Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Thu, 8 Oct 2026 22:33:24 +0530 Subject: [PATCH 06/33] UTS: package the translator skill procedure as an agent skill Replaces the single uts-to-lang-skill-creator.md procedure doc with an agent skill at uts/skills/uts-to-lang-skill-creator/ that loads in both Claude Code and Codex: a short SKILL.md (portable frontmatter, ground rules and stop points first), per-phase references, record templates, and runnable scripts (corpus scanner, spec-clone locator, repo survey). Also: - the guide recommends a skill format that works in both tools: portable frontmatter, script paths relative to the skill directory, and install paths for Claude Code and Codex - correct the guide's description of __PASSTHROUGH__, which uts-proxy passes through unchanged - link the translator skill guide from writing-derived-tests.md - update the uts/README.md tree and spec counts (adding objects, the REST proxy tier and skills), and document installing the skill --- uts/README.md | 44 +- .../uts-to-lang-skill-creator.md | 1205 ----------------- .../writing-translator-skills.md | 72 +- uts/docs/writing-derived-tests.md | 2 + uts/skills/uts-to-lang-skill-creator/SKILL.md | 253 ++++ .../assets/templates/acceptance-checklist.md | 60 + .../assets/templates/design-record.md | 54 + .../assets/templates/final-report.md | 51 + .../assets/templates/repo-profile.md | 63 + .../assets/templates/uts-infra-design.md | 66 + .../references/failure-modes.md | 34 + .../references/glossary.md | 35 + .../references/maintenance.md | 21 + .../references/phase-1-understand-repo.md | 77 ++ .../references/phase-2-design-harness.md | 113 ++ .../references/phase-3-build-harness.md | 56 + .../references/phase-4-design-record.md | 42 + .../references/phase-5-generate-skill.md | 174 +++ .../references/phase-6-validate-skill.md | 121 ++ .../scripts/scan_constructs.py | 39 + .../scripts/spec_clone_info.py | 142 ++ .../scripts/survey_repo.py | 86 ++ 22 files changed, 1572 insertions(+), 1238 deletions(-) delete mode 100644 uts/docs/translator-skills/uts-to-lang-skill-creator.md create mode 100644 uts/skills/uts-to-lang-skill-creator/SKILL.md create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/failure-modes.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/glossary.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/maintenance.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/scan_constructs.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/survey_repo.py diff --git a/uts/README.md b/uts/README.md index a24ba5f6e..252312331 100644 --- a/uts/README.md +++ b/uts/README.md @@ -17,6 +17,7 @@ uts/ │ │ ├── push/ # RSH — push admin │ │ └── types/ # T* — type definitions │ └── integration/ # REST integration tests (Ably sandbox) +│ └── proxy/ # Proxy-based fault injection tests ├── realtime/ │ ├── unit/ # Realtime unit tests (mocked WebSocket) │ │ ├── helpers/ @@ -29,16 +30,24 @@ uts/ │ │ └── presence/ # RTP — realtime presence │ └── integration/ # Realtime integration tests │ ├── proxy/ # Proxy-based fault injection tests +│ ├── auth/, channels/, connection/, presence/ # Direct sandbox tests, by area │ └── *.md # Direct sandbox tests +├── objects/ +│ ├── helpers/ +│ │ └── standard_test_pool.md # Standard LiveObjects test pool and fixtures +│ ├── unit/ # LiveObjects unit tests (mocked WebSocket) +│ └── integration/ # LiveObjects integration tests +│ └── proxy/ # Proxy-based fault injection tests ├── docs/ # Guides and reference │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests │ ├── translator-skills/ # Building uts-to- translator skills -│ │ ├── writing-translator-skills.md # What a translator skill and its harness must contain -│ │ └── uts-to-lang-skill-creator.md # Agent procedure to build one +│ │ └── writing-translator-skills.md # What a translator skill and its harness must contain │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix +├── skills/ # Agent skills (Claude Code, Codex) +│ └── uts-to-lang-skill-creator/ # Builds a uts-to- skill and its harness (SKILL.md, references/, assets/, scripts/) └── README.md # This file ``` @@ -46,13 +55,19 @@ uts/ | Category | Count | Description | |----------|-------|-------------| -| REST unit | 40 | Mocked HTTP client tests | -| REST integration | 11 | Ably sandbox tests | +| REST unit | 41 | Mocked HTTP client tests | +| REST integration (direct) | 11 | Ably sandbox tests | +| REST integration (proxy) | 1 | Fault injection via Go proxy | | Realtime unit | 54 | Mocked WebSocket tests | | Realtime integration (direct) | 13 | Direct sandbox tests | | Realtime integration (proxy) | 7 | Fault injection via Go proxy | -| Helper specs | 4 | Mock infrastructure definitions | -| **Total** | **129** | | +| Objects unit | 15 | LiveObjects, mocked WebSocket | +| Objects integration (direct) | 3 | LiveObjects sandbox tests | +| Objects integration (proxy) | 1 | LiveObjects fault injection | +| Helper specs | 4 | Mock and fixture infrastructure definitions | +| **Total** | **150** | | + +Counts as of 2026-10-08; `find uts/ -name '*.md'` is authoritative (it also lists `objects/PLAN.md`, which is not a spec). ## Three Test Tiers @@ -141,11 +156,24 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations -- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` Claude Code skill and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync -- **[UTS-to-Lang Skill Creator](docs/translator-skills/uts-to-lang-skill-creator.md)** — Step-by-step procedure for an LLM agent to create a `uts-to-` skill for an SDK repo: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations +- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync +- **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that, given an SDK repo, follows a step-by-step procedure to create its `uts-to-` skill: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations. See [Installing the skill creator](#installing-the-skill-creator) - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs +### Installing the skill creator + +Install it once, at user level, by symlinking (if an earlier copy is there, remove it first: `ln -sfn` would create the link inside it) the skill directory from your local clone of this repo into both tools' skill directories. Both Claude Code and Codex follow directory symlinks, and a `git pull` of the clone updates both: + +```sh +SPEC=~/src/specification # your local clone of ably/specification +mkdir -p ~/.claude/skills ~/.agents/skills +ln -sfn "$SPEC/uts/skills/uts-to-lang-skill-creator" ~/.claude/skills/uts-to-lang-skill-creator # Claude Code +ln -sfn "$SPEC/uts/skills/uts-to-lang-skill-creator" ~/.agents/skills/uts-to-lang-skill-creator # Codex +``` + +Then, in the SDK repo, run `/uts-to-lang-skill-creator ` (Claude Code) or `$uts-to-lang-skill-creator ` (Codex), or ask for a `uts-to-` skill in your own words. Run it on the most capable model tier available (Opus-class). If symlinks aren't available (for example on Windows without Developer Mode), copy the directory instead and re-copy after each pull; the skill detects a stale copy and asks you to refresh it. Without installing, any agent can be told: "Read `/uts/skills/uts-to-lang-skill-creator/SKILL.md` and follow it." + ## Go Test Proxy The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection. diff --git a/uts/docs/translator-skills/uts-to-lang-skill-creator.md b/uts/docs/translator-skills/uts-to-lang-skill-creator.md deleted file mode 100644 index 504ba99d9..000000000 --- a/uts/docs/translator-skills/uts-to-lang-skill-creator.md +++ /dev/null @@ -1,1205 +0,0 @@ -# UTS-to-Lang Skill Creator - -This is a step-by-step procedure for an LLM agent (for example Claude Code) to create a `uts-to-` translator skill for one Ably SDK repository. [Writing UTS Translator Skills](writing-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why. This document says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this document links to it, and you must read the linked section. - -The order is fixed, because each step stands on the one before it: - -1. **Understand the repo:** how the SDK is built, tested and mocked today (Phase 1). -2. **Design the UTS test infrastructure (the harness)** for this language from what you found: what to reuse, what to wrap, what to build (Phase 2). -3. **Build and verify the harness** (Phase 3). -4. **Create the `uts-to-` skill on top of it** (Phases 4 and 5). -5. **Validate the skill with pilot translations** (Phase 6), then report (Phase 7). - -The harness and the skill are **one deliverable** (guide [section 2](writing-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). - -A harness designed without studying the repo's existing tests duplicates or fights them. A human may skim this document to see what the agent will do and where it will ask for decisions. - -## Contents - -1. [Purpose and how to use this procedure](#1-purpose-and-how-to-use-this-procedure) -2. [Ground rules](#2-ground-rules) -3. [Phase 1: understand the repo](#3-phase-1-understand-the-repo) -4. [Phase 2: design the harness](#4-phase-2-design-the-harness) -5. [Phase 3: build and verify the harness](#5-phase-3-build-and-verify-the-harness) -6. [Phase 4: skill design decisions](#6-phase-4-skill-design-decisions) -7. [Phase 5: generate the skill files](#7-phase-5-generate-the-skill-files) -8. [Phase 6: validate the skill](#8-phase-6-validate-the-skill) -9. [Phase 7: final report and handover](#9-phase-7-final-report-and-handover) -10. [Maintenance](#10-maintenance) -- [Appendix A: Repo profile template](#appendix-a-repo-profile-template) -- [Appendix B: Gap table and harness design template](#appendix-b-gap-table-and-harness-design-template) -- [Appendix C: Design record template](#appendix-c-design-record-template) -- [Appendix D: Final report template](#appendix-d-final-report-template) -- [Appendix E: Acceptance checklist (guide section 13)](#appendix-e-acceptance-checklist-guide-section-13) -- [Appendix F: Glossary](#appendix-f-glossary) -- [Appendix G: Common failure modes and remedies](#appendix-g-common-failure-modes-and-remedies) - ---- - -## 1. Purpose and how to use this procedure - -### 1.1 Who reads this - -- **The executor** is an LLM agent working inside the target SDK repository, with shell access, file read/write access, and the ability to ask the user questions, running on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). Every instruction below is addressed to it ("you"). -- **The user** is an SDK engineer who owns the target repository. They answer the questions at the stop points and approve every change outside the skill directory. -- **The lead reviewer** may skim the working records (Repo profile, Harness design, Design record, Final report) instead of the whole conversation. - -### 1.2 Invocation - -The user starts a run with one sentence, for example: - -``` -Follow /uts/docs/translator-skills/uts-to-lang-skill-creator.md to create a uts-to- skill for this repo. -``` - -`` is the user's local clone of `ably/specification`. If the user also states goals ("objects unit tier first", "rest and realtime only"), record them as inputs. - -### 1.3 Required inputs - -| Input | Example | If missing | -|---|---|---| -| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo; confirm it has SDK sources and tests | -| Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | -| Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | **Stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | -| User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | -| Model tier | The session's model, Opus-class ([2.8](#28-model-tier-and-sub-agents)) | Take it from your environment if it states your model; otherwise ask. If the model isn't Opus-class: **STOP-1** | -| Local clones of ably-cocoa and ably-java (optional) | Clones that contain commits `b5074d9b` and `0ff24017` | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | - -### 1.4 Outputs - -1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). -2. **The skill**, at `.claude/skills/uts-to-/` unless the Design record says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](writing-translator-skills.md#31-layout)). -3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. -4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): - - `repo-profile.md` ([Appendix A](#appendix-a-repo-profile-template)); - - `uts-infra-design.md`: the gap table, the harness design and the build log ([Appendix B](#appendix-b-gap-table-and-harness-design-template)); - - `design-record.md` ([Appendix C](#appendix-c-design-record-template)), which also holds the decision log; - - `final-report.md` ([Appendix D](#appendix-d-final-report-template)), with the acceptance checklist ([Appendix E](#appendix-e-acceptance-checklist-guide-section-13)). - -Write them at the default location from Phase 1 on; if D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. - -The working records are what a later agent reads to maintain or regenerate the skill ([section 10](#10-maintenance)), so keep them accurate as you go, not only at the end. - -### 1.5 Intent - -The guide's intent ([introduction](writing-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the harness and regenerate. Never hand-fix a pilot test and call the skill done. - -### 1.6 Which document wins - -| Question | Authority | -|---|---| -| What a faithful translation is; how a failure is diagnosed; the `deviations.md` format | [`writing-derived-tests.md`](../writing-derived-tests.md) | -| What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../writing-test-specs.md) | -| Integration and proxy tiers | [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | -| Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | -| What the harness and the skill must contain, and why | the guide ([How it relates to the other UTS docs](writing-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | -| How and in what order you build them | this document | -| The SDK's actual API | the SDK source (guide [3.1](writing-translator-skills.md#31-layout), "The notes are a map, not an authority") | -| Questions none of the above answer, as a last resort only | The reference implementations, under [2.1](#21-your-inputs): never an authority | - -If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. A disagreement between a reference implementation and the guide or the UTS docs is never a STOP-11: follow the guide, and record the disagreement with the read ([2.1](#21-your-inputs)). - -**Procedure recommendations.** Where this document adds detail the guide doesn't have (for example: the working-records files and their location, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this document's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). - ---- - -## 2. Ground rules - -### 2.1 Your inputs - -You work from four primary sources: - -1. this document; -2. the guide; -3. the local spec clone: the UTS docs (`uts/README.md`, `uts/docs/`), the helper specs, the UTS corpus (`uts/rest`, `uts/realtime`, `uts/objects`) and, for API and spec-error questions, the features specs under `specifications/`; -4. the target SDK repo. - -and, only as a last resort, the reference implementations below. - -**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](writing-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: - -1. You have searched this document, the guide, the UTS docs and the helper specs, and they don't answer the question. -2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) with `git -C show :` and `git -C ls-tree -r --name-only -- `, never checking out, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. If a clone lacks the pinned commit, fetching it is network access too. -3. You take a **pattern**, not text: how something was structured or solved. Never copy, port or translate code, scripts, prose, tables or examples verbatim. Write every hook, helper, script, rule and example yourself, in this language's idioms, from the guide, the UTS docs and the target repo. -4. You check what you took against the guide's [Patterns to avoid](writing-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. -5. You record each consultation in the Design record ("Reference-implementation reads": repo, commit, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. - -A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](#83-pilot-translate-one-spec-per-available-tier)). - -### 2.2 What you may change - -| Action | Allowed? | -|---|---| -| Create and edit files under the skill directory and the working-records directory | Yes | -| Create new test-only harness code (shared test library, module helpers, smoke tests, UTS test-project wiring) | Only after the user approves the harness design at [STOP-3](#23-stop-points), and only within the scope they approved | -| Change existing test-support code that the SDK's own tests use (shared fakes, fixtures, base classes) | Only with explicit approval for each change, and never in a way that changes behaviour for the existing tests. Prefer wrapping or extending to editing | -| Change SDK production code, including adding test hooks or raising visibility | **Only with explicit approval for each change** ([STOP-4](#23-stop-points)). Present the proposal first ([4.6](#46-step-2f-propose-the-sdk-test-hooks)) | -| Change CI workflows, build scripts that CI runs, or dependency manifests (including adding a test dependency) | **Only with explicit approval** ([STOP-5](#23-stop-points)). Otherwise write the change as a suggestion in the Final report | -| Edit project or solution files needed to register new test files or projects (`.sln`, `.csproj`, `.projitems`, an Xcode project, etc.) | Only with explicit approval ([STOP-5](#23-stop-points)), recorded in the Design record (D-20). If the user declines, give them the exact entries to add, wait until they confirm, and list them in the Final report | -| Stage, commit, push, branch, or open a PR | **No, unless the user asks** | -| Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | -| Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | -| Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | -| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies, reference-implementation reads under [2.1](#21-your-inputs)) | No | - -### 2.3 Stop points - -At a stop point, present what you found and the options, then wait for the user's answer. Record the question and the answer in the decision log ([Appendix C](#appendix-c-design-record-template)). - -| ID | Phase | When | What you present | -|---|---|---|---| -| **STOP-1** | Start | A required input is missing, or the session's model isn't Opus-class ([1.3](#13-required-inputs), [2.8](#28-model-tier-and-sub-agents)) | The missing input (or the model found) and why it's needed | -| **STOP-2** | 1 (end) | The Repo profile is written | The profile, for confirmation or correction | -| **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | -| **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](#46-step-2f-propose-the-sdk-test-hooks)) | -| **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | -| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases, reading a reference implementation on GitHub or fetching its commit, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | -| **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | -| **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | -| **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | -| **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](#52-step-3b-build-in-order-with-acceptance-checks), [8.4](#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | -| **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | -| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, commit, path) and what they showed; the reading you propose | - -### 2.4 Never guess - -- **Spec content:** quote the spec, a helper spec or a UTS doc from the clone, with its path and heading. Don't rely on memory of the Ably specification. -- **API signatures:** read the SDK source and the harness source before writing any call. A signature you haven't read is a guess. -- **Commands:** run every build, test, lint and script command you record, and record its exit status. A command you haven't run is marked "unverified" in the working records. -- **Facts about the repo:** every Repo profile entry has evidence (a file path, a symbol or a command and its output). "Unknown" is allowed only with the reason and a question for the user. -- **Undocumented corpus constructs:** the guide marks some constructs *(undocumented)*; their meaning is inferred from usage. Read each use in the corpus before choosing a rendering, and say in the catalogue that the meaning is inferred. -- **Reference-implementation reads:** what another SDK's skill or harness does is evidence of one SDK's choice, not of the spec or of this SDK. Confirm it against the guide or the UTS docs before relying on it. Where they are silent, record the choice as yours in the decision log, citing the read. - -### 2.5 Record every decision - -Every choice you make that isn't forced by the guide or the UTS docs goes into the decision log in the Design record, with the options, the choice, who made it (you or the user) and the reason. That includes harness choices in Phases 2 and 3 (reuse vs build, hook shapes, placement), not only the skill decisions in Phase 4. If a later phase changes an earlier decision, add a new log entry that supersedes the old one; don't edit history. - -### 2.6 Pin the spec clone - -At the start of the run: - -```sh -git -C rev-parse HEAD # full SHA -git -C status --porcelain -- uts specifications -git -C log -1 --format='%H %cd' -- uts/docs/translator-skills -``` - -Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), and the guide's last-change commit. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. - -Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. - -### 2.7 Working style - -- Do Phase 1 before anything else, and don't write harness code before the user approves the harness design (STOP-3), or skill files before the harness is verified (STOP-7). -- Escalate verification in the repo's own order (compile, then build tests, then a filtered run, then wider runs), as recorded in the Repo profile. -- Keep the working records up to date as you go; they are also your memory if the session is interrupted. On resuming, read them first. -- Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. - -### 2.8 Model tier and sub-agents - -- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model): MUST.** The guide's [Model tier](writing-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. -- **Sub-agents** you spawn for review, validation or the pilot ([8.3](#83-pilot-translate-one-spec-per-available-tier)) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, running the [8.2](#82-test-the-audit-itself) corpus sweep and collecting its JSON), if at all, and record which steps used one. -- **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. -- **Pin** the generated skill's model where the tooling allows (D-26). - ---- - -## 3. Phase 1: understand the repo - -**Goal:** a written, user-confirmed Repo profile that explains how the SDK is built, tested and mocked today, in enough depth to design the harness from it. This phase is mandatory and comes before anything else: the harness you design in Phase 2 should extend what the repo already has, follow its test style, and plug into its build and CI, and the skill's file locations, idioms and commands all derive from this profile. - -### 3.1 Step 1a: read the repo's own instructions - -Read these first, if they exist, and follow them for the rest of the run: `CLAUDE.md` (root and nested), `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `.editorconfig`, any `docs/` or `Docs/` pages about testing, and any existing `.claude/` directory (settings, skills, commands). Record any instruction that constrains you (for example "don't edit project file X", "use helper Y instead of Z"). - -### 3.2 Step 1b: answer the discovery checklist - -Answer every item. For each, record the answer and its evidence in the Repo profile. The "how to find it" column lists starting points; follow the evidence wherever it leads. - -**Build, test and CI** - -| # | Find | How to find it | -|---|---|---| -| P-01 | **Language(s) and version(s)**: SDK language, test language (they can differ), toolchain version, language mode or edition | Toolchain pins (`global.json`, `.python-version`, `.nvmrc`, `go.mod`, `rust-toolchain*`, `Package.swift` tools version, Gradle `jvmToolchain`); `git ls-files` extension counts | -| P-02 | **Build system and commands**: build the SDK; build the tests; run all tests; run one class; run one test; run by filter; target-qualified filters | Build files (`*.sln`/`*.csproj`, `pyproject.toml`/`tox.ini`/`noxfile.py`, `package.json` scripts, `build.gradle*`, `Package.swift`, `Makefile`, wrapper scripts); CI workflow steps; CONTRIBUTING. **Run** the build-tests and single-class commands and record the exit status. For interpreted languages, "build the tests" means collection (e.g. `pytest --collect-only`) plus the repo's type checker. Record the command that lists collected tests (e.g. `pytest --collect-only -q`, `dotnet test --list-tests`) | -| P-03 | **Test framework(s)**: runner, assertion library, parameterisation, suite-level fixtures (before/after all), teardown hooks, **static and runtime skip idioms**, unconditional-fail idiom, **collection rules** (what names and files the runner picks up) | Test dependencies in the build files; existing tests; the runner's config (`pytest.ini`, `conftest.py`, `xunit.runner.json`, `jest.config.*`). Check whether a runtime skip needs an extra package (guide [7.1](writing-translator-skills.md#71-three-acceptable-end-states)) | -| P-04 | **Test layout and naming**: test projects/targets/source sets, their directories, naming conventions for files, classes and methods, shared test-support projects, and which test targets can see which code | Directory listing of test roots; build files for test targets | -| P-05 | **CI**: every workflow that builds or tests; which job runs which suites; OS and runtime matrix; network access; submodule checkout; secrets; how a new suite or test target would be picked up | `.github/workflows/*`, other CI configs; follow each test step to the command it runs | -| P-06 | **Lint, format and strictness gates**: EditorConfig, formatters, linters, static analysis, type checkers; whether CI treats warnings as errors; whether the local build is laxer than CI | Lint configs, `Makefile` lint targets, CI lint jobs, compiler flags in build files | -| P-07 | **Developer and CI platforms**: which OSes developers and CI use; whether the script runtime the skill will use (e.g. `python3`) is available on all of them; where a proxy binary would have to run | CONTRIBUTING, CI matrix, existing scripts | -| P-08 | **Repo conventions**: licence or file headers, comment style, line endings, commit message rules, files kept in sync (e.g. several dependency manifests), project files not discovered by the build | CLAUDE.md, CONTRIBUTING, `.editorconfig`, `.gitattributes`, recent commits | - -**How the SDK is structured for testing** - -| # | Find | How to find it | -|---|---|---| -| P-09 | **Async model**: callbacks, futures/promises, `async`/`await`, coroutines; whether SDK APIs are callback-based, async, or both; the test runner's async support and its mode; whether the runner **virtualises time** | SDK public API signatures; runner plugins (e.g. pytest-asyncio mode); existing async tests (guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)) | -| P-10 | **Concurrency and threading**: which thread, queue or loop SDK callbacks run on; whether listeners can run concurrently with the test body; how to drain the SDK's internal work queue (this becomes `process_pending_events()`); whether the language checks concurrency safety at compile time | SDK source: dispatch, executor or event-loop code; how existing tests wait for callbacks | -| P-11 | **Time model**: **every time source** the SDK uses: timers, timed blocking waits, scheduled or delayed callbacks, wall-clock and monotonic reads | Grep the SDK source for the language's sleep, timer, wait, delay and now APIs; list each call site's owner (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)) | -| P-12 | **Runner parallelism**: are tests run in parallel by default, and how is it turned off per suite? | Runner docs and config; existing serialisation attributes or collection settings | -| P-13 | **Injection points (test hooks)** for each of: WebSocket transport, HTTP client, clock/timers, reachability/network monitor, randomness (retry jitter, fallback-host shuffle). For each: symbol, how it is set (option, constructor argument, global, subclass), whether it is set at client construction, per-client or process-global, test-only or public | SDK client-options and any debug/test-options types; constructors of the transport, HTTP, timer and network layers; existing tests that replace them | -| P-14 | **Internal-access mechanism**: how tests reach non-public code, and where the repo already uses it | e.g. `InternalsVisibleTo`, `@testable import`, private headers and module maps, same-package placement, `_`-prefixed conventions; grep for existing test-only accessors | -| P-15 | **SDK API surface vs the pseudocode, per module**: client constructors and options, async return types, the error type and how `ErrorInfo` fields (`code`, `statusCode`, `message`, cause) are read, enum renderings, protocol-format option (JSON vs msgpack), `endpoint` vs explicit hosts, the plugin mechanism. For `objects` (LiveObjects): is it implemented, where, as a plugin or built in, typed or dynamic, and which spec IDL it follows | Public API; compare with the pseudocode in two or three specs per module and with `specifications/features.md` / `objects-features.md` in the clone. Produce a short divergence list per module | - -**Existing native test support** - -| # | Find | How to find it | -|---|---|---| -| P-16 | **Existing mocks, fakes and test doubles**: for HTTP, WebSocket/transport, clock/timers, network monitor, randomness, delta decoding, plugins. For each: file and symbol, what it can do (capture requests or frames, inject responses or server messages, simulate refusal, timeout, DNS failure, disconnect), its style (handler/callback, queued responses, await), whether it is process-global, and which native tests use it | Search the test roots for `Mock`, `Fake`, `Stub`, `Test*Transport`, `*Handler`, clock or timer types; follow the hook symbols from P-13 to their test-side implementations | -| P-17 | **Existing test helpers**: state waits, polling helpers, async-to-sync bridges, event capture or recorder types, log capture, custom assertions, client factories, test-options builders | Shared test-support code; base test classes; helpers used by many test files | -| P-18 | **Existing sandbox and integration support**: how native integration tests provision a sandbox app (from `test-app-setup.json`?), point clients at the sandbox, retry, clean up; any proxy or fault-injection tooling | Integration test directories and their fixtures; environment variables they read | -| P-19 | **`ably-common`**: submodule path and name (it may not be called `ably-common`), whether it is initialised, and whether `test-resources/test-app-setup.json`, `msgpack_test_fixtures.json` and `encoding.json` are present | `.gitmodules`; `git submodule status`; list the files | -| P-20 | **Existing UTS assets and their state**: any UTS harness, UTS-derived tests (`UTS:` tags), `deviations.md` files, previous `uts-to-*` skills in this repo. For each asset: does it compile, do its tests pass, which spec SHA do headers name, which helper-spec symbols exist | `grep -rn "UTS:" `; search for `MockWebSocket`, `MockHttpClient`, `SandboxApp`, `uts-proxy`, `deviations.md`, `.claude/skills/uts-*` | -| P-21 | **Sandbox access**: whether tests can reach `sandbox.realtime.ably-nonprod.net` from developer machines and CI | Existing integration tests; CI network settings (guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)). Probing the sandbox, or running existing integration tests for P-20, is network access: **STOP-6** first | - -Useful generic commands (adapt to the platform): - -```sh -git -C rev-parse HEAD; git -C status --porcelain -git -C ls-files | sed 's/.*\.//' | sort | uniq -c | sort -rn | head -20 # extension counts -git -C ls-files | grep -iE '(^|/)(test|tests|spec)s?/' | head -50 # test roots -git -C submodule status -grep -rnE "class (Mock|Fake|Stub)|(Mock|Fake)[A-Z][A-Za-z]*(Transport|Http|Clock|Timer|Socket)" | head -40 -grep -rn "UTS:" | head # previous UTS ports -``` - -### 3.3 Step 1c: study how the existing tests drive the SDK - -The checklist tells you what exists; this step tells you how it is used. Read, in full, at least one existing native test of each kind below (if the repo has it), and write down the pattern each follows: how the client is created, how the double is installed, how the test waits, how it asserts, how it cleans up. - -| Kind | What to learn from it | -|---|---| -| A REST test that checks an outgoing request | How HTTP is intercepted; how requests are captured and responses injected | -| A realtime test that drives connection state | How the transport is replaced; how server messages are injected; how state changes are awaited | -| A test of timer-driven behaviour (retry, timeout, heartbeat) | Whether time is faked or real, and how it is advanced | -| A test that checks something did **not** happen | How the test flushes pending work before a negative assertion | -| A test that reaches internal state | The internal-access mechanism in practice | -| An integration test against the sandbox | Provisioning, client options, timeouts, cleanup | -| A test from each plugin or optional module (e.g. LiveObjects) | How the plugin is registered in tests; its own test helpers | - -Then answer, in the Repo profile: - -- Which existing doubles behave like the helper specs need (see the semantics in [4.3](#43-step-2c-map-each-helper-spec-to-native-code)), and which differ? -- Which existing waits are race-free (subscribe first, then check), and which sample state or sleep? -- Where do the existing tests flake or carry workarounds (retries, sleeps, `@Ignore`/skip with a reason, comments about races)? These mark concurrency or timing problems the harness must avoid. - -### 3.4 Step 1d: write and confirm the Repo profile - -Write `repo-profile.md` from [Appendix A](#appendix-a-repo-profile-template). Record the repo's HEAD SHA and whether it is dirty. Then **STOP-2**: show the user the profile, highlight every "unknown" and every inference, and ask them to confirm or correct it. Apply their corrections before continuing. - -**Done when:** every P-row has an answer with evidence or an explicit "unknown" with a question; the build-tests and single-class commands have been run (or marked unverified with the reason); the existing-test patterns of 3.3 are written down; the user has confirmed the profile. - ---- - -## 4. Phase 2: design the harness - -**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](writing-translator-skills.md#2-the-harness-uts-test-infrastructure) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. - -Design principles: - -- **Follow guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)**: implement each helper spec once, and build the harness to look like the pseudocode. When an existing double has the right behaviour but a different shape, **wrap it in a pseudocode-shaped facade** rather than teaching the skill to translate into the old shape. -- **Reuse what is correct; don't reuse what is subtly wrong.** An existing double that misses a helper-spec guarantee (for example a synchronous `close()`) will make the UTS ports flaky. Extend or replace it for the UTS ports, leaving the native tests' behaviour unchanged. -- **Hooks** are injected at client construction, per-client where the SDK allows (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)). - -### 4.1 Step 2a: list the required capabilities - -Start the gap table from rows G-01 to G-20 of [Appendix B](#appendix-b-gap-table-and-harness-design-template), which cover every requirement of guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) to [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. - -**Check:** every bullet and table row of guide section 2 maps to a gap-table row. - -### 4.2 Step 2b: assess each capability: reuse, wrap, extend or build - -This is the guide's "Reuse before you build" ([2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)) applied to every row. For each row, record the status (`present`, `partial`, `missing`, `n/a`), the evidence (file and symbol, or what you searched for), and the **approach**: - -| Approach | When | -|---|---| -| **Reuse** | An existing component already meets the requirement's semantics and shape (verified, not assumed) | -| **Wrap** | The semantics are right but the shape differs from the pseudocode; add a facade with the spec's names | -| **Extend** | The component is close but lacks capabilities (e.g. no WS-level timeout, no raw frames); add them without changing existing behaviour | -| **Build** | Nothing suitable exists, or what exists is semantically wrong for UTS | -| **n/a** | The SDK lacks the feature (e.g. `MockVCDiff*` when the SDK has no delta support); confirm against the features spec in the clone and give the reason | - -Also record which tiers and modules each row blocks, whether closing it needs a production-code change, and a size (S/M/L). "Partial" always says what is missing. - -If harness code already exists, **check it, don't trust it**: compile it, run its tests by filter, and compare its behaviour with the semantics in 4.3 and 4.4. - -### 4.3 Step 2c: map each helper spec to native code - -Read each helper spec in full, then design its native implementation. For each, produce a symbol table in `uts-infra-design.md`: every symbol the helper spec declares **and** every harness symbol the corpus uses (find them with the scanner in [7.4](#74-step-5d-fill-the-construct-catalogue); the corpus uses members the helper specs don't declare) → native type and member → approach (reuse, wrap, extend, build) → the existing component it builds on, if any. - -| Helper spec | Design questions to answer | Requirements to check against | -|---|---|---| -| [`mock_http.md`](../../rest/unit/helpers/mock_http.md) | Which HTTP hook does `install_mock` use? Does an existing HTTP double capture method, URL, headers and body? Can it fail at connection level separately from request level? Will the legacy `queue_*` calls be implemented, or mapped onto `onRequest` handlers in the skill? | The helper spec; guide 2.2 `mock_http.md` row | -| [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md) | Which transport hook installs it? Does an existing transport double exist, and is its `close()` asynchronous? How are typed messages and raw untyped frames sent to the client? How are client frames captured? | The helper spec (including "Async Behavior and Event Loop Considerations"); guide 2.2 `mock_websocket.md` row and the notes below it | -| [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md) | Does the SDK support deltas, and how is a decoder registered? | The helper spec; guide 2.2 | -| [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | Which channel and plugin setup does `setup_synced_channel` need? Where do module helpers live (where internals are visible, if white-box specs need them)? How is the plugin registered in a client-options builder? | The helper spec; guide 2.2 `standard_test_pool.md` row and "Document each fixture helper's scope" | -| *(no helper spec)* `MockNetworkListener` | Is there a network-monitor hook? | guide 2.2 last table row; `realtime/unit/connection/network_change_test.md` | - -Where a design question can only be answered by a production change, it becomes a hook proposal (4.6). - -### 4.4 Step 2d: design the wait and assertion helpers - -Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the self-tests that prove them (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are in [5.2](#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. - -| Helper | Requirements | -|---|---| -| `AWAIT_STATE` (connection and channel) | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | -| `poll_until`, `poll_until_success` | `writing-test-specs.md`; `uts/README.md`; guide 5.7; `writing-derived-tests.md` "No real timers in unit tests" (deadlines) | -| `process_pending_events()` | `uts/README.md`; guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | -| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | -| Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must), [5.6](writing-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | -| Thread-safe capture, log sink, caller-attributed failures | guide 2.2 | -| `assertContainsInOrder` | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | - -### 4.5 Step 2e: design the sandbox and proxy helpers - -- **Sandbox** (guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). -- **Proxy** (guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Include every operational requirement in guide 2.4. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). - -### 4.6 Step 2f: propose the SDK test hooks - -For every hook row that isn't `present`, write a proposal in `uts-infra-design.md`: - -- the API (name, type, where it is set), modelled on the repo's existing test options if it has any (P-13); -- the default behaviour (unchanged for users); -- its visibility (test-only if the language allows); -- every production file it touches; -- how the harness installs it at client construction, and whether it is per-client; -- for the clock hook, the list of time sources from P-11 it covers, including timed blocking waits and the async runtime's timers. - -These proposals are presented at STOP-3 and need individual approval at **STOP-4** before you implement them. - -### 4.7 Step 2g: plan placement, harness tests and build order - -- **Placement** (guide 2.2, "Placement" and "Recommended layout"): name the harness library and its per-tier directories, the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). -- **Harness tests:** the per-tier smoke tests and helper self-tests of guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must), item by item, for every tier you plan to support. They are permanent and run in CI; they carry no `UTS:` tag and are never copied by the skill; they live outside every directory the mapping will give the resolver as a `targetDir`. Name the command that runs each tier's set plus the common self-tests (guide 2.7 "Selectable"); it goes into the mapping's `harness` entry (D-03). Name the CI job that runs it (5.2 row 13). The acceptance checks in [5.2](#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. -- **Build order:** follow the order in 5.2, dropping rows that are `present` or out of scope. - -### 4.8 Step 2h: derive tier feasibility - -Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "not possible until the SDK has an objects API (placeholder notes; the skill refuses the module)"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). A tier is **ready** only when its G-19 and G-20 rows (smoke tests and self-tests, guide 2.7) are present, green and wired into CI (or the CI change was declined at STOP-5 and is reported as unmet, so the skill doesn't yet conform to guide 2.5/2.7; see [5.4](#54-step-3d-stop-7-confirm-the-harness)), or will be built in Phase 3. - -### 4.9 Step 2i: STOP-3, approve the design and choose the scope - -Present `uts-infra-design.md`: the gap table, the helper-spec mappings, the wait-helper design, the sandbox and proxy design, the hook proposals, placement, the build order and the tier matrix. Ask the user to choose: - -- **(a) Build the missing harness now**, all or a named subset. Recommend this option (the skill depends on the harness), but the user chooses. Continue with Phase 3. -- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4, step C](writing-translator-skills.md#phase-1-selection-steps-0-and-af) requires (see the readiness data in [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). -- **(c) Scope the skill to the tiers possible today.** As (b), without the plan. - -Under (b) and (c), G-19 and G-20 for every tier the skill will offer can't be planned or scoped out; Phase 3 and STOP-7 still run for them. - -Record the answer and any design changes the user asked for. Under (b) and (c), the skill's tier step (C) must offer only ready tiers, and the Final report lists the rest as known limitations. - ---- - -## 5. Phase 3: build and verify the harness - -**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has green smoke tests and helper self-tests (guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)), wired into CI. This is part of the skill's deliverable, not a preliminary. Skip the rows the user didn't choose to build at STOP-3. - -### 5.1 Step 3a: rules while building - -- **Production hooks only after STOP-4 approval**, one proposal at a time, in the smallest form that works. -- **Don't change existing behaviour** of doubles or helpers the native tests use; extend, wrap or add new code instead (see [2.2](#22-what-you-may-change)). -- Follow the repo's test style (P-03, P-04, P-08): naming, file headers, assertion library, formatting. -- Run the existing tests that touch anything you changed, by filter, before and after the change. -- Log every design change you make while building in the decision log; update `uts-infra-design.md`. - -### 5.2 Step 3b: build in order, with acceptance checks - -Each step has an acceptance check; don't start the next step until it passes (or the user accepts a documented exception). **Bound:** if a row's check still fails after three fix attempts, stop (**STOP-10**) with a summary: the failure, what you tried, your diagnosis, and the options (change the design and log it, accept a documented exception, or descope the tiers the row blocks). - -The smoke tests and self-tests in the checks below are the permanent harness tests of guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) (MUST): build them as this phase's gates, and keep them in the repo and in CI afterwards, where the generated skill's preflight runs them. Each row's check is an explicit assertion in a harness test, not a manual observation. - -Rows 4–7 depend on each other: build them in order, but run an acceptance item that needs a component from a later row as soon as that component exists (the fake-clock check needs a transport mock; the REST row's realtime-driven request needs `MockWebSocket`; the realtime row's state waits need row 7), and don't leave row 7 until every row 4–7 item passes. Common items that need an integration component (the integration state-wait helper, integration teardown) run at row 9, once that component exists; they are part of row 9's acceptance. - -| Order | Build | Acceptance check | -|---|---|---| -| 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal harness test (one trivial passing test; delete it once row 2's self-tests exist) compiles, is collected and passes under the single-class filter and under the tier's harness-test command | -| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken, and a repeated state passes (guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)); capture lists are safe under concurrent appends; a failing helper reports the caller's line | -| 3 | Approved SDK hooks | Each hook replaces the real implementation for one client without affecting another client created afterwards (or, if process-global, that fact is recorded for decision D-14); the existing tests still pass | -| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it, and nothing waits on the real clock (guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must)); one advance runs cascaded work to quiescence (zero-delay reschedules and timers created mid-advance) | -| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, REST entry, each as an explicit assertion | -| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, realtime entry, each as an explicit assertion | -| 7 | Wait helpers per [4.4](#44-step-2d-design-the-wait-and-assertion-helpers) | **Helper self-tests:** every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Common and Unit, fake clock entries, each as an explicit assertion; in addition, `AWAIT_STATE` completes for a state reached before or during the wait (the teardown self-test follows your SDK's `close()` behaviour, guide 5.4) | -| 8 | `MockNetworkListener` (if the SDK has a network monitor), `MockVCDiff*` (if the SDK supports deltas) | A smoke test drives each one | -| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Direct sandbox entry, each as an explicit assertion | -| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Proxy entry, each as an explicit assertion; clients authenticate as decided in D-18 | -| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test and self-tests:** Every item in guide [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, objects entry, each as an explicit assertion. The smoke test lives beside the module helpers (outside every `targetDir`) and is selected by the unit tier's harness command | -| 12 | Harness README describing what exists, with its Known gaps | See [5.3](#53-step-3c-write-the-harness-readme) | -| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing; each tier's smoke tests and self-tests run, ungated, in that tier's job (guide [2.5](writing-translator-skills.md#25-build-and-ci-wiring-must), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)); the repo's existing test commands and jobs (e.g. bare `pytest`, `dotnet test` on the solution) don't also collect the UTS or harness tests (exclude them via `testpaths`/`--ignore`, a solution filter or a separate project), unless that job is deliberately their one CI home | - -Then compile all test targets as strictly as CI does (P-06) and run the repo's lint gates over the new code. - -### 5.3 Step 3c: write the harness README - -Write a README next to the harness covering everything guide [3.1](writing-translator-skills.md#31-layout) and [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: - -- take the symbol map from the [4.3](#43-step-2c-map-each-helper-spec-to-native-code) tables as built, and the known gaps from the gap table's `partial` and `missing` rows; -- add the layout (shared, port-only, module helpers, test targets) and the complete command that runs each tier's smoke tests and self-tests; -- write the **Known gaps** section (MUST): every helper-spec member, contract point or corpus construct the harness doesn't implement, by tier, as a table whose first column is the exact pseudocode token, matched by grepping the selected specs' `pseudo` fences (guide 3.1), then status and workaround, so the skill's preflight can match it against selected specs; -- index every module's fixture helpers in one place, even those in a separate test-support module; -- link each fixture helper's scope to the module notes (item 9), where guide 2.2 puts it; -- record the README's full repo path in the Design record. - -### 5.4 Step 3d: STOP-7, confirm the harness - -Re-run the assessment of [4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test and self-test results per tier (each run by its tier's filter), their CI wiring, the files created or changed, and the updated tier matrix. Only then continue to the skill. - -**Done when:** every row the user chose to build is `present`; each supported tier has green smoke tests and self-tests covering guide 2.7, wired into CI (or, if the user declined the CI change at STOP-5, listed in the Final report as an unmet requirement: the skill does not conform to guide 2.5/2.7 until CI runs the harness tests, so mark the Appendix E rows "Harness smoke tests per tier and helper self-tests …" and "One CI home per suite …" ✗ with that reason; the tier stays ready, because the skill's preflight still runs the harness tests locally on every run, and the harness README states that CI doesn't yet run them); the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. If the user declines network access at STOP-6, the integration and proxy tiers aren't supported: mark them unready (`blockedBy`: harness tests not run) and list them in the Final report. - ---- - -## 6. Phase 4: skill design decisions - -**Goal:** every choice the skill bakes in is made once, with the user, and written down. - -### 6.1 Step 4a: draft the Design record - -For each decision below, propose an answer derived from the Repo profile, the built harness and the guide, then present them together. Write them into `design-record.md` ([Appendix C](#appendix-c-design-record-template)). - -| ID | Decision | Guide reference | How to propose it | -|---|---|---|---| -| D-01 | Skill name and location | [3.1](writing-translator-skills.md#31-layout), [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/` | -| D-02 | Script language and minimum runtime version | [3.1](writing-translator-skills.md#31-layout), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [10](writing-translator-skills.md#10-verification-and-ci) | Python 3 unless P-07 shows it isn't available on every developer platform | -| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](writing-translator-skills.md#32-the-mapping-file), [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](#47-step-2g-plan-placement-harness-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that. Record the mapping's `harness` entry (root, README, per-tier sources and harness-test commands from 4.7) | -| D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](writing-translator-skills.md#51-traceability), [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | -| D-05 | Name collisions with native tests | [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | -| D-06 | **Tag syntax** | [5.1](writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | -| D-07 | File header format | [5.1](writing-translator-skills.md#51-traceability), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@` (plus `(locally modified; blob )` for a locally modified spec, guide 9.1), disclosures; integration/proxy also the corresponding unit spec | -| D-08 | **`deviations.md` location** per owning test module | [7.3](writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | -| D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | -| D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](writing-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | -| D-11 | Pending idiom for a missing API (registered, skipped, not commented out) | [7.1](writing-translator-skills.md#71-three-acceptable-end-states), [7.4](writing-translator-skills.md#74-language-inapplicable-inputs) | The framework's static skip or pending marker | -| D-12 | **Internal-access ladder**: the existing exposure to check first, the sanctioned way to add a test-only exposure, who to escalate to; where the white-box spec list lives | [5.9](writing-translator-skills.md#59-internal-access-white-box-unit-specs) | From P-14. List white-box specs in the module notes or the mapping file | -| D-13 | **Delegation policy** for delegating specs and tests | [5.3](writing-translator-skills.md#53-structural-variants) | The guide's recommendation (one test with the delegating tag, running the referenced REST cases through a shared helper; else pending with a `NOTE`). The guide calls this new policy: confirm with the user | -| D-14 | Parallelism: serialise UTS suites, or rely on per-client hooks | [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | From P-12 and the hooks as built | -| D-15 | Teardown mechanism (scope, hook, `finally`) and suite-level fixtures for `BEFORE ALL TESTS` | [5.4](writing-translator-skills.md#54-setup-and-teardown) | From P-03 and the harness | -| D-16 | Protocol-variant rendering (`PROTOCOL`) | [5.3](writing-translator-skills.md#53-structural-variants) | The framework's parameterisation over `json`/`msgpack`; tag stays singular | -| D-17 | Default timeouts and poll interval | [5.6](writing-translator-skills.md#56-time-and-waiting) | As guide 5.6 "Every wait is bounded": the spec's timeout where it states one; otherwise an explicit 10–30 s timeout on every integration `poll_until` and 10–15 s helper defaults for state waits. Record the chosen values | -| D-18 | Auth through the proxy | [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | As designed in [4.5](#45-step-2e-design-the-sandbox-and-proxy-helpers) | -| D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](#74-step-5d-fill-the-construct-catalogue)) | -| D-20 | **Commit policy**; project/solution-file edits | [10](writing-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | -| D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](writing-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | -| D-22 | `allowed-tools` and remote access | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash, Read, Edit, Write`; no `WebFetch`, because everything is read from the local clone (the skill never consults the reference implementations, [2.1](#21-your-inputs)) | -| D-23 | Which modules get notes files, and which are placeholders | [3.1](writing-translator-skills.md#31-layout), [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | -| D-24 | Location of the working records, and whether they are kept in the repo | [1.4](#14-outputs) of this document | Default `.claude/skills/uts-to-/generation/`, kept for maintenance | -| D-25 | Pilot specs and mode per tier | [8.3](#83-pilot-translate-one-spec-per-available-tier) of this document | One per ready module × tier in scope, and at least one per module with a full notes file, chosen by the criteria there | -| D-26 | Model tier for skill runs: pin the model, or state it | [Model tier](writing-translator-skills.md#model-tier), [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | Pin to an Opus-class model if the tooling supports it (verify the mechanism in the current Claude Code documentation; don't assume a frontmatter field); otherwise state it in the opening lines of `SKILL.md`. Either way, the skill's final report records the model | - -### 6.2 Step 4b: STOP-8, confirm - -Show the Design record. Make clear which decisions follow directly from the guide (the user may still override them, but should know they are departing from it) and which are genuine choices. Record the answers. Don't generate any skill file before this confirmation. - ---- - -## 7. Phase 5: generate the skill files - -Generate the files in this order: each one depends on the previous ones. `SKILL.md` comes last because it references all the others; its outline is fixed by the guide, so you can draft it early, but finish it last. - -### 7.1 Step 5a: create the layout and the mapping file - -1. Create the directory layout from guide [3.1](writing-translator-skills.md#31-layout) (plus the working-records directory, D-24). -2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](writing-translator-skills.md#32-the-mapping-file). -3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data, and guide [4, step C](writing-translator-skills.md#phase-1-selection-steps-0-and-af) refuses unready tiers), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. - -**Acceptance:** - -- [ ] Every module the user wants has an entry; every tier value is one path relative to the repo (or a declared root), never machine-absolute. -- [ ] Every derived namespace is unique (a module segment after the tier, or a module-specific prefix). -- [ ] `notes` paths are relative to the skill directory and the files exist (placeholders at this point are fine). -- [ ] Hand-maintained entries are marked (e.g. in `_comment`) and listed in the Design record. -- [ ] Every tier the harness design marks unready has an `unready` entry with the blocking gap-table row. -- [ ] The mapping has a `harness` entry (root, README, per-tier sources and the complete smoke-test and self-test command), every path in it exists, no harness test lies inside any tier `targetDir`, and no `targetDir` lies inside the harness root (guide [3.2](writing-translator-skills.md#32-the-mapping-file), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)). -- [ ] The file is valid JSON, UTF-8, LF line endings. - -### 7.2 Step 5b: write the resolver - -Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. - -Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty, dirtyFiles}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: - -```json -{ - "ok": true, - "sourceModule": "realtime", - "specRepo": { "sha": "", "dirty": false, "dirtyFiles": [] }, - "mapped": true, - "translationNotes": "", - "harness": { - "root": "", - "readme": "", - "tiers": { "unit": { "sources": ["..."], "tests": "" }, "integration": { "...": "..." }, "proxy": { "...": "..." } } - }, - "tiers": { - "unit": { - "present": true, - "sourceDir": "/uts/realtime/unit", - "targetDir": "", - "namespace": "", - "buildTarget": "", - "ready": true, - "specs": [ { "file": "connection/connection_failures_test.md", "className": "ConnectionFailuresTests" } ] - }, - "integration": { "present": true, "...": "..." }, - "proxy": { "present": true, "ready": false, "blockedBy": "G-16: ProxySession not built", "...": "..." } - } -} -``` - -Errors are `{"ok": false, "code": "", "message": ""}`, with a distinct code for each failure guide 8.1 describes (not a UTS module path, directory not found, no tier directories, mapping missing or malformed, bad target name; the guide's example codes are a fine naming), plus one for a declared notes file that is missing (e.g. `NOTES_NOT_FOUND`) and one for a name collision (e.g. `NAME_COLLISION`). - -**Acceptance** (each verified by running the script in [8.1](#81-run-the-resolver-on-every-module)): - -- [ ] Every row of guide 8.1 is implemented: validation, error objects, output contract, path-only tier detection (including specs in tier sub-directories, e.g. `realtime/integration/channels/`), exclusions relative to the tier base, deterministic runner-collectable naming (D-04), collision detection (D-05, including file-name uniqueness where the runner needs it), validate-then-write `--create` writing the D-03 default layout, an error for a missing notes file. -- [ ] `harness` matches the mapping, with every path validated (repo-relative) and existing; `ready`/`blockedBy` match the mapping's `unready` data; `testFile` is present wherever the file name isn't derived from `className`. -- [ ] Never crashes: any exception becomes `ok: false` with a code. - -### 7.3 Step 5c: write the audit - -Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](#21-your-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: - -| Test-side pattern | Source | -|---|---| -| Tag marker (`UTS:` after the line-comment marker) | D-06 | -| Assertion calls (every assertion form the catalogue maps `ASSERT …`, `FAILS WITH`, `THROWS`, `EXPECT THROW` and `AWAIT_ERROR` to) | [7.4](#74-step-5d-fill-the-construct-catalogue) | -| Wait calls (every harness helper the catalogue maps `AWAIT`, `AWAIT_STATE`, `poll_until`, … to) | [7.4](#74-step-5d-fill-the-construct-catalogue); the harness README | -| Comment syntax (line and block), so commented-out assertions are ignored | P-01 | - -Write these as data at the top of the script, so a catalogue change is a one-line edit. Match the longest keyword first on both sides. - -Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifiable` for a spec with no `**Test ID**` lines (not zero tests); if you depart from it, document your shape in `SKILL.md`. Use one exit code for clean, a distinct one for ID problems (missing, orphan, duplicate) and another for "couldn't run". - -**Acceptance** (verified in [8.2](#82-test-the-audit-itself)): - -- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including the end-at-heading rule (a guide SHOULD that this procedure requires: *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `FAILS WITH` / `THROWS` / `EXPECT THROW` / `AWAIT_ERROR` lines counted as assertions, awaited or not, and `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. -- [ ] Where your parsing differs from the contract, `SKILL.md` says so (guide 8.2 "Implement exactly this, or say where yours differs"). -- [ ] SHOULD items you implemented are listed in `SKILL.md`; the ones you didn't are listed in the Final report. - -### 7.4 Step 5d: fill the construct catalogue - -This step produces the construct table that goes into `SKILL.md` (and, for module-specific rows, into the notes). - -1. **Start from every row of guide [section 6](writing-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](writing-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. -2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells, or anything seen in a reference-implementation read ([2.1](#21-your-inputs)); they illustrate other SDKs. -3. **Scan the corpus for constructs the guide doesn't list** (guide section 6 says the catalogue isn't exhaustive). Run the scanner below over each module the skill covers, and add a row for every uppercase keyword or snake_case call you can't account for. -4. **Give every row a status:** - - `mapped`: a direct rendering; - - `mapped (inferred)`: an *(undocumented)* construct, meaning inferred from its uses (list one use); - - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](writing-translator-skills.md#72-harness-stand-ins-are-not-deviations)); - - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation; never substitute a similar mock outcome, guide [5.2](writing-translator-skills.md#52-fidelity)), and the Final report has a plan; - - `n/a`: no spec in the covered modules uses it (say so after scanning). -5. **No blank cells.** Collect every row the skill can't render and has no plan for, from one scan, into a single **STOP-9**. - -Corpus scanner (prints the uppercase keywords and snake_case calls used in `pseudo` fences, outside comments and string literals, with counts; compare its output with the catalogue, and use it in Phase 2 to list the harness symbols the corpus uses): - -```python -# scan_constructs.py /uts/ -import collections, pathlib, re, sys - -root = pathlib.Path(sys.argv[1]) -keywords, calls = collections.Counter(), collections.Counter() -for path in root.rglob("*.md"): - in_pseudo = False - for line in path.read_text(encoding="utf-8").splitlines(): - if line.startswith("```"): - in_pseudo = (line.strip() == "```pseudo") if not in_pseudo else False - continue - if not in_pseudo or line.lstrip().startswith(("#", "//")): - continue - line = re.sub(r'"[^"]*"', '""', line) # ignore string literals - line = re.split(r"\s(?:#|//)\s", line, maxsplit=1)[0] # drop trailing comments - keywords.update(re.findall(r"\b[A-Z][A-Z_]+\b", line)) - calls.update(re.findall(r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\s*\(", line)) -for title, counter in (("KEYWORDS", keywords), ("SNAKE_CASE CALLS", calls)): - print("==", title) - for name, count in sorted(counter.items()): - print(f"{count:6d} {name}") -``` - -The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_SET`, `LWW`); classify those under the guide's "wire constants" and enum rows rather than as new constructs. The scanner sees only `snake_case(` calls; method-style harness calls such as `mock_ws.active_connection.send_to_client(…)` are caught, but camelCase members (`onConnectionAttempt`) are not, so also read the helper specs' interfaces. Ship it as `scripts/scan_constructs.py` in the skill, as guide [section 6](writing-translator-skills.md#6-pseudocode-construct-catalogue) and [3.1](writing-translator-skills.md#31-layout) recommend (SHOULD), because a re-sync re-scans the corpus (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 2); record the choice in the Design record (D-19). - -**Acceptance:** - -- [ ] Every guide section 6 row is present with a rendering and a status; none blank. -- [ ] Every scanner hit in the covered modules is either a catalogue row, a wire constant or enum value, a spec-local name, or internal pseudocode of a helper spec or `objects/PLAN.md` (implemented as harness or ignored, never translated). -- [ ] Every rendering names a symbol that exists in the harness or the SDK (you read it), or is marked `not available` with a plan. -- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide [5.7](writing-translator-skills.md#57-waits-must-catch-events), [6.7](writing-translator-skills.md#67-notes-on-the-catalogue)). - -### 7.5 Step 5e: write the module notes - -Write `references/-mapping.md` for every module chosen in D-23. Cover the thirteen items in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should) ("A module notes file should cover"), in that order. For each item, either fill it or write "Not applicable: ". The items that most need evidence: - -- **Item 1 (source of truth and runtime status):** name the IDL the notes apply (for `objects`, check whether the SDK follows `objects-features.md` in the clone or another variant; guide 3.4 item 1 explains the typed-SDK case) and whether the module is implemented, which decides whether evaluate mode is possible. -- **Item 5 (type mapping):** for every polymorphic or union type in the pseudocode, the SDK type and which conversions throw or return null. Read the SDK source for each. -- **Item 8 (internal access):** the ladder from D-12 and the list of white-box specs. To find candidates, look for specs that construct internal classes or call members the public API doesn't have (the guide's white-box row in [6.6](writing-translator-skills.md#66-mocks-fixtures-and-harness) lists examples), then confirm each against the SDK's public surface. -- **Item 9 (helper-spec coverage):** the module's rows of the symbol tables from [4.3](#43-step-2c-map-each-helper-spec-to-native-code), as built, with each fixture helper's scope and every sanctioned stand-in. -- **Item 12 (worked example):** translate one short real test from the module by hand, name each mechanical rewrite, and make sure it compiles (it becomes one of the skill's examples; [8.7](#87-verify-the-skills-own-examples)). - -A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), "If the module's notes file exists but is only a placeholder"). - -**Acceptance:** - -- [ ] All thirteen items present or marked not applicable with a reason. -- [ ] Every SDK and harness symbol named in the notes exists at the recorded repo SHA. -- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide [4, step 1](writing-translator-skills.md#phase-2-per-spec-steps-17)). -- [ ] No line numbers; cite symbols and headings (guide [12](writing-translator-skills.md#12-lessons-learned)). - -### 7.6 Step 5f: write `SKILL.md` - -Follow the outline in guide [3.4](writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide ["What the skill owns, and what it defers"](writing-translator-skills.md#what-the-skill-owns-and-what-it-defers)). - -| Part of `SKILL.md` | Content | Guide | -|---|---|---| -| Frontmatter | `name: uts-to-`; a pushy `description` with trigger phrases; `argument-hint`; `allowed-tools` from D-22; the model pin from D-26, where the tooling supports one (otherwise the model statement in the opening lines) | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | -| Usage guard | Empty argument → print the usage line and stop; placeholder paths only | [3.3](writing-translator-skills.md#33-frontmatter-and-arguments) | -| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state (the resolver's `specRepo`); for each selected spec in `dirtyFiles`, stop and ask, and if the user proceeds stamp its blob (guide 9.1) | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.1](writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](writing-translator-skills.md#94-local-clone-vs-fetching-main) | -| Skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | -| Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | -| Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](writing-translator-skills.md#34-recommended-outlines-should), [2.7](writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | -| Skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README and helper sources from the resolver's `harness` output; for the reference test per tier, write `none yet` until [8.8](#88-name-the-reference-tests) names one | [4](writing-translator-skills.md#phase-2-per-spec-steps-17), [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit) | -| Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](writing-translator-skills.md#5-translation-rules) | -| Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](writing-translator-skills.md#6-pseudocode-construct-catalogue) | -| File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](writing-translator-skills.md#34-recommended-outlines-should) | -| Evaluation | Guide 7 rendered with D-08 to D-11 and D-21 | [7](writing-translator-skills.md#7-evaluation-and-deviations) | -| Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](writing-translator-skills.md#55-integration-tier-hygiene) | -| Verification | Guide 10 with the exact commands from the Repo profile | [10](writing-translator-skills.md#10-verification-and-ci) | -| Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](writing-translator-skills.md#92-a-re-sync-mode-should), [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Final report | The format, including the model line | [11](writing-translator-skills.md#11-final-report-format) | -| Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](writing-translator-skills.md#10-verification-and-ci) | - -**Acceptance:** - -- [ ] Every part in the table is present. -- [ ] The skill is module-generic: module-specific rules live in the notes. -- [ ] No inline copy of a semantic rule where a link to its source would do; in particular the `deviations.md` format is referenced, not redefined. -- [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills (reference-implementation reads during creation are recorded in the working records, never cited in the skill). -- [ ] None of the guide's [Patterns to avoid](writing-translator-skills.md#patterns-to-avoid) appears in the skill, its notes, its scripts or the harness. -- [ ] The Harness reference names only files, filters and commands that exist, and doesn't inline helper signatures. -- [ ] Every command in it is one the Repo profile records as run. (Running each through the skill, and compiling every example, are Phase 6 checks: [8.5](#85-exercise-the-skills-other-paths), [8.7](#87-verify-the-skills-own-examples).) -- [ ] Every helper or SDK symbol it names exists. - ---- - -## 8. Phase 6: validate the skill - -**Goal:** evidence that the skill works on this repo, on top of the harness you built. This phase is mandatory. Record every result in the Final report. - -### 8.1 Run the resolver on every module - -Run the resolver on `/uts/rest`, `/uts/realtime` and `/uts/objects`, whether or not they are in scope (an unmapped module must resolve with `mapped: false`, not crash). For each, check: - -- [ ] `ok: true`; tiers `present` match `ls /uts/`. -- [ ] Spec counts match the files on disk minus the exclusions. A cross-check: `find /uts// -name '*.md' -not -path '*/helpers/*'` (add the proxy exclusion for the integration tier). -- [ ] No helper spec, `README.md`, `PLAN.md` or `*_SUMMARY.md` appears in `specs`. -- [ ] Every `className` follows D-04 and is collected by the runner; no collisions (or the collision is reported). -- [ ] `targetDir`, namespace and build target match the mapping and the Design record. -- [ ] The same module given as a `~`-prefixed path, and (on Windows) with backslash separators, resolves identically (guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) "Validate the module"). - -Then run the negative cases and check each returns `ok: false` with the right code: a path whose parent isn't `uts`; a missing directory; a module with no tier directories (a scratch directory named `uts/` will do); a mapping entry whose notes file is missing (use a scratch copy of the mapping); `--create` with an invalid name; `--create` on a scratch copy of a deliberately corrupt mapping (the file must be left unchanged). - -### 8.2 Test the audit itself - -Use one known-good test file (the pilot output from [8.3](#83-pilot-translate-one-spec-per-available-tier) once it is reviewed; until then, a hand-written file with the correct tags) and make scratch copies for each mutation. If you started with the hand-written file, run the mutations again once the pilot output passes review: - -| Mutation | Expected audit result | -|---|---| -| Delete one assertion | A positive assertion shortfall for that test | -| Comment out one assertion | The same shortfall (commented assertions don't count) | -| Duplicate one test's tag | `duplicate` lists the ID; distinct non-zero exit | -| Add a tag that isn't in the spec | `orphan` lists it | -| Remove one tag | `missing` lists it | -| Add surplus waits to a test with a deleted assertion | The assertion shortfall is still reported (counts aren't summed) | -| Delete the native assertion that renders a spec `FAILS WITH` (awaited or not) | A positive assertion shortfall for that test | -| Delete the poll call that renders a spec `poll_until` (with no `AWAIT` on the spec line) | An await shortfall for that test (spec polls count as waits) | - -**Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l`` (as of `12540dcf`: rest 571, realtime 554, objects 339). - -### 8.3 Pilot-translate one spec per available tier - -Choose one spec per ready module × tier in scope, and at least one per module with a full notes file (D-25). A good pilot is short, representative of the tier, and uses the harness features most likely to be wrong: - -| Tier | Choose a spec that uses | Example candidates in the corpus | -|---|---|---| -| REST unit | `MockHttpClient`, request capture, `FAILS WITH` | `rest/unit/time.md` | -| Realtime unit | `MockWebSocket`, `enable_fake_timers`, `ADVANCE_TIME`, `AWAIT_STATE` | `realtime/unit/connection/when_state_test.md` | -| Objects unit | `setup_synced_channel`, builders, serial helpers | `objects/unit/internal_live_counter_api.md` (then a mutation spec) | -| Direct integration | `SandboxApp`, `BEFORE ALL TESTS`, `## Protocol Variants` | `rest/integration/publish.md` (has `## Protocol Variants`), or `rest/integration/time_stats.md` then `realtime/integration/channel_history_test.md` | -| Proxy | `create_proxy_session`, rules, the event log | `rest/integration/proxy/rest_fallback.md` or `realtime/integration/proxy/connection_resume.md` | - -The candidates are examples only. Before choosing one, confirm it exists in your clone (`ls /uts/`) and still uses the features listed (grep for them); check it against the module's scope and the SDK's features. Prefer specs with no existing UTS-derived test in the repo (P-20); if the best candidate has one, ask the user before the skill adds to or regenerates it. - -**Run the pilot the way a user would.** - -- **Loading the skill.** A skill created during a session may not be listed until a new session starts. Ask the user to start a new Claude Code session in the repo, on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)) (or to reload skills, if their client supports it), and confirm `/uts-to-` is listed. A fresh session also tests that the skill is context-complete on its own (guide intent), so the pilot never consults the reference implementations. -- **Answering its questions.** The answers are D-03 (mapping confirmation) and D-25 (tier, specs, mode), confirmed at STOP-8. Either the user runs `/uts-to- /uts/` in the new session and answers with them, or, if the skill is listed in a session you control, you invoke it and answer with exactly those choices. If you use a subagent, run it on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)), and give it only the invocation and the D-25 answers, none of this run's context. It must not consult the reference implementations or any other SDK's skill, because the pilot tests the skill alone. Record the pilot session's or subagent's model with the pilot results. Any answer not covered by D-25 goes to the user. -- **Record each question the skill asked.** More than the four questions, other than a stop the guide requires (an unmapped module's name at step B, a known gap or red harness at step F, an unmapped construct, the fix-attempt bound), or a question it should have answered from its own files, is a skill defect ([8.4](#84-fix-the-cause-then-regenerate)). - -**A green preflight comes first.** The skill's step F (after **STOP-6** for network tiers) must pass for the pilot's tier before the pilot translates anything; a pilot run on a red harness doesn't count. Record the preflight result with the pilot. If network access is declined, that tier's pilot isn't run and is reported as unvalidated. - -Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: - -1. Compile (and collect: every tagged test function is collected; count a parameterised or protocol-variant test once). -2. Audit; walk the review checklist (guide [8.3](writing-translator-skills.md#83-review-checklist-must-after-the-audit)). -3. In evaluate mode, and after **STOP-6** for network tiers, run the class by filter and diagnose per `writing-derived-tests.md` Phase 2. -4. Review the generated file line by line against the spec yourself, and show it to the user. - -### 8.4 Fix the cause, then regenerate - -For every defect in a pilot test: - -1. Classify it: a rule in `SKILL.md`, a mapping in the notes, a catalogue row, a resolver or audit bug, or a harness gap. (A UTS spec error is not a skill defect: record it, emit the fail-fast test per D-10, and **STOP-10**.) -2. Fix that cause. **Never edit only the generated test.** A harness fix follows the rules of [5.1](#51-step-3a-rules-while-building), adds or extends a self-test that would have caught the defect, re-runs the affected tiers' smoke tests and self-tests, and updates the harness README (and its Known gaps). -3. Delete the generated test and regenerate it with the skill. -4. Re-audit, recompile and, in evaluate mode, re-run. -5. Log the defect and the fix in the Design record's changelog. - -**Bound:** once a pilot spec has been regenerated three times (or the D-21 count) without a clean audit and, in evaluate mode, one of the three end states, stop (**STOP-10**) with a summary of every defect found and fixed so far. Count all regenerations of that spec together, not per defect. - -### 8.5 Exercise the skill's other paths - -Run each path through the skill (in the session from [8.3](#83-pilot-translate-one-spec-per-available-tier)) and record the result: - -1. **Usage guard:** no argument → the usage line, then stop. -2. **Bad module path:** a path whose parent isn't `uts` → the resolver's error relayed, then stop. -3. **Placeholder notes:** a module whose notes file is a placeholder → refused, with the reason. -4. **Unready tier:** a tier the mapping marks unready → not offered at step C, with its `blockedBy`. -5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. -6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification (including a change outside the pseudocode, such as `## Protocol Variants`, reported as **changed**, and, run against a scratch clone of the spec repo (`git clone `; the spec clone itself is never edited, 2.2), an uncommitted edit to that spec reported as **changed**), that the header SHA is updated, and that only the survivors in guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. -7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. -8. **Red harness:** on a scratch copy (or a temporary working-tree edit you revert afterwards), break one harness test (for example, make the WebSocket mock refuse every connection), or run an integration tier with the network unavailable → the skill stops at step F before generating anything, reports a harness or environment problem (not an SDK deviation), and names the failing test. Restore the change and confirm the preflight is green again. -9. **Known gap:** select a spec that uses a construct the harness README lists under Known gaps → the preflight reports the conflict and stops to ask (guide 4 step F), rather than generating a substituted rendering. If Known gaps is empty, use a scratch copy of the README with an entry for a construct the selected spec uses. - -### 8.6 Lint and CI-strict compile - -Run the repo's lint and format gates (P-06) over every file you created or changed, including the harness, the skill's scripts and Markdown where the repo's EditorConfig covers them. Compile the test targets as strictly as CI does (warnings as errors, static analysis, type checkers). Run the skill's scripts on every developer platform you can, or note which you couldn't. - -### 8.7 Verify the skill's own examples - -Every code example in `SKILL.md` and the notes must compile (guide [12](writing-translator-skills.md#12-lessons-learned)). For each example, either: - -- name the real, compiled file it was copied from (and check it still matches), or -- paste it into a scratch file in the test target, compile, and delete the scratch file. - -Record which examples were verified and how. - -### 8.8 Name the reference tests - -After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4, step 3](writing-translator-skills.md#phase-2-per-spec-steps-17)). Never name a smoke test. - -**Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected (including the red-harness stop) and every command in `SKILL.md` has been run; every pilot ran after a green preflight; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. - ---- - -## 9. Phase 7: final report and handover - -1. Write `final-report.md` from [Appendix D](#appendix-d-final-report-template). -2. Fill the acceptance checklist in [Appendix E](#appendix-e-acceptance-checklist-guide-section-13): tick each item, or explain why it isn't met (scoped out, planned, not applicable). -3. Show the user the report, the list of changed and new files (harness, harness tests, hooks, skill, pilot tests), and the suggested CI changes. The run is complete only if the harness tests are green, and in CI, at every tier the skill supports; otherwise the report says which tiers aren't and why. **Don't commit** unless asked. -4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. - ---- - -## 10. Maintenance - -The harness and the skill must stay in step with five things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. - -| What changed | How to detect it | What to update | -|---|---|---| -| **The guide** | `git -C diff -- uts/docs/translator-skills/` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](#61-step-4a-draft-the-design-record) and [7.6](#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | -| **The UTS docs or helper specs** | `git -C diff -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | -| **The UTS corpus** | The skill's own re-sync mode (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | -| **The SDK** (API, hooks, test layout, CI) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | - -Rules for every maintenance run: - -- Follow the same ground rules ([section 2](#2-ground-rules)) and stop points, including the model tier ([2.8](#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](#21-your-inputs)). If the guide's "as of" commits have moved since a recorded read, re-check that read's conclusion against the updated Patterns to avoid before relying on it again. -- Start from the working records; update the Repo profile and the harness design if anything they describe has changed. -- Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](writing-translator-skills.md#92-a-re-sync-mode-should)). -- After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](#81-run-the-resolver-on-every-module) and [8.2](#82-test-the-audit-itself). -- Record the new spec SHA and repo SHA in the Design record and append a changelog entry. - ---- - -## Appendix A: Repo profile template - -```markdown -# Repo profile: - -- Repo: , HEAD (clean | dirty: ) -- Spec clone: , HEAD (clean | dirty: ); guide last changed in -- Date: ; confirmed by the user on (STOP-2) -- User goals: - -## Build, test and CI -| # | Item | Answer | Evidence (path, symbol or command → result) | Verified? | -|---|---|---|---|---| -| P-01 | Language(s) and version(s) | | | | -| P-02 | Build and test commands (build, build tests, all, class, single test, filter, target-qualified filter) | | | ran / unverified: | -| P-03 | Test framework, runner, assertions, parameterisation, fixtures, teardown, static and runtime skip, fail idiom, collection rules | | | | -| P-04 | Test layout, naming, targets, shared test-support code | | | | -| P-05 | CI workflows, jobs → suites, matrix, network, submodules, secrets | | | | -| P-06 | Lint/format gates, static analysis, type checkers, warnings-as-errors | | | | -| P-07 | Developer and CI platforms; script runtime availability | | | | -| P-08 | Repo conventions; hand-maintained project files; synced manifests | | | | - -## SDK structure for testing -| # | Item | Answer | Evidence | Verified? | -|---|---|---|---|---| -| P-09 | Async model; runner async support; virtual time | | | | -| P-10 | Threading, callback queues, internal-queue drain, compile-time concurrency checks | | | | -| P-11 | Every time source (timers, timed waits, delayed callbacks, clock reads) | | | | -| P-12 | Runner parallelism and how to serialise | | | | -| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global) | | | | -| P-14 | Internal-access mechanism and current use | | | | -| P-15 | API surface vs pseudocode, per module (rest / realtime / objects) | | | | - -## Existing native test support -| # | Item | Answer | Evidence | Verified? | -|---|---|---|---|---| -| P-16 | Existing mocks, fakes, test doubles (capabilities, style, global?, users) | | | | -| P-17 | Existing test helpers (waits, polls, capture, log capture, assertions, factories) | | | | -| P-18 | Existing sandbox/integration support; fault-injection tooling | | | | -| P-19 | ably-common submodule (path, initialised, fixture files) | | | | -| P-20 | Existing UTS assets and their state | | | | -| P-21 | Sandbox access from developer machines and CI | | | | - -## How the existing tests drive the SDK (step 1c) -| Kind | Test read (path, test name) | Pattern: create client / install double / wait / assert / clean up | -|---|---|---| -| REST request | | | -| Realtime connection state | | | -| Timer-driven behaviour | | | -| Negative assertion | | | -| Internal state | | | -| Sandbox integration | | | -| Plugin module | | | - -- Doubles that already meet helper-spec semantics: … -- Doubles that differ (and how): … -- Race-free waits vs sampling or sleeping waits: … -- Known flakiness and workarounds: … - -## API divergences per module -- rest: … -- realtime: … -- objects: … (implemented? plugin? typed? IDL followed?) - -## Unknowns and questions for the user -- … -``` - -## Appendix B: Gap table and harness design template - -```markdown -# Harness design: @ - -## Gap table -| # | Capability | Guide | Status | Evidence | Approach (reuse / wrap / extend / build / n/a) | Builds on | Blocks (module/tier) | Production change? | Size | Acceptance check | -|---|---|---|---|---|---|---|---|---|---|---| -| G-01 | WebSocket transport factory hook | 2.1 | present / partial / missing / n/a | | | | realtime/unit, objects/unit | yes / no | S/M/L | | -| G-02 | HTTP client hook | 2.1 | | | | | rest/unit | | | | -| G-03 | Clock hook (timers, timed waits, async-runtime timers) | 2.1 | | | | | | | | | -| G-04 | Network-monitor hook | 2.1 | | | | | | | | | -| G-05 | Randomness hooks (jitter, host shuffle) | 2.1 | | | | | | | | | -| G-06 | MockHttpClient (+ legacy queue_*) | 2.2, mock_http.md | | | | | | | | | -| G-07 | MockWebSocket (+ alternative API, undocumented members, raw frames, ping) | 2.2, mock_websocket.md | | | | | | | | | -| G-08 | MockVCDiff encoder/decoders | 2.2, mock_vcdiff.md | | | | | | | | | -| G-09 | MockNetworkListener | 2.2 | | | | | | | | | -| G-10 | standard_test_pool.md implementation; plugin options builder | 2.2, standard_test_pool.md | | | | | | | | | -| G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | -| G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | -| G-13 | Placement and recommended harness layout (shared / port-only / module helpers); fixture-helper scope documented in the module notes | 2.2 | | | | | | | | | -| G-14 | Harness README with Known gaps; mapping `harness` entry (root, README, per-tier sources and harness-test commands) | 3.1, 3.2 | | | | | | | | | -| G-15 | SandboxApp; ably-common | 2.3 | | | | | */integration | | | | -| G-16 | ProxyManager (pinned, every OS); ProxySession; rule builders; proxy auth | 2.4 | | | | | */proxy | | | | -| G-17 | UTS test target; per-tier selection and filters; one CI home per suite | 2.5 | | | | | | | | | -| G-18 | Concurrency, time, parallelism and visibility models known | 2.6 | | | | | | | | | -| G-19 | Per-tier smoke tests (permanent, in CI, outside generated dirs) | 2.7 | | | | | | | | | -| G-20 | Helper self-tests (common, unit, fake clock, REST, objects, sandbox, proxy) | 2.7 | | | | | | | | | - -## Helper-spec symbol maps (one table per helper spec) -### mock_http.md -| Spec symbol (declared or corpus-only) | Native type.member | Approach | Builds on | Notes | -|---|---|---|---|---| -### mock_websocket.md -### mock_vcdiff.md -### standard_test_pool.md -### MockNetworkListener (no helper spec) - -## Wait and assertion helpers -| Helper | Native symbol | Approach | Guarantees checked by | -|---|---|---|---| - -## Sandbox and proxy -- Sandbox provisioning: …; client targeting (endpoint or hosts): …; suite fixture mapping: … -- Proxy: version …; binaries per OS …; local override …; auth through the proxy …; platform gating … - -## Hook proposals (each needs STOP-4 approval) -| Hook | API | Default behaviour | Visibility | Files touched | Installed how | Per-client? | Approved on | -|---|---|---|---|---|---|---|---| - -## Placement -- Shared test-support: ; port-only harness: ; module helpers: - -## Tier feasibility -| Module | unit | integration | proxy | -|---|---|---|---| -| rest | ready / after G-.. / not possible: / n/a | | | -| realtime | | | | -| objects | | | | - -## Decision at STOP-3 -(a) build now: | (b) plan only | (c) scope to ready tiers — chosen by on ; design changes requested: … - -## Build log (Phase 3) / plan (option b) -| Order | Row | Work | Acceptance check | Result | Date | -|---|---|---|---|---|---| - -## Confirmation at STOP-7 -Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier matrix: …; confirmed on -``` - -## Appendix C: Design record template - -```markdown -# Design record: uts-to- - -## Inputs at generation time -- Spec clone: (clean | dirty: ); guide: -- Repo: -- Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: -- Harness README: -- Model: (Opus-class: yes | no, departure in decision log #); sub-agents: , or none - -## Decisions -| ID | Decision | Choice | Guide-default? | Decided by | Rationale | -|---|---|---|---|---|---| -| D-01 | Skill name and location | | yes / no | user / agent | | -| D-02 | Script language and runtime | | | | | -| D-03 | Mapping: module × tier → target dir (hand-maintained entries marked) | (table below) | | | | -| D-04 | Naming and collection rules (with three worked examples) | | | | | -| D-05 | Collision handling | | | | | -| D-06 | Tag syntax | | | | | -| D-07 | File header format | | | | | -| D-08 | deviations.md location(s) | | | | | -| D-09 | Runtime skip idiom; RUN_DEVIATIONS command | | | | | -| D-10 | Fail-fast idiom | | | | | -| D-11 | Pending idiom for missing APIs | | | | | -| D-12 | Internal-access ladder; white-box spec list location | | | | | -| D-13 | Delegation policy | | | | | -| D-14 | Parallelism policy | | | | | -| D-15 | Teardown; suite-level fixtures | | | | | -| D-16 | Protocol-variant rendering | | | | | -| D-17 | Default timeouts and poll interval | | | | | -| D-18 | Auth through the proxy | | | | | -| D-19 | Re-sync flag; manifest; corpus scanner shipped? | | | | | -| D-20 | Commit policy; project/solution-file edits (approved / left to the user) | | | | | -| D-21 | Fix-attempt bound | | | | | -| D-22 | allowed-tools; remote access | | | | | -| D-23 | Notes files per module (full / placeholder) | | | | | -| D-24 | Working-records location | | | | | -| D-25 | Pilot specs and mode per tier | | | | | -| D-26 | Model tier and pinning for skill runs | | | | | - -### D-03 mapping -| Module | unit | integration | proxy | notes | hand-maintained? | -|---|---|---|---|---|---| - -## Decision log -| # | Date | Phase | Question | Options | Answer | By | Supersedes | -|---|---|---|---|---|---|---|---| - -## Reference-implementation reads (last resort, 2.1) -| # | Date | Question | Guide/UTS sections searched | Repo @ commit : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | -|---|---|---|---|---|---|---|---| - -## Changelog (fixes made to the skill, notes or harness) -| # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Tests regenerated | -|---|---|---|---|---|---| -``` - -## Appendix D: Final report template - -```markdown -# Final report: uts-to- for - -- Spec clone (clean | dirty: ); repo ; date -- Model: (Opus-class: yes | no, and why); sub-agents: , or none -- Scope: ; out of scope and why - -## Files created or changed -| Path | New / changed | Kind (hook, harness, smoke test, skill, script, notes, pilot test, CI suggestion) | Approved at | -|---|---|---|---| -Hand-maintained project files the user must update: … - -## Repo profile -Link to repo-profile.md; corrections made at STOP-2. - -## Harness -Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test and self-test results per tier, and their CI jobs; preflight results from the pilot runs; harness changes made during Phase 6 (with the self-tests added); the README's Known gaps; the plan for any remaining rows. Link to uts-infra-design.md and the harness README. - -## Design decisions -Link to design-record.md; decisions that depart from the guide's defaults, and why. - -## Validation results -- Resolver: rest / realtime / objects → ok, spec counts; negative cases → codes -- Audit self-test: each mutation → result; corpus sweep → N specs, 0 crashes, unverifiable specs listed -- Lint: …; CI-strict compile: …; examples verified: N of N (how) - -## Pilot results -(each pilot run's full section-11 report as the skill printed it: the header line, the table row, and every labelled line, including not-applicable omissions, missing APIs, mock-capability gaps, harness stand-ins and constructs it couldn't map; the table below collects the rows) -| Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | -|---|---|---|---|---|---|---|---| -Fixes made to the skill, notes or harness during the pilot (from the changelog): … -Reference tests named in SKILL.md: - -## Known limitations -Tiers not ready; `not available` catalogue rows; harness capabilities missing; audit SHOULD items not implemented; platforms the scripts weren't run on. - -## Deviations and spec issues found -- SDK deviations (recorded in deviations.md): … -- Suspected UTS spec errors or gaps: — ; draft fix: … -- Spec-repo doc issues (contradictions between guide and UTS docs; guide gaps revealed by reference-implementation reads): … - -## Reference-implementation reads -Each read from the Design record (question, repo @ commit : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. - -## Suggested CI changes - (not applied unless approved at STOP-5) - -## Next steps -Which modules and tiers to translate next, in order, and why; harness work remaining. - -## Acceptance checklist -(Appendix E, filled) -``` - -## Appendix E: Acceptance checklist (guide section 13) - -Fill this in the Final report. Each row is one item of the guide's [checklist](writing-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill), with where this procedure produces or verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). - -| Guide §13 item | Produced in | Verified in | Status | -|---|---|---|---| -| **Harness** | | | | -| SDK test hooks (5), clock covering blocking waits and async-runtime timers, injected at construction | 4.6; 5.2 rows 3–4 | 5.2 acceptance; G-01…G-05 | | -| Shared test library per helper spec (incl. legacy `queue_*` and `mock_http.captured_requests`; `mock_websocket.md` async close, CONNECTED ordering, raw frames, alternative API and undocumented members; vcdiff; test pool; `MockNetworkListener`) | 4.3; 5.2 rows 5–6, 8, 11 | smoke tests; G-06…G-10 | | -| Wait helpers (latched, fail-fast `AWAIT_STATE`; value-returning `poll_until`; `poll_until_success`; `process_pending_events`; wall-clock wrapper; fake-time-safe deadlines) | 4.4; 5.2 row 7 | helper tests; G-11 | | -| Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | -| Harness designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; recommended harness layout; fixture-helper scope documented | 4.2; 4.7; 5.3; notes item 9 | G-13 | | -| Harness README with a Known gaps section (MUST) | 5.3; 5.2 row 12 | G-14; 8.5 path 9 | | -| Harness smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.5 path 8 | | -| `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | -| `ProxyManager` (pinned, every OS, verified download, health check, port handling); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | -| One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | -| Concurrency, time, parallelism, visibility models documented | Phase 1 (P-09 to P-14) | STOP-2; G-18 | | -| **Skill files** | | | | -| `SKILL.md`: name, pushy description, `argument-hint`, usage guard, placeholder paths, 3.4 outline incl. the Harness reference | 7.6 | 7.6 acceptance; 8.3 | | -| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked, `harness` entry | 7.1 | 7.1 acceptance; 8.1 | | -| `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries, `harness` output | 7.2 | 8.1 | | -| `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | -| Module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | -| Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | -| **Workflow** | | | | -| Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | -| Steps 0 and A–F: the four questions and the harness preflight (stop on red) | 7.6 | 8.3 pilot (green preflight first); 8.5 paths 8–9 | | -| Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 8.3 pilot; 8.8 | | -| Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 8.3 pilot; 8.5 (re-sync) | | -| Re-sync mode; fix the cause, then regenerate | 7.6; D-19 | 8.5 (re-sync); 8.4 (fix and regenerate) | | -| Mid-run harness changes re-run the smoke tests and self-tests and are reported separately | 7.6 (Harness reference) | 8.4 step 2; final report | | -| **Translation rules in `SKILL.md`** | | | | -| One verbatim `UTS: ` per test; no invented tags; spec point in a collectable name | 7.6; D-04, D-06 | 8.2, 8.3 | | -| File header: source spec, tier, spec SHA, disclosures, corresponding unit spec | 7.6; D-07 | 8.3 | | -| Verbatim comments, spec variable names, section headings, spec order | 7.6 | 8.3 review | | -| Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome | 7.6; 7.4 | 8.3 audit | | -| Policies for sub-cases, groupers, test-case tables, delegating tests, ID-less tests | 7.6; D-13 | 8.3 (where the pilot has them) | | -| Protocol variants parameterised; tag singular; proxy JSON only | 7.6; D-16 | 8.3 integration pilot | | -| Clients closed in teardown; mocks restored; stop on a failed wait | 7.6; D-15 | 8.3 | | -| Unique channel names; own proxy session per test | 7.6 | 8.3 integration/proxy pilots | | -| Unit: fake time and flushes, no real timers. Integration: wall-clock | 7.6; 7.4; D-17 | 8.3 | | -| Record-and-verify for transient states; polls return the settled value | 7.6; 7.4 | 8.3 | | -| No accommodate-both assertions; `IS ` per typing discipline | 7.6 | 8.3 review | | -| Public-API specs stay public; internal-access ladder with escalation stop | 7.6; 7.5 item 8; D-12 | 8.3 | | -| Construct table filled for every row | 7.4 | 7.4 acceptance | | -| **Evaluation** | | | | -| Three end states; runtime skip and fail-fast idioms; `RUN_DEVIATIONS` command | 7.6; D-09, D-10 | 8.3 (evaluate) | | -| `deviations.md` location; format deferred; written at the right time | 7.6; D-08 | 8.3 (evaluate) | | -| Harness stand-ins in comments, not `deviations.md` | 7.4; 7.6 | 8.3 review | | -| Language-inapplicable inputs (cases a–c) noted in the test file | 7.6; D-11 | 8.3 (where they occur) | | -| Stopping rules and spec-error escalation | 7.6; D-21 | 8.4 | | -| **Verification** | | | | -| Compile (or collect), lint, CI-strict compile, filtered run | 7.6 | 8.3, 8.6 | | -| No commits by default; changed files listed | 7.6; D-20 | Phase 7 | | -| Final report format | 7.6 | 8.3 (the skill's report) | | -| The skill's own examples compile | 7.5 item 12; 7.6 | 8.7 | | -| **Creation process** | | | | -| Skill and harness created on an Opus-class model (MUST); model pinned or stated for runs (SHOULD) and recorded in the skill's final report | 2.8; D-26; 7.6 (frontmatter, final report) | Design record "Inputs at generation time" (Model line); 8.3 (pilot model recorded); Final report header | | -| Reference-implementation reads, if any, last resort only, recorded and reported as guide gaps | 2.1; Appendix C | Final report "Reference-implementation reads" | | - -## Appendix F: Glossary - -| Term | Meaning | -|---|---| -| **UTS** | Universal Test Suite: language-neutral pseudocode test specs in `uts/` of the spec repo | -| **Module** | A top-level corpus directory: `rest`, `realtime`, `objects` | -| **Tier** | `unit` (mocked transports), `integration` (direct sandbox), `proxy` (sandbox through `uts-proxy`), detected by path | -| **Spec** | One Markdown file of UTS tests; **features spec** means the Ably specification under `specifications/` | -| **Test ID / tag** | The `**Test ID**` of a spec test; the `UTS: ` comment above the derived test | -| **Helper spec** | A spec of harness behaviour (`mock_http.md`, `mock_websocket.md`, `mock_vcdiff.md`, `standard_test_pool.md`); implemented once, never translated | -| **Hook** | An injection point in SDK production code that the harness uses (transport, HTTP, clock, network monitor, randomness) | -| **Harness (UTS test infrastructure)** | The native test library implementing the helper specs, wait helpers, sandbox and proxy helpers and fixtures | -| **Reuse / wrap / extend / build** | The four ways to provide a harness capability from what the repo has ([4.2](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)) | -| **Shared / port-only** | Harness code used by native tests and UTS ports alike / used only by UTS ports | -| **Smoke test** | A per-tier harness test that drives one tier end to end through the real hooks (guide 2.7); permanent, in CI, not spec-derived, no tag, never a reference test | -| **Self-test** | A harness test that checks one helper obeys its helper-spec contract (guide 2.7); permanent, in CI, no tag | -| **Preflight** | The generated skill's step F: compile, run the tier's smoke tests and self-tests, check Known gaps; stop on red | -| **Reference test** | A reviewed, spec-derived test per tier that the skill reads before generating | -| **Reference implementations** | The existing `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java) skills and their harnesses, at the commits the guide pins: a last-resort, read-only reference for patterns, never copied and never an authority ([2.1](#21-your-inputs)). Not a reference test | -| **Opus-class model** | The most capable model tier available (Claude Opus or an equivalent); required for this procedure and its sub-agents ([2.8](#28-model-tier-and-sub-agents)) | -| **Notes** | `references/-mapping.md`: how ably-js-shaped pseudocode maps to this SDK for one module | -| **Resolver** | `resolve_uts.py`: validates a module and derives targets and names; its output is the single source of truth | -| **Audit** | `audit_translation.py`: compares a spec with its test file (IDs, ledger, counts) | -| **Ledger / shortfall** | The per-test list of spec lines by kind / spec count minus test count, for assertions and waits separately | -| **Translate-only / evaluate** | Generate and statically check / also run and diagnose | -| **Deviation** | An SDK behaviour that differs from a correct spec, recorded in `deviations.md` | -| **Stand-in** | A harness-level difference in how a test drives the SDK; disclosed in a comment, not a deviation | -| **White-box spec** | A unit spec that drives internal APIs; listed in the notes | -| **Delegating spec/test** | One that says to run another module's tests against this client | -| **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | -| **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | -| **Phase N / skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Skill Phase 1" (selection, steps 0 and A–F) and "skill Phase 2" (per spec, steps 1–7) are the generated skill's workflow (guide [4](writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | -| **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | - -## Appendix G: Common failure modes and remedies - -All from the guide and the UTS docs; follow the link for the detail. - -| Symptom | Likely cause | Remedy | Source | -|---|---|---|---| -| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the distinct collected test functions (a parameterised test counted once) with the tag count | guide [10](writing-translator-skills.md#10-verification-and-ci), [5.1](writing-translator-skills.md#51-traceability) | -| Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | -| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests"; guide [5.6](writing-translator-skills.md#56-time-and-waiting) | -| Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](writing-translator-skills.md#21-sdk-test-hooks-must) | -| A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | -| `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](writing-translator-skills.md#67-notes-on-the-catalogue) | -| A state wait runs out the timeout after the connection failed | No fail-fast on FAILED | Fail immediately on FAILED, except when the test just requested a transition out of it | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | -| Assertions after a poll see fewer items | The test re-reads after polling an eventually-consistent store | Use the value the poll returned | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events) | -| `poll_until_success` times out with no useful error | Emulated with a reader that swallows errors | A separate helper that rethrows the last error | guide [5.7](writing-translator-skills.md#57-waits-must-catch-events); `uts/README.md` | -| Negative assertions pass or fail at random | One zero-delay yield doesn't drain chained callbacks | Drain until the internal queue is empty; or the quiescence control-listener | guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must); `standard_test_pool.md` "Negative-assertion quiescence" | -| Tests pass alone, fail together | Process-global hooks with a parallel runner; leaked clients or timers | Serialise or make hooks per-client (D-14); close clients and restore mocks in teardown | guide [2.6](writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must), [5.4](writing-translator-skills.md#54-setup-and-teardown) | -| State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | -| An objects test's operation is silently ignored | Hand-written serial literal sorts before the pool serial | Use the serial helpers | `standard_test_pool.md`; guide 2.2 | -| Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | -| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions"; guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | -| Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | -| Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | -| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning"; guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | -| Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | -| Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](writing-translator-skills.md#82-audit-audit_translationpy-must) | -| Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must) | -| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](writing-translator-skills.md#phase-1-selection-steps-0-and-af) | -| Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](writing-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | -| A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](writing-translator-skills.md#5-translation-rules) | -| Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](writing-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](writing-translator-skills.md#12-lessons-learned) | diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/translator-skills/writing-translator-skills.md index 71bd54407..eaebbc7b3 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/translator-skills/writing-translator-skills.md @@ -1,13 +1,13 @@ # Writing UTS Translator Skills -This guide explains how to build a per-language Claude Code skill — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). +This guide explains how to build a per-language agent skill (Claude Code and Codex) — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). -If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, follow the step-by-step procedure in [UTS-to-Lang Skill Creator](uts-to-lang-skill-creator.md), on an Opus-class model ([Model tier](#model-tier)); this guide remains the authority on what the skill must contain. +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, use the [`uts-to-lang-skill-creator` skill](../../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building the skill and its harness; this guide remains the authority on what the skill must contain. **Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. - **Harness and skill are one deliverable.** The skill translates onto a harness built in the same repo from its existing test setup, and it isn't complete until that harness's smoke tests and self-tests are green, in CI, at every tier the skill supports ([section 2](#2-the-harness-uts-test-infrastructure)). -- **Easy to use.** One command (`/uts-to- `) and four questions; the skill does the rest. +- **Easy to use.** One command (`/uts-to- ` in Claude Code, `$uts-to- ` in Codex) and four questions; the skill does the rest. - **Context-complete.** Every run gathers the full context: the spec, the translation manual, the sources and README of the UTS test infrastructure (the **harness**), the module notes and the resolver output. A good test doesn't depend on anyone's memory. - **Tests are reproducible artifacts.** The UTS spec plus the skill is the source of truth; the generated test is an output. When a translation is wrong or the spec changes, regenerate it ([section 9](#9-keeping-in-sync-with-spec-changes)) rather than hand-maintain it. - **Fix the cause, then regenerate.** Fix the skill, the module notes or the harness, never only the generated file. Only DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders and `deviations.md` entries survive regeneration. @@ -43,7 +43,7 @@ Facts about the existing skills and SDKs are **as of 2026-10-07**: spec commit ` ### What a translator skill is -A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that Claude Code follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a harness in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). +A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that an agent (Claude Code or Codex) follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a harness in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). ### How it relates to the other UTS docs @@ -53,7 +53,7 @@ A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and no | [`writing-test-specs.md`](../writing-test-specs.md) | The pseudocode reference: Test IDs, mock patterns, `AWAIT_STATE`, `poll_until`, record-and-verify, error pattern, unique channel names | | [`writing-derived-tests.md`](../writing-derived-tests.md) | **Translation and evaluation semantics.** Faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | | [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | -| **This guide** | How to package all of the above into a repeatable, scripted Claude Code skill for one SDK | +| **This guide** | How to package all of the above into a repeatable, scripted agent skill (Claude Code and Codex) for one SDK | | Existing skills and harnesses ([Reference implementations](#reference-implementations-last-resort)) | Nothing normative: a last-resort, read-only reference for patterns when creating a skill, below every document above | `writing-derived-tests.md` is the **source of truth for semantics**. This guide never redefines what a faithful translation is, or how a deviation is classified and recorded; it says how a skill enforces those rules. If the two ever disagree, `writing-derived-tests.md` wins. @@ -72,15 +72,15 @@ A skill **SHOULD** keep only the language rendering of a rule and link to the so ### Model tier -Creating a skill and its harness, and running the skill, are long, multi-file reasoning tasks in which a silent omission costs far more than compute. Use **the most capable model tier available (Claude Opus or an equivalent Opus-class model)**. "Opus-class" means that tier, whatever its current version, not a particular release. +Creating a skill and its harness, and running the skill, are long, multi-file reasoning tasks in which a silent omission costs far more than compute. Use **the most capable model tier available (Claude Opus or an equivalent Opus-class model)**. "Opus-class" means that tier, whatever its current version, not a particular release; in another tool (for example Codex), it means that tool's most capable tier, recorded by name. | Task | Level | Why | |---|---|---| -| Creating the skill and building its harness (the [procedure](uts-to-lang-skill-creator.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | -| Translating and evaluating with the finished skill (`/uts-to-`) | **SHOULD** | The resolver and the audit ([section 8](#8-deterministic-tooling)) make paths, names, ID coverage and assertion counts mechanical, but not the rest. Matching wait predicates and timeouts, copying comments verbatim, setup fidelity, and diagnosing a failure as SDK, spec or translation ([7](#7-evaluation-and-deviations), [8.3](#83-review-checklist-must-after-the-audit)) remain model judgement, and a count can't catch a changed predicate ([8.2](#82-audit-audit_translationpy-must)) | +| Creating the skill and building its harness (the [procedure](../../skills/uts-to-lang-skill-creator/SKILL.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | +| Translating and evaluating with the finished skill (`/uts-to-` or `$uts-to-`) | **SHOULD** | The resolver and the audit ([section 8](#8-deterministic-tooling)) make paths, names, ID coverage and assertion counts mechanical, but not the rest. Matching wait predicates and timeouts, copying comments verbatim, setup fidelity, and diagnosing a failure as SDK, spec or translation ([7](#7-evaluation-and-deviations), [8.3](#83-review-checklist-must-after-the-audit)) remain model judgement, and a count can't catch a changed predicate ([8.2](#82-audit-audit_translationpy-must)) | | Sub-agents spawned during either (review, validation; pilot runs during creation serve creation) | **The level of the task they serve** | A reviewer weaker than the author misses what the author missed. A cheaper model MAY be used only for a clearly mechanical step whose result a script verifies (for example, running the audit over the corpus and collecting its JSON), if at all | -The deterministic scripts reduce the dependence on the model; they don't remove it. Pin the model where the tooling allows ([3.3](#33-frontmatter-and-arguments)), and record the model used in every run's final report ([section 11](#11-final-report-format)). +The deterministic scripts reduce the dependence on the model; they don't remove it. State the model tier in the skill, pin it only where a tool supports that ([3.3](#33-frontmatter-and-arguments)), and record the model used in every run's final report ([section 11](#11-final-report-format)). --- @@ -310,7 +310,7 @@ The harness has two kinds of test of its own, both permanent and both required: ### 3.1 Layout ``` -.claude/skills/uts-to-/ +/ # .claude/skills/uts-to-/ (linked from .agents/skills/uts-to-/) ├── SKILL.md # the procedure ├── uts-package-mapping.json # spec module → target dir/package per tier, + notes pointer ├── scripts/ @@ -332,6 +332,16 @@ The harness has two kinds of test of its own, both permanent and both required: Both existing skills use Python for scripts. Any language works, but the scripts must run on every developer platform (see [8.1](#81-resolver-resolve_utspy-must) and [section 10](#10-verification-and-ci)). +**Install for both Claude Code and Codex (SHOULD).** Claude Code loads project skills from `.claude/skills/`; Codex loads them from `.agents/skills/`. Keep one copy and link the other location to it, so both tools load the same files: + +```sh +mkdir -p .agents/skills +ln -s ../../.claude/skills/uts-to- .agents/skills/uts-to- +git add .agents/skills/uts-to- +``` + +Both tools follow directory symlinks. On Windows, Git needs `core.symlinks=true` (and Developer Mode); if that isn't available to your contributors, commit a copy instead, and add a CI check that the two directories are identical. Everything in the skill refers to its own files relative to the skill directory ([3.3](#33-frontmatter-and-arguments)), so either location works. Users invoke it as `/uts-to- ` in Claude Code and `$uts-to- ` in Codex. + Keep two documents with separate roles: the test library's README describes **what exists** (helpers, seams, layout); `SKILL.md` describes **how to author** tests. Refer to the README by its full repo path, so it can't be confused with the spec repo's `uts/README.md`. The harness README MUST include a **Known gaps** section: every helper-spec member or contract point, and every corpus construct, that the harness doesn't implement, by tier, in a form the skill's preflight can match against the selected specs (construct or helper name → status and workaround). Write it as a table whose first column is the exact pseudocode token as it appears in `pseudo` fences (for example `respond_with_timeout`, `PING_MESSAGE`, `AWAIT_ALL`), so the preflight matches mechanically: for each row, grep the selected specs' `pseudo` fences for the token; any hit is a conflict. The corpus scanner (section 6) MAY accept spec files as well as a module directory for this. It SHOULD also cover: the SDK hooks used, each helper-spec symbol → harness symbol, the wait helpers and their default timeouts, the threading and time model, how to run each tier and its smoke tests and self-tests (2.7), and one index of every module's fixture helpers (even those in a separate test-support module). **The notes are a map, not an authority.** Where the notes, the spec's IDL and the SDK disagree about the SDK's API, the SDK source is the ground truth: fix the notes. @@ -368,21 +378,30 @@ Each tier value is **one path**, relative to the repo or to a declared root (Swi ### 3.3 Frontmatter and arguments +Write the skill so the same directory loads in Claude Code and in OpenAI Codex. Both implement the open [Agent Skills](https://agentskills.io/specification) format. Put only the portable fields in `SKILL.md`: + ```yaml --- name: uts-to- -description: "Translate Ably UTS (Universal Test Suite) pseudocode specs into runnable tests in . Use when asked to translate, port, derive, generate or re-sync tests from a UTS spec or module (uts/rest, uts/realtime, uts/objects), to evaluate UTS-derived tests against the SDK, or to check UTS coverage. Takes a UTS module directory, validates it, resolves target test dirs, lets you pick a tier (unit/integration/proxy) and specs, then derives one test per spec Test ID. Usage: /uts-to- " -argument-hint: -allowed-tools: Bash, Read, Edit, Write +description: "Translates Ably UTS (Universal Test Suite) pseudocode specs into runnable LANG tests in REPO. Use when asked to translate, port, derive, generate or re-sync tests from a UTS spec or module (uts/rest, uts/realtime, uts/objects), to evaluate UTS-derived tests against the SDK, or to check UTS coverage. Takes a UTS module directory, validates it, resolves the target test directories, asks for a tier (unit, integration, proxy) and specs, then derives one LANG test per spec Test ID. Not for creating the skill or its harness (use uts-to-lang-skill-creator), or for writing UTS specs." +license: "" +compatibility: "Claude Code or Codex. Needs git, python3 and a local clone of ably/specification; run on an Opus-class model." +allowed-tools: Bash Read Edit Write +metadata: + version: "1.0.0" + short-description: "Translate UTS specs into LANG tests" --- ``` -- **`name`** SHOULD be `uts-to-`. (`uts-to-kotlin` omits `name` and relies on the directory name; that works but is less explicit.) -- **`description`** SHOULD be "pushy": list the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to ", "re-sync UTS tests"). Both existing skills' descriptions are procedural, which suits explicit `/` invocation but triggers poorly otherwise. -- **Argument:** a single UTS **module** directory, directly under `uts/`. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. If the argument is empty, the skill MUST print the usage line and stop. -- **`allowed-tools`:** both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-local-clone-vs-fetching-main)). -- **Model (SHOULD):** if your Claude Code version lets a skill declare the model it runs on, pin the skill to an Opus-class model ([Model tier](#model-tier)). Check the current Claude Code documentation for the mechanism rather than assuming a frontmatter field. If it can't, say in the opening lines of `SKILL.md` that the skill should be run on an Opus-class model. Either way, the skill records the model in its final report ([section 11](#11-final-report-format)). -- **Placeholders, not personal paths,** in examples and usage text: `/uts/objects`. +- **`name`** MUST be `uts-to-`, equal to the skill's directory name: lowercase letters, digits and hyphens, at most 64 characters. (`uts-to-kotlin` omits `name` and relies on the directory name; both tools accept that, but the Agent Skills format requires the field.) +- **`description`** MUST be at most 1,024 characters and contain no `<` or `>`. Write `LANG` and `REPO`, not ``: skill packaging validators (for example skill-creator's `quick_validate.py`) reject angle brackets, and Anthropic's skill-authoring best practices also say no XML tags in descriptions. It SHOULD be "pushy" and written in the third person: front-load the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to LANG", "re-sync UTS tests") and end with a "Not for …" boundary. Put the usage line in the body, not in the description. Both existing skills' descriptions are procedural, which suits explicit invocation but triggers poorly otherwise. +- **`metadata`** values MUST be strings (quote `"1.0.0"` and `"false"`). Codex shows `metadata.short-description` in its skill list. +- **`allowed-tools`** is space-separated. Claude Code pre-approves the listed tools while the skill runs; Codex ignores the field. Both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-local-clone-vs-fetching-main)). +- **Claude Code-only fields** (`argument-hint`, `model`, `disable-model-invocation`, `when_to_use`, `context`, …) MAY be added only if the team accepts that they aren't portable: Codex ignores them, and they make claude.ai upload and standard packaging fail. Put Codex-only settings (display name, implicit-invocation policy) in an optional `agents/openai.yaml` in the skill directory. +- **Argument:** a single UTS **module** directory, directly under `uts/`. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. Don't depend on argument substitution (`$ARGUMENTS` works only in Claude Code): say in words where the argument comes from ("the module directory given after `/uts-to-LANG` in Claude Code or `$uts-to-LANG` in Codex, or named in the user's message"). If there is none, the skill MUST print the usage line and stop. +- **Paths to the skill's own files** MUST be relative to the skill directory (the directory containing `SKILL.md`), for example "run `scripts/resolve_uts.py` from this skill's directory". Never write `.claude/skills/uts-to-/scripts/…`: Codex loads the skill from `.agents/skills/`, and `${CLAUDE_SKILL_DIR}` is Claude Code-only. The scripts follow the same rule: they locate the mapping file and the notes relative to their own file (for example `Path(__file__).parent.parent`), never relative to the working directory. +- **Model (SHOULD):** no portable frontmatter field pins a model. State in the opening lines of `SKILL.md` (and in `compatibility`) that the skill should run on an Opus-class model ([Model tier](#model-tier)). Claude Code's `model` field is a non-portable option (see above). Either way, the skill records the model in its final report ([section 11](#11-final-report-format)). +- **Placeholders, not personal paths,** in examples and usage text: `/uts/objects` in the body (angle brackets are fine in the body, but not in `description`). ### 3.4 Recommended outlines (SHOULD) @@ -684,7 +703,7 @@ Fill in the last column for your language; the skill MUST map every construct it | `BEFORE ALL TESTS`, `AFTER ALL TESTS`, `BEFORE EACH TEST`, `AFTER EACH TEST`, `AFTER TEST:` | Fixtures | `with…` scopes, per test | `@BeforeAll` / `@AfterAll` | | | `POST https://…/apps WITH body from …`, `DELETE … WITH Authorization: Basic …` | Sandbox provisioning | `SandboxApp` | `SandboxApp.create()` / `delete()` | | | `create_proxy_session(endpoint:, rules:)` (or `rules:` only; see 6.7), `add_rules`, `trigger_action`, `get_log()` / `session.getLog()`, `proxy_port` | `proxy.md` | `withProxySession(rules:)`; typed `ProxyEvent` log | `ProxySession.create(…)`, `finally { session.close() }`; typed log | | -| `"__PASSTHROUGH__"` as a field value in a `replace` action's message *(undocumented)* | A uts-proxy sentinel that `proxy.md` doesn't define (used in `realtime/integration/proxy/connection_resume.md`); confirm its behaviour against the pinned uts-proxy release before mapping it | — | — | | +| `"__PASSTHROUGH__"` as a field value in a `replace` action's message *(undocumented)* | A placeholder string, not a uts-proxy feature: uts-proxy v0.3.0 sends a `replace` message verbatim, so the SDK receives the literal value (used in `realtime/integration/proxy/connection_resume.md` Test 22, whose assertions don't depend on it). Treat it as an ordinary string; don't implement a substitution | — | — | | | `MockVCDiffEncoder()`, `MockVCDiffDecoder`, `FailingMockVCDiffDecoder` | `mock_vcdiff.md` | — | — | | | `setup_synced_channel`, `setup_synced_channel_no_ack`, `STANDARD_POOL_OBJECTS`, `build_*`, serial helpers, `provision_objects_via_rest` | `standard_test_pool.md` | Unit tier seeds the pool directly (sanctioned stand-in); helpers in the test-support target | `setupSyncedChannel("test")`, `build*` in module `Helpers.kt` | | | White-box access: `applyOperation`, `channel.object.objectsPool`, `processChannelState`, direct construction, `get_backoff_coefficient(n)`, `get_jitter_coefficient()`, `encode_recovery_key(…)`, `CLEAR channel._lastPayload.messageId` | Internal access ([5.9](#59-internal-access-white-box-unit-specs)). `CLEAR …` in `delta_decoding_test.md` is a private-field write inside an *integration* spec | Core: `import Ably.Private` + private headers; objects: `@testable import` + `testsOnly_` accessors | `internal` within `:liveobjects` | | @@ -853,11 +872,11 @@ A per-module manifest (spec file → SHA → test file) MAY be kept as well. Lin ### 9.2 A re-sync mode (SHOULD) -Neither existing skill has one; both have stale suites as a result. Add a mode (for example `/uts-to- --resync`) that: +Neither existing skill has one; both have stale suites as a result. Add a mode (for example `/uts-to- --resync` in Claude Code, `$uts-to- --resync` in Codex) that: 1. Runs the audit over **every** mapped spec in the module, not only the ones selected. 2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as **new**; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. Then re-scan the corpus for constructs the skill can't map (section 6). -3. Classifies each spec as **new**, **changed** (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), **unchanged** or **removed**. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](uts-to-lang-skill-creator.md#10-maintenance)). +3. Classifies each spec as **new**, **changed** (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), **unchanged** or **removed**. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](../../skills/uts-to-lang-skill-creator/references/maintenance.md#10-maintenance)). 4. Regenerates the affected tests, **preserving only** DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. 5. Updates the SHA in each regenerated file's header, and its blob note (9.1): re-stamped if the spec is still locally modified, removed if not (and in the manifest, if kept). 6. Reports the classification ([section 11](#11-final-report-format)). @@ -973,7 +992,9 @@ The skill and its harness are one deliverable: the skill isn't done until every **Skill files** -- [ ] `SKILL.md` with `name: uts-to-`, a pushy description with trigger phrases, `argument-hint`, usage guard, placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline, including the Harness reference +- [ ] `SKILL.md` frontmatter portable across Claude Code and Codex: `name` equal to the directory; a pushy third-person description of at most 1,024 characters with no `<` or `>` and a "Not for" boundary; string-valued `metadata`; Claude Code-only fields only if accepted as non-portable ([3.3](#33-frontmatter-and-arguments)) +- [ ] `SKILL.md`: a usage guard that doesn't rely on `$ARGUMENTS`; the skill's own files referred to relative to the skill directory, in `SKILL.md` and in the scripts; placeholder paths; follows the [3.4](#34-recommended-outlines-should) outline, including the Harness reference +- [ ] Installed for both tools from one source: `.claude/skills/` and `.agents/skills/`, by in-repo symlink or a CI-checked copy ([3.1](#31-layout)) (SHOULD) - [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked, `harness` entry - [ ] `resolve_uts.py`: validation, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, collision detection, validate-then-write `--create` preserving entries, `harness` output - [ ] `audit_translation.py`: the parsing contract, ID coverage, duplicates, separate assert/await counts, ignores commented assertions, never crashes, flags unverifiable specs; itself mutation-tested @@ -1022,7 +1043,7 @@ The skill and its harness are one deliverable: the skill isn't done until every **Creation process** -- [ ] Skill and harness created on an Opus-class model (MUST); the skill pins the model or states it for its runs (SHOULD), and records it in its final report ([Model tier](#model-tier)) +- [ ] Skill and harness created on an Opus-class model (MUST); the skill states the model for its runs, or pins it where the tool supports that (SHOULD), and records it in its final report ([Model tier](#model-tier)) - [ ] Reads of the [reference implementations](#reference-implementations-last-resort), if any, made only as a last resort, recorded, and reported as guide gaps --- @@ -1056,7 +1077,7 @@ Rules: 5. **Record each consultation**: the repo, commit and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. 6. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. -Reading GitHub is network access: the [procedure](uts-to-lang-skill-creator.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. +Reading GitHub is network access: the [procedure](../../skills/uts-to-lang-skill-creator/SKILL.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. ### Patterns to avoid @@ -1068,6 +1089,7 @@ Observed in the existing skills and their harnesses as of 2026-10-07 (the review - **Two sources of tier truth**: path-based in the resolver, content-based in the `SKILL.md` integration section. - **A `CONTAINS_IN_ORDER` mapping that isn't a subsequence check** ([6.7](#67-notes-on-the-catalogue)). - **No re-sync mode, no final report, no lint step, no collision detection**, and a silent `null` when a declared notes file is missing. +- **Claude Code-only packaging** ([3.3](#33-frontmatter-and-arguments)): script paths hard-coded as `python3 .claude/skills//scripts/…` and reliance on `$ARGUMENTS`, both of which break under Codex; angle-bracket placeholders (`<…>`) in `description`, which skill packaging validators reject. **In the harnesses** diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index e5dd1f321..3f3c952cc 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -13,6 +13,8 @@ The process has two phases: Not every situation has an existing implementation. Tests may be written ahead of the implementation (test-first development), or for a new SDK that doesn't yet exist. In those cases, only the translation phase applies. +To automate translation for one SDK with an agent skill (`uts-to-`), see [Writing UTS Translator Skills](translator-skills/writing-translator-skills.md). That guide packages this document's rules into a repeatable procedure. This document remains the authority on translation and evaluation semantics. + --- ## Phase 1: Translation diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md new file mode 100644 index 000000000..d985fe00e --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -0,0 +1,253 @@ +--- +name: uts-to-lang-skill-creator +description: "Builds a uts-to-LANG translator skill, together with the UTS test harness it runs on, for one Ably SDK repository (C#, Python, Go, JavaScript, Ruby, Dart, Rust, ...). Use it whenever asked to create, bootstrap, scaffold or generate a uts-to-csharp, uts-to-python, uts-to-go or other uts-to-* skill; to set up UTS (Universal Test Suite) test infrastructure in an SDK repo (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, harness smoke tests and self-tests); to bring the UTS specs in uts/rest, uts/realtime and uts/objects to a new SDK; or to maintain, regenerate or re-validate an existing uts-to-* skill after the guide, UTS docs, corpus or SDK changed. Runs a procedure with user stop points: profile the repo, design and build the harness, generate the skill, pilot-validate it, report. Needs a local ably/specification clone. Not for translating specs once a uts-to-* skill exists (use that skill) or for writing UTS specs." +license: Apache-2.0 +compatibility: "Claude Code or Codex, on an Opus-class model. Needs shell and file access, git, Python 3.8+, a local clone of ably/specification, and the target SDK's build toolchain." +allowed-tools: Read Grep Glob +metadata: + team: "engineering" + version: "1.0.0" + tags: "testing, uts, translator-skill, skill-creator, harness, ably-specification" + marketplace: "false" + short-description: "Build a uts-to-LANG skill and its UTS harness" +--- + +# UTS-to-Lang Skill Creator + +Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create a `uts-to-` translator skill, and the UTS test infrastructure (the **harness**) it runs on, for one Ably SDK repository. [Writing UTS Translator Skills](../../docs/translator-skills/writing-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why; this skill says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this skill links to it, and you must read the linked section. + +The harness and the skill are **one deliverable** (guide [section 2](../../docs/translator-skills/writing-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](references/phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). The phases run in a fixed order, because each stands on the one before it; a harness designed without studying the repo's existing tests duplicates or fights them. + +"This skill" means this `SKILL.md` together with its `references/` and `assets/templates/`. The ground rules (section 2) and the inputs and outputs (section 1) are below; each phase's steps are in `references/`, and you read each one in full before its phase starts ([Phases at a glance](#phases-at-a-glance)). Section numbers (for example "4.6") are stable across the files ([Reference index](#reference-index)). + +## Path convention + +- **Files of this skill** (`references/`, `assets/templates/`, `scripts/`) are relative to the directory that contains this `SKILL.md`, wherever it is installed. Run scripts as `python3 /scripts/.py`, where `` is the directory containing this `SKILL.md`. +- **Files of the spec repo** (the guide, the UTS docs, the helper specs, the corpus) are linked relative to this skill's place in a spec clone, `uts/skills/uts-to-lang-skill-creator/`, so the links work on GitHub. A link that climbs out of the skill directory (`../../…` from `SKILL.md`, `../../../…` from `references/`) is a spec-repo path; every other link is a file of this skill. At run time, never resolve a spec-repo path against the directory you loaded this skill from. Run `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and read each file at `/uts/…`: a link to `…/docs/translator-skills/writing-translator-skills.md#31-layout` means `/uts/docs/translator-skills/writing-translator-skills.md`, section 3.1. Never fetch them from GitHub. +- **Inputs** come from the user's words: the text after `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex), or their message. Don't rely on argument substitution. + +## 2. Ground rules + +### 2.1 Your inputs + +You work from four primary sources: + +1. this skill; +2. the guide; +3. the local spec clone: the UTS docs (`uts/README.md`, `uts/docs/`), the helper specs, the UTS corpus (`uts/rest`, `uts/realtime`, `uts/objects`) and, for API and spec-error questions, the features specs under `specifications/`; +4. the target SDK repo. + +and, only as a last resort, the reference implementations below. + +**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](../../docs/translator-skills/writing-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: + +1. You have searched this skill, the guide, the UTS docs and the helper specs, and they don't answer the question. +2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) with `git -C show :` and `git -C ls-tree -r --name-only -- `, never checking out, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. If a clone lacks the pinned commit, fetching it is network access too. +3. You take a **pattern**, not text: how something was structured or solved. Never copy, port or translate code, scripts, prose, tables or examples verbatim. Write every hook, helper, script, rule and example yourself, in this language's idioms, from the guide, the UTS docs and the target repo. +4. You check what you took against the guide's [Patterns to avoid](../../docs/translator-skills/writing-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. +5. You record each consultation in the Design record ("Reference-implementation reads": repo, commit, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. + +A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)). + +### 2.2 What you may change + +| Action | Allowed? | +|---|---| +| Create and edit files under the skill directory, its `.agents/skills` link or copy (D-01), and the working-records directory | Yes | +| Create new test-only harness code (shared test library, module helpers, smoke tests, UTS test-project wiring) | Only after the user approves the harness design at [STOP-3](#23-stop-points), and only within the scope they approved | +| Change existing test-support code that the SDK's own tests use (shared fakes, fixtures, base classes) | Only with explicit approval for each change, and never in a way that changes behaviour for the existing tests. Prefer wrapping or extending to editing | +| Change SDK production code, including adding test hooks or raising visibility | **Only with explicit approval for each change** ([STOP-4](#23-stop-points)). Present the proposal first ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | +| Change CI workflows, build scripts that CI runs, or dependency manifests (including adding a test dependency) | **Only with explicit approval** ([STOP-5](#23-stop-points)). Otherwise write the change as a suggestion in the Final report | +| Edit project or solution files needed to register new test files or projects (`.sln`, `.csproj`, `.projitems`, an Xcode project, etc.) | Only with explicit approval ([STOP-5](#23-stop-points)), recorded in the Design record (D-20). If the user declines, give them the exact entries to add, wait until they confirm, and list them in the Final report | +| Stage, commit, push, branch, or open a PR | **No, unless the user asks** | +| Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | +| Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | +| Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | +| Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies, reference-implementation reads under [2.1](#21-your-inputs)) | No | + +### 2.3 Stop points + +At a stop point, present what you found and the options, then wait for the user's answer. Record the question and the answer in the decision log ([design-record template](assets/templates/design-record.md)). + +| ID | Phase | When | What you present | +|---|---|---|---| +| **STOP-1** | Start | A required input is missing, the session's model isn't Opus-class ([1.3](#13-required-inputs), [2.8](#28-model-tier-and-sub-agents)), or `spec_clone_info.py` fails or reports `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | +| **STOP-2** | 1 (end) | The Repo profile is written | The profile, for confirmation or correction | +| **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | +| **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | +| **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | +| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases, reading a reference implementation on GitHub or fetching its commit, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | +| **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | +| **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | +| **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | +| **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](references/phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), [8.4](references/phase-6-validate-skill.md#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | +| **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | +| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, commit, path) and what they showed; the reading you propose | + +### 2.4 Never guess + +- **Spec content:** quote the spec, a helper spec or a UTS doc from the clone, with its path and heading. Don't rely on memory of the Ably specification. +- **API signatures:** read the SDK source and the harness source before writing any call. A signature you haven't read is a guess. +- **Commands:** run every build, test, lint and script command you record, and record its exit status. A command you haven't run is marked "unverified" in the working records. +- **Facts about the repo:** every Repo profile entry has evidence (a file path, a symbol or a command and its output). "Unknown" is allowed only with the reason and a question for the user. +- **Undocumented corpus constructs:** the guide marks some constructs *(undocumented)*; their meaning is inferred from usage. Read each use in the corpus before choosing a rendering, and say in the catalogue that the meaning is inferred. +- **Reference-implementation reads:** what another SDK's skill or harness does is evidence of one SDK's choice, not of the spec or of this SDK. Confirm it against the guide or the UTS docs before relying on it. Where they are silent, record the choice as yours in the decision log, citing the read. + +### 2.5 Record every decision + +Every choice you make that isn't forced by the guide or the UTS docs goes into the decision log in the Design record, with the options, the choice, who made it (you or the user) and the reason. That includes harness choices in Phases 2 and 3 (reuse vs build, hook shapes, placement), not only the skill decisions in Phase 4. If a later phase changes an earlier decision, add a new log entry that supersedes the old one; don't edit history. + +### 2.6 Pin the spec clone + +At the start of the run, run `python3 /scripts/spec_clone_info.py []`, passing the path the user gave, if any. It locates the clone (the path given; else `UTS_SPEC_CLONE`; else the clone this skill is symlinked from; a path inside a clone is walked up to its root), checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd' -- uts/docs/translator-skills uts/skills/uts-to-lang-skill-creator`. It prints one JSON object. + +- Any `ok: false` (for example `SPEC_CLONE_NOT_FOUND`, `NOT_A_SPEC_CLONE`, `GIT_ERROR`): **STOP-1**; ask for the clone's path. +- `foundBy` other than `argument`: tell the user which clone was found and how, and confirm it before Phase 1. +- `skillMatchesClone: false` (an installed copy that differs from this clone's copy of the skill): **STOP-1**; ask the user to refresh the install, or to confirm which version to follow. +- `skillMatchesClone: null` (the clone has no copy of this skill, for example an older checkout): tell the user and confirm before Phase 1. + +Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), the guide's and this skill's last-change commit (`guideLastChange`) and `skillVersion`. Read every spec-repo file at the absolute paths the script reports (`paths`), or under `specClone`. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. + +Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. + +### 2.7 Working style + +- Do Phase 1 before anything else, and don't write harness code before the user approves the harness design (STOP-3), or skill files before the harness is verified (STOP-7). +- Escalate verification in the repo's own order (compile, then build tests, then a filtered run, then wider runs), as recorded in the Repo profile. +- Keep the working records up to date as you go; they are also your memory if the session is interrupted. On resuming, read them first. +- Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. + +### 2.8 Model tier and sub-agents + +- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name): MUST.** The guide's [Model tier](../../docs/translator-skills/writing-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. +- **Sub-agents** you spawn for review, validation or the pilot ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, running the [8.2](references/phase-6-validate-skill.md#82-test-the-audit-itself) corpus sweep and collecting its JSON), if at all, and record which steps used one. +- **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. +- **State** the generated skill's model tier in its opening lines and `compatibility`; pin it only as D-26 decides. + +## 1. Purpose and how to use this procedure + +### 1.1 Who reads this + +- **The executor** is an LLM agent working inside the target SDK repository, with shell access, file read/write access, and the ability to ask the user questions, running on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). Every instruction below is addressed to it ("you"). +- **The user** is an SDK engineer who owns the target repository. They answer the questions at the stop points and approve every change outside the skill directory. +- **The lead reviewer** may skim the working records (Repo profile, Harness design, Design record, Final report) instead of the whole conversation. Any human may skim this skill to see what the agent will do and where it will ask for decisions. + +### 1.2 Invocation + +The user starts a run in the target SDK repo with one sentence, for example: + +``` +/uts-to-lang-skill-creator ~/src/specification realtime unit first (Claude Code) +$uts-to-lang-skill-creator ~/src/specification realtime unit first (Codex) +Create a uts-to-csharp skill for this repo; the spec clone is ~/src/specification. +Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to create a uts-to- skill for this repo. (any agent, skill not installed) +``` + +`` is the user's local clone of `ably/specification`. Take it, and any goals, from the user's words; argument substitution isn't available in every tool. Installation (a symlink into `~/.claude/skills/` and `~/.agents/skills/`) is described in [`uts/README.md`](../../README.md#installing-the-skill-creator). If the user also states goals ("objects unit tier first", "rest and realtime only"), record them as inputs. + +### 1.3 Required inputs + +| Input | Example | If missing | +|---|---|---| +| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo; confirm it has SDK sources and tests | +| Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | +| Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | Located by `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and confirmed with the user; otherwise **stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | +| User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | +| Model tier | The session's model, Opus-class ([2.8](#28-model-tier-and-sub-agents)) | Take it from your environment if it states your model; otherwise ask. If the model isn't Opus-class: **STOP-1** | +| Local clones of ably-cocoa and ably-java (optional) | Clones that contain commits `b5074d9b` and `0ff24017` | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | + +### 1.4 Outputs + +1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). +2. **The skill**, at `.claude/skills/uts-to-/` (linked from `.agents/skills/uts-to-/` for Codex, guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout)) unless the Design record says otherwise (D-01): `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout)). +3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. +4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): + - `repo-profile.md` ([repo-profile template](assets/templates/repo-profile.md)); + - `uts-infra-design.md`: the gap table, the harness design and the build log ([uts-infra-design template](assets/templates/uts-infra-design.md)); + - `design-record.md` ([design-record template](assets/templates/design-record.md)), which also holds the decision log; + - `final-report.md` ([final-report template](assets/templates/final-report.md)), with the acceptance checklist ([acceptance-checklist template](assets/templates/acceptance-checklist.md)) pasted into its "Acceptance checklist" section. + +Create each record by copying its template from `assets/templates/` (the repo-profile, uts-infra-design, design-record and final-report templates) to the working-records directory under the same file name, then fill it in. + +Write them at the default location from Phase 1 on, using the language inferred so far for ``; if STOP-2 changes the language or D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. + +The working records are what a later agent reads to maintain or regenerate the skill ([section 10](references/maintenance.md#10-maintenance)), so keep them accurate as you go, not only at the end. + +### 1.5 Intent + +The guide's intent ([introduction](../../docs/translator-skills/writing-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the harness and regenerate. Never hand-fix a pilot test and call the skill done. + +### 1.6 Which document wins + +| Question | Authority | +|---|---| +| What a faithful translation is; how a failure is diagnosed; the `deviations.md` format | [`writing-derived-tests.md`](../../docs/writing-derived-tests.md) | +| What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../../docs/writing-test-specs.md) | +| Integration and proxy tiers | [`integration-testing.md`](../../docs/integration-testing.md), [`proxy.md`](../../docs/proxy.md) | +| Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | +| What the harness and the skill must contain, and why | the guide ([How it relates to the other UTS docs](../../docs/translator-skills/writing-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | +| How and in what order you build them | this skill | +| The SDK's actual API | the SDK source (guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout), "The notes are a map, not an authority") | +| Questions none of the above answer, as a last resort only | The reference implementations, under [2.1](#21-your-inputs): never an authority | + +If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. A disagreement between a reference implementation and the guide or the UTS docs is never a STOP-11: follow the guide, and record the disagreement with the read ([2.1](#21-your-inputs)). + +**Procedure recommendations.** Where this skill adds detail the guide doesn't have (for example: the working-records files and their location, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this skill's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). + +## Phases at a glance + +Before starting a phase, read its reference file in full: this table is a map, not enough to act on. On resuming an interrupted run, read the working records first, then the reference of the phase you are in. + +| Phase | Goal | Read first | Writes | Ends at | +|---|---|---|---|---| +| 1. Understand the repo | A confirmed Repo profile: how the SDK is built, tested and mocked today | [references/phase-1-understand-repo.md](references/phase-1-understand-repo.md) | `repo-profile.md` (repo-profile template) | STOP-2 | +| 2. Design the harness | Reuse, wrap, extend or build each capability; hooks; placement; tier feasibility | [references/phase-2-design-harness.md](references/phase-2-design-harness.md) | `uts-infra-design.md` (uts-infra-design template) | STOP-3 | +| 3. Build and verify the harness | The approved harness, with green per-tier smoke tests and self-tests, wired into CI | [references/phase-3-build-harness.md](references/phase-3-build-harness.md) | harness code and README; build log | STOP-7 | +| 4. Skill design decisions | Decisions D-01 to D-26, confirmed | [references/phase-4-design-record.md](references/phase-4-design-record.md) | `design-record.md` (design-record template) | STOP-8 | +| 5. Generate the skill files | Mapping, resolver, audit, construct catalogue, module notes, `SKILL.md` | [references/phase-5-generate-skill.md](references/phase-5-generate-skill.md) | the `uts-to-` skill | (STOP-9 if constructs are unmapped) | +| 6. Validate the skill | Resolver, audit mutations, pilots, other paths, lint, examples | [references/phase-6-validate-skill.md](references/phase-6-validate-skill.md) | pilot tests; results | (STOP-10 at a bound) | +| 7. Final report and handover | Report and acceptance checklist | [section 9](#9-phase-7-final-report-and-handover) below | `final-report.md` (final-report and acceptance-checklist templates) | — | + +## 9. Phase 7: final report and handover + +1. Write `final-report.md` from the [final-report template](assets/templates/final-report.md). +2. Fill the [acceptance checklist](assets/templates/acceptance-checklist.md): tick each item, or explain why it isn't met (scoped out, planned, not applicable). +3. Show the user the report, the list of changed and new files (harness, harness tests, hooks, skill, pilot tests), and the suggested CI changes. The run is complete only if the harness tests are green, and in CI, at every tier the skill supports; otherwise the report says which tiers aren't and why. **Don't commit** unless asked. +4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. + +The acceptance checklist maps each item of the guide's [section 13 checklist](../../docs/translator-skills/writing-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill) to where this procedure produces and verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). + +## Maintenance runs + +When the user asks to update, regenerate or re-validate an existing `uts-to-` skill or its harness (the guide, the UTS docs, the corpus, the harness or the SDK changed), read [references/maintenance.md](references/maintenance.md) first, then the working records. The ground rules and stop points above apply unchanged. + +## Reference index + +| File | Read it when | Sections | +|---|---|---| +| [references/phase-1-understand-repo.md](references/phase-1-understand-repo.md) | Starting Phase 1 | 3 | +| [references/phase-2-design-harness.md](references/phase-2-design-harness.md) | Starting Phase 2 | 4 | +| [references/phase-3-build-harness.md](references/phase-3-build-harness.md) | Starting Phase 3 | 5 | +| [references/phase-4-design-record.md](references/phase-4-design-record.md) | Starting Phase 4; mapping a guide change to decisions (maintenance) | 6 | +| [references/phase-5-generate-skill.md](references/phase-5-generate-skill.md) | Starting Phase 5; listing corpus harness symbols in Phase 2 (7.4) | 7 | +| [references/phase-6-validate-skill.md](references/phase-6-validate-skill.md) | Starting Phase 6 | 8 | +| [references/maintenance.md](references/maintenance.md) | A maintenance run | 10 | +| [assets/templates/repo-profile.md](assets/templates/repo-profile.md) | Copy to the working-records directory at the start of Phase 1 | — | +| [assets/templates/uts-infra-design.md](assets/templates/uts-infra-design.md) | Copy at the start of Phase 2 | — | +| [assets/templates/design-record.md](assets/templates/design-record.md) | Copy at the start of the run (it holds the decision log from Phase 1 on) | — | +| [assets/templates/final-report.md](assets/templates/final-report.md) | Copy in Phase 7 | — | +| [assets/templates/acceptance-checklist.md](assets/templates/acceptance-checklist.md) | Fill into the Final report in Phase 7 | — | +| [references/glossary.md](references/glossary.md) | A term is unclear | — | +| [references/failure-modes.md](references/failure-modes.md) | A harness check, a pilot test or a command fails in a way you don't recognise | — | + +The templates are copied, not linked: they contain no links, so they stay valid in the SDK repo. + +## Scripts + +All read-only, Python 3, printing to stdout; run them from anywhere as `python3 /scripts/.py`. + +| Script | Use | Output | +|---|---|---| +| `spec_clone_info.py [SPEC_CLONE]` | Locate, validate and pin the spec clone ([2.6](#26-pin-the-spec-clone)). Discovery order: the path the user gave; `UTS_SPEC_CLONE`; the clone this skill is symlinked from | One JSON object: `specClone`, `foundBy`, `sha`, `dirty`, `dirtyFiles`, `guideLastChange`, `skillDir`, `skillRealpath`, `skillVersion`, `skillMatchesClone`, `warning` (if any), `paths` (guide, UTS docs, helper specs); or `ok: false` with a `code` | +| `survey_repo.py [ …]` | A first pass over the target repo for the discovery checklist ([3.2](references/phase-1-understand-repo.md#32-step-1b-answer-the-discovery-checklist)) | HEAD and status, extension counts, test roots, submodules, candidate mocks, `UTS:` tags, existing skills | +| `scan_constructs.py /uts/` | List the corpus's pseudocode constructs and harness calls ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); copy it into the generated skill's `scripts/` (D-19) | Keyword and snake_case call counts | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md new file mode 100644 index 000000000..d3e424d5b --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md @@ -0,0 +1,60 @@ + + +| Guide §13 item | Produced in | Verified in | Status | +|---|---|---|---| +| **Harness** | | | | +| SDK test hooks (5), clock covering blocking waits and async-runtime timers, injected at construction | 4.6; 5.2 rows 3–4 | 5.2 acceptance; G-01…G-05 | | +| Shared test library per helper spec (incl. legacy `queue_*` and `mock_http.captured_requests`; `mock_websocket.md` async close, CONNECTED ordering, raw frames, alternative API and undocumented members; vcdiff; test pool; `MockNetworkListener`) | 4.3; 5.2 rows 5–6, 8, 11 | smoke tests; G-06…G-10 | | +| Wait helpers (latched, fail-fast `AWAIT_STATE`; value-returning `poll_until`; `poll_until_success`; `process_pending_events`; wall-clock wrapper; fake-time-safe deadlines) | 4.4; 5.2 row 7 | helper tests; G-11 | | +| Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | +| Harness designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; recommended harness layout; fixture-helper scope documented | 4.2; 4.7; 5.3; notes item 9 | G-13 | | +| Harness README with a Known gaps section (MUST) | 5.3; 5.2 row 12 | G-14; 8.5 path 9 | | +| Harness smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.5 path 8 | | +| `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | +| `ProxyManager` (pinned, every OS, verified download, health check, port handling); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | +| One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | +| Concurrency, time, parallelism, visibility models documented | Phase 1 (P-09 to P-14) | STOP-2; G-18 | | +| **Skill files** | | | | +| `SKILL.md` frontmatter portable across Claude Code and Codex: `name` equal to the directory; pushy third-person description of at most 1,024 characters, no `<` or `>`, a "Not for" boundary; string-valued `metadata`; Claude Code-only fields only if accepted as non-portable | 7.6; D-22; D-26 | 7.6 acceptance; 8.6 | | +| `SKILL.md`: usage guard that doesn't rely on `$ARGUMENTS`; own files referred to relative to the skill directory, in `SKILL.md` and the scripts; placeholder paths; 3.4 outline incl. the Harness reference | 7.6 | 7.6 acceptance; 8.3; 8.5 paths 1 and 10 | | +| Installed for both tools from one source (`.claude/skills/` and `.agents/skills/`, by in-repo symlink or a CI-checked copy) (SHOULD) | D-01; 7.1 | 8.3; 8.5 path 10 | | +| `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked, `harness` entry | 7.1 | 7.1 acceptance; 8.1 | | +| `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries, `harness` output | 7.2 | 8.1 | | +| `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | +| Module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | +| Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | +| **Workflow** | | | | +| Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | +| Steps 0 and A–F: the four questions and the harness preflight (stop on red) | 7.6 | 8.3 pilot (green preflight first); 8.5 paths 8–9 | | +| Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 8.3 pilot; 8.8 | | +| Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 8.3 pilot; 8.5 (re-sync) | | +| Re-sync mode; fix the cause, then regenerate | 7.6; D-19 | 8.5 (re-sync); 8.4 (fix and regenerate) | | +| Mid-run harness changes re-run the smoke tests and self-tests and are reported separately | 7.6 (Harness reference) | 8.4 step 2; final report | | +| **Translation rules in `SKILL.md`** | | | | +| One verbatim `UTS: ` per test; no invented tags; spec point in a collectable name | 7.6; D-04, D-06 | 8.2, 8.3 | | +| File header: source spec, tier, spec SHA, disclosures, corresponding unit spec | 7.6; D-07 | 8.3 | | +| Verbatim comments, spec variable names, section headings, spec order | 7.6 | 8.3 review | | +| Annotate, never drop, every `ASSERT`/`AWAIT`; never substitute a similar mock outcome | 7.6; 7.4 | 8.3 audit | | +| Policies for sub-cases, groupers, test-case tables, delegating tests, ID-less tests | 7.6; D-13 | 8.3 (where the pilot has them) | | +| Protocol variants parameterised; tag singular; proxy JSON only | 7.6; D-16 | 8.3 integration pilot | | +| Clients closed in teardown; mocks restored; stop on a failed wait | 7.6; D-15 | 8.3 | | +| Unique channel names; own proxy session per test | 7.6 | 8.3 integration/proxy pilots | | +| Unit: fake time and flushes, no real timers. Integration: wall-clock | 7.6; 7.4; D-17 | 8.3 | | +| Record-and-verify for transient states; polls return the settled value | 7.6; 7.4 | 8.3 | | +| No accommodate-both assertions; `IS ` per typing discipline | 7.6 | 8.3 review | | +| Public-API specs stay public; internal-access ladder with escalation stop | 7.6; 7.5 item 8; D-12 | 8.3 | | +| Construct table filled for every row | 7.4 | 7.4 acceptance | | +| **Evaluation** | | | | +| Three end states; runtime skip and fail-fast idioms; `RUN_DEVIATIONS` command | 7.6; D-09, D-10 | 8.3 (evaluate) | | +| `deviations.md` location; format deferred; written at the right time | 7.6; D-08 | 8.3 (evaluate) | | +| Harness stand-ins in comments, not `deviations.md` | 7.4; 7.6 | 8.3 review | | +| Language-inapplicable inputs (cases a–c) noted in the test file | 7.6; D-11 | 8.3 (where they occur) | | +| Stopping rules and spec-error escalation | 7.6; D-21 | 8.4 | | +| **Verification** | | | | +| Compile (or collect), lint, CI-strict compile, filtered run | 7.6 | 8.3, 8.6 | | +| No commits by default; changed files listed | 7.6; D-20 | Phase 7 | | +| Final report format | 7.6 | 8.3 (the skill's report) | | +| The skill's own examples compile | 7.5 item 12; 7.6 | 8.7 | | +| **Creation process** | | | | +| Skill and harness created on an Opus-class model (MUST); model stated for runs, or pinned where the tool supports that (SHOULD), and recorded in the skill's final report | 2.8; D-26; 7.6 (frontmatter, final report) | Design record "Inputs at generation time" (Model line); 8.3 (pilot model recorded); Final report header | | +| Reference-implementation reads, if any, last resort only, recorded and reported as guide gaps | 2.1; design-record template | Final report "Reference-implementation reads" | | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md new file mode 100644 index 000000000..82da43241 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md @@ -0,0 +1,54 @@ +# Design record: uts-to- + +## Inputs at generation time +- Spec clone: (clean | dirty: ); guide/skill: ; uts-to-lang-skill-creator version +- Repo: +- Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: +- Harness README: +- Model: (Opus-class: yes | no, departure in decision log #); sub-agents: , or none + +## Decisions +| ID | Decision | Choice | Guide-default? | Decided by | Rationale | +|---|---|---|---|---|---| +| D-01 | Skill name and location | | yes / no | user / agent | | +| D-02 | Script language and runtime | | | | | +| D-03 | Mapping: module × tier → target dir (hand-maintained entries marked) | (table below) | | | | +| D-04 | Naming and collection rules (with three worked examples) | | | | | +| D-05 | Collision handling | | | | | +| D-06 | Tag syntax | | | | | +| D-07 | File header format | | | | | +| D-08 | deviations.md location(s) | | | | | +| D-09 | Runtime skip idiom; RUN_DEVIATIONS command | | | | | +| D-10 | Fail-fast idiom | | | | | +| D-11 | Pending idiom for missing APIs | | | | | +| D-12 | Internal-access ladder; white-box spec list location | | | | | +| D-13 | Delegation policy | | | | | +| D-14 | Parallelism policy | | | | | +| D-15 | Teardown; suite-level fixtures | | | | | +| D-16 | Protocol-variant rendering | | | | | +| D-17 | Default timeouts and poll interval | | | | | +| D-18 | Auth through the proxy | | | | | +| D-19 | Re-sync flag; manifest; corpus scanner shipped? | | | | | +| D-20 | Commit policy; project/solution-file edits (approved / left to the user) | | | | | +| D-21 | Fix-attempt bound | | | | | +| D-22 | allowed-tools; remote access | | | | | +| D-23 | Notes files per module (full / placeholder) | | | | | +| D-24 | Working-records location | | | | | +| D-25 | Pilot specs and mode per tier | | | | | +| D-26 | Model tier and pinning for skill runs | | | | | + +### D-03 mapping +| Module | unit | integration | proxy | notes | hand-maintained? | +|---|---|---|---|---|---| + +## Decision log +| # | Date | Phase | Question | Options | Answer | By | Supersedes | +|---|---|---|---|---|---|---|---| + +## Reference-implementation reads (last resort, 2.1) +| # | Date | Question | Guide/UTS sections searched | Repo @ commit : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | +|---|---|---|---|---|---|---|---| + +## Changelog (fixes made to the skill, notes or harness) +| # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Tests regenerated | +|---|---|---|---|---|---| diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md new file mode 100644 index 000000000..e7d48702b --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md @@ -0,0 +1,51 @@ +# Final report: uts-to- for + +- Spec clone (clean | dirty: ); repo ; date +- Model: (Opus-class: yes | no, and why); sub-agents: , or none +- Scope: ; out of scope and why + +## Files created or changed +| Path | New / changed | Kind (hook, harness, smoke test, skill, script, notes, pilot test, CI suggestion) | Approved at | +|---|---|---|---| +Hand-maintained project files the user must update: … + +## Repo profile +Link to repo-profile.md; corrections made at STOP-2. + +## Harness +Gap table status after this run (rows reused, wrapped, extended, built, planned, open); option chosen at STOP-3; hooks added (with approvals); smoke-test and self-test results per tier, and their CI jobs; preflight results from the pilot runs; harness changes made during Phase 6 (with the self-tests added); the README's Known gaps; the plan for any remaining rows. Link to uts-infra-design.md and the harness README. + +## Design decisions +Link to design-record.md; decisions that depart from the guide's defaults, and why. + +## Validation results +- Resolver: rest / realtime / objects → ok, spec counts; negative cases → codes +- Audit self-test: each mutation → result; corpus sweep → N specs, 0 crashes, unverifiable specs listed +- Lint: …; CI-strict compile: …; examples verified: N of N (how) + +## Pilot results +(each pilot run's full section-11 report as the skill printed it: the header line, the table row, and every labelled line, including not-applicable omissions, missing APIs, mock-capability gaps, harness stand-ins and constructs it couldn't map; the table below collects the rows) +| Spec file | Test file | IDs (spec/test) | Audit | Shortfall accounted | Compile | Run | Deviations added | +|---|---|---|---|---|---|---|---| +Fixes made to the skill, notes or harness during the pilot (from the changelog): … +Reference tests named in SKILL.md: + +## Known limitations +Tiers not ready; `not available` catalogue rows; harness capabilities missing; audit SHOULD items not implemented; platforms the scripts weren't run on. + +## Deviations and spec issues found +- SDK deviations (recorded in deviations.md): … +- Suspected UTS spec errors or gaps: — ; draft fix: … +- Spec-repo doc issues (contradictions between guide and UTS docs; guide gaps revealed by reference-implementation reads): … + +## Reference-implementation reads +Each read from the Design record (question, repo @ commit : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. + +## Suggested CI changes + (not applied unless approved at STOP-5) + +## Next steps +Which modules and tiers to translate next, in order, and why; harness work remaining. + +## Acceptance checklist +(the acceptance-checklist template, filled) diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md new file mode 100644 index 000000000..d769f70f7 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md @@ -0,0 +1,63 @@ +# Repo profile: + +- Repo: , HEAD (clean | dirty: ) +- Spec clone: , HEAD (clean | dirty: ); guide/skill last changed in ; skill version +- Date: ; confirmed by the user on (STOP-2) +- User goals: + +## Build, test and CI +| # | Item | Answer | Evidence (path, symbol or command → result) | Verified? | +|---|---|---|---|---| +| P-01 | Language(s) and version(s) | | | | +| P-02 | Build and test commands (build, build tests, all, class, single test, filter, target-qualified filter) | | | ran / unverified: | +| P-03 | Test framework, runner, assertions, parameterisation, fixtures, teardown, static and runtime skip, fail idiom, collection rules | | | | +| P-04 | Test layout, naming, targets, shared test-support code | | | | +| P-05 | CI workflows, jobs → suites, matrix, network, submodules, secrets | | | | +| P-06 | Lint/format gates, static analysis, type checkers, warnings-as-errors | | | | +| P-07 | Developer and CI platforms; script runtime availability | | | | +| P-08 | Repo conventions; hand-maintained project files; synced manifests | | | | + +## SDK structure for testing +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-09 | Async model; runner async support; virtual time | | | | +| P-10 | Threading, callback queues, internal-queue drain, compile-time concurrency checks | | | | +| P-11 | Every time source (timers, timed waits, delayed callbacks, clock reads) | | | | +| P-12 | Runner parallelism and how to serialise | | | | +| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global) | | | | +| P-14 | Internal-access mechanism and current use | | | | +| P-15 | API surface vs pseudocode, per module (rest / realtime / objects) | | | | + +## Existing native test support +| # | Item | Answer | Evidence | Verified? | +|---|---|---|---|---| +| P-16 | Existing mocks, fakes, test doubles (capabilities, style, global?, users) | | | | +| P-17 | Existing test helpers (waits, polls, capture, log capture, assertions, factories) | | | | +| P-18 | Existing sandbox/integration support; fault-injection tooling | | | | +| P-19 | ably-common submodule (path, initialised, fixture files) | | | | +| P-20 | Existing UTS assets and their state | | | | +| P-21 | Sandbox access from developer machines and CI | | | | + +## How the existing tests drive the SDK (step 1c) +| Kind | Test read (path, test name) | Pattern: create client / install double / wait / assert / clean up | +|---|---|---| +| REST request | | | +| Realtime connection state | | | +| Timer-driven behaviour | | | +| Negative assertion | | | +| Internal state | | | +| Sandbox integration | | | +| Plugin module | | | + +- Doubles that already meet helper-spec semantics: … +- Doubles that differ (and how): … +- Race-free waits vs sampling or sleeping waits: … +- Known flakiness and workarounds: … + +## API divergences per module +- rest: … +- realtime: … +- objects: … (implemented? plugin? typed? IDL followed?) + +## Unknowns and questions for the user +- … diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md new file mode 100644 index 000000000..73b3f29d0 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md @@ -0,0 +1,66 @@ +# Harness design: @ + +## Gap table +| # | Capability | Guide | Status | Evidence | Approach (reuse / wrap / extend / build / n/a) | Builds on | Blocks (module/tier) | Production change? | Size | Acceptance check | +|---|---|---|---|---|---|---|---|---|---|---| +| G-01 | WebSocket transport factory hook | 2.1 | present / partial / missing / n/a | | | | realtime/unit, objects/unit | yes / no | S/M/L | | +| G-02 | HTTP client hook | 2.1 | | | | | rest/unit | | | | +| G-03 | Clock hook (timers, timed waits, async-runtime timers) | 2.1 | | | | | | | | | +| G-04 | Network-monitor hook | 2.1 | | | | | | | | | +| G-05 | Randomness hooks (jitter, host shuffle) | 2.1 | | | | | | | | | +| G-06 | MockHttpClient (+ legacy queue_*) | 2.2, mock_http.md | | | | | | | | | +| G-07 | MockWebSocket (+ alternative API, undocumented members, raw frames, ping) | 2.2, mock_websocket.md | | | | | | | | | +| G-08 | MockVCDiff encoder/decoders | 2.2, mock_vcdiff.md | | | | | | | | | +| G-09 | MockNetworkListener | 2.2 | | | | | | | | | +| G-10 | standard_test_pool.md implementation; plugin options builder | 2.2, standard_test_pool.md | | | | | | | | | +| G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | +| G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | +| G-13 | Placement and recommended harness layout (shared / port-only / module helpers); fixture-helper scope documented in the module notes | 2.2 | | | | | | | | | +| G-14 | Harness README with Known gaps; mapping `harness` entry (root, README, per-tier sources and harness-test commands) | 3.1, 3.2 | | | | | | | | | +| G-15 | SandboxApp; ably-common | 2.3 | | | | | */integration | | | | +| G-16 | ProxyManager (pinned, every OS); ProxySession; rule builders; proxy auth | 2.4 | | | | | */proxy | | | | +| G-17 | UTS test target; per-tier selection and filters; one CI home per suite | 2.5 | | | | | | | | | +| G-18 | Concurrency, time, parallelism and visibility models known | 2.6 | | | | | | | | | +| G-19 | Per-tier smoke tests (permanent, in CI, outside generated dirs) | 2.7 | | | | | | | | | +| G-20 | Helper self-tests (common, unit, fake clock, REST, objects, sandbox, proxy) | 2.7 | | | | | | | | | + +## Helper-spec symbol maps (one table per helper spec) +### mock_http.md +| Spec symbol (declared or corpus-only) | Native type.member | Approach | Builds on | Notes | +|---|---|---|---|---| +### mock_websocket.md +### mock_vcdiff.md +### standard_test_pool.md +### MockNetworkListener (no helper spec) + +## Wait and assertion helpers +| Helper | Native symbol | Approach | Guarantees checked by | +|---|---|---|---| + +## Sandbox and proxy +- Sandbox provisioning: …; client targeting (endpoint or hosts): …; suite fixture mapping: … +- Proxy: version …; binaries per OS …; local override …; auth through the proxy …; platform gating … + +## Hook proposals (each needs STOP-4 approval) +| Hook | API | Default behaviour | Visibility | Files touched | Installed how | Per-client? | Approved on | +|---|---|---|---|---|---|---|---| + +## Placement +- Shared test-support: ; port-only harness: ; module helpers: + +## Tier feasibility +| Module | unit | integration | proxy | +|---|---|---|---| +| rest | ready / after G-.. / not possible: / n/a | | | +| realtime | | | | +| objects | | | | + +## Decision at STOP-3 +(a) build now: | (b) plan only | (c) scope to ready tiers — chosen by on ; design changes requested: … + +## Build log (Phase 3) / plan (option b) +| Order | Row | Work | Acceptance check | Result | Date | +|---|---|---|---|---|---| + +## Confirmation at STOP-7 +Smoke tests and self-tests per tier (filter, result, CI job): …; updated tier matrix: …; confirmed on diff --git a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md new file mode 100644 index 000000000..9f8e384db --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md @@ -0,0 +1,34 @@ +# Common failure modes and remedies + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +All from the guide and the UTS docs; follow the link for the detail. + +| Symptom | Likely cause | Remedy | Source | +|---|---|---|---| +| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the distinct collected test functions (a parameterised test counted once) with the tag count | guide [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci), [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability) | +| Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | +| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | +| Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must) | +| A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | +| `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue) | +| A state wait runs out the timeout after the connection failed | No fail-fast on FAILED | Fail immediately on FAILED, except when the test just requested a transition out of it | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | +| Assertions after a poll see fewer items | The test re-reads after polling an eventually-consistent store | Use the value the poll returned | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | +| `poll_until_success` times out with no useful error | Emulated with a reader that swallows errors | A separate helper that rethrows the last error | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events); `uts/README.md` | +| Negative assertions pass or fail at random | One zero-delay yield doesn't drain chained callbacks | Drain until the internal queue is empty; or the quiescence control-listener | guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must); `standard_test_pool.md` "Negative-assertion quiescence" | +| Tests pass alone, fail together | Process-global hooks with a parallel runner; leaked clients or timers | Serialise or make hooks per-client (D-14); close clients and restore mocks in teardown | guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must), [5.4](../../../docs/translator-skills/writing-translator-skills.md#54-setup-and-teardown) | +| State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | +| An objects test's operation is silently ignored | Hand-written serial literal sorts before the pool serial | Use the serial helpers | `standard_test_pool.md`; guide 2.2 | +| Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | +| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions"; guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | +| Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | +| Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | +| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning"; guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | +| Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | +| Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](../../../docs/translator-skills/writing-translator-skills.md#82-audit-audit_translationpy-must) | +| Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | +| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | +| Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | +| A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](../../../docs/translator-skills/writing-translator-skills.md#5-translation-rules) | +| Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](../../../docs/translator-skills/writing-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/glossary.md b/uts/skills/uts-to-lang-skill-creator/references/glossary.md new file mode 100644 index 000000000..4e54dcfff --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/glossary.md @@ -0,0 +1,35 @@ +# Glossary + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +| Term | Meaning | +|---|---| +| **UTS** | Universal Test Suite: language-neutral pseudocode test specs in `uts/` of the spec repo | +| **Module** | A top-level corpus directory: `rest`, `realtime`, `objects` | +| **Tier** | `unit` (mocked transports), `integration` (direct sandbox), `proxy` (sandbox through `uts-proxy`), detected by path | +| **Spec** | One Markdown file of UTS tests; **features spec** means the Ably specification under `specifications/` | +| **Test ID / tag** | The `**Test ID**` of a spec test; the `UTS: ` comment above the derived test | +| **Helper spec** | A spec of harness behaviour (`mock_http.md`, `mock_websocket.md`, `mock_vcdiff.md`, `standard_test_pool.md`); implemented once, never translated | +| **Hook** | An injection point in SDK production code that the harness uses (transport, HTTP, clock, network monitor, randomness) | +| **Harness (UTS test infrastructure)** | The native test library implementing the helper specs, wait helpers, sandbox and proxy helpers and fixtures | +| **Reuse / wrap / extend / build** | The four ways to provide a harness capability from what the repo has ([4.2](phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)) | +| **Shared / port-only** | Harness code used by native tests and UTS ports alike / used only by UTS ports | +| **Smoke test** | A per-tier harness test that drives one tier end to end through the real hooks (guide 2.7); permanent, in CI, not spec-derived, no tag, never a reference test | +| **Self-test** | A harness test that checks one helper obeys its helper-spec contract (guide 2.7); permanent, in CI, no tag | +| **Preflight** | The generated skill's step F: compile, run the tier's smoke tests and self-tests, check Known gaps; stop on red | +| **Reference test** | A reviewed, spec-derived test per tier that the skill reads before generating | +| **Reference implementations** | The existing `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java) skills and their harnesses, at the commits the guide pins: a last-resort, read-only reference for patterns, never copied and never an authority ([2.1](../SKILL.md#21-your-inputs)). Not a reference test | +| **Opus-class model** | The most capable model tier available (Claude Opus or an equivalent); required for this procedure and its sub-agents ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) | +| **Notes** | `references/-mapping.md`: how ably-js-shaped pseudocode maps to this SDK for one module | +| **Resolver** | `resolve_uts.py`: validates a module and derives targets and names; its output is the single source of truth | +| **Audit** | `audit_translation.py`: compares a spec with its test file (IDs, ledger, counts) | +| **Ledger / shortfall** | The per-test list of spec lines by kind / spec count minus test count, for assertions and waits separately | +| **Translate-only / evaluate** | Generate and statically check / also run and diagnose | +| **Deviation** | An SDK behaviour that differs from a correct spec, recorded in `deviations.md` | +| **Stand-in** | A harness-level difference in how a test drives the SDK; disclosed in a comment, not a deviation | +| **White-box spec** | A unit spec that drives internal APIs; listed in the notes | +| **Delegating spec/test** | One that says to run another module's tests against this client | +| **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | +| **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | +| **Phase N / generated-skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Generated-skill Phase 1" (selection, steps 0 and A–F) and "generated-skill Phase 2" (per spec, steps 1–7) are the generated `uts-to-` skill's workflow (guide [4](../../../docs/translator-skills/writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | +| **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | diff --git a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md new file mode 100644 index 000000000..29c2ecc60 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md @@ -0,0 +1,21 @@ +# 10. Maintenance + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +The harness and the skill must stay in step with five things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. + +| What changed | How to detect it | What to update | +|---|---|---| +| **The guide** | `git -C diff -- uts/docs/translator-skills/ uts/skills/uts-to-lang-skill-creator/` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](phase-4-design-record.md#61-step-4a-draft-the-design-record) and [7.6](phase-5-generate-skill.md#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | +| **The UTS docs or helper specs** | `git -C diff -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | +| **The UTS corpus** | The skill's own re-sync mode (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | +| **The SDK** (API, hooks, test layout, CI) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | + +Rules for every maintenance run: + +- Follow the same ground rules ([section 2](../SKILL.md#2-ground-rules)) and stop points, including the model tier ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](../SKILL.md#21-your-inputs)). If the guide's "as of" commits have moved since a recorded read, re-check that read's conclusion against the updated Patterns to avoid before relying on it again. +- Start from the working records; update the Repo profile and the harness design if anything they describe has changed. +- Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should)). +- After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](phase-6-validate-skill.md#81-run-the-resolver-on-every-module) and [8.2](phase-6-validate-skill.md#82-test-the-audit-itself). +- Record the new spec SHA and repo SHA in the Design record and append a changelog entry. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md b/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md new file mode 100644 index 000000000..83d24bc68 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md @@ -0,0 +1,77 @@ +# 3. Phase 1: understand the repo + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +**Goal:** a written, user-confirmed Repo profile that explains how the SDK is built, tested and mocked today, in enough depth to design the harness from it. This phase is mandatory and comes before anything else: the harness you design in Phase 2 should extend what the repo already has, follow its test style, and plug into its build and CI, and the skill's file locations, idioms and commands all derive from this profile. + +## 3.1 Step 1a: read the repo's own instructions + +Read these first, if they exist, and follow them for the rest of the run: `CLAUDE.md` (root and nested), `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `.editorconfig`, any `docs/` or `Docs/` pages about testing, and any existing `.claude/`, `.agents/` or `.codex/` directory (settings, skills, commands). Record any instruction that constrains you (for example "don't edit project file X", "use helper Y instead of Z"). + +## 3.2 Step 1b: answer the discovery checklist + +Answer every item. For each, record the answer and its evidence in the Repo profile. The "how to find it" column lists starting points; follow the evidence wherever it leads. + +**Build, test and CI** + +| # | Find | How to find it | +|---|---|---| +| P-01 | **Language(s) and version(s)**: SDK language, test language (they can differ), toolchain version, language mode or edition | Toolchain pins (`global.json`, `.python-version`, `.nvmrc`, `go.mod`, `rust-toolchain*`, `Package.swift` tools version, Gradle `jvmToolchain`); `git ls-files` extension counts | +| P-02 | **Build system and commands**: build the SDK; build the tests; run all tests; run one class; run one test; run by filter; target-qualified filters | Build files (`*.sln`/`*.csproj`, `pyproject.toml`/`tox.ini`/`noxfile.py`, `package.json` scripts, `build.gradle*`, `Package.swift`, `Makefile`, wrapper scripts); CI workflow steps; CONTRIBUTING. **Run** the build-tests and single-class commands and record the exit status. For interpreted languages, "build the tests" means collection (e.g. `pytest --collect-only`) plus the repo's type checker. Record the command that lists collected tests (e.g. `pytest --collect-only -q`, `dotnet test --list-tests`) | +| P-03 | **Test framework(s)**: runner, assertion library, parameterisation, suite-level fixtures (before/after all), teardown hooks, **static and runtime skip idioms**, unconditional-fail idiom, **collection rules** (what names and files the runner picks up) | Test dependencies in the build files; existing tests; the runner's config (`pytest.ini`, `conftest.py`, `xunit.runner.json`, `jest.config.*`). Check whether a runtime skip needs an extra package (guide [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states)) | +| P-04 | **Test layout and naming**: test projects/targets/source sets, their directories, naming conventions for files, classes and methods, shared test-support projects, and which test targets can see which code | Directory listing of test roots; build files for test targets | +| P-05 | **CI**: every workflow that builds or tests; which job runs which suites; OS and runtime matrix; network access; submodule checkout; secrets; how a new suite or test target would be picked up | `.github/workflows/*`, other CI configs; follow each test step to the command it runs | +| P-06 | **Lint, format and strictness gates**: EditorConfig, formatters, linters, static analysis, type checkers; whether CI treats warnings as errors; whether the local build is laxer than CI | Lint configs, `Makefile` lint targets, CI lint jobs, compiler flags in build files | +| P-07 | **Developer and CI platforms**: which OSes developers and CI use; whether the script runtime the skill will use (e.g. `python3`) is available on all of them; where a proxy binary would have to run | CONTRIBUTING, CI matrix, existing scripts | +| P-08 | **Repo conventions**: licence or file headers, comment style, line endings, commit message rules, files kept in sync (e.g. several dependency manifests), project files not discovered by the build | CLAUDE.md, CONTRIBUTING, `.editorconfig`, `.gitattributes`, recent commits | + +**How the SDK is structured for testing** + +| # | Find | How to find it | +|---|---|---| +| P-09 | **Async model**: callbacks, futures/promises, `async`/`await`, coroutines; whether SDK APIs are callback-based, async, or both; the test runner's async support and its mode; whether the runner **virtualises time** | SDK public API signatures; runner plugins (e.g. pytest-asyncio mode); existing async tests (guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)) | +| P-10 | **Concurrency and threading**: which thread, queue or loop SDK callbacks run on; whether listeners can run concurrently with the test body; how to drain the SDK's internal work queue (this becomes `process_pending_events()`); whether the language checks concurrency safety at compile time | SDK source: dispatch, executor or event-loop code; how existing tests wait for callbacks | +| P-11 | **Time model**: **every time source** the SDK uses: timers, timed blocking waits, scheduled or delayed callbacks, wall-clock and monotonic reads | Grep the SDK source for the language's sleep, timer, wait, delay and now APIs; list each call site's owner (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must)) | +| P-12 | **Runner parallelism**: are tests run in parallel by default, and how is it turned off per suite? | Runner docs and config; existing serialisation attributes or collection settings | +| P-13 | **Injection points (test hooks)** for each of: WebSocket transport, HTTP client, clock/timers, reachability/network monitor, randomness (retry jitter, fallback-host shuffle). For each: symbol, how it is set (option, constructor argument, global, subclass), whether it is set at client construction, per-client or process-global, test-only or public | SDK client-options and any debug/test-options types; constructors of the transport, HTTP, timer and network layers; existing tests that replace them | +| P-14 | **Internal-access mechanism**: how tests reach non-public code, and where the repo already uses it | e.g. `InternalsVisibleTo`, `@testable import`, private headers and module maps, same-package placement, `_`-prefixed conventions; grep for existing test-only accessors | +| P-15 | **SDK API surface vs the pseudocode, per module**: client constructors and options, async return types, the error type and how `ErrorInfo` fields (`code`, `statusCode`, `message`, cause) are read, enum renderings, protocol-format option (JSON vs msgpack), `endpoint` vs explicit hosts, the plugin mechanism. For `objects` (LiveObjects): is it implemented, where, as a plugin or built in, typed or dynamic, and which spec IDL it follows | Public API; compare with the pseudocode in two or three specs per module and with `specifications/features.md` / `objects-features.md` in the clone. Produce a short divergence list per module | + +**Existing native test support** + +| # | Find | How to find it | +|---|---|---| +| P-16 | **Existing mocks, fakes and test doubles**: for HTTP, WebSocket/transport, clock/timers, network monitor, randomness, delta decoding, plugins. For each: file and symbol, what it can do (capture requests or frames, inject responses or server messages, simulate refusal, timeout, DNS failure, disconnect), its style (handler/callback, queued responses, await), whether it is process-global, and which native tests use it | Search the test roots for `Mock`, `Fake`, `Stub`, `Test*Transport`, `*Handler`, clock or timer types; follow the hook symbols from P-13 to their test-side implementations | +| P-17 | **Existing test helpers**: state waits, polling helpers, async-to-sync bridges, event capture or recorder types, log capture, custom assertions, client factories, test-options builders | Shared test-support code; base test classes; helpers used by many test files | +| P-18 | **Existing sandbox and integration support**: how native integration tests provision a sandbox app (from `test-app-setup.json`?), point clients at the sandbox, retry, clean up; any proxy or fault-injection tooling | Integration test directories and their fixtures; environment variables they read | +| P-19 | **`ably-common`**: submodule path and name (it may not be called `ably-common`), whether it is initialised, and whether `test-resources/test-app-setup.json`, `msgpack_test_fixtures.json` and `encoding.json` are present | `.gitmodules`; `git submodule status`; list the files | +| P-20 | **Existing UTS assets and their state**: any UTS harness, UTS-derived tests (`UTS:` tags), `deviations.md` files, previous `uts-to-*` skills in this repo. For each asset: does it compile, do its tests pass, which spec SHA do headers name, which helper-spec symbols exist | `grep -rn "UTS:" `; search for `MockWebSocket`, `MockHttpClient`, `SandboxApp`, `uts-proxy`, `deviations.md`, `.claude/skills/uts-*`, `.agents/skills/uts-*`, `.codex/skills/uts-*` | +| P-21 | **Sandbox access**: whether tests can reach `sandbox.realtime.ably-nonprod.net` from developer machines and CI | Existing integration tests; CI network settings (guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)). Probing the sandbox, or running existing integration tests for P-20, is network access: **STOP-6** first | + +For a first pass, run `python3 /scripts/survey_repo.py [ …]` (read-only; it prints the HEAD SHA and dirty state, extension counts, likely test roots, submodule status, candidate mocks and fakes, existing `UTS:` tags and existing skills). It is a starting point, not an answer: adapt or extend the searches for the repo's language, and record each answer's evidence. + +## 3.3 Step 1c: study how the existing tests drive the SDK + +The checklist tells you what exists; this step tells you how it is used. Read, in full, at least one existing native test of each kind below (if the repo has it), and write down the pattern each follows: how the client is created, how the double is installed, how the test waits, how it asserts, how it cleans up. + +| Kind | What to learn from it | +|---|---| +| A REST test that checks an outgoing request | How HTTP is intercepted; how requests are captured and responses injected | +| A realtime test that drives connection state | How the transport is replaced; how server messages are injected; how state changes are awaited | +| A test of timer-driven behaviour (retry, timeout, heartbeat) | Whether time is faked or real, and how it is advanced | +| A test that checks something did **not** happen | How the test flushes pending work before a negative assertion | +| A test that reaches internal state | The internal-access mechanism in practice | +| An integration test against the sandbox | Provisioning, client options, timeouts, cleanup | +| A test from each plugin or optional module (e.g. LiveObjects) | How the plugin is registered in tests; its own test helpers | + +Then answer, in the Repo profile: + +- Which existing doubles behave like the helper specs need (see the semantics in [4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code)), and which differ? +- Which existing waits are race-free (subscribe first, then check), and which sample state or sleep? +- Where do the existing tests flake or carry workarounds (retries, sleeps, `@Ignore`/skip with a reason, comments about races)? These mark concurrency or timing problems the harness must avoid. + +## 3.4 Step 1d: write and confirm the Repo profile + +Write `repo-profile.md` from the [repo-profile template](../assets/templates/repo-profile.md). Record the repo's HEAD SHA and whether it is dirty. Then **STOP-2**: show the user the profile, highlight every "unknown" and every inference, and ask them to confirm or correct it. Apply their corrections before continuing. + +**Done when:** every P-row has an answer with evidence or an explicit "unknown" with a question; the build-tests and single-class commands have been run (or marked unverified with the reason); the existing-test patterns of 3.3 are written down; the user has confirmed the profile. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md b/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md new file mode 100644 index 000000000..ea17a6ab9 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md @@ -0,0 +1,113 @@ +# 4. Phase 2: design the harness + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +Contents: + +- [4.1 Step 2a: list the required capabilities](#41-step-2a-list-the-required-capabilities) +- [4.2 Step 2b: assess each capability: reuse, wrap, extend or build](#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) +- [4.3 Step 2c: map each helper spec to native code](#43-step-2c-map-each-helper-spec-to-native-code) +- [4.4 Step 2d: design the wait and assertion helpers](#44-step-2d-design-the-wait-and-assertion-helpers) +- [4.5 Step 2e: design the sandbox and proxy helpers](#45-step-2e-design-the-sandbox-and-proxy-helpers) +- [4.6 Step 2f: propose the SDK test hooks](#46-step-2f-propose-the-sdk-test-hooks) +- [4.7 Step 2g: plan placement, harness tests and build order](#47-step-2g-plan-placement-harness-tests-and-build-order) +- [4.8 Step 2h: derive tier feasibility](#48-step-2h-derive-tier-feasibility) +- [4.9 Step 2i: STOP-3, approve the design and choose the scope](#49-step-2i-stop-3-approve-the-design-and-choose-the-scope) + +**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](../../../docs/translator-skills/writing-translator-skills.md#2-the-harness-uts-test-infrastructure) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. + +Design principles: + +- **Follow guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)**: implement each helper spec once, and build the harness to look like the pseudocode. When an existing double has the right behaviour but a different shape, **wrap it in a pseudocode-shaped facade** rather than teaching the skill to translate into the old shape. +- **Reuse what is correct; don't reuse what is subtly wrong.** An existing double that misses a helper-spec guarantee (for example a synchronous `close()`) will make the UTS ports flaky. Extend or replace it for the UTS ports, leaving the native tests' behaviour unchanged. +- **Hooks** are injected at client construction, per-client where the SDK allows (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must), [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)). + +## 4.1 Step 2a: list the required capabilities + +Start the gap table from rows G-01 to G-20 of the [uts-infra-design template](../assets/templates/uts-infra-design.md), which cover every requirement of guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must) to [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. + +**Check:** every bullet and table row of guide section 2 maps to a gap-table row. + +## 4.2 Step 2b: assess each capability: reuse, wrap, extend or build + +This is the guide's "Reuse before you build" ([2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)) applied to every row. For each row, record the status (`present`, `partial`, `missing`, `n/a`), the evidence (file and symbol, or what you searched for), and the **approach**: + +| Approach | When | +|---|---| +| **Reuse** | An existing component already meets the requirement's semantics and shape (verified, not assumed) | +| **Wrap** | The semantics are right but the shape differs from the pseudocode; add a facade with the spec's names | +| **Extend** | The component is close but lacks capabilities (e.g. no WS-level timeout, no raw frames); add them without changing existing behaviour | +| **Build** | Nothing suitable exists, or what exists is semantically wrong for UTS | +| **n/a** | The SDK lacks the feature (e.g. `MockVCDiff*` when the SDK has no delta support); confirm against the features spec in the clone and give the reason | + +Also record which tiers and modules each row blocks, whether closing it needs a production-code change, and a size (S/M/L). "Partial" always says what is missing. + +If harness code already exists, **check it, don't trust it**: compile it, run its tests by filter, and compare its behaviour with the semantics in 4.3 and 4.4. + +## 4.3 Step 2c: map each helper spec to native code + +Read each helper spec in full, then design its native implementation. For each, produce a symbol table in `uts-infra-design.md`: every symbol the helper spec declares **and** every harness symbol the corpus uses (find them with the scanner in [7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue); the corpus uses members the helper specs don't declare) → native type and member → approach (reuse, wrap, extend, build) → the existing component it builds on, if any. + +| Helper spec | Design questions to answer | Requirements to check against | +|---|---|---| +| [`mock_http.md`](../../../rest/unit/helpers/mock_http.md) | Which HTTP hook does `install_mock` use? Does an existing HTTP double capture method, URL, headers and body? Can it fail at connection level separately from request level? Will the legacy `queue_*` calls be implemented, or mapped onto `onRequest` handlers in the skill? | The helper spec; guide 2.2 `mock_http.md` row | +| [`mock_websocket.md`](../../../realtime/unit/helpers/mock_websocket.md) | Which transport hook installs it? Does an existing transport double exist, and is its `close()` asynchronous? How are typed messages and raw untyped frames sent to the client? How are client frames captured? | The helper spec (including "Async Behavior and Event Loop Considerations"); guide 2.2 `mock_websocket.md` row and the notes below it | +| [`mock_vcdiff.md`](../../../realtime/unit/helpers/mock_vcdiff.md) | Does the SDK support deltas, and how is a decoder registered? | The helper spec; guide 2.2 | +| [`standard_test_pool.md`](../../../objects/helpers/standard_test_pool.md) | Which channel and plugin setup does `setup_synced_channel` need? Where do module helpers live (where internals are visible, if white-box specs need them)? How is the plugin registered in a client-options builder? | The helper spec; guide 2.2 `standard_test_pool.md` row and "Document each fixture helper's scope" | +| *(no helper spec)* `MockNetworkListener` | Is there a network-monitor hook? | guide 2.2 last table row; `realtime/unit/connection/network_change_test.md` | + +Where a design question can only be answered by a production change, it becomes a hook proposal (4.6). + +## 4.4 Step 2d: design the wait and assertion helpers + +Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the self-tests that prove them (guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. + +| Helper | Requirements | +|---|---| +| `AWAIT_STATE` (connection and channel) | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | +| `poll_until`, `poll_until_success` | `writing-test-specs.md`; `uts/README.md`; guide 5.7; `writing-derived-tests.md` "No real timers in unit tests" (deadlines) | +| `process_pending_events()` | `uts/README.md`; guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | +| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | +| Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must), [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | +| Thread-safe capture, log sink, caller-attributed failures | guide 2.2 | +| `assertContainsInOrder` | guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue) | + +## 4.5 Step 2e: design the sandbox and proxy helpers + +- **Sandbox** (guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). +- **Proxy** (guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../../../docs/proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Include every operational requirement in guide 2.4. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). + +## 4.6 Step 2f: propose the SDK test hooks + +For every hook row that isn't `present`, write a proposal in `uts-infra-design.md`: + +- the API (name, type, where it is set), modelled on the repo's existing test options if it has any (P-13); +- the default behaviour (unchanged for users); +- its visibility (test-only if the language allows); +- every production file it touches; +- how the harness installs it at client construction, and whether it is per-client; +- for the clock hook, the list of time sources from P-11 it covers, including timed blocking waits and the async runtime's timers. + +These proposals are presented at STOP-3 and need individual approval at **STOP-4** before you implement them. + +## 4.7 Step 2g: plan placement, harness tests and build order + +- **Placement** (guide 2.2, "Placement" and "Recommended layout"): name the harness library and its per-tier directories, the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). +- **Harness tests:** the per-tier smoke tests and helper self-tests of guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must), item by item, for every tier you plan to support. They are permanent and run in CI; they carry no `UTS:` tag and are never copied by the skill; they live outside every directory the mapping will give the resolver as a `targetDir`. Name the command that runs each tier's set plus the common self-tests (guide 2.7 "Selectable"); it goes into the mapping's `harness` entry (D-03). Name the CI job that runs it (5.2 row 13). The acceptance checks in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. +- **Build order:** follow the order in 5.2, dropping rows that are `present` or out of scope. + +## 4.8 Step 2h: derive tier feasibility + +Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each of unit, integration, proxy, mark **ready**, **ready after building rows X, Y**, or **not possible** (with the reason, e.g. "not possible until the SDK has an objects API (placeholder notes; the skill refuses the module)"). A tier the corpus doesn't have for a module is `n/a` (check with `ls /uts/`). A tier is **ready** only when its G-19 and G-20 rows (smoke tests and self-tests, guide 2.7) are present, green and wired into CI (or the CI change was declined at STOP-5 and is reported as unmet, so the skill doesn't yet conform to guide 2.5/2.7; see [5.4](phase-3-build-harness.md#54-step-3d-stop-7-confirm-the-harness)), or will be built in Phase 3. + +## 4.9 Step 2i: STOP-3, approve the design and choose the scope + +Present `uts-infra-design.md`: the gap table, the helper-spec mappings, the wait-helper design, the sandbox and proxy design, the hook proposals, placement, the build order and the tier matrix. Ask the user to choose: + +- **(a) Build the missing harness now**, all or a named subset. Recommend this option (the skill depends on the harness), but the user chooses. Continue with Phase 3. +- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4, step C](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) requires (see the readiness data in [7.1](phase-5-generate-skill.md#71-step-5a-create-the-layout-and-the-mapping-file)). +- **(c) Scope the skill to the tiers possible today.** As (b), without the plan. + +Under (b) and (c), G-19 and G-20 for every tier the skill will offer can't be planned or scoped out; Phase 3 and STOP-7 still run for them. + +Record the answer and any design changes the user asked for. Under (b) and (c), the skill's tier step (C) must offer only ready tiers, and the Final report lists the rest as known limitations. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md b/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md new file mode 100644 index 000000000..1ac6edb7a --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md @@ -0,0 +1,56 @@ +# 5. Phase 3: build and verify the harness + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has green smoke tests and helper self-tests (guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)), wired into CI. This is part of the skill's deliverable, not a preliminary. Skip the rows the user didn't choose to build at STOP-3. + +## 5.1 Step 3a: rules while building + +- **Production hooks only after STOP-4 approval**, one proposal at a time, in the smallest form that works. +- **Don't change existing behaviour** of doubles or helpers the native tests use; extend, wrap or add new code instead (see [2.2](../SKILL.md#22-what-you-may-change)). +- Follow the repo's test style (P-03, P-04, P-08): naming, file headers, assertion library, formatting. +- Run the existing tests that touch anything you changed, by filter, before and after the change. +- Log every design change you make while building in the decision log; update `uts-infra-design.md`. + +## 5.2 Step 3b: build in order, with acceptance checks + +Each step has an acceptance check; don't start the next step until it passes (or the user accepts a documented exception). **Bound:** if a row's check still fails after three fix attempts, stop (**STOP-10**) with a summary: the failure, what you tried, your diagnosis, and the options (change the design and log it, accept a documented exception, or descope the tiers the row blocks). + +The smoke tests and self-tests in the checks below are the permanent harness tests of guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) (MUST): build them as this phase's gates, and keep them in the repo and in CI afterwards, where the generated skill's preflight runs them. Each row's check is an explicit assertion in a harness test, not a manual observation. + +Rows 4–7 depend on each other: build them in order, but run an acceptance item that needs a component from a later row as soon as that component exists (the fake-clock check needs a transport mock; the REST row's realtime-driven request needs `MockWebSocket`; the realtime row's state waits need row 7), and don't leave row 7 until every row 4–7 item passes. Common items that need an integration component (the integration state-wait helper, integration teardown) run at row 9, once that component exists; they are part of row 9's acceptance. + +| Order | Build | Acceptance check | +|---|---|---| +| 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal harness test (one trivial passing test; delete it once row 2's self-tests exist) compiles, is collected and passes under the single-class filter and under the tier's harness-test command | +| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken, and a repeated state passes (guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)); capture lists are safe under concurrent appends; a failing helper reports the caller's line | +| 3 | Approved SDK hooks | Each hook replaces the real implementation for one client without affecting another client created afterwards (or, if process-global, that fact is recorded for decision D-14); the existing tests still pass | +| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it, and nothing waits on the real clock (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must)); one advance runs cascaded work to quiescence (zero-delay reschedules and timers created mid-advance) | +| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, REST entry, each as an explicit assertion | +| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, realtime entry, each as an explicit assertion | +| 7 | Wait helpers per [4.4](phase-2-design-harness.md#44-step-2d-design-the-wait-and-assertion-helpers) | **Helper self-tests:** every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Common and Unit, fake clock entries, each as an explicit assertion; in addition, `AWAIT_STATE` completes for a state reached before or during the wait (the teardown self-test follows your SDK's `close()` behaviour, guide 5.4) | +| 8 | `MockNetworkListener` (if the SDK has a network monitor), `MockVCDiff*` (if the SDK supports deltas) | A smoke test drives each one | +| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Direct sandbox entry, each as an explicit assertion | +| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Proxy entry, each as an explicit assertion; clients authenticate as decided in D-18 | +| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, objects entry, each as an explicit assertion. The smoke test lives beside the module helpers (outside every `targetDir`) and is selected by the unit tier's harness command | +| 12 | Harness README describing what exists, with its Known gaps | See [5.3](#53-step-3c-write-the-harness-readme) | +| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing; each tier's smoke tests and self-tests run, ungated, in that tier's job (guide [2.5](../../../docs/translator-skills/writing-translator-skills.md#25-build-and-ci-wiring-must), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)); the repo's existing test commands and jobs (e.g. bare `pytest`, `dotnet test` on the solution) don't also collect the UTS or harness tests (exclude them via `testpaths`/`--ignore`, a solution filter or a separate project), unless that job is deliberately their one CI home | + +Then compile all test targets as strictly as CI does (P-06) and run the repo's lint gates over the new code. + +## 5.3 Step 3c: write the harness README + +Write a README next to the harness covering everything guide [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) and [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: + +- take the symbol map from the [4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code) tables as built, and the known gaps from the gap table's `partial` and `missing` rows; +- add the layout (shared, port-only, module helpers, test targets) and the complete command that runs each tier's smoke tests and self-tests; +- write the **Known gaps** section (MUST): every helper-spec member, contract point or corpus construct the harness doesn't implement, by tier, as a table whose first column is the exact pseudocode token, matched by grepping the selected specs' `pseudo` fences (guide 3.1), then status and workaround, so the skill's preflight can match it against selected specs; +- index every module's fixture helpers in one place, even those in a separate test-support module; +- link each fixture helper's scope to the module notes (item 9), where guide 2.2 puts it; +- record the README's full repo path in the Design record. + +## 5.4 Step 3d: STOP-7, confirm the harness + +Re-run the assessment of [4.2](phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build) and the tier matrix of [4.8](phase-2-design-harness.md#48-step-2h-derive-tier-feasibility), and update `uts-infra-design.md` with the build log. Show the user the smoke-test and self-test results per tier (each run by its tier's filter), their CI wiring, the files created or changed, and the updated tier matrix. Only then continue to the skill. + +**Done when:** every row the user chose to build is `present`; each supported tier has green smoke tests and self-tests covering guide 2.7, wired into CI (or, if the user declined the CI change at STOP-5, listed in the Final report as an unmet requirement: the skill does not conform to guide 2.5/2.7 until CI runs the harness tests, so mark the acceptance-checklist rows "Harness smoke tests per tier and helper self-tests …" and "One CI home per suite …" ✗ with that reason; the tier stays ready, because the skill's preflight still runs the harness tests locally on every run, and the harness README states that CI doesn't yet run them); the strict compile and lint pass; the harness README exists; the user has confirmed at STOP-7. If the user declines network access at STOP-6, the integration and proxy tiers aren't supported: mark them unready (`blockedBy`: harness tests not run) and list them in the Final report. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md b/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md new file mode 100644 index 000000000..ab444353b --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md @@ -0,0 +1,42 @@ +# 6. Phase 4: skill design decisions + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +**Goal:** every choice the skill bakes in is made once, with the user, and written down. + +## 6.1 Step 4a: draft the Design record + +For each decision below, propose an answer derived from the Repo profile, the built harness and the guide, then present them together. Write them into `design-record.md` ([design-record template](../assets/templates/design-record.md)). + +| ID | Decision | Guide reference | How to propose it | +|---|---|---|---| +| D-01 | Skill name and location | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-` so Codex loads it too (guide 3.1); record the link or copy method | +| D-02 | Script language and minimum runtime version | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | Python 3 unless P-07 shows it isn't available on every developer platform | +| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file), [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](phase-2-design-harness.md#47-step-2g-plan-placement-harness-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that. Record the mapping's `harness` entry (root, README, per-tier sources and harness-test commands from 4.7) | +| D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability), [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | +| D-05 | Name collisions with native tests | [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | +| D-06 | **Tag syntax** | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | +| D-07 | File header format | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@` (plus `(locally modified; blob )` for a locally modified spec, guide 9.1), disclosures; integration/proxy also the corresponding unit spec | +| D-08 | **`deviations.md` location** per owning test module | [7.3](../../../docs/translator-skills/writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | +| D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | +| D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | +| D-11 | Pending idiom for a missing API (registered, skipped, not commented out) | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states), [7.4](../../../docs/translator-skills/writing-translator-skills.md#74-language-inapplicable-inputs) | The framework's static skip or pending marker | +| D-12 | **Internal-access ladder**: the existing exposure to check first, the sanctioned way to add a test-only exposure, who to escalate to; where the white-box spec list lives | [5.9](../../../docs/translator-skills/writing-translator-skills.md#59-internal-access-white-box-unit-specs) | From P-14. List white-box specs in the module notes or the mapping file | +| D-13 | **Delegation policy** for delegating specs and tests | [5.3](../../../docs/translator-skills/writing-translator-skills.md#53-structural-variants) | The guide's recommendation (one test with the delegating tag, running the referenced REST cases through a shared helper; else pending with a `NOTE`). The guide calls this new policy: confirm with the user | +| D-14 | Parallelism: serialise UTS suites, or rely on per-client hooks | [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | From P-12 and the hooks as built | +| D-15 | Teardown mechanism (scope, hook, `finally`) and suite-level fixtures for `BEFORE ALL TESTS` | [5.4](../../../docs/translator-skills/writing-translator-skills.md#54-setup-and-teardown) | From P-03 and the harness | +| D-16 | Protocol-variant rendering (`PROTOCOL`) | [5.3](../../../docs/translator-skills/writing-translator-skills.md#53-structural-variants) | The framework's parameterisation over `json`/`msgpack`; tag stays singular | +| D-17 | Default timeouts and poll interval | [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | As guide 5.6 "Every wait is bounded": the spec's timeout where it states one; otherwise an explicit 10–30 s timeout on every integration `poll_until` and 10–15 s helper defaults for state waits. Record the chosen values | +| D-18 | Auth through the proxy | [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | As designed in [4.5](phase-2-design-harness.md#45-step-2e-design-the-sandbox-and-proxy-helpers) | +| D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | +| D-20 | **Commit policy**; project/solution-file edits | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | +| D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](../../../docs/translator-skills/writing-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | +| D-22 | `allowed-tools` and remote access | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash Read Edit Write` (space-separated, guide 3.3); no `WebFetch`, because everything is read from the local clone (the skill never consults the reference implementations, [2.1](../SKILL.md#21-your-inputs)) | +| D-23 | Which modules get notes files, and which are placeholders | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | +| D-24 | Location of the working records, and whether they are kept in the repo | [1.4](../SKILL.md#14-outputs) of this skill | Default `.claude/skills/uts-to-/generation/`, kept for maintenance | +| D-25 | Pilot specs and mode per tier | [8.3](phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier) of this skill | One per ready module × tier in scope, and at least one per module with a full notes file, chosen by the criteria there | +| D-26 | Model tier for skill runs: pin the model, or state it | [Model tier](../../../docs/translator-skills/writing-translator-skills.md#model-tier), [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | No portable frontmatter field pins a model: state the Opus-class requirement in the opening lines of `SKILL.md` and in `compatibility` (guide 3.3). Add Claude Code's `model` field only if the user accepts that it isn't portable. Either way, the skill's final report records the model | + +## 6.2 Step 4b: STOP-8, confirm + +Show the Design record. Make clear which decisions follow directly from the guide (the user may still override them, but should know they are departing from it) and which are genuine choices. Record the answers. Don't generate any skill file before this confirmation. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md b/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md new file mode 100644 index 000000000..d3144ec68 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md @@ -0,0 +1,174 @@ +# 7. Phase 5: generate the skill files + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +Contents: + +- [7.1 Step 5a: create the layout and the mapping file](#71-step-5a-create-the-layout-and-the-mapping-file) +- [7.2 Step 5b: write the resolver](#72-step-5b-write-the-resolver) +- [7.3 Step 5c: write the audit](#73-step-5c-write-the-audit) +- [7.4 Step 5d: fill the construct catalogue](#74-step-5d-fill-the-construct-catalogue) +- [7.5 Step 5e: write the module notes](#75-step-5e-write-the-module-notes) +- [7.6 Step 5f: write `SKILL.md`](#76-step-5f-write-skillmd) + +Generate the files in this order: each one depends on the previous ones. `SKILL.md` comes last because it references all the others; its outline is fixed by the guide, so you can draft it early, but finish it last. + +## 7.1 Step 5a: create the layout and the mapping file + +1. Create the directory layout from guide [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) (plus the working-records directory, D-24). Then create the `.agents/skills/uts-to-` link (or the copy and its CI check) that D-01 chose, so Codex loads the skill too. +2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file). +3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data, and guide [4, step C](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) refuses unready tiers), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. + +**Acceptance:** + +- [ ] Every module the user wants has an entry; every tier value is one path relative to the repo (or a declared root), never machine-absolute. +- [ ] Every derived namespace is unique (a module segment after the tier, or a module-specific prefix). +- [ ] `notes` paths are relative to the skill directory and the files exist (placeholders at this point are fine). +- [ ] Hand-maintained entries are marked (e.g. in `_comment`) and listed in the Design record. +- [ ] Every tier the harness design marks unready has an `unready` entry with the blocking gap-table row. +- [ ] The mapping has a `harness` entry (root, README, per-tier sources and the complete smoke-test and self-test command), every path in it exists, no harness test lies inside any tier `targetDir`, and no `targetDir` lies inside the harness root (guide [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)). +- [ ] The file is valid JSON, UTF-8, LF line endings. + +## 7.2 Step 5b: write the resolver + +Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. + +Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty, dirtyFiles}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: + +```json +{ + "ok": true, + "sourceModule": "realtime", + "specRepo": { "sha": "", "dirty": false, "dirtyFiles": [] }, + "mapped": true, + "translationNotes": "", + "harness": { + "root": "", + "readme": "", + "tiers": { "unit": { "sources": ["..."], "tests": "" }, "integration": { "...": "..." }, "proxy": { "...": "..." } } + }, + "tiers": { + "unit": { + "present": true, + "sourceDir": "/uts/realtime/unit", + "targetDir": "", + "namespace": "", + "buildTarget": "", + "ready": true, + "specs": [ { "file": "connection/connection_failures_test.md", "className": "ConnectionFailuresTests" } ] + }, + "integration": { "present": true, "...": "..." }, + "proxy": { "present": true, "ready": false, "blockedBy": "G-16: ProxySession not built", "...": "..." } + } +} +``` + +Errors are `{"ok": false, "code": "", "message": ""}`, with a distinct code for each failure guide 8.1 describes (not a UTS module path, directory not found, no tier directories, mapping missing or malformed, bad target name; the guide's example codes are a fine naming), plus one for a declared notes file that is missing (e.g. `NOTES_NOT_FOUND`) and one for a name collision (e.g. `NAME_COLLISION`). + +**Acceptance** (each verified by running the script in [8.1](phase-6-validate-skill.md#81-run-the-resolver-on-every-module)): + +- [ ] Every row of guide 8.1 is implemented: validation, error objects, output contract, path-only tier detection (including specs in tier sub-directories, e.g. `realtime/integration/channels/`), exclusions relative to the tier base, deterministic runner-collectable naming (D-04), collision detection (D-05, including file-name uniqueness where the runner needs it), validate-then-write `--create` writing the D-03 default layout, an error for a missing notes file. +- [ ] `harness` matches the mapping, with every path validated (repo-relative) and existing; `ready`/`blockedBy` match the mapping's `unready` data; `testFile` is present wherever the file name isn't derived from `className`. +- [ ] Never crashes: any exception becomes `ok: false` with a code. + +## 7.3 Step 5c: write the audit + +Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](../../../docs/translator-skills/writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](../SKILL.md#21-your-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: + +| Test-side pattern | Source | +|---|---| +| Tag marker (`UTS:` after the line-comment marker) | D-06 | +| Assertion calls (every assertion form the catalogue maps `ASSERT …`, `FAILS WITH`, `THROWS`, `EXPECT THROW` and `AWAIT_ERROR` to) | [7.4](#74-step-5d-fill-the-construct-catalogue) | +| Wait calls (every harness helper the catalogue maps `AWAIT`, `AWAIT_STATE`, `poll_until`, … to) | [7.4](#74-step-5d-fill-the-construct-catalogue); the harness README | +| Comment syntax (line and block), so commented-out assertions are ignored | P-01 | + +Write these as data at the top of the script, so a catalogue change is a one-line edit. Match the longest keyword first on both sides. + +Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifiable` for a spec with no `**Test ID**` lines (not zero tests); if you depart from it, document your shape in `SKILL.md`. Use one exit code for clean, a distinct one for ID problems (missing, orphan, duplicate) and another for "couldn't run". + +**Acceptance** (verified in [8.2](phase-6-validate-skill.md#82-test-the-audit-itself)): + +- [ ] Every MUST row and the whole parsing contract of guide 8.2 are implemented, including the end-at-heading rule (a guide SHOULD that this procedure requires: *procedure recommendation*): ID coverage (`missing`, `orphan`, `duplicate`, distinct non-zero exit), per-test ledger, separate assert and await counts (spec-side `FAILS WITH` / `THROWS` / `EXPECT THROW` / `AWAIT_ERROR` lines counted as assertions, awaited or not, and `poll_until` / `poll_until_success` / `POLL_UNTIL` / `AWAIT UNTIL` / `WAIT_FOR` lines counted as waits, matching the test side), commented-out assertions ignored, never crashes, unverifiable specs reported. +- [ ] Where your parsing differs from the contract, `SKILL.md` says so (guide 8.2 "Implement exactly this, or say where yours differs"). +- [ ] SHOULD items you implemented are listed in `SKILL.md`; the ones you didn't are listed in the Final report. + +## 7.4 Step 5d: fill the construct catalogue + +This step produces the construct table that goes into `SKILL.md` (and, for module-specific rows, into the notes). + +1. **Start from every row of guide [section 6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. +2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells, or anything seen in a reference-implementation read ([2.1](../SKILL.md#21-your-inputs)); they illustrate other SDKs. +3. **Scan the corpus for constructs the guide doesn't list** (guide section 6 says the catalogue isn't exhaustive). Run the scanner below over each module the skill covers, and add a row for every uppercase keyword or snake_case call you can't account for. +4. **Give every row a status:** + - `mapped`: a direct rendering; + - `mapped (inferred)`: an *(undocumented)* construct, meaning inferred from its uses (list one use); + - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](../../../docs/translator-skills/writing-translator-skills.md#72-harness-stand-ins-are-not-deviations)); + - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation; never substitute a similar mock outcome, guide [5.2](../../../docs/translator-skills/writing-translator-skills.md#52-fidelity)), and the Final report has a plan; + - `n/a`: no spec in the covered modules uses it (say so after scanning). +5. **No blank cells.** Collect every row the skill can't render and has no plan for, from one scan, into a single **STOP-9**. + +Corpus scanner: run `python3 /scripts/scan_constructs.py /uts/`. It prints the uppercase keywords and snake_case calls used in `pseudo` fences, outside comments and string literals, with counts; compare its output with the catalogue, and use it in Phase 2 to list the harness symbols the corpus uses. + +The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_SET`, `LWW`); classify those under the guide's "wire constants" and enum rows rather than as new constructs. The scanner sees only `snake_case(` calls; method-style harness calls such as `mock_ws.active_connection.send_to_client(…)` are caught, but camelCase members (`onConnectionAttempt`) are not, so also read the helper specs' interfaces. Ship it in the generated skill as `scripts/scan_constructs.py` (copy this skill's script; it is language-independent), as guide [section 6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) and [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) recommend (SHOULD), because a re-sync re-scans the corpus (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) step 2); record the choice in the Design record (D-19). + +**Acceptance:** + +- [ ] Every guide section 6 row is present with a rendering and a status; none blank. +- [ ] Every scanner hit in the covered modules is either a catalogue row, a wire constant or enum value, a spec-local name, or internal pseudocode of a helper spec or `objects/PLAN.md` (implemented as harness or ignored, never translated). +- [ ] Every rendering names a symbol that exists in the harness or the SDK (you read it), or is marked `not available` with a plan. +- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events), [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)). + +## 7.5 Step 5e: write the module notes + +Write `references/-mapping.md` for every module chosen in D-23. Cover the thirteen items in guide [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should) ("A module notes file should cover"), in that order. For each item, either fill it or write "Not applicable: ". The items that most need evidence: + +- **Item 1 (source of truth and runtime status):** name the IDL the notes apply (for `objects`, check whether the SDK follows `objects-features.md` in the clone or another variant; guide 3.4 item 1 explains the typed-SDK case) and whether the module is implemented, which decides whether evaluate mode is possible. +- **Item 5 (type mapping):** for every polymorphic or union type in the pseudocode, the SDK type and which conversions throw or return null. Read the SDK source for each. +- **Item 8 (internal access):** the ladder from D-12 and the list of white-box specs. To find candidates, look for specs that construct internal classes or call members the public API doesn't have (the guide's white-box row in [6.6](../../../docs/translator-skills/writing-translator-skills.md#66-mocks-fixtures-and-harness) lists examples), then confirm each against the SDK's public surface. +- **Item 9 (helper-spec coverage):** the module's rows of the symbol tables from [4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code), as built, with each fixture helper's scope and every sanctioned stand-in. +- **Item 12 (worked example):** translate one short real test from the module by hand, name each mechanical rewrite, and make sure it compiles (it becomes one of the skill's examples; [8.7](phase-6-validate-skill.md#87-verify-the-skills-own-examples)). + +A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), "If the module's notes file exists but is only a placeholder"). + +**Acceptance:** + +- [ ] All thirteen items present or marked not applicable with a reason. +- [ ] Every SDK and harness symbol named in the notes exists at the recorded repo SHA. +- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide [4, step 1](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17)). +- [ ] No line numbers; cite symbols and headings (guide [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned)). + +## 7.6 Step 5f: write `SKILL.md` + +Follow the outline in guide [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide ["What the skill owns, and what it defers"](../../../docs/translator-skills/writing-translator-skills.md#what-the-skill-owns-and-what-it-defers)). + +| Part of `SKILL.md` | Content | Guide | +|---|---|---| +| Frontmatter | The portable fields of guide 3.3: `name: uts-to-`, equal to the directory name; a pushy, third-person `description` of at most 1,024 characters with trigger phrases, a "Not for …" boundary and no `<` or `>`; `license`; `compatibility`; `allowed-tools` from D-22; string-valued `metadata` with `short-description`. Claude Code-only fields only as decided in D-26; the model statement in the opening lines | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | +| Usage guard | No argument → print the usage line and stop, without relying on `$ARGUMENTS` (read the argument from the user's words); the skill's own scripts and files named relative to the skill directory, never `.claude/skills/…`; placeholder paths only | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | +| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state (the resolver's `specRepo`); for each selected spec in `dirtyFiles`, stop and ask, and if the user proceeds stamp its blob (guide 9.1) | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main) | +| Generated-skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | +| Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Generated-skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README and helper sources from the resolver's `harness` output; for the reference test per tier, write `none yet` until [8.8](phase-6-validate-skill.md#88-name-the-reference-tests) names one | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17), [8.3](../../../docs/translator-skills/writing-translator-skills.md#83-review-checklist-must-after-the-audit) | +| Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](../../../docs/translator-skills/writing-translator-skills.md#5-translation-rules) | +| Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) | +| File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should) | +| Evaluation | Guide 7 rendered with D-08 to D-11 and D-21 | [7](../../../docs/translator-skills/writing-translator-skills.md#7-evaluation-and-deviations) | +| Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](../../../docs/translator-skills/writing-translator-skills.md#55-integration-tier-hygiene) | +| Verification | Guide 10 with the exact commands from the Repo profile | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | +| Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should), [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| Final report | The format, including the model line | [11](../../../docs/translator-skills/writing-translator-skills.md#11-final-report-format) | +| Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | + +**Acceptance:** + +- [ ] Every part in the table is present. +- [ ] The skill is module-generic: module-specific rules live in the notes. +- [ ] No inline copy of a semantic rule where a link to its source would do; in particular the `deviations.md` format is referenced, not redefined. +- [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills (reference-implementation reads during creation are recorded in the working records, never cited in the skill). +- [ ] None of the guide's [Patterns to avoid](../../../docs/translator-skills/writing-translator-skills.md#patterns-to-avoid) appears in the skill, its notes, its scripts or the harness. +- [ ] The Harness reference names only files, filters and commands that exist, and doesn't inline helper signatures. +- [ ] Every command in it is one the Repo profile records as run. (Running each through the skill, and compiling every example, are Phase 6 checks: [8.5](phase-6-validate-skill.md#85-exercise-the-skills-other-paths), [8.7](phase-6-validate-skill.md#87-verify-the-skills-own-examples).) +- [ ] Every helper or SDK symbol it names exists. +- [ ] The frontmatter passes a standard validator (for example skill-creator's `quick_validate.py`): only portable fields unless D-26 says otherwise, `name` equal to the directory, description of at most 1,024 characters with no `<` or `>`, string `metadata` values. +- [ ] No `.claude/skills/` path in `SKILL.md`, the notes or the scripts; the scripts find the mapping and notes relative to their own file, not the working directory. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md new file mode 100644 index 000000000..5aef032db --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md @@ -0,0 +1,121 @@ +# 8. Phase 6: validate the skill + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +Contents: + +- [8.1 Run the resolver on every module](#81-run-the-resolver-on-every-module) +- [8.2 Test the audit itself](#82-test-the-audit-itself) +- [8.3 Pilot-translate one spec per available tier](#83-pilot-translate-one-spec-per-available-tier) +- [8.4 Fix the cause, then regenerate](#84-fix-the-cause-then-regenerate) +- [8.5 Exercise the skill's other paths](#85-exercise-the-skills-other-paths) +- [8.6 Lint and CI-strict compile](#86-lint-and-ci-strict-compile) +- [8.7 Verify the skill's own examples](#87-verify-the-skills-own-examples) +- [8.8 Name the reference tests](#88-name-the-reference-tests) + +**Goal:** evidence that the skill works on this repo, on top of the harness you built. This phase is mandatory. Record every result in the Final report. + +## 8.1 Run the resolver on every module + +Run the resolver on `/uts/rest`, `/uts/realtime` and `/uts/objects`, whether or not they are in scope (an unmapped module must resolve with `mapped: false`, not crash). For each, check: + +- [ ] `ok: true`; tiers `present` match `ls /uts/`. +- [ ] Spec counts match the files on disk minus the exclusions. A cross-check: `find /uts// -name '*.md' -not -path '*/helpers/*'` (add the proxy exclusion for the integration tier). +- [ ] No helper spec, `README.md`, `PLAN.md` or `*_SUMMARY.md` appears in `specs`. +- [ ] Every `className` follows D-04 and is collected by the runner; no collisions (or the collision is reported). +- [ ] `targetDir`, namespace and build target match the mapping and the Design record. +- [ ] The same module given as a `~`-prefixed path, and (on Windows) with backslash separators, resolves identically (guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) "Validate the module"). + +Then run the negative cases and check each returns `ok: false` with the right code: a path whose parent isn't `uts`; a missing directory; a module with no tier directories (a scratch directory named `uts/` will do); a mapping entry whose notes file is missing (use a scratch copy of the mapping); `--create` with an invalid name; `--create` on a scratch copy of a deliberately corrupt mapping (the file must be left unchanged). + +## 8.2 Test the audit itself + +Use one known-good test file (the pilot output from [8.3](#83-pilot-translate-one-spec-per-available-tier) once it is reviewed; until then, a hand-written file with the correct tags) and make scratch copies for each mutation. If you started with the hand-written file, run the mutations again once the pilot output passes review: + +| Mutation | Expected audit result | +|---|---| +| Delete one assertion | A positive assertion shortfall for that test | +| Comment out one assertion | The same shortfall (commented assertions don't count) | +| Duplicate one test's tag | `duplicate` lists the ID; distinct non-zero exit | +| Add a tag that isn't in the spec | `orphan` lists it | +| Remove one tag | `missing` lists it | +| Add surplus waits to a test with a deleted assertion | The assertion shortfall is still reported (counts aren't summed) | +| Delete the native assertion that renders a spec `FAILS WITH` (awaited or not) | A positive assertion shortfall for that test | +| Delete the poll call that renders a spec `poll_until` (with no `AWAIT` on the spec line) | An await shortfall for that test (spec polls count as waits) | + +**Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l`` (as of `12540dcf`: rest 571, realtime 554, objects 339). + +## 8.3 Pilot-translate one spec per available tier + +Choose one spec per ready module × tier in scope, and at least one per module with a full notes file (D-25). A good pilot is short, representative of the tier, and uses the harness features most likely to be wrong: + +| Tier | Choose a spec that uses | Example candidates in the corpus | +|---|---|---| +| REST unit | `MockHttpClient`, request capture, `FAILS WITH` | `rest/unit/time.md` | +| Realtime unit | `MockWebSocket`, `enable_fake_timers`, `ADVANCE_TIME`, `AWAIT_STATE` | `realtime/unit/connection/when_state_test.md` | +| Objects unit | `setup_synced_channel`, builders, serial helpers | `objects/unit/internal_live_counter_api.md` (then a mutation spec) | +| Direct integration | `SandboxApp`, `BEFORE ALL TESTS`, `## Protocol Variants` | `rest/integration/publish.md` (has `## Protocol Variants`), or `rest/integration/time_stats.md` then `realtime/integration/channel_history_test.md` | +| Proxy | `create_proxy_session`, rules, the event log | `rest/integration/proxy/rest_fallback.md` or `realtime/integration/proxy/connection_resume.md` | + +The candidates are examples only. Before choosing one, confirm it exists in your clone (`ls /uts/`) and still uses the features listed (grep for them); check it against the module's scope and the SDK's features. Prefer specs with no existing UTS-derived test in the repo (P-20); if the best candidate has one, ask the user before the skill adds to or regenerates it. + +**Run the pilot the way a user would.** + +- **Loading the skill.** A skill created during a session may not be listed until a new session starts. Ask the user to start a new session of the same tool (Claude Code or Codex) in the repo, on an Opus-class model ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) (or to reload skills, if their client supports it), and confirm `/uts-to-` is listed (and, if D-01 links the skill for Codex, that `$uts-to-` is listed in a new Codex session). A fresh session also tests that the skill is context-complete on its own (guide intent), so the pilot never consults the reference implementations. +- **Answering its questions.** The answers are D-03 (mapping confirmation) and D-25 (tier, specs, mode), confirmed at STOP-8. Either the user runs `/uts-to- /uts/` (Claude Code) or `$uts-to- /uts/` (Codex) in the new session and answers with them, or, if the skill is listed in a session you control, you invoke it and answer with exactly those choices. If you use a subagent, run it on an Opus-class model ([2.8](../SKILL.md#28-model-tier-and-sub-agents)), and give it only the invocation and the D-25 answers, none of this run's context. It must not consult the reference implementations or any other SDK's skill, because the pilot tests the skill alone. Record the pilot session's or subagent's model with the pilot results. Any answer not covered by D-25 goes to the user. +- **Record each question the skill asked.** More than the four questions, other than a stop the guide requires (an unmapped module's name at step B, a known gap or red harness at step F, an unmapped construct, the fix-attempt bound), or a question it should have answered from its own files, is a skill defect ([8.4](#84-fix-the-cause-then-regenerate)). + +**A green preflight comes first.** The skill's step F (after **STOP-6** for network tiers) must pass for the pilot's tier before the pilot translates anything; a pilot run on a red harness doesn't count. Record the preflight result with the pilot. If network access is declined, that tier's pilot isn't run and is reported as unvalidated. + +Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: + +1. Compile (and collect: every tagged test function is collected; count a parameterised or protocol-variant test once). +2. Audit; walk the review checklist (guide [8.3](../../../docs/translator-skills/writing-translator-skills.md#83-review-checklist-must-after-the-audit)). +3. In evaluate mode, and after **STOP-6** for network tiers, run the class by filter and diagnose per `writing-derived-tests.md` Phase 2. +4. Review the generated file line by line against the spec yourself, and show it to the user. + +## 8.4 Fix the cause, then regenerate + +For every defect in a pilot test: + +1. Classify it: a rule in `SKILL.md`, a mapping in the notes, a catalogue row, a resolver or audit bug, or a harness gap. (A UTS spec error is not a skill defect: record it, emit the fail-fast test per D-10, and **STOP-10**.) +2. Fix that cause. **Never edit only the generated test.** A harness fix follows the rules of [5.1](phase-3-build-harness.md#51-step-3a-rules-while-building), adds or extends a self-test that would have caught the defect, re-runs the affected tiers' smoke tests and self-tests, and updates the harness README (and its Known gaps). +3. Delete the generated test and regenerate it with the skill. +4. Re-audit, recompile and, in evaluate mode, re-run. +5. Log the defect and the fix in the Design record's changelog. + +**Bound:** once a pilot spec has been regenerated three times (or the D-21 count) without a clean audit and, in evaluate mode, one of the three end states, stop (**STOP-10**) with a summary of every defect found and fixed so far. Count all regenerations of that spec together, not per defect. + +## 8.5 Exercise the skill's other paths + +Run each path through the skill (in the session from [8.3](#83-pilot-translate-one-spec-per-available-tier)) and record the result: + +1. **Usage guard:** no argument → the usage line, then stop. +2. **Bad module path:** a path whose parent isn't `uts` → the resolver's error relayed, then stop. +3. **Placeholder notes:** a module whose notes file is a placeholder → refused, with the reason. +4. **Unready tier:** a tier the mapping marks unready → not offered at step C, with its `blockedBy`. +5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. +6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification (including a change outside the pseudocode, such as `## Protocol Variants`, reported as **changed**, and, run against a scratch clone of the spec repo (`git clone `; the spec clone itself is never edited, 2.2), an uncommitted edit to that spec reported as **changed**), that the header SHA is updated, and that only the survivors in guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. +7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. +8. **Red harness:** on a scratch copy (or a temporary working-tree edit you revert afterwards), break one harness test (for example, make the WebSocket mock refuse every connection), or run an integration tier with the network unavailable → the skill stops at step F before generating anything, reports a harness or environment problem (not an SDK deviation), and names the failing test. Restore the change and confirm the preflight is green again. +9. **Known gap:** select a spec that uses a construct the harness README lists under Known gaps → the preflight reports the conflict and stops to ask (guide 4 step F), rather than generating a substituted rendering. If Known gaps is empty, use a scratch copy of the README with an entry for a construct the selected spec uses. +10. **Second tool:** if D-01 links the skill for Codex, start a Codex session in the repo, invoke `$uts-to-` with no argument (usage line, then stop) and with one module (step A runs the resolver from the skill's directory, not from `.claude/skills/`). Record whether Codex was available; if it wasn't, report this path as not run. + +## 8.6 Lint and CI-strict compile + +Run the frontmatter validation of [7.6](phase-5-generate-skill.md#76-step-5f-write-skillmd). Run the repo's lint and format gates (P-06) over every file you created or changed, including the harness, the skill's scripts and Markdown where the repo's EditorConfig covers them. Compile the test targets as strictly as CI does (warnings as errors, static analysis, type checkers). Run the skill's scripts on every developer platform you can, or note which you couldn't. + +## 8.7 Verify the skill's own examples + +Every code example in `SKILL.md` and the notes must compile (guide [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned)). For each example, either: + +- name the real, compiled file it was copied from (and check it still matches), or +- paste it into a scratch file in the test target, compile, and delete the scratch file. + +Record which examples were verified and how. + +## 8.8 Name the reference tests + +After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4, step 3](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17)). Never name a smoke test. + +**Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected (including the red-harness stop) and every command in `SKILL.md` has been run; every pilot ran after a green preflight; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. diff --git a/uts/skills/uts-to-lang-skill-creator/scripts/scan_constructs.py b/uts/skills/uts-to-lang-skill-creator/scripts/scan_constructs.py new file mode 100755 index 000000000..2cc9c339c --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/scripts/scan_constructs.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python3 +"""Corpus construct scanner for UTS pseudocode. + +Usage: scan_constructs.py /uts/ + +Prints the uppercase keywords and snake_case calls used in `pseudo` fences, +outside comments and string literals, with counts. Compare its output with the +construct catalogue; also use it to list the harness symbols the corpus uses. +""" +import collections, pathlib, re, sys + + +def main(argv): + if len(argv) != 2 or not pathlib.Path(argv[1]).is_dir(): + print("usage: scan_constructs.py /uts/", file=sys.stderr) + return 2 + root = pathlib.Path(argv[1]) + keywords, calls = collections.Counter(), collections.Counter() + for path in root.rglob("*.md"): + in_pseudo = False + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("```"): + in_pseudo = (line.strip() == "```pseudo") if not in_pseudo else False + continue + if not in_pseudo or line.lstrip().startswith(("#", "//")): + continue + line = re.sub(r'"[^"]*"', '""', line) # ignore string literals + line = re.split(r"\s(?:#|//)\s", line, maxsplit=1)[0] # drop trailing comments + keywords.update(re.findall(r"\b[A-Z][A-Z_]+\b", line)) + calls.update(re.findall(r"\b([a-z][a-z0-9]*(?:_[a-z0-9]+)+)\s*\(", line)) + for title, counter in (("KEYWORDS", keywords), ("SNAKE_CASE CALLS", calls)): + print("==", title) + for name, count in sorted(counter.items()): + print(f"{count:6d} {name}") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py b/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py new file mode 100755 index 000000000..f581637dc --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py @@ -0,0 +1,142 @@ +#!/usr/bin/env python3 +"""Locate, validate and pin the local ably/specification clone. + +Usage: spec_clone_info.py [SPEC_CLONE] + +Discovery order: + 1. the SPEC_CLONE argument (the path the user gave); + 2. the UTS_SPEC_CLONE environment variable; + 3. the clone this skill lives in, when it is installed by symlink + (the real path of this skill's directory, three levels up); + otherwise: {"ok": false, "code": "SPEC_CLONE_NOT_FOUND"}. +A path inside a clone (e.g. /uts/objects) is walked up to the clone root. + +Prints exactly one JSON object. Read-only; never touches the network. +""" +import filecmp, json, os, pathlib, subprocess, sys + +SKILL_NAME = "uts-to-lang-skill-creator" +SKILL_IN_CLONE = pathlib.Path("uts", "skills", SKILL_NAME) +REQUIRED = { + "guide": "uts/docs/translator-skills/writing-translator-skills.md", + "utsReadme": "uts/README.md", + "writingDerivedTests": "uts/docs/writing-derived-tests.md", + "writingTestSpecs": "uts/docs/writing-test-specs.md", + "integrationTesting": "uts/docs/integration-testing.md", + "proxy": "uts/docs/proxy.md", + "mockHttp": "uts/rest/unit/helpers/mock_http.md", + "mockWebsocket": "uts/realtime/unit/helpers/mock_websocket.md", + "mockVcdiff": "uts/realtime/unit/helpers/mock_vcdiff.md", + "standardTestPool": "uts/objects/helpers/standard_test_pool.md", + "featuresSpecs": "specifications", +} + + +def fail(code, message, **extra): + print(json.dumps({"ok": False, "code": code, "message": message, **extra}, indent=2)) + return 1 + + +def git(clone, *args): + out = subprocess.run(["git", "-C", str(clone), *args], capture_output=True, text=True, encoding="utf-8") + if out.returncode != 0: + raise RuntimeError(out.stderr.strip() or f"git {' '.join(args)} failed") + return out.stdout + + +def is_spec_clone(path): + return all((path / rel).exists() for rel in REQUIRED.values()) + + +def skill_files(root): + return sorted(p.relative_to(root).as_posix() for p in root.rglob("*") + if p.is_file() and "__pycache__" not in p.parts) + + +def skill_version(skill_dir): + try: + text = (skill_dir / "SKILL.md").read_text(encoding="utf-8") + except OSError: + return None + for line in text.split("\n---", 1)[0].splitlines(): + if line.strip().startswith("version:"): + return line.split(":", 1)[1].strip().strip('"') + return None + + +def main(argv): + skill_dir = pathlib.Path(__file__).absolute().parent.parent + skill_real = skill_dir.resolve() + candidates = [] + if len(argv) > 1 and argv[1].strip(): + candidates.append(("argument", argv[1])) + if os.environ.get("UTS_SPEC_CLONE"): + candidates.append(("UTS_SPEC_CLONE", os.environ["UTS_SPEC_CLONE"])) + candidates.append(("skill location", str(skill_real.parent.parent.parent))) + + clone, source = None, None + for source, raw in candidates: + path = pathlib.Path(os.path.expanduser(raw)).resolve() + # accept a path inside a clone (e.g. /uts/objects): walk up to the root + found = next((p for p in [path, *path.parents] if is_spec_clone(p)), None) + if found is not None: + clone = found + break + if source != "skill location": + missing = [rel for rel in REQUIRED.values() if not (path / rel).exists()] + return fail("NOT_A_SPEC_CLONE", f"{path} (from {source}) is not an ably/specification clone", + missing=missing) + if clone is None: + return fail("SPEC_CLONE_NOT_FOUND", + "No spec clone given and the skill is not installed from one. " + "Ask the user for the path of their local ably/specification clone (STOP-1).") + + try: + sha = git(clone, "rev-parse", "HEAD").strip() + status = git(clone, "status", "--porcelain", "--untracked-files=all", "--", "uts", "specifications").splitlines() + last = git(clone, "log", "-1", "--format=%H %cd", "--", + "uts/docs/translator-skills", SKILL_IN_CLONE.as_posix()).strip() + except (RuntimeError, OSError) as exc: + return fail("GIT_ERROR", str(exc), specClone=str(clone)) + + clone_skill = clone / SKILL_IN_CLONE + matches = None + if clone_skill.is_dir(): + if clone_skill.resolve() == skill_real: + matches = True + else: + files = skill_files(skill_real) + matches = files == skill_files(clone_skill) and all( + filecmp.cmp(skill_real / f, clone_skill / f, shallow=False) for f in files) + + result = { + "ok": True, + "specClone": str(clone), + "foundBy": source, + "sha": sha, + "dirty": bool(status), + "dirtyFiles": [line[3:] for line in status], + "guideLastChange": { + "sha": last.split(" ", 1)[0] if last else None, + "date": last.split(" ", 1)[1] if " " in last else None, + }, + "skillDir": str(skill_dir), + "skillRealpath": str(skill_real), + "skillVersion": skill_version(skill_real), + "skillMatchesClone": matches, + "paths": {key: str(clone / rel) for key, rel in REQUIRED.items()}, + } + if matches is False: + result["warning"] = ( + "SKILL_MISMATCH: the installed skill differs from the copy in this spec clone. " + "Ask the user to refresh the install, or to confirm which version to follow (STOP-1)." + ) + print(json.dumps(result, indent=2)) + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main(sys.argv)) + except Exception as exc: # never crash: report as JSON + sys.exit(fail("INTERNAL_ERROR", repr(exc))) diff --git a/uts/skills/uts-to-lang-skill-creator/scripts/survey_repo.py b/uts/skills/uts-to-lang-skill-creator/scripts/survey_repo.py new file mode 100755 index 000000000..eccd3e0c7 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/scripts/survey_repo.py @@ -0,0 +1,86 @@ +#!/usr/bin/env python3 +"""First-pass, read-only survey of a target SDK repository (Phase 1, step 1b). + +Usage: survey_repo.py [ ...] + +Prints the HEAD SHA and dirty state, file-extension counts, likely test roots, +submodule status, candidate mocks and fakes, and existing UTS tags. It is a +starting point for the discovery checklist, not an answer to it: follow the +evidence, and adapt or extend the searches for the repo's language. +""" +import collections, pathlib, re, subprocess, sys + +MOCK_RE = re.compile(r"class (Mock|Fake|Stub)|(Mock|Fake)[A-Z][A-Za-z]*(Transport|Http|Clock|Timer|Socket)") +TEST_ROOT_RE = re.compile(r"(^|/)(test|tests|spec|specs)/", re.IGNORECASE) + + +def git(repo, *args): + out = subprocess.run(["git", "-C", str(repo), *args], capture_output=True, text=True, encoding="utf-8") + return out.stdout if out.returncode == 0 else f"(git {' '.join(args)} failed: {out.stderr.strip()})\n" + + +def grep(roots, pattern, limit): + hits = [] + for root in roots: + for path in sorted(pathlib.Path(root).rglob("*")): + if not path.is_file() or any(part.startswith(".") for part in path.relative_to(root).parts): + continue + try: + lines = path.read_text(encoding="utf-8").splitlines() + except (UnicodeDecodeError, OSError): + continue + for n, line in enumerate(lines, 1): + if pattern.search(line): + hits.append(f"{path}:{n}: {line.strip()[:160]}") + if len(hits) >= limit: + return hits + return hits + + +def display(path, repo): + try: + return str(path.relative_to(repo)) + except ValueError: + return str(path) + + +def main(argv): + if len(argv) < 2 or not pathlib.Path(argv[1]).is_dir(): + print("usage: survey_repo.py [ ...]", file=sys.stderr) + return 2 + repo = pathlib.Path(argv[1]).resolve() + files = git(repo, "ls-files").splitlines() + + print("== HEAD and status") + print(git(repo, "rev-parse", "HEAD").strip()) + print(git(repo, "status", "--porcelain").strip() or "(clean)") + + print("\n== Extension counts (top 20)") + exts = collections.Counter(f.rsplit(".", 1)[-1] if "." in f.rsplit("/", 1)[-1] else "(none)" for f in files) + for ext, count in exts.most_common(20): + print(f"{count:7d} {ext}") + + print("\n== Likely test roots (first 50 files under test/tests/spec dirs)") + test_files = [f for f in files if TEST_ROOT_RE.search(f)] + for f in test_files[:50]: + print(f) + roots = [repo / r for r in argv[2:]] or sorted({repo / f.split("/")[0] for f in test_files}) + print("\n== Roots searched below: " + (", ".join(display(r, repo) for r in roots) or "(none)")) + + print("\n== Submodules") + print(git(repo, "submodule", "status").strip() or "(none)") + + print("\n== Candidate mocks, fakes and stubs (first 40)") + print("\n".join(grep(roots, MOCK_RE, 40)) or "(none)") + + print("\n== Existing UTS tags (first 10)") + print("\n".join(grep(roots, re.compile(r"UTS:"), 10)) or "(none)") + + print("\n== Existing skills") + print("\n".join(f for f in files if f.startswith((".claude/skills/", ".agents/skills/", ".codex/skills/")) and f.endswith("SKILL.md")) + or "(none)") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) From 9857c3f257e3d8c795666bbd96a959407c8b1d75 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Fri, 9 Oct 2026 15:26:32 +0530 Subject: [PATCH 07/33] UTS: move the translator skill guide to uts/docs The guide is now the only file left in uts/docs/translator-skills/, so move it next to the other writing-* guides it builds on (writing-test-specs.md and writing-derived-tests.md) as uts/docs/writing-uts-spec-translator-skills.md, and retitle it "Writing UTS Spec Translator Skills". Update every relative link to and from the guide (the skill, the README and writing-derived-tests.md), the plain-text paths in the skill (SKILL.md, the maintenance diff, the checklist template and spec_clone_info.py), and the uts/README.md tree and Guides entry. No other content changes. --- uts/README.md | 5 +- uts/docs/writing-derived-tests.md | 2 +- ... => writing-uts-spec-translator-skills.md} | 20 +++--- uts/skills/uts-to-lang-skill-creator/SKILL.md | 28 ++++---- .../assets/templates/acceptance-checklist.md | 2 +- .../references/failure-modes.md | 50 ++++++------- .../references/glossary.md | 2 +- .../references/maintenance.md | 6 +- .../references/phase-1-understand-repo.md | 8 +-- .../references/phase-2-design-harness.md | 30 ++++---- .../references/phase-3-build-harness.md | 24 +++---- .../references/phase-4-design-record.md | 48 ++++++------- .../references/phase-5-generate-skill.md | 70 +++++++++---------- .../references/phase-6-validate-skill.md | 10 +-- .../scripts/spec_clone_info.py | 4 +- 15 files changed, 154 insertions(+), 155 deletions(-) rename uts/docs/{translator-skills/writing-translator-skills.md => writing-uts-spec-translator-skills.md} (98%) diff --git a/uts/README.md b/uts/README.md index 252312331..b347fedc9 100644 --- a/uts/README.md +++ b/uts/README.md @@ -41,8 +41,7 @@ uts/ ├── docs/ # Guides and reference │ ├── writing-test-specs.md # How to write UTS specs │ ├── writing-derived-tests.md # How to translate specs into SDK tests -│ ├── translator-skills/ # Building uts-to- translator skills -│ │ └── writing-translator-skills.md # What a translator skill and its harness must contain +│ ├── writing-uts-spec-translator-skills.md # What a uts-to- translator skill and its harness must contain │ ├── integration-testing.md # Integration testing policy │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix @@ -156,7 +155,7 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations -- **[Writing UTS Translator Skills](docs/translator-skills/writing-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync +- **[Writing UTS Spec Translator Skills](docs/writing-uts-spec-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync - **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that, given an SDK repo, follows a step-by-step procedure to create its `uts-to-` skill: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations. See [Installing the skill creator](#installing-the-skill-creator) - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs diff --git a/uts/docs/writing-derived-tests.md b/uts/docs/writing-derived-tests.md index 3f3c952cc..d505eadf3 100644 --- a/uts/docs/writing-derived-tests.md +++ b/uts/docs/writing-derived-tests.md @@ -13,7 +13,7 @@ The process has two phases: Not every situation has an existing implementation. Tests may be written ahead of the implementation (test-first development), or for a new SDK that doesn't yet exist. In those cases, only the translation phase applies. -To automate translation for one SDK with an agent skill (`uts-to-`), see [Writing UTS Translator Skills](translator-skills/writing-translator-skills.md). That guide packages this document's rules into a repeatable procedure. This document remains the authority on translation and evaluation semantics. +To automate translation for one SDK with an agent skill (`uts-to-`), see [Writing UTS Spec Translator Skills](writing-uts-spec-translator-skills.md). That guide packages this document's rules into a repeatable procedure. This document remains the authority on translation and evaluation semantics. --- diff --git a/uts/docs/translator-skills/writing-translator-skills.md b/uts/docs/writing-uts-spec-translator-skills.md similarity index 98% rename from uts/docs/translator-skills/writing-translator-skills.md rename to uts/docs/writing-uts-spec-translator-skills.md index eaebbc7b3..436e190c1 100644 --- a/uts/docs/translator-skills/writing-translator-skills.md +++ b/uts/docs/writing-uts-spec-translator-skills.md @@ -1,8 +1,8 @@ -# Writing UTS Translator Skills +# Writing UTS Spec Translator Skills This guide explains how to build a per-language agent skill (Claude Code and Codex) — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). -If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, use the [`uts-to-lang-skill-creator` skill](../../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building the skill and its harness; this guide remains the authority on what the skill must contain. +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, use the [`uts-to-lang-skill-creator` skill](../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building the skill and its harness; this guide remains the authority on what the skill must contain. **Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. @@ -49,10 +49,10 @@ A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and no | Document | Owns | |---|---| -| [`uts/README.md`](../../README.md) | Pseudocode conventions: identifier naming, absent values, property access, enum values, `poll_until_success`, `process_pending_events()`, language-inapplicable inputs | -| [`writing-test-specs.md`](../writing-test-specs.md) | The pseudocode reference: Test IDs, mock patterns, `AWAIT_STATE`, `poll_until`, record-and-verify, error pattern, unique channel names | -| [`writing-derived-tests.md`](../writing-derived-tests.md) | **Translation and evaluation semantics.** Faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | -| [`integration-testing.md`](../integration-testing.md), [`proxy.md`](../proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | +| [`uts/README.md`](../README.md) | Pseudocode conventions: identifier naming, absent values, property access, enum values, `poll_until_success`, `process_pending_events()`, language-inapplicable inputs | +| [`writing-test-specs.md`](writing-test-specs.md) | The pseudocode reference: Test IDs, mock patterns, `AWAIT_STATE`, `poll_until`, record-and-verify, error pattern, unique channel names | +| [`writing-derived-tests.md`](writing-derived-tests.md) | **Translation and evaluation semantics.** Faithful translation, the evaluation decision tree, the three test patterns, `deviations.md` format, idiomatic translation vs deviation, timer and timeout traps | +| [`integration-testing.md`](integration-testing.md), [`proxy.md`](proxy.md) | Integration and proxy tiers: sandbox setup, timeouts, protocol variants, the proxy session and rule API | | **This guide** | How to package all of the above into a repeatable, scripted agent skill (Claude Code and Codex) for one SDK | | Existing skills and harnesses ([Reference implementations](#reference-implementations-last-resort)) | Nothing normative: a last-resort, read-only reference for patterns when creating a skill, below every document above | @@ -76,7 +76,7 @@ Creating a skill and its harness, and running the skill, are long, multi-file re | Task | Level | Why | |---|---|---| -| Creating the skill and building its harness (the [procedure](../../skills/uts-to-lang-skill-creator/SKILL.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | +| Creating the skill and building its harness (the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md)) | **MUST** | The agent must hold this guide, the procedure, the UTS docs, the helper specs and the SDK's source in context at once; design hooks, mocks, fixtures and CI wiring that span many files; and decide rules that every later run inherits, so one silent omission becomes a defect in every generated test | | Translating and evaluating with the finished skill (`/uts-to-` or `$uts-to-`) | **SHOULD** | The resolver and the audit ([section 8](#8-deterministic-tooling)) make paths, names, ID coverage and assertion counts mechanical, but not the rest. Matching wait predicates and timeouts, copying comments verbatim, setup fidelity, and diagnosing a failure as SDK, spec or translation ([7](#7-evaluation-and-deviations), [8.3](#83-review-checklist-must-after-the-audit)) remain model judgement, and a count can't catch a changed predicate ([8.2](#82-audit-audit_translationpy-must)) | | Sub-agents spawned during either (review, validation; pilot runs during creation serve creation) | **The level of the task they serve** | A reviewer weaker than the author misses what the author missed. A cheaper model MAY be used only for a clearly mechanical step whose result a script verifies (for example, running the audit over the corpus and collecting its JSON), if at all | @@ -173,7 +173,7 @@ Where the SDK already has a test-options type, add the hooks to it, and implemen ### 2.4 The proxy (MUST for the proxy tier) - A `ProxyManager` that downloads, caches and starts [`ably/uts-proxy`](https://github.com/ably/uts-proxy). Pin the version, and check that the pinned release publishes a binary for every developer OS (its GitHub release assets). An environment override pointing at a locally built binary (`UTS_PROXY_LOCAL_PATH` in both existing harnesses; ably-java also accepts `-Duts.proxy.localPath`) is useful; keep it a configuration override, not a test gate. Operationally, the manager also verifies the downloaded archive (a checksum), serialises the download across concurrent test processes, waits for the proxy's control endpoint to report healthy before returning, reaps the proxy process when the test process exits, and handles the port: proxy suites share one control port, so run them in one process (one fork or worker), or give each worker its own port. -- A native `ProxySession` implementing [`proxy.md`](../proxy.md): `create_proxy_session`, `add_rules`, `trigger_action`, `get_log`, `close`, plus rule-builder helpers and a typed event log. +- A native `ProxySession` implementing [`proxy.md`](proxy.md): `create_proxy_session`, `add_rules`, `trigger_action`, `get_log`, `close`, plus rule-builder helpers and a typed event log. - **The `action` inside a frame `match` (`ws_frame_to_client` / `ws_frame_to_server`) must be a JSON string**: a name (`"ATTACHED"`) or a numeric string (`"11"`), never a number (the proxy rejects a number with HTTP 400). uts-proxy v0.3.0 resolves names only up to `AUTH` (17), so use numeric strings for `OBJECT` (`"19"`), `OBJECT_SYNC` (`"20"`) and `ANNOTATION` (`"21"`); an unresolvable name silently never matches. This applies only to `match.action`: a rule's own `action` is an object (`{"type": "suppress"}`), and the `action` inside an injected message stays a number. Rule builders should take an integer and stringify it, as both existing harnesses do. - Proxy tests always use JSON: the proxy supports text frames only (`integration-testing.md`, "Protocol Variants"). - **Auth through the proxy.** The proxy hop is plain HTTP/WS (`tls: false`), and RSA1 and RSC18 say Basic Auth over HTTP without TLS results in an error (`writing-test-specs.md` "RSC18 only applies to Basic auth configurations": checked at construction). `proxy.md`, `integration-testing.md` and `writing-test-specs.md` nonetheless show `key: api_key` with `tls: false`. In practice, ably-cocoa authenticates proxied clients with a locally signed `TokenRequest`; ably-java does the same where the spec observes auth (`AuthReauthTest`), but its objects proxy suite (`ObjectsFaultsTest`) uses `key` over the plain hop, which ably-java permits. Token auth is the option valid under RSA1 for every SDK; whichever you use, say why in a comment. @@ -876,7 +876,7 @@ Neither existing skill has one; both have stale suites as a result. Add a mode ( 1. Runs the audit over **every** mapped spec in the module, not only the ones selected. 2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as **new**; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. Then re-scan the corpus for constructs the skill can't map (section 6). -3. Classifies each spec as **new**, **changed** (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), **unchanged** or **removed**. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](../../skills/uts-to-lang-skill-creator/references/maintenance.md#10-maintenance)). +3. Classifies each spec as **new**, **changed** (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), **unchanged** or **removed**. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](../skills/uts-to-lang-skill-creator/references/maintenance.md#10-maintenance)). 4. Regenerates the affected tests, **preserving only** DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. 5. Updates the SHA in each regenerated file's header, and its blob note (9.1): re-stamped if the spec is still locally modified, removed if not (and in the manifest, if kept). 6. Reports the classification ([section 11](#11-final-report-format)). @@ -1077,7 +1077,7 @@ Rules: 5. **Record each consultation**: the repo, commit and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. 6. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. -Reading GitHub is network access: the [procedure](../../skills/uts-to-lang-skill-creator/SKILL.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. +Reading GitHub is network access: the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. ### Patterns to avoid diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md index d985fe00e..10f6e3e09 100644 --- a/uts/skills/uts-to-lang-skill-creator/SKILL.md +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -14,16 +14,16 @@ metadata: # UTS-to-Lang Skill Creator -Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create a `uts-to-` translator skill, and the UTS test infrastructure (the **harness**) it runs on, for one Ably SDK repository. [Writing UTS Translator Skills](../../docs/translator-skills/writing-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why; this skill says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this skill links to it, and you must read the linked section. +Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create a `uts-to-` translator skill, and the UTS test infrastructure (the **harness**) it runs on, for one Ably SDK repository. [Writing UTS Spec Translator Skills](../../docs/writing-uts-spec-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why; this skill says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this skill links to it, and you must read the linked section. -The harness and the skill are **one deliverable** (guide [section 2](../../docs/translator-skills/writing-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](references/phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). The phases run in a fixed order, because each stands on the one before it; a harness designed without studying the repo's existing tests duplicates or fights them. +The harness and the skill are **one deliverable** (guide [section 2](../../docs/writing-uts-spec-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](references/phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). The phases run in a fixed order, because each stands on the one before it; a harness designed without studying the repo's existing tests duplicates or fights them. "This skill" means this `SKILL.md` together with its `references/` and `assets/templates/`. The ground rules (section 2) and the inputs and outputs (section 1) are below; each phase's steps are in `references/`, and you read each one in full before its phase starts ([Phases at a glance](#phases-at-a-glance)). Section numbers (for example "4.6") are stable across the files ([Reference index](#reference-index)). ## Path convention - **Files of this skill** (`references/`, `assets/templates/`, `scripts/`) are relative to the directory that contains this `SKILL.md`, wherever it is installed. Run scripts as `python3 /scripts/.py`, where `` is the directory containing this `SKILL.md`. -- **Files of the spec repo** (the guide, the UTS docs, the helper specs, the corpus) are linked relative to this skill's place in a spec clone, `uts/skills/uts-to-lang-skill-creator/`, so the links work on GitHub. A link that climbs out of the skill directory (`../../…` from `SKILL.md`, `../../../…` from `references/`) is a spec-repo path; every other link is a file of this skill. At run time, never resolve a spec-repo path against the directory you loaded this skill from. Run `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and read each file at `/uts/…`: a link to `…/docs/translator-skills/writing-translator-skills.md#31-layout` means `/uts/docs/translator-skills/writing-translator-skills.md`, section 3.1. Never fetch them from GitHub. +- **Files of the spec repo** (the guide, the UTS docs, the helper specs, the corpus) are linked relative to this skill's place in a spec clone, `uts/skills/uts-to-lang-skill-creator/`, so the links work on GitHub. A link that climbs out of the skill directory (`../../…` from `SKILL.md`, `../../../…` from `references/`) is a spec-repo path; every other link is a file of this skill. At run time, never resolve a spec-repo path against the directory you loaded this skill from. Run `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and read each file at `/uts/…`: a link to `…/docs/writing-uts-spec-translator-skills.md#31-layout` means `/uts/docs/writing-uts-spec-translator-skills.md`, section 3.1. Never fetch them from GitHub. - **Inputs** come from the user's words: the text after `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex), or their message. Don't rely on argument substitution. ## 2. Ground rules @@ -39,12 +39,12 @@ You work from four primary sources: and, only as a last resort, the reference implementations below. -**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](../../docs/translator-skills/writing-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: +**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](../../docs/writing-uts-spec-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: 1. You have searched this skill, the guide, the UTS docs and the helper specs, and they don't answer the question. 2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) with `git -C show :` and `git -C ls-tree -r --name-only -- `, never checking out, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. If a clone lacks the pinned commit, fetching it is network access too. 3. You take a **pattern**, not text: how something was structured or solved. Never copy, port or translate code, scripts, prose, tables or examples verbatim. Write every hook, helper, script, rule and example yourself, in this language's idioms, from the guide, the UTS docs and the target repo. -4. You check what you took against the guide's [Patterns to avoid](../../docs/translator-skills/writing-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. +4. You check what you took against the guide's [Patterns to avoid](../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. 5. You record each consultation in the Design record ("Reference-implementation reads": repo, commit, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)). @@ -99,14 +99,14 @@ Every choice you make that isn't forced by the guide or the UTS docs goes into t ### 2.6 Pin the spec clone -At the start of the run, run `python3 /scripts/spec_clone_info.py []`, passing the path the user gave, if any. It locates the clone (the path given; else `UTS_SPEC_CLONE`; else the clone this skill is symlinked from; a path inside a clone is walked up to its root), checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd' -- uts/docs/translator-skills uts/skills/uts-to-lang-skill-creator`. It prints one JSON object. +At the start of the run, run `python3 /scripts/spec_clone_info.py []`, passing the path the user gave, if any. It locates the clone (the path given; else `UTS_SPEC_CLONE`; else the clone this skill is symlinked from; a path inside a clone is walked up to its root), checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd' -- uts/docs/writing-uts-spec-translator-skills.md uts/skills/uts-to-lang-skill-creator`. It prints one JSON object. - Any `ok: false` (for example `SPEC_CLONE_NOT_FOUND`, `NOT_A_SPEC_CLONE`, `GIT_ERROR`): **STOP-1**; ask for the clone's path. - `foundBy` other than `argument`: tell the user which clone was found and how, and confirm it before Phase 1. - `skillMatchesClone: false` (an installed copy that differs from this clone's copy of the skill): **STOP-1**; ask the user to refresh the install, or to confirm which version to follow. - `skillMatchesClone: null` (the clone has no copy of this skill, for example an older checkout): tell the user and confirm before Phase 1. -Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), the guide's and this skill's last-change commit (`guideLastChange`) and `skillVersion`. Read every spec-repo file at the absolute paths the script reports (`paths`), or under `specClone`. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. +Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), the guide's and this skill's last-change commit (`guideLastChange`) and `skillVersion`. Read every spec-repo file at the absolute paths the script reports (`paths`), or under `specClone`. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). You MAY compare with `origin/main` if the user agrees to a `git fetch`; never pull or check out on their behalf. Read every doc and spec from this clone for the whole run. If the clone changes during the run, stop and ask whether to restart from the new SHA. Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. @@ -119,7 +119,7 @@ Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) a ### 2.8 Model tier and sub-agents -- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name): MUST.** The guide's [Model tier](../../docs/translator-skills/writing-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. +- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name): MUST.** The guide's [Model tier](../../docs/writing-uts-spec-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. - **Sub-agents** you spawn for review, validation or the pilot ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, running the [8.2](references/phase-6-validate-skill.md#82-test-the-audit-itself) corpus sweep and collecting its JSON), if at all, and record which steps used one. - **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. - **State** the generated skill's model tier in its opening lines and `compatibility`; pin it only as D-26 decides. @@ -158,8 +158,8 @@ Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to ### 1.4 Outputs -1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). -2. **The skill**, at `.claude/skills/uts-to-/` (linked from `.agents/skills/uts-to-/` for Codex, guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout)) unless the Design record says otherwise (D-01): `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout)). +1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). +2. **The skill**, at `.claude/skills/uts-to-/` (linked from `.agents/skills/uts-to-/` for Codex, guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)) unless the Design record says otherwise (D-01): `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). 3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. 4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): - `repo-profile.md` ([repo-profile template](assets/templates/repo-profile.md)); @@ -175,7 +175,7 @@ The working records are what a later agent reads to maintain or regenerate the s ### 1.5 Intent -The guide's intent ([introduction](../../docs/translator-skills/writing-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the harness and regenerate. Never hand-fix a pilot test and call the skill done. +The guide's intent ([introduction](../../docs/writing-uts-spec-translator-skills.md)) applies to this procedure too: if the skill you generate produces a wrong test, fix the skill, its notes or the harness and regenerate. Never hand-fix a pilot test and call the skill done. ### 1.6 Which document wins @@ -185,9 +185,9 @@ The guide's intent ([introduction](../../docs/translator-skills/writing-translat | What pseudocode means | [`uts/README.md`](../../README.md) and [`writing-test-specs.md`](../../docs/writing-test-specs.md) | | Integration and proxy tiers | [`integration-testing.md`](../../docs/integration-testing.md), [`proxy.md`](../../docs/proxy.md) | | Mock and fixture behaviour | the helper specs: [`mock_http.md`](../../rest/unit/helpers/mock_http.md), [`mock_websocket.md`](../../realtime/unit/helpers/mock_websocket.md), [`mock_vcdiff.md`](../../realtime/unit/helpers/mock_vcdiff.md), [`standard_test_pool.md`](../../objects/helpers/standard_test_pool.md) | -| What the harness and the skill must contain, and why | the guide ([How it relates to the other UTS docs](../../docs/translator-skills/writing-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | +| What the harness and the skill must contain, and why | the guide ([How it relates to the other UTS docs](../../docs/writing-uts-spec-translator-skills.md#how-it-relates-to-the-other-uts-docs)) | | How and in what order you build them | this skill | -| The SDK's actual API | the SDK source (guide [3.1](../../docs/translator-skills/writing-translator-skills.md#31-layout), "The notes are a map, not an authority") | +| The SDK's actual API | the SDK source (guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout), "The notes are a map, not an authority") | | Questions none of the above answer, as a last resort only | The reference implementations, under [2.1](#21-your-inputs): never an authority | If two documents disagree, follow the higher one in the table (and **STOP-11** if the conflict changes what you would build or generate), record the conflict in the Design record, and list it in the Final report under "Spec-repo doc issues". Don't patch the spec repo. A disagreement between a reference implementation and the guide or the UTS docs is never a STOP-11: follow the guide, and record the disagreement with the read ([2.1](#21-your-inputs)). @@ -215,7 +215,7 @@ Before starting a phase, read its reference file in full: this table is a map, n 3. Show the user the report, the list of changed and new files (harness, harness tests, hooks, skill, pilot tests), and the suggested CI changes. The run is complete only if the harness tests are green, and in CI, at every tier the skill supports; otherwise the report says which tiers aren't and why. **Don't commit** unless asked. 4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. -The acceptance checklist maps each item of the guide's [section 13 checklist](../../docs/translator-skills/writing-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill) to where this procedure produces and verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). +The acceptance checklist maps each item of the guide's [section 13 checklist](../../docs/writing-uts-spec-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill) to where this procedure produces and verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). ## Maintenance runs diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md index d3e424d5b..0d20bed8d 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md @@ -1,4 +1,4 @@ - + | Guide §13 item | Produced in | Verified in | Status | |---|---|---|---| diff --git a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md index 9f8e384db..71cbf23ab 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md +++ b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md @@ -6,29 +6,29 @@ All from the guide and the UTS docs; follow the link for the detail. | Symptom | Likely cause | Remedy | Source | |---|---|---|---| -| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the distinct collected test functions (a parameterised test counted once) with the tag count | guide [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci), [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability) | -| Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | -| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | -| Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must) | -| A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | -| `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue) | -| A state wait runs out the timeout after the connection failed | No fail-fast on FAILED | Fail immediately on FAILED, except when the test just requested a transition out of it | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | -| Assertions after a poll see fewer items | The test re-reads after polling an eventually-consistent store | Use the value the poll returned | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events) | -| `poll_until_success` times out with no useful error | Emulated with a reader that swallows errors | A separate helper that rethrows the last error | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events); `uts/README.md` | -| Negative assertions pass or fail at random | One zero-delay yield doesn't drain chained callbacks | Drain until the internal queue is empty; or the quiescence control-listener | guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must); `standard_test_pool.md` "Negative-assertion quiescence" | -| Tests pass alone, fail together | Process-global hooks with a parallel runner; leaked clients or timers | Serialise or make hooks per-client (D-14); close clients and restore mocks in teardown | guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must), [5.4](../../../docs/translator-skills/writing-translator-skills.md#54-setup-and-teardown) | -| State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | +| Tests compile but the runner collects none (or fewer than the tags) | Names don't match the runner's collection rules | Fix the resolver's naming (D-04); compare the distinct collected test functions (a parameterised test counted once) with the tag count | guide [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci), [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability) | +| Integration waits time out instantly with a misleading timeout message | The runner virtualises time; the timeout measures virtual time | Wrap every real-network wait in a wall-clock helper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/writing-uts-spec-translator-skills.md#56-time-and-waiting) | +| A poll under fake timers never expires | The poll deadline reads the faked clock | Restructure to avoid stubbing the wall clock, or read a monotonic clock | `writing-derived-tests.md` "No real timers in unit tests"; guide [5.6](../../../docs/writing-uts-spec-translator-skills.md#56-time-and-waiting) | +| Retries fire on wall-clock time despite `ADVANCE_TIME` | A timed blocking wait or a runtime timer isn't on the fake clock | Extend the clock hook to every time source (P-11) | guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must) | +| A wait misses a transient state | Sampling wait, or check-before-subscribe race | Event-latched `AWAIT_STATE`; record-and-verify with `CONTAINS_IN_ORDER` | guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events) | +| `CONTAINS_IN_ORDER` fails although the states occurred | Rendered as a contiguous prefix or an exact filtered list | Subsequence helper | guide [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue) | +| A state wait runs out the timeout after the connection failed | No fail-fast on FAILED | Fail immediately on FAILED, except when the test just requested a transition out of it | guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events) | +| Assertions after a poll see fewer items | The test re-reads after polling an eventually-consistent store | Use the value the poll returned | guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events) | +| `poll_until_success` times out with no useful error | Emulated with a reader that swallows errors | A separate helper that rethrows the last error | guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events); `uts/README.md` | +| Negative assertions pass or fail at random | One zero-delay yield doesn't drain chained callbacks | Drain until the internal queue is empty; or the quiescence control-listener | guide [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must); `standard_test_pool.md` "Negative-assertion quiescence" | +| Tests pass alone, fail together | Process-global hooks with a parallel runner; leaked clients or timers | Serialise or make hooks per-client (D-14); close clients and restore mocks in teardown | guide [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must), [5.4](../../../docs/writing-uts-spec-translator-skills.md#54-setup-and-teardown) | +| State machine misbehaves only under the mock | Mock `close()` calls `onClose` synchronously, or CONNECTED arrives before the connection completes (often an existing double reused unchanged) | Fix or wrap the mock per `mock_websocket.md` | guide [2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | | An objects test's operation is silently ignored | Hand-written serial literal sorts before the pool serial | Use the serial helpers | `standard_test_pool.md`; guide 2.2 | -| Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | -| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions"; guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | -| Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | -| Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | -| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning"; guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | -| Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | -| Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](../../../docs/translator-skills/writing-translator-skills.md#82-audit-audit_translationpy-must) | -| Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | -| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | -| Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | -| A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](../../../docs/translator-skills/writing-translator-skills.md#5-translation-rules) | -| Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](../../../docs/translator-skills/writing-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned) | +| Proxy session creation fails with HTTP 400 | Numeric `match.action` | Stringify it in the rule builder | guide [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier); `proxy.md` | +| A proxy rule never fires | Action name beyond the proxy's name table (`OBJECT`, `OBJECT_SYNC`, `ANNOTATION`) | Use numeric strings (`"19"`, `"20"`, `"21"`) | `proxy.md` "Match Conditions"; guide [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | +| Proxied client fails to authenticate | Basic auth over the plain proxy hop (RSA1) | Token auth through the proxy, with a comment | guide [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | +| Duplicate sandbox apps; flaky provisioning | `POST /apps` retried | Retry only idempotent requests; check the status before parsing | guide [2.3](../../../docs/writing-uts-spec-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers) | +| Resource leaks after objects integration tests | REST provisioning client not closed | Close it | `standard_test_pool.md` "REST Fixture Provisioning"; guide [2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) | +| Under virtual-time advances, the idle timer disconnects an objects fixture | The fixture's idle interval is shorter than the advance | Set a large idle interval in the fake-clock fixture variant and document it in the notes | guide [2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must), "Document each fixture helper's scope" | +| Audit says clean but an assertion is missing | Counts summed with waits; commented assertions counted; helpers after the last test attributed to it | Fix the audit; keep shared helpers before the first test or in a separate file | guide [8.2](../../../docs/writing-uts-spec-translator-skills.md#82-audit-audit_translationpy-must) | +| Generated class clashes with a native test class | Name collision across targets | Collision detection or target-qualified filters | guide [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must) | +| Module translated with a guessed mapping | Notes file missing (silent `null`) or a placeholder | Resolver error for a missing notes file; refuse placeholder modules | guide [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must), [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) | +| Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | +| A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](../../../docs/writing-uts-spec-translator-skills.md#5-translation-rules) | +| Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](../../../docs/writing-uts-spec-translator-skills.md#93-renamed-merged-and-added-ids) | +| Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](../../../docs/writing-uts-spec-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](../../../docs/writing-uts-spec-translator-skills.md#12-lessons-learned) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/glossary.md b/uts/skills/uts-to-lang-skill-creator/references/glossary.md index 4e54dcfff..c98355179 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/glossary.md +++ b/uts/skills/uts-to-lang-skill-creator/references/glossary.md @@ -31,5 +31,5 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs | **Delegating spec/test** | One that says to run another module's tests against this client | | **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | | **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | -| **Phase N / generated-skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Generated-skill Phase 1" (selection, steps 0 and A–F) and "generated-skill Phase 2" (per spec, steps 1–7) are the generated `uts-to-` skill's workflow (guide [4](../../../docs/translator-skills/writing-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | +| **Phase N / generated-skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Generated-skill Phase 1" (selection, steps 0 and A–F) and "generated-skill Phase 2" (per spec, steps 1–7) are the generated `uts-to-` skill's workflow (guide [4](../../../docs/writing-uts-spec-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | | **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | diff --git a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md index 29c2ecc60..f2a2474e0 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md +++ b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md @@ -6,9 +6,9 @@ The harness and the skill must stay in step with five things that change indepen | What changed | How to detect it | What to update | |---|---|---| -| **The guide** | `git -C diff -- uts/docs/translator-skills/ uts/skills/uts-to-lang-skill-creator/` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](phase-4-design-record.md#61-step-4a-draft-the-design-record) and [7.6](phase-5-generate-skill.md#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | +| **The guide** | `git -C diff -- uts/docs/writing-uts-spec-translator-skills.md uts/skills/uts-to-lang-skill-creator/` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `; for a SHA recorded before the guide reached this path, also diff its earlier path, from `git log --follow --name-only -- uts/docs/writing-uts-spec-translator-skills.md`) | Map each changed guide section to the artifacts that implement it, using the "Guide" columns in [6.1](phase-4-design-record.md#61-step-4a-draft-the-design-record) and [7.6](phase-5-generate-skill.md#76-step-5f-write-skillmd) and the gap table; update them; re-run the Phase 3 and Phase 6 checks for those artifacts | | **The UTS docs or helper specs** | `git -C diff -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | -| **The UTS corpus** | The skill's own re-sync mode (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | +| **The UTS corpus** | The skill's own re-sync mode (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](../../../docs/writing-uts-spec-translator-skills.md#93-renamed-merged-and-added-ids) | | **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | | **The SDK** (API, hooks, test layout, CI) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | @@ -16,6 +16,6 @@ Rules for every maintenance run: - Follow the same ground rules ([section 2](../SKILL.md#2-ground-rules)) and stop points, including the model tier ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](../SKILL.md#21-your-inputs)). If the guide's "as of" commits have moved since a recorded read, re-check that read's conclusion against the updated Patterns to avoid before relying on it again. - Start from the working records; update the Repo profile and the harness design if anything they describe has changed. -- Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should)). +- Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should)). - After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](phase-6-validate-skill.md#81-run-the-resolver-on-every-module) and [8.2](phase-6-validate-skill.md#82-test-the-audit-itself). - Record the new spec SHA and repo SHA in the Design record and append a changelog entry. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md b/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md index 83d24bc68..c9f6be9f6 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-1-understand-repo.md @@ -18,7 +18,7 @@ Answer every item. For each, record the answer and its evidence in the Repo prof |---|---|---| | P-01 | **Language(s) and version(s)**: SDK language, test language (they can differ), toolchain version, language mode or edition | Toolchain pins (`global.json`, `.python-version`, `.nvmrc`, `go.mod`, `rust-toolchain*`, `Package.swift` tools version, Gradle `jvmToolchain`); `git ls-files` extension counts | | P-02 | **Build system and commands**: build the SDK; build the tests; run all tests; run one class; run one test; run by filter; target-qualified filters | Build files (`*.sln`/`*.csproj`, `pyproject.toml`/`tox.ini`/`noxfile.py`, `package.json` scripts, `build.gradle*`, `Package.swift`, `Makefile`, wrapper scripts); CI workflow steps; CONTRIBUTING. **Run** the build-tests and single-class commands and record the exit status. For interpreted languages, "build the tests" means collection (e.g. `pytest --collect-only`) plus the repo's type checker. Record the command that lists collected tests (e.g. `pytest --collect-only -q`, `dotnet test --list-tests`) | -| P-03 | **Test framework(s)**: runner, assertion library, parameterisation, suite-level fixtures (before/after all), teardown hooks, **static and runtime skip idioms**, unconditional-fail idiom, **collection rules** (what names and files the runner picks up) | Test dependencies in the build files; existing tests; the runner's config (`pytest.ini`, `conftest.py`, `xunit.runner.json`, `jest.config.*`). Check whether a runtime skip needs an extra package (guide [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states)) | +| P-03 | **Test framework(s)**: runner, assertion library, parameterisation, suite-level fixtures (before/after all), teardown hooks, **static and runtime skip idioms**, unconditional-fail idiom, **collection rules** (what names and files the runner picks up) | Test dependencies in the build files; existing tests; the runner's config (`pytest.ini`, `conftest.py`, `xunit.runner.json`, `jest.config.*`). Check whether a runtime skip needs an extra package (guide [7.1](../../../docs/writing-uts-spec-translator-skills.md#71-three-acceptable-end-states)) | | P-04 | **Test layout and naming**: test projects/targets/source sets, their directories, naming conventions for files, classes and methods, shared test-support projects, and which test targets can see which code | Directory listing of test roots; build files for test targets | | P-05 | **CI**: every workflow that builds or tests; which job runs which suites; OS and runtime matrix; network access; submodule checkout; secrets; how a new suite or test target would be picked up | `.github/workflows/*`, other CI configs; follow each test step to the command it runs | | P-06 | **Lint, format and strictness gates**: EditorConfig, formatters, linters, static analysis, type checkers; whether CI treats warnings as errors; whether the local build is laxer than CI | Lint configs, `Makefile` lint targets, CI lint jobs, compiler flags in build files | @@ -29,9 +29,9 @@ Answer every item. For each, record the answer and its evidence in the Repo prof | # | Find | How to find it | |---|---|---| -| P-09 | **Async model**: callbacks, futures/promises, `async`/`await`, coroutines; whether SDK APIs are callback-based, async, or both; the test runner's async support and its mode; whether the runner **virtualises time** | SDK public API signatures; runner plugins (e.g. pytest-asyncio mode); existing async tests (guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)) | +| P-09 | **Async model**: callbacks, futures/promises, `async`/`await`, coroutines; whether SDK APIs are callback-based, async, or both; the test runner's async support and its mode; whether the runner **virtualises time** | SDK public API signatures; runner plugins (e.g. pytest-asyncio mode); existing async tests (guide [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must)) | | P-10 | **Concurrency and threading**: which thread, queue or loop SDK callbacks run on; whether listeners can run concurrently with the test body; how to drain the SDK's internal work queue (this becomes `process_pending_events()`); whether the language checks concurrency safety at compile time | SDK source: dispatch, executor or event-loop code; how existing tests wait for callbacks | -| P-11 | **Time model**: **every time source** the SDK uses: timers, timed blocking waits, scheduled or delayed callbacks, wall-clock and monotonic reads | Grep the SDK source for the language's sleep, timer, wait, delay and now APIs; list each call site's owner (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must)) | +| P-11 | **Time model**: **every time source** the SDK uses: timers, timed blocking waits, scheduled or delayed callbacks, wall-clock and monotonic reads | Grep the SDK source for the language's sleep, timer, wait, delay and now APIs; list each call site's owner (guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must)) | | P-12 | **Runner parallelism**: are tests run in parallel by default, and how is it turned off per suite? | Runner docs and config; existing serialisation attributes or collection settings | | P-13 | **Injection points (test hooks)** for each of: WebSocket transport, HTTP client, clock/timers, reachability/network monitor, randomness (retry jitter, fallback-host shuffle). For each: symbol, how it is set (option, constructor argument, global, subclass), whether it is set at client construction, per-client or process-global, test-only or public | SDK client-options and any debug/test-options types; constructors of the transport, HTTP, timer and network layers; existing tests that replace them | | P-14 | **Internal-access mechanism**: how tests reach non-public code, and where the repo already uses it | e.g. `InternalsVisibleTo`, `@testable import`, private headers and module maps, same-package placement, `_`-prefixed conventions; grep for existing test-only accessors | @@ -46,7 +46,7 @@ Answer every item. For each, record the answer and its evidence in the Repo prof | P-18 | **Existing sandbox and integration support**: how native integration tests provision a sandbox app (from `test-app-setup.json`?), point clients at the sandbox, retry, clean up; any proxy or fault-injection tooling | Integration test directories and their fixtures; environment variables they read | | P-19 | **`ably-common`**: submodule path and name (it may not be called `ably-common`), whether it is initialised, and whether `test-resources/test-app-setup.json`, `msgpack_test_fixtures.json` and `encoding.json` are present | `.gitmodules`; `git submodule status`; list the files | | P-20 | **Existing UTS assets and their state**: any UTS harness, UTS-derived tests (`UTS:` tags), `deviations.md` files, previous `uts-to-*` skills in this repo. For each asset: does it compile, do its tests pass, which spec SHA do headers name, which helper-spec symbols exist | `grep -rn "UTS:" `; search for `MockWebSocket`, `MockHttpClient`, `SandboxApp`, `uts-proxy`, `deviations.md`, `.claude/skills/uts-*`, `.agents/skills/uts-*`, `.codex/skills/uts-*` | -| P-21 | **Sandbox access**: whether tests can reach `sandbox.realtime.ably-nonprod.net` from developer machines and CI | Existing integration tests; CI network settings (guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)). Probing the sandbox, or running existing integration tests for P-20, is network access: **STOP-6** first | +| P-21 | **Sandbox access**: whether tests can reach `sandbox.realtime.ably-nonprod.net` from developer machines and CI | Existing integration tests; CI network settings (guide [2.3](../../../docs/writing-uts-spec-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)). Probing the sandbox, or running existing integration tests for P-20, is network access: **STOP-6** first | For a first pass, run `python3 /scripts/survey_repo.py [ …]` (read-only; it prints the HEAD SHA and dirty state, extension counts, likely test roots, submodule status, candidate mocks and fakes, existing `UTS:` tags and existing skills). It is a starting point, not an answer: adapt or extend the searches for the repo's language, and record each answer's evidence. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md b/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md index ea17a6ab9..f34150d51 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-2-design-harness.md @@ -14,23 +14,23 @@ Contents: - [4.8 Step 2h: derive tier feasibility](#48-step-2h-derive-tier-feasibility) - [4.9 Step 2i: STOP-3, approve the design and choose the scope](#49-step-2i-stop-3-approve-the-design-and-choose-the-scope) -**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](../../../docs/translator-skills/writing-translator-skills.md#2-the-harness-uts-test-infrastructure) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. +**Goal:** a written design for the harness in this language, derived from the Repo profile: for every capability guide [section 2](../../../docs/writing-uts-spec-translator-skills.md#2-the-harness-uts-test-infrastructure) requires, whether to **reuse** what exists, **wrap** it, **extend** it or **build** it new; how each helper spec maps to native code; which SDK hooks are still needed; where the code lives; and how each piece will be accepted. The user approves the design before you build anything. Design principles: -- **Follow guide [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)**: implement each helper spec once, and build the harness to look like the pseudocode. When an existing double has the right behaviour but a different shape, **wrap it in a pseudocode-shaped facade** rather than teaching the skill to translate into the old shape. +- **Follow guide [2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)**: implement each helper spec once, and build the harness to look like the pseudocode. When an existing double has the right behaviour but a different shape, **wrap it in a pseudocode-shaped facade** rather than teaching the skill to translate into the old shape. - **Reuse what is correct; don't reuse what is subtly wrong.** An existing double that misses a helper-spec guarantee (for example a synchronous `close()`) will make the UTS ports flaky. Extend or replace it for the UTS ports, leaving the native tests' behaviour unchanged. -- **Hooks** are injected at client construction, per-client where the SDK allows (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must), [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must)). +- **Hooks** are injected at client construction, per-client where the SDK allows (guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must), [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must)). ## 4.1 Step 2a: list the required capabilities -Start the gap table from rows G-01 to G-20 of the [uts-infra-design template](../assets/templates/uts-infra-design.md), which cover every requirement of guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must) to [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. +Start the gap table from rows G-01 to G-20 of the [uts-infra-design template](../assets/templates/uts-infra-design.md), which cover every requirement of guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must) to [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must). Split a row where its parts get different approaches (e.g. `MockWebSocket` reused but its alternative API built). Rows for concurrency knowledge (G-18) come from the Repo profile (P-09 to P-14) and are "missing" if the profile left them unknown. **Check:** every bullet and table row of guide section 2 maps to a gap-table row. ## 4.2 Step 2b: assess each capability: reuse, wrap, extend or build -This is the guide's "Reuse before you build" ([2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)) applied to every row. For each row, record the status (`present`, `partial`, `missing`, `n/a`), the evidence (file and symbol, or what you searched for), and the **approach**: +This is the guide's "Reuse before you build" ([2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must)) applied to every row. For each row, record the status (`present`, `partial`, `missing`, `n/a`), the evidence (file and symbol, or what you searched for), and the **approach**: | Approach | When | |---|---| @@ -60,22 +60,22 @@ Where a design question can only be answered by a production change, it becomes ## 4.4 Step 2d: design the wait and assertion helpers -Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the self-tests that prove them (guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. +Design these against the async and concurrency model from P-09 to P-12 and the existing helpers from P-17. For each, record the native symbol and whether you reuse an existing helper (only if it already meets the requirements), wrap it, or build it. The requirements are in the sources below; the self-tests that prove them (guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), rows 2, 4 and 7. | Helper | Requirements | |---|---| -| `AWAIT_STATE` (connection and channel) | guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | +| `AWAIT_STATE` (connection and channel) | guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events); `writing-test-specs.md` "State Transitions" | | `poll_until`, `poll_until_success` | `writing-test-specs.md`; `uts/README.md`; guide 5.7; `writing-derived-tests.md` "No real timers in unit tests" (deadlines) | -| `process_pending_events()` | `uts/README.md`; guide [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | -| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | -| Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must), [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | +| `process_pending_events()` | `uts/README.md`; guide [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | +| Wall-clock timeout wrapper | `writing-derived-tests.md` "Integration timeouts are wall-clock"; guide [5.6](../../../docs/writing-uts-spec-translator-skills.md#56-time-and-waiting) | +| Fake clock driver (`enable_fake_timers`, `ADVANCE_TIME`) | guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must), [5.6](../../../docs/writing-uts-spec-translator-skills.md#56-time-and-waiting); covers every time source in P-11 | | Thread-safe capture, log sink, caller-attributed failures | guide 2.2 | -| `assertContainsInOrder` | guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue) | +| `assertContainsInOrder` | guide [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue) | ## 4.5 Step 2e: design the sandbox and proxy helpers -- **Sandbox** (guide [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). -- **Proxy** (guide [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../../../docs/proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Include every operational requirement in guide 2.4. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). +- **Sandbox** (guide [2.3](../../../docs/writing-uts-spec-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers)): check the native integration tests' provisioning (P-18) against guide 2.3; reuse or wrap it if it meets every point, otherwise build `SandboxApp`. Decide how clients are pointed at the sandbox and how per-run vs per-test provisioning maps to the runner's suite fixtures (P-03). +- **Proxy** (guide [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [`proxy.md`](../../../docs/proxy.md)): design `ProxyManager` and `ProxySession` to every point of guide 2.4. Checking that the pinned release has a binary for every developer OS in P-07 is a network read: ask first (**STOP-6**), or mark it unverified until Phase 3 row 10 and plan platform gating where it can't run. Include every operational requirement in guide 2.4. Default the session's endpoint to the sandbox, because two corpus specs pass only `rules:` (guide [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue)). Decide auth through the proxy (D-18). ## 4.6 Step 2f: propose the SDK test hooks @@ -93,7 +93,7 @@ These proposals are presented at STOP-3 and need individual approval at **STOP-4 ## 4.7 Step 2g: plan placement, harness tests and build order - **Placement** (guide 2.2, "Placement" and "Recommended layout"): name the harness library and its per-tier directories, the directory and target for shared test-support code (importing no test framework), port-only harness code, and each module's helpers. Put white-box suites, and module helpers they need, where internals are visible (P-14; guide 2.6). -- **Harness tests:** the per-tier smoke tests and helper self-tests of guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must), item by item, for every tier you plan to support. They are permanent and run in CI; they carry no `UTS:` tag and are never copied by the skill; they live outside every directory the mapping will give the resolver as a `targetDir`. Name the command that runs each tier's set plus the common self-tests (guide 2.7 "Selectable"); it goes into the mapping's `harness` entry (D-03). Name the CI job that runs it (5.2 row 13). The acceptance checks in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. +- **Harness tests:** the per-tier smoke tests and helper self-tests of guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must), item by item, for every tier you plan to support. They are permanent and run in CI; they carry no `UTS:` tag and are never copied by the skill; they live outside every directory the mapping will give the resolver as a `targetDir`. Name the command that runs each tier's set plus the common self-tests (guide 2.7 "Selectable"); it goes into the mapping's `harness` entry (D-03). Name the CI job that runs it (5.2 row 13). The acceptance checks in [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks) say what each must prove. - **Build order:** follow the order in 5.2, dropping rows that are `present` or out of scope. ## 4.8 Step 2h: derive tier feasibility @@ -105,7 +105,7 @@ Fill a module × tier matrix: for each of `rest`, `realtime`, `objects` and each Present `uts-infra-design.md`: the gap table, the helper-spec mappings, the wait-helper design, the sandbox and proxy design, the hook proposals, placement, the build order and the tier matrix. Ask the user to choose: - **(a) Build the missing harness now**, all or a named subset. Recommend this option (the skill depends on the harness), but the user chooses. Continue with Phase 3. -- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4, step C](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) requires (see the readiness data in [7.1](phase-5-generate-skill.md#71-step-5a-create-the-layout-and-the-mapping-file)). +- **(b) Plan it only.** Keep the build order, with each row's acceptance check and size, as the plan in `uts-infra-design.md`. Phase 3 is skipped for the planned rows; the skill is generated for the tiers that are already ready, and the unready tiers stay in the mapping but the skill must refuse them with a message naming the missing harness piece, as guide [4, step C](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) requires (see the readiness data in [7.1](phase-5-generate-skill.md#71-step-5a-create-the-layout-and-the-mapping-file)). - **(c) Scope the skill to the tiers possible today.** As (b), without the plan. Under (b) and (c), G-19 and G-20 for every tier the skill will offer can't be planned or scoped out; Phase 3 and STOP-7 still run for them. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md b/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md index 1ac6edb7a..b11c84a03 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-3-build-harness.md @@ -2,7 +2,7 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. -**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has green smoke tests and helper self-tests (guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)), wired into CI. This is part of the skill's deliverable, not a preliminary. Skip the rows the user didn't choose to build at STOP-3. +**Goal:** the approved harness exists, compiles under the repo's strictest settings, and each supported tier has green smoke tests and helper self-tests (guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)), wired into CI. This is part of the skill's deliverable, not a preliminary. Skip the rows the user didn't choose to build at STOP-3. ## 5.1 Step 3a: rules while building @@ -16,31 +16,31 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs Each step has an acceptance check; don't start the next step until it passes (or the user accepts a documented exception). **Bound:** if a row's check still fails after three fix attempts, stop (**STOP-10**) with a summary: the failure, what you tried, your diagnosis, and the options (change the design and log it, accept a documented exception, or descope the tiers the row blocks). -The smoke tests and self-tests in the checks below are the permanent harness tests of guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) (MUST): build them as this phase's gates, and keep them in the repo and in CI afterwards, where the generated skill's preflight runs them. Each row's check is an explicit assertion in a harness test, not a manual observation. +The smoke tests and self-tests in the checks below are the permanent harness tests of guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) (MUST): build them as this phase's gates, and keep them in the repo and in CI afterwards, where the generated skill's preflight runs them. Each row's check is an explicit assertion in a harness test, not a manual observation. Rows 4–7 depend on each other: build them in order, but run an acceptance item that needs a component from a later row as soon as that component exists (the fake-clock check needs a transport mock; the REST row's realtime-driven request needs `MockWebSocket`; the realtime row's state waits need row 7), and don't leave row 7 until every row 4–7 item passes. Common items that need an integration component (the integration state-wait helper, integration teardown) run at row 9, once that component exists; they are part of row 9's acceptance. | Order | Build | Acceptance check | |---|---|---| | 1 | UTS test target/project and its tier selection (directories or filters per tier), wired so that `build-tests` compiles it | A minimal harness test (one trivial passing test; delete it once row 2's self-tests exist) compiles, is collected and passes under the single-class filter and under the tier's harness-test command | -| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken, and a repeated state passes (guide [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)); capture lists are safe under concurrent appends; a failing helper reports the caller's line | +| 2 | Library core: thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failure helper | Tests of the helpers themselves: `assertContainsInOrder` passes on `[connecting, disconnected, connecting, connected]` vs `[disconnected, connecting, connected]` and fails when the order is broken, and a repeated state passes (guide [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue)); capture lists are safe under concurrent appends; a failing helper reports the caller's line | | 3 | Approved SDK hooks | Each hook replaces the real implementation for one client without affecting another client created afterwards (or, if process-global, that fact is recorded for decision D-14); the existing tests still pass | -| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it, and nothing waits on the real clock (guide [2.1](../../../docs/translator-skills/writing-translator-skills.md#21-sdk-test-hooks-must)); one advance runs cascaded work to quiescence (zero-delay reschedules and timers created mid-advance) | -| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, REST entry, each as an explicit assertion | -| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, realtime entry, each as an explicit assertion | -| 7 | Wait helpers per [4.4](phase-2-design-harness.md#44-step-2d-design-the-wait-and-assertion-helpers) | **Helper self-tests:** every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Common and Unit, fake clock entries, each as an explicit assertion; in addition, `AWAIT_STATE` completes for a state reached before or during the wait (the teardown self-test follows your SDK's `close()` behaviour, guide 5.4) | +| 4 | Fake clock wired through the clock hook | A timer-driven SDK behaviour doesn't happen before `ADVANCE_TIME` and does happen after it; a timed blocking wait in the SDK (if P-11 found any) is also driven by the fake clock; the async runtime's timers fire under it, and nothing waits on the real clock (guide [2.1](../../../docs/writing-uts-spec-translator-skills.md#21-sdk-test-hooks-must)); one advance runs cascaded work to quiescence (zero-delay reschedules and timers created mid-advance) | +| 5 | `MockHttpClient` per `mock_http.md`, plus the legacy `queue_*` API or its documented mapping | **REST unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, REST entry, each as an explicit assertion | +| 6 | `MockWebSocket` per `mock_websocket.md`, including the alternative API, undocumented members and raw frames | **Realtime unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, realtime entry, each as an explicit assertion | +| 7 | Wait helpers per [4.4](phase-2-design-harness.md#44-step-2d-design-the-wait-and-assertion-helpers) | **Helper self-tests:** every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Common and Unit, fake clock entries, each as an explicit assertion; in addition, `AWAIT_STATE` completes for a state reached before or during the wait (the teardown self-test follows your SDK's `close()` behaviour, guide 5.4) | | 8 | `MockNetworkListener` (if the SDK has a network monitor), `MockVCDiff*` (if the SDK supports deltas) | A smoke test drives each one | -| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Direct sandbox entry, each as an explicit assertion | -| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Proxy entry, each as an explicit assertion; clients authenticate as decided in D-18 | -| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, objects entry, each as an explicit assertion. The smoke test lives beside the module helpers (outside every `targetDir`) and is selected by the unit tier's harness command | +| 9 | `SandboxApp`, `ably-common` access (after **STOP-6**) | **Direct integration smoke test and self-tests:** Every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Direct sandbox entry, each as an explicit assertion | +| 10 | `ProxyManager` and `ProxySession` (pinned version; every developer OS) (after **STOP-6**) | **Proxy smoke test and self-tests:** Every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Proxy entry, each as an explicit assertion; clients authenticate as decided in D-18 | +| 11 | Module helpers, e.g. the `standard_test_pool.md` implementation and a plugin client-options builder | **Objects unit smoke test and self-tests:** Every item in guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)'s Unit, objects entry, each as an explicit assertion. The smoke test lives beside the module helpers (outside every `targetDir`) and is selected by the unit tier's harness command | | 12 | Harness README describing what exists, with its Known gaps | See [5.3](#53-step-3c-write-the-harness-readme) | -| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing; each tier's smoke tests and self-tests run, ungated, in that tier's job (guide [2.5](../../../docs/translator-skills/writing-translator-skills.md#25-build-and-ci-wiring-must), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)); the repo's existing test commands and jobs (e.g. bare `pytest`, `dotnet test` on the solution) don't also collect the UTS or harness tests (exclude them via `testpaths`/`--ignore`, a solution filter or a separate project), unless that job is deliberately their one CI home | +| 13 | CI wiring (needs **STOP-5** approval), or a suggested change for the Final report | Each suite runs in exactly one CI job per platform it targets; nothing reports green without executing; each tier's smoke tests and self-tests run, ungated, in that tier's job (guide [2.5](../../../docs/writing-uts-spec-translator-skills.md#25-build-and-ci-wiring-must), [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)); the repo's existing test commands and jobs (e.g. bare `pytest`, `dotnet test` on the solution) don't also collect the UTS or harness tests (exclude them via `testpaths`/`--ignore`, a solution filter or a separate project), unless that job is deliberately their one CI home | Then compile all test targets as strictly as CI does (P-06) and run the repo's lint gates over the new code. ## 5.3 Step 3c: write the harness README -Write a README next to the harness covering everything guide [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) and [2.2](../../../docs/translator-skills/writing-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: +Write a README next to the harness covering everything guide [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout) and [2.2](../../../docs/writing-uts-spec-translator-skills.md#22-a-shared-test-library-implementing-the-helper-specs-must) say it should cover (3.1's list, and 2.2's threading facts for `mock_websocket.md`). Procedure detail: - take the symbol map from the [4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code) tables as built, and the known gaps from the gap table's `partial` and `missing` rows; - add the layout (shared, port-only, module helpers, test targets) and the complete command that runs each tier's smoke tests and self-tests; diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md b/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md index ab444353b..fcaf4a89d 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-4-design-record.md @@ -10,32 +10,32 @@ For each decision below, propose an answer derived from the Repo profile, the bu | ID | Decision | Guide reference | How to propose it | |---|---|---|---| -| D-01 | Skill name and location | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-` so Codex loads it too (guide 3.1); record the link or copy method | -| D-02 | Script language and minimum runtime version | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | Python 3 unless P-07 shows it isn't available on every developer platform | -| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file), [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](phase-2-design-harness.md#47-step-2g-plan-placement-harness-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that. Record the mapping's `harness` entry (root, README, per-tier sources and harness-test commands from 4.7) | -| D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability), [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | -| D-05 | Name collisions with native tests | [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | -| D-06 | **Tag syntax** | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | -| D-07 | File header format | [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@` (plus `(locally modified; blob )` for a locally modified spec, guide 9.1), disclosures; integration/proxy also the corresponding unit spec | -| D-08 | **`deviations.md` location** per owning test module | [7.3](../../../docs/translator-skills/writing-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | -| D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | -| D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | -| D-11 | Pending idiom for a missing API (registered, skipped, not commented out) | [7.1](../../../docs/translator-skills/writing-translator-skills.md#71-three-acceptable-end-states), [7.4](../../../docs/translator-skills/writing-translator-skills.md#74-language-inapplicable-inputs) | The framework's static skip or pending marker | -| D-12 | **Internal-access ladder**: the existing exposure to check first, the sanctioned way to add a test-only exposure, who to escalate to; where the white-box spec list lives | [5.9](../../../docs/translator-skills/writing-translator-skills.md#59-internal-access-white-box-unit-specs) | From P-14. List white-box specs in the module notes or the mapping file | -| D-13 | **Delegation policy** for delegating specs and tests | [5.3](../../../docs/translator-skills/writing-translator-skills.md#53-structural-variants) | The guide's recommendation (one test with the delegating tag, running the referenced REST cases through a shared helper; else pending with a `NOTE`). The guide calls this new policy: confirm with the user | -| D-14 | Parallelism: serialise UTS suites, or rely on per-client hooks | [2.6](../../../docs/translator-skills/writing-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | From P-12 and the hooks as built | -| D-15 | Teardown mechanism (scope, hook, `finally`) and suite-level fixtures for `BEFORE ALL TESTS` | [5.4](../../../docs/translator-skills/writing-translator-skills.md#54-setup-and-teardown) | From P-03 and the harness | -| D-16 | Protocol-variant rendering (`PROTOCOL`) | [5.3](../../../docs/translator-skills/writing-translator-skills.md#53-structural-variants) | The framework's parameterisation over `json`/`msgpack`; tag stays singular | -| D-17 | Default timeouts and poll interval | [5.6](../../../docs/translator-skills/writing-translator-skills.md#56-time-and-waiting) | As guide 5.6 "Every wait is bounded": the spec's timeout where it states one; otherwise an explicit 10–30 s timeout on every integration `poll_until` and 10–15 s helper defaults for state waits. Record the chosen values | -| D-18 | Auth through the proxy | [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | As designed in [4.5](phase-2-design-harness.md#45-step-2e-design-the-sandbox-and-proxy-helpers) | -| D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | -| D-20 | **Commit policy**; project/solution-file edits | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | -| D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](../../../docs/translator-skills/writing-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | -| D-22 | `allowed-tools` and remote access | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash Read Edit Write` (space-separated, guide 3.3); no `WebFetch`, because everything is read from the local clone (the skill never consults the reference implementations, [2.1](../SKILL.md#21-your-inputs)) | -| D-23 | Which modules get notes files, and which are placeholders | [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout), [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | +| D-01 | Skill name and location | [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout), [3.3](../../../docs/writing-uts-spec-translator-skills.md#33-frontmatter-and-arguments) | `uts-to-` at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-` so Codex loads it too (guide 3.1); record the link or copy method | +| D-02 | Script language and minimum runtime version | [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout), [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must), [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci) | Python 3 unless P-07 shows it isn't available on every developer platform | +| D-03 | **Target directory per module × tier** (the mapping), unique namespaces, hand-maintained entries | [3.2](../../../docs/writing-uts-spec-translator-skills.md#32-the-mapping-file), [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must) (placement follows visibility) | Follow the UTS test target from Phase 3 and the placement from [4.7](phase-2-design-harness.md#47-step-2g-plan-placement-harness-tests-and-build-order). Keep a module segment after the tier. Also record the default layout `--create ` scaffolds for a new module (e.g. `//`); the resolver implements exactly that. Record the mapping's `harness` entry (root, README, per-tier sources and harness-test commands from 4.7) | +| D-04 | **Naming**: file, class/suite and test-function names; the resolver's name derivation; runner collection rules | [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability), [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must) | Give three worked examples (a `_test.md` spec, a plain spec, one in a sub-directory) and confirm the runner collects each name (P-03) | +| D-05 | Name collisions with native tests | [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must) | Detect in the resolver, or always run with a target-qualified filter. Where the runner imports test files by basename, also apply guide 8.1's file-name uniqueness rule across the whole mapping | +| D-06 | **Tag syntax** | [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability) | `// UTS: `, or the language's line-comment marker (`# UTS: `), immediately above the test, before its attributes or decorators (guide 3.4 file outline) | +| D-07 | File header format | [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability), [9.1](../../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must) | Source spec path, tier, `ably/specification@` (plus `(locally modified; blob )` for a locally modified spec, guide 9.1), disclosures; integration/proxy also the corresponding unit spec | +| D-08 | **`deviations.md` location** per owning test module | [7.3](../../../docs/writing-uts-spec-translator-skills.md#73-deviationsmd) | One per owning test module, next to its suites | +| D-09 | **Runtime skip idiom** (env-gated on `RUN_DEVIATIONS`) and the reproduction command | [7.1](../../../docs/writing-uts-spec-translator-skills.md#71-three-acceptable-end-states) | From P-03; must read the environment at run time | +| D-10 | **Fail-fast idiom** for UTS spec errors | [7.1](../../../docs/writing-uts-spec-translator-skills.md#71-three-acceptable-end-states) | The framework's unconditional failure with the message `UTS spec error — fix the spec first; see deviations.md` | +| D-11 | Pending idiom for a missing API (registered, skipped, not commented out) | [7.1](../../../docs/writing-uts-spec-translator-skills.md#71-three-acceptable-end-states), [7.4](../../../docs/writing-uts-spec-translator-skills.md#74-language-inapplicable-inputs) | The framework's static skip or pending marker | +| D-12 | **Internal-access ladder**: the existing exposure to check first, the sanctioned way to add a test-only exposure, who to escalate to; where the white-box spec list lives | [5.9](../../../docs/writing-uts-spec-translator-skills.md#59-internal-access-white-box-unit-specs) | From P-14. List white-box specs in the module notes or the mapping file | +| D-13 | **Delegation policy** for delegating specs and tests | [5.3](../../../docs/writing-uts-spec-translator-skills.md#53-structural-variants) | The guide's recommendation (one test with the delegating tag, running the referenced REST cases through a shared helper; else pending with a `NOTE`). The guide calls this new policy: confirm with the user | +| D-14 | Parallelism: serialise UTS suites, or rely on per-client hooks | [2.6](../../../docs/writing-uts-spec-translator-skills.md#26-understand-the-sdks-concurrency-model-must) | From P-12 and the hooks as built | +| D-15 | Teardown mechanism (scope, hook, `finally`) and suite-level fixtures for `BEFORE ALL TESTS` | [5.4](../../../docs/writing-uts-spec-translator-skills.md#54-setup-and-teardown) | From P-03 and the harness | +| D-16 | Protocol-variant rendering (`PROTOCOL`) | [5.3](../../../docs/writing-uts-spec-translator-skills.md#53-structural-variants) | The framework's parameterisation over `json`/`msgpack`; tag stays singular | +| D-17 | Default timeouts and poll interval | [5.6](../../../docs/writing-uts-spec-translator-skills.md#56-time-and-waiting) | As guide 5.6 "Every wait is bounded": the spec's timeout where it states one; otherwise an explicit 10–30 s timeout on every integration `poll_until` and 10–15 s helper defaults for state waits. Record the chosen values | +| D-18 | Auth through the proxy | [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier) | As designed in [4.5](phase-2-design-harness.md#45-step-2e-design-the-sandbox-and-proxy-helpers) | +| D-19 | **Re-sync mode**: flag name, whether to keep a per-module manifest | [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should) | `--resync`; manifest optional; ship the corpus scanner in `scripts/` (guide [6](../../../docs/writing-uts-spec-translator-skills.md#6-pseudocode-construct-catalogue) SHOULD; [7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | +| D-20 | **Commit policy**; project/solution-file edits | [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci) | No commits by default; list changed files. Record each project/solution-file edit approved at STOP-5, and any the user chose to make by hand | +| D-21 | Fix-attempt bounds: per test in the skill's evaluate mode; per pilot spec in Phase 6 | [7.5](../../../docs/writing-uts-spec-translator-skills.md#75-stopping-rules-for-evaluate-mode-should) | Three in both cases; in the pilot, count every regeneration of a spec, whatever the defect | +| D-22 | `allowed-tools` and remote access | [3.3](../../../docs/writing-uts-spec-translator-skills.md#33-frontmatter-and-arguments), [9.4](../../../docs/writing-uts-spec-translator-skills.md#94-local-clone-vs-fetching-main) | `Bash Read Edit Write` (space-separated, guide 3.3); no `WebFetch`, because everything is read from the local clone (the skill never consults the reference implementations, [2.1](../SKILL.md#21-your-inputs)) | +| D-23 | Which modules get notes files, and which are placeholders | [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout), [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) | A notes file for every module whose surface diverges from the pseudocode (P-15); at least `objects`, if the SDK implements it (a placeholder otherwise) | | D-24 | Location of the working records, and whether they are kept in the repo | [1.4](../SKILL.md#14-outputs) of this skill | Default `.claude/skills/uts-to-/generation/`, kept for maintenance | | D-25 | Pilot specs and mode per tier | [8.3](phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier) of this skill | One per ready module × tier in scope, and at least one per module with a full notes file, chosen by the criteria there | -| D-26 | Model tier for skill runs: pin the model, or state it | [Model tier](../../../docs/translator-skills/writing-translator-skills.md#model-tier), [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | No portable frontmatter field pins a model: state the Opus-class requirement in the opening lines of `SKILL.md` and in `compatibility` (guide 3.3). Add Claude Code's `model` field only if the user accepts that it isn't portable. Either way, the skill's final report records the model | +| D-26 | Model tier for skill runs: pin the model, or state it | [Model tier](../../../docs/writing-uts-spec-translator-skills.md#model-tier), [3.3](../../../docs/writing-uts-spec-translator-skills.md#33-frontmatter-and-arguments) | No portable frontmatter field pins a model: state the Opus-class requirement in the opening lines of `SKILL.md` and in `compatibility` (guide 3.3). Add Claude Code's `model` field only if the user accepts that it isn't portable. Either way, the skill's final report records the model | ## 6.2 Step 4b: STOP-8, confirm diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md b/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md index d3144ec68..5ce16ce06 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-5-generate-skill.md @@ -15,9 +15,9 @@ Generate the files in this order: each one depends on the previous ones. `SKILL. ## 7.1 Step 5a: create the layout and the mapping file -1. Create the directory layout from guide [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) (plus the working-records directory, D-24). Then create the `.agents/skills/uts-to-` link (or the copy and its CI check) that D-01 chose, so Codex loads the skill too. -2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file). -3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data, and guide [4, step C](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) refuses unready tiers), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. +1. Create the directory layout from guide [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout) (plus the working-records directory, D-24). Then create the `.agents/skills/uts-to-` link (or the copy and its CI check) that D-01 chose, so Codex loads the skill too. +2. Write `uts-package-mapping.json` from D-03, in the shape of guide [3.2](../../../docs/writing-uts-spec-translator-skills.md#32-the-mapping-file). +3. Record tier readiness as data in the mapping (guide 3.2 allows per-module defaults as data, and guide [4, step C](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) refuses unready tiers), so the skill can read it without the working records. *Procedure recommendation:* an `unready` object per module, e.g. `"unready": {"proxy": "G-16: ProxySession not built"}`, which the resolver reports as `tiers..ready: false` with `blockedBy`. **Acceptance:** @@ -26,14 +26,14 @@ Generate the files in this order: each one depends on the previous ones. `SKILL. - [ ] `notes` paths are relative to the skill directory and the files exist (placeholders at this point are fine). - [ ] Hand-maintained entries are marked (e.g. in `_comment`) and listed in the Design record. - [ ] Every tier the harness design marks unready has an `unready` entry with the blocking gap-table row. -- [ ] The mapping has a `harness` entry (root, README, per-tier sources and the complete smoke-test and self-test command), every path in it exists, no harness test lies inside any tier `targetDir`, and no `targetDir` lies inside the harness root (guide [3.2](../../../docs/translator-skills/writing-translator-skills.md#32-the-mapping-file), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)). +- [ ] The mapping has a `harness` entry (root, README, per-tier sources and the complete smoke-test and self-test command), every path in it exists, no harness test lies inside any tier `targetDir`, and no `targetDir` lies inside the harness root (guide [3.2](../../../docs/writing-uts-spec-translator-skills.md#32-the-mapping-file), [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)). - [ ] The file is valid JSON, UTF-8, LF line endings. ## 7.2 Step 5b: write the resolver -Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. +Implement `scripts/resolve_uts.py` (or the language chosen in D-02) to every row of guide [8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must). It prints exactly one JSON object. -Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty, dirtyFiles}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](../../../docs/translator-skills/writing-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: +Output contract: the fields of guide 8.1, including `harness` (the mapping's harness entry with paths validated, still repo-relative), `testRoot` when the mapping declares a root, including `testFile` wherever the target file name isn't `.`, plus `ready` and `blockedBy` per tier (*procedure recommendation*: from the mapping's `unready` data, [7.1](#71-step-5a-create-the-layout-and-the-mapping-file)). Also implement the optional `specRepo` (`{sha, dirty, dirtyFiles}`; making it mandatory is a *procedure recommendation*), so the skill takes the clone's state for headers and reports ([8.1](../../../docs/writing-uts-spec-translator-skills.md#81-resolver-resolve_utspy-must), [9.1](../../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/writing-uts-spec-translator-skills.md#94-local-clone-vs-fetching-main)) from the script rather than recomputing it. Choose the namespace and build-target field names and document them in `SKILL.md`. For example: ```json { @@ -73,7 +73,7 @@ Errors are `{"ok": false, "code": "", "message": ""}`, with a distin ## 7.3 Step 5c: write the audit -Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](../../../docs/translator-skills/writing-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](../SKILL.md#21-your-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: +Implement `scripts/audit_translation.py` to every row and the parsing contract of guide [8.2](../../../docs/writing-uts-spec-translator-skills.md#82-audit-audit_translationpy-must). The spec-side parser is language-independent; write it yourself from the parsing contract (never by porting another SDK's audit, [2.1](../SKILL.md#21-your-inputs)). The test side needs four language-specific patterns, taken from the Design record, the harness and the construct catalogue: | Test-side pattern | Source | |---|---| @@ -96,69 +96,69 @@ Output: use the shape in guide 8.2 "Output shape (SHOULD)", including `notVerifi This step produces the construct table that goes into `SKILL.md` (and, for module-specific rows, into the notes). -1. **Start from every row of guide [section 6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. +1. **Start from every row of guide [section 6](../../../docs/writing-uts-spec-translator-skills.md#6-pseudocode-construct-catalogue)** (6.1 to 6.6) and the notes in [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue). Copy the "Construct" and "Meaning" columns; replace the Swift and Kotlin columns with one column for this repo. 2. **Fill the rendering for each row** from the harness you built and the SDK: the exact helper or framework call, with its real signature (read it from the source). Don't copy the guide's Swift or Kotlin cells, or anything seen in a reference-implementation read ([2.1](../SKILL.md#21-your-inputs)); they illustrate other SDKs. 3. **Scan the corpus for constructs the guide doesn't list** (guide section 6 says the catalogue isn't exhaustive). Run the scanner below over each module the skill covers, and add a row for every uppercase keyword or snake_case call you can't account for. 4. **Give every row a status:** - `mapped`: a direct rendering; - `mapped (inferred)`: an *(undocumented)* construct, meaning inferred from its uses (list one use); - - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](../../../docs/translator-skills/writing-translator-skills.md#72-harness-stand-ins-are-not-deviations)); - - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation; never substitute a similar mock outcome, guide [5.2](../../../docs/translator-skills/writing-translator-skills.md#52-fidelity)), and the Final report has a plan; + - `stand-in`: a sanctioned harness stand-in, disclosed in the file header (guide [7.2](../../../docs/writing-uts-spec-translator-skills.md#72-harness-stand-ins-are-not-deviations)); + - `not available`: the harness lacks the capability; the rendering says what the skill does (extend the mock, or record a Mock Infrastructure Limitation; never substitute a similar mock outcome, guide [5.2](../../../docs/writing-uts-spec-translator-skills.md#52-fidelity)), and the Final report has a plan; - `n/a`: no spec in the covered modules uses it (say so after scanning). 5. **No blank cells.** Collect every row the skill can't render and has no plan for, from one scan, into a single **STOP-9**. Corpus scanner: run `python3 /scripts/scan_constructs.py /uts/`. It prints the uppercase keywords and snake_case calls used in `pseudo` fences, outside comments and string literals, with counts; compare its output with the catalogue, and use it in Phase 2 to list the harness symbols the corpus uses. -The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_SET`, `LWW`); classify those under the guide's "wire constants" and enum rows rather than as new constructs. The scanner sees only `snake_case(` calls; method-style harness calls such as `mock_ws.active_connection.send_to_client(…)` are caught, but camelCase members (`onConnectionAttempt`) are not, so also read the helper specs' interfaces. Ship it in the generated skill as `scripts/scan_constructs.py` (copy this skill's script; it is language-independent), as guide [section 6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) and [3.1](../../../docs/translator-skills/writing-translator-skills.md#31-layout) recommend (SHOULD), because a re-sync re-scans the corpus (guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) step 2); record the choice in the Design record (D-19). +The keyword list also contains wire constants and enum values (`ATTACHED`, `MAP_SET`, `LWW`); classify those under the guide's "wire constants" and enum rows rather than as new constructs. The scanner sees only `snake_case(` calls; method-style harness calls such as `mock_ws.active_connection.send_to_client(…)` are caught, but camelCase members (`onConnectionAttempt`) are not, so also read the helper specs' interfaces. Ship it in the generated skill as `scripts/scan_constructs.py` (copy this skill's script; it is language-independent), as guide [section 6](../../../docs/writing-uts-spec-translator-skills.md#6-pseudocode-construct-catalogue) and [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout) recommend (SHOULD), because a re-sync re-scans the corpus (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should) step 2); record the choice in the Design record (D-19). **Acceptance:** - [ ] Every guide section 6 row is present with a rendering and a status; none blank. - [ ] Every scanner hit in the covered modules is either a catalogue row, a wire constant or enum value, a spec-local name, or internal pseudocode of a helper spec or `objects/PLAN.md` (implemented as harness or ignored, never translated). - [ ] Every rendering names a symbol that exists in the harness or the SDK (you read it), or is marked `not available` with a plan. -- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide [5.7](../../../docs/translator-skills/writing-translator-skills.md#57-waits-must-catch-events), [6.7](../../../docs/translator-skills/writing-translator-skills.md#67-notes-on-the-catalogue)). +- [ ] The guide's flagged traps are rendered correctly: `CONTAINS_IN_ORDER` as a subsequence; `poll_until` returns the settled value; `poll_until_success` as a separate helper; `AWAIT_STATE` race-free and failing fast on FAILED; both `AWAIT_STATE` spellings; `create_proxy_session(endpoint:)` per `proxy.md`, with the endpoint defaulted when a spec passes only `rules:` (guide [5.7](../../../docs/writing-uts-spec-translator-skills.md#57-waits-must-catch-events), [6.7](../../../docs/writing-uts-spec-translator-skills.md#67-notes-on-the-catalogue)). ## 7.5 Step 5e: write the module notes -Write `references/-mapping.md` for every module chosen in D-23. Cover the thirteen items in guide [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should) ("A module notes file should cover"), in that order. For each item, either fill it or write "Not applicable: ". The items that most need evidence: +Write `references/-mapping.md` for every module chosen in D-23. Cover the thirteen items in guide [3.4](../../../docs/writing-uts-spec-translator-skills.md#34-recommended-outlines-should) ("A module notes file should cover"), in that order. For each item, either fill it or write "Not applicable: ". The items that most need evidence: - **Item 1 (source of truth and runtime status):** name the IDL the notes apply (for `objects`, check whether the SDK follows `objects-features.md` in the clone or another variant; guide 3.4 item 1 explains the typed-SDK case) and whether the module is implemented, which decides whether evaluate mode is possible. - **Item 5 (type mapping):** for every polymorphic or union type in the pseudocode, the SDK type and which conversions throw or return null. Read the SDK source for each. -- **Item 8 (internal access):** the ladder from D-12 and the list of white-box specs. To find candidates, look for specs that construct internal classes or call members the public API doesn't have (the guide's white-box row in [6.6](../../../docs/translator-skills/writing-translator-skills.md#66-mocks-fixtures-and-harness) lists examples), then confirm each against the SDK's public surface. +- **Item 8 (internal access):** the ladder from D-12 and the list of white-box specs. To find candidates, look for specs that construct internal classes or call members the public API doesn't have (the guide's white-box row in [6.6](../../../docs/writing-uts-spec-translator-skills.md#66-mocks-fixtures-and-harness) lists examples), then confirm each against the SDK's public surface. - **Item 9 (helper-spec coverage):** the module's rows of the symbol tables from [4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code), as built, with each fixture helper's scope and every sanctioned stand-in. - **Item 12 (worked example):** translate one short real test from the module by hand, name each mechanical rewrite, and make sure it compiles (it becomes one of the skill's examples; [8.7](phase-6-validate-skill.md#87-verify-the-skills-own-examples)). -A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), "If the module's notes file exists but is only a placeholder"). +A module whose mapping you can't author yet gets a **placeholder** notes file that says so; the skill must then refuse that module (guide [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af), "If the module's notes file exists but is only a placeholder"). **Acceptance:** - [ ] All thirteen items present or marked not applicable with a reason. - [ ] Every SDK and harness symbol named in the notes exists at the recorded repo SHA. -- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](../../../docs/translator-skills/writing-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide [4, step 1](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17)). -- [ ] No line numbers; cite symbols and headings (guide [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned)). +- [ ] Overrides of the generic flow (reading list, naming rendering, deviations location, expected audit shortfalls) are explicit, and no naming override drops the spec point from test names (guide [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability)), so the skill can say "the notes win" (guide [4, step 1](../../../docs/writing-uts-spec-translator-skills.md#phase-2-per-spec-steps-17)). +- [ ] No line numbers; cite symbols and headings (guide [12](../../../docs/writing-uts-spec-translator-skills.md#12-lessons-learned)). ## 7.6 Step 5f: write `SKILL.md` -Follow the outline in guide [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide ["What the skill owns, and what it defers"](../../../docs/translator-skills/writing-translator-skills.md#what-the-skill-owns-and-what-it-defers)). +Follow the outline in guide [3.4](../../../docs/writing-uts-spec-translator-skills.md#34-recommended-outlines-should). Write each part as the language rendering of the guide's rule, and link to the UTS doc that defines the meaning rather than copying it (guide ["What the skill owns, and what it defers"](../../../docs/writing-uts-spec-translator-skills.md#what-the-skill-owns-and-what-it-defers)). | Part of `SKILL.md` | Content | Guide | |---|---|---| -| Frontmatter | The portable fields of guide 3.3: `name: uts-to-`, equal to the directory name; a pushy, third-person `description` of at most 1,024 characters with trigger phrases, a "Not for …" boundary and no `<` or `>`; `license`; `compatibility`; `allowed-tools` from D-22; string-valued `metadata` with `short-description`. Claude Code-only fields only as decided in D-26; the model statement in the opening lines | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | -| Usage guard | No argument → print the usage line and stop, without relying on `$ARGUMENTS` (read the argument from the user's words); the skill's own scripts and files named relative to the skill directory, never `.claude/skills/…`; placeholder paths only | [3.3](../../../docs/translator-skills/writing-translator-skills.md#33-frontmatter-and-arguments) | -| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state (the resolver's `specRepo`); for each selected spec in `dirtyFiles`, stop and ask, and if the user proceeds stamp its blob (guide 9.1) | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), [9.1](../../../docs/translator-skills/writing-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/translator-skills/writing-translator-skills.md#94-local-clone-vs-fetching-main) | -| Generated-skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af) | -| Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | -| Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should), [2.7](../../../docs/translator-skills/writing-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | -| Generated-skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README and helper sources from the resolver's `harness` output; for the reference test per tier, write `none yet` until [8.8](phase-6-validate-skill.md#88-name-the-reference-tests) names one | [4](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17), [8.3](../../../docs/translator-skills/writing-translator-skills.md#83-review-checklist-must-after-the-audit) | -| Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](../../../docs/translator-skills/writing-translator-skills.md#5-translation-rules) | -| Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](../../../docs/translator-skills/writing-translator-skills.md#6-pseudocode-construct-catalogue) | -| File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](../../../docs/translator-skills/writing-translator-skills.md#34-recommended-outlines-should) | -| Evaluation | Guide 7 rendered with D-08 to D-11 and D-21 | [7](../../../docs/translator-skills/writing-translator-skills.md#7-evaluation-and-deviations) | -| Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](../../../docs/translator-skills/writing-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](../../../docs/translator-skills/writing-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](../../../docs/translator-skills/writing-translator-skills.md#55-integration-tier-hygiene) | -| Verification | Guide 10 with the exact commands from the Repo profile | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | -| Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should), [9.3](../../../docs/translator-skills/writing-translator-skills.md#93-renamed-merged-and-added-ids) | -| Final report | The format, including the model line | [11](../../../docs/translator-skills/writing-translator-skills.md#11-final-report-format) | -| Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](../../../docs/translator-skills/writing-translator-skills.md#10-verification-and-ci) | +| Frontmatter | The portable fields of guide 3.3: `name: uts-to-`, equal to the directory name; a pushy, third-person `description` of at most 1,024 characters with trigger phrases, a "Not for …" boundary and no `<` or `>`; `license`; `compatibility`; `allowed-tools` from D-22; string-valued `metadata` with `short-description`. Claude Code-only fields only as decided in D-26; the model statement in the opening lines | [3.3](../../../docs/writing-uts-spec-translator-skills.md#33-frontmatter-and-arguments) | +| Usage guard | No argument → print the usage line and stop, without relying on `$ARGUMENTS` (read the argument from the user's words); the skill's own scripts and files named relative to the skill directory, never `.claude/skills/…`; placeholder paths only | [3.3](../../../docs/writing-uts-spec-translator-skills.md#33-frontmatter-and-arguments) | +| Required reading (step 0) | As guide 4 step 0 (including the "Pseudocode Conventions" section of `uts/README.md`), from the same clone as the module; record its SHA and dirty state (the resolver's `specRepo`); for each selected spec in `dirtyFiles`, stop and ask, and if the user proceeds stamp its blob (guide 9.1) | [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af), [9.1](../../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must), [9.4](../../../docs/writing-uts-spec-translator-skills.md#94-local-clone-vs-fetching-main) | +| Generated-skill Phase 1 (selection), steps A–E | As guide 4, with this repo's resolver commands; step C offers only tiers the resolver reports `present` and `ready`, and names `blockedBy` for the others | [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) | +| Harness preflight (step F) | As guide 4 step F: compile (or collect) the harness and the test target; run the chosen tier's smoke tests and self-tests with the resolver's `harness` command; check the harness README's Known gaps against the selected specs; on red, stop and report a harness or environment problem, never an SDK deviation; record the result for the final report | [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af), [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Harness reference | Per tier, from the resolver's `harness` output: harness root and README (full repo path), helper sources to read, smoke-test and self-test locations and run commands, the generated-test run command, and a pointer to the README's Known gaps; no inlined helper signatures; smoke tests named as wiring examples only. Also the mid-run harness-change rule (guide 5) | [3.4](../../../docs/writing-uts-spec-translator-skills.md#34-recommended-outlines-should), [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) | +| Generated-skill Phase 2 (per spec), steps 1–7 | As guide 4, with this repo's commands; the harness README and helper sources from the resolver's `harness` output; for the reference test per tier, write `none yet` until [8.8](phase-6-validate-skill.md#88-name-the-reference-tests) names one | [4](../../../docs/writing-uts-spec-translator-skills.md#phase-2-per-spec-steps-17), [8.3](../../../docs/writing-uts-spec-translator-skills.md#83-review-checklist-must-after-the-audit) | +| Translation rules | The language rendering of every rule in 5.1–5.9, including the decisions D-04 to D-18 | [5](../../../docs/writing-uts-spec-translator-skills.md#5-translation-rules) | +| Construct table | The catalogue from [7.4](#74-step-5d-fill-the-construct-catalogue), with the stop-and-ask rule for anything unmapped | [6](../../../docs/writing-uts-spec-translator-skills.md#6-pseudocode-construct-catalogue) | +| File template per tier | Header, imports, suite (serialised if D-14 says so), per-test shape; from the outline in 3.4 | [3.4](../../../docs/writing-uts-spec-translator-skills.md#34-recommended-outlines-should) | +| Evaluation | Guide 7 rendered with D-08 to D-11 and D-21 | [7](../../../docs/writing-uts-spec-translator-skills.md#7-evaluation-and-deviations) | +| Integration tiers | Direct-sandbox and proxy wiring with this harness's symbols (D-15, D-18); file templates | [2.3](../../../docs/writing-uts-spec-translator-skills.md#23-sandbox-provisioning-and-fixtures-must-for-integration-tiers), [2.4](../../../docs/writing-uts-spec-translator-skills.md#24-the-proxy-must-for-the-proxy-tier), [5.5](../../../docs/writing-uts-spec-translator-skills.md#55-integration-tier-hygiene) | +| Verification | Guide 10 with the exact commands from the Repo profile | [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci) | +| Re-sync mode | Guide 9.2 and 9.3, with D-19 | [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should), [9.3](../../../docs/writing-uts-spec-translator-skills.md#93-renamed-merged-and-added-ids) | +| Final report | The format, including the model line | [11](../../../docs/writing-uts-spec-translator-skills.md#11-final-report-format) | +| Commits | No commits by default; list changed files and hand-maintained project files (D-20) | [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci) | **Acceptance:** @@ -166,7 +166,7 @@ Follow the outline in guide [3.4](../../../docs/translator-skills/writing-transl - [ ] The skill is module-generic: module-specific rules live in the notes. - [ ] No inline copy of a semantic rule where a link to its source would do; in particular the `deviations.md` format is referenced, not redefined. - [ ] No line numbers; no personal or machine-absolute paths; no reference to other SDKs' skills (reference-implementation reads during creation are recorded in the working records, never cited in the skill). -- [ ] None of the guide's [Patterns to avoid](../../../docs/translator-skills/writing-translator-skills.md#patterns-to-avoid) appears in the skill, its notes, its scripts or the harness. +- [ ] None of the guide's [Patterns to avoid](../../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid) appears in the skill, its notes, its scripts or the harness. - [ ] The Harness reference names only files, filters and commands that exist, and doesn't inline helper signatures. - [ ] Every command in it is one the Repo profile records as run. (Running each through the skill, and compiling every example, are Phase 6 checks: [8.5](phase-6-validate-skill.md#85-exercise-the-skills-other-paths), [8.7](phase-6-validate-skill.md#87-verify-the-skills-own-examples).) - [ ] Every helper or SDK symbol it names exists. diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md index 5aef032db..55901d75a 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md @@ -24,7 +24,7 @@ Run the resolver on `/uts/rest`, `/uts/realtime` and `` will do); a mapping entry whose notes file is missing (use a scratch copy of the mapping); `--create` with an invalid name; `--create` on a scratch copy of a deliberately corrupt mapping (the file must be left unchanged). @@ -70,7 +70,7 @@ The candidates are examples only. Before choosing one, confirm it exists in your Use translate-only mode if the SDK lacks the feature; otherwise evaluate. Then: 1. Compile (and collect: every tagged test function is collected; count a parameterised or protocol-variant test once). -2. Audit; walk the review checklist (guide [8.3](../../../docs/translator-skills/writing-translator-skills.md#83-review-checklist-must-after-the-audit)). +2. Audit; walk the review checklist (guide [8.3](../../../docs/writing-uts-spec-translator-skills.md#83-review-checklist-must-after-the-audit)). 3. In evaluate mode, and after **STOP-6** for network tiers, run the class by filter and diagnose per `writing-derived-tests.md` Phase 2. 4. Review the generated file line by line against the spec yourself, and show it to the user. @@ -95,7 +95,7 @@ Run each path through the skill (in the session from [8.3](#83-pilot-translate-o 3. **Placeholder notes:** a module whose notes file is a placeholder → refused, with the reason. 4. **Unready tier:** a tier the mapping marks unready → not offered at step C, with its `blockedBy`. 5. **Unmapped module:** on a scratch copy of the mapping, a module without an entry → step B asks for a name and runs `--create`, which writes the D-03 default layout and preserves the other entries. -6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification (including a change outside the pseudocode, such as `## Protocol Variants`, reported as **changed**, and, run against a scratch clone of the spec repo (`git clone `; the spec clone itself is never edited, 2.2), an uncommitted edit to that spec reported as **changed**), that the header SHA is updated, and that only the survivors in guide [9.2](../../../docs/translator-skills/writing-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. +6. **Re-sync** (if D-19 ships it): set a pilot file's header SHA to an older spec commit that changed that spec (`git -C log --format=%H -- uts/`), run `--resync`, and check the classification (including a change outside the pseudocode, such as `## Protocol Variants`, reported as **changed**, and, run against a scratch clone of the spec repo (`git clone `; the spec clone itself is never edited, 2.2), an uncommitted edit to that spec reported as **changed**), that the header SHA is updated, and that only the survivors in guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should) step 4 are preserved. Restore the file afterwards. 7. **Commands:** every build, run, lint and audit command `SKILL.md` names has now been run through the skill; fix any that failed. 8. **Red harness:** on a scratch copy (or a temporary working-tree edit you revert afterwards), break one harness test (for example, make the WebSocket mock refuse every connection), or run an integration tier with the network unavailable → the skill stops at step F before generating anything, reports a harness or environment problem (not an SDK deviation), and names the failing test. Restore the change and confirm the preflight is green again. 9. **Known gap:** select a spec that uses a construct the harness README lists under Known gaps → the preflight reports the conflict and stops to ask (guide 4 step F), rather than generating a substituted rendering. If Known gaps is empty, use a scratch copy of the README with an entry for a construct the selected spec uses. @@ -107,7 +107,7 @@ Run the frontmatter validation of [7.6](phase-5-generate-skill.md#76-step-5f-wri ## 8.7 Verify the skill's own examples -Every code example in `SKILL.md` and the notes must compile (guide [12](../../../docs/translator-skills/writing-translator-skills.md#12-lessons-learned)). For each example, either: +Every code example in `SKILL.md` and the notes must compile (guide [12](../../../docs/writing-uts-spec-translator-skills.md#12-lessons-learned)). For each example, either: - name the real, compiled file it was copied from (and check it still matches), or - paste it into a scratch file in the test target, compile, and delete the scratch file. @@ -116,6 +116,6 @@ Record which examples were verified and how. ## 8.8 Name the reference tests -After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4, step 3](../../../docs/translator-skills/writing-translator-skills.md#phase-2-per-spec-steps-17)). Never name a smoke test. +After review, name one spec-derived test file per tier in `SKILL.md` as the file to read first (guide [4, step 3](../../../docs/writing-uts-spec-translator-skills.md#phase-2-per-spec-steps-17)). Never name a smoke test. **Phase 6 is done when:** the resolver passes on all three modules and all negative cases; every path in 8.5 behaves as expected (including the red-harness stop) and every command in `SKILL.md` has been run; every pilot ran after a green preflight; the audit passes every mutation and the sweep; each pilot compiles, audits clean (or every shortfall is accounted for) and, in evaluate mode, ends in one of the three end states; lint and the CI-strict compile pass; every example is verified. diff --git a/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py b/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py index f581637dc..81b4870ac 100755 --- a/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py +++ b/uts/skills/uts-to-lang-skill-creator/scripts/spec_clone_info.py @@ -18,7 +18,7 @@ SKILL_NAME = "uts-to-lang-skill-creator" SKILL_IN_CLONE = pathlib.Path("uts", "skills", SKILL_NAME) REQUIRED = { - "guide": "uts/docs/translator-skills/writing-translator-skills.md", + "guide": "uts/docs/writing-uts-spec-translator-skills.md", "utsReadme": "uts/README.md", "writingDerivedTests": "uts/docs/writing-derived-tests.md", "writingTestSpecs": "uts/docs/writing-test-specs.md", @@ -95,7 +95,7 @@ def main(argv): sha = git(clone, "rev-parse", "HEAD").strip() status = git(clone, "status", "--porcelain", "--untracked-files=all", "--", "uts", "specifications").splitlines() last = git(clone, "log", "-1", "--format=%H %cd", "--", - "uts/docs/translator-skills", SKILL_IN_CLONE.as_posix()).strip() + "uts/docs/writing-uts-spec-translator-skills.md", SKILL_IN_CLONE.as_posix()).strip() except (RuntimeError, OSError) as exc: return fail("GIT_ERROR", str(exc), specClone=str(clone)) From b56c81932b4465c0e33ca342130f9426137ad59d Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Fri, 9 Oct 2026 15:45:31 +0530 Subject: [PATCH 08/33] UTS: drop provenance notes from the translator skill guide and skill Remove notes about when and against which commits the guide was written (dates, reviewed SHAs, "as of" stamps, commit ids in examples) from the guide and the uts-to-lang-skill-creator skill. They read as noise and go stale. The reference-implementation links now point at main. The last-resort rules no longer refer to pinned commits: whatever is taken from the existing skills is checked against Patterns to avoid, which describes them when the guide was written. Local clones of those repos are read at their current HEAD. No requirements, IDs or checklist items change. --- .../writing-uts-spec-translator-skills.md | 37 +++++++++---------- uts/skills/uts-to-lang-skill-creator/SKILL.md | 14 +++---- .../assets/templates/design-record.md | 2 +- .../assets/templates/final-report.md | 2 +- .../references/glossary.md | 2 +- .../references/maintenance.md | 2 +- .../references/phase-6-validate-skill.md | 2 +- 7 files changed, 30 insertions(+), 31 deletions(-) diff --git a/uts/docs/writing-uts-spec-translator-skills.md b/uts/docs/writing-uts-spec-translator-skills.md index 436e190c1..75542cd5a 100644 --- a/uts/docs/writing-uts-spec-translator-skills.md +++ b/uts/docs/writing-uts-spec-translator-skills.md @@ -1,6 +1,6 @@ # Writing UTS Spec Translator Skills -This guide explains how to build a per-language agent skill (Claude Code and Codex) — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Its rules were distilled from two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), which appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). +This guide explains how to build a per-language agent skill (Claude Code and Codex) — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, use the [`uts-to-lang-skill-creator` skill](../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building the skill and its harness; this guide remains the authority on what the skill must contain. @@ -15,10 +15,10 @@ If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, **How to read this guide.** Requirements use three levels: - **MUST**: required for a skill that conforms to this guide. Most MUSTs protect translation fidelity (without them a skill produces tests that silently check less than the spec); the rest protect repeatability and reviewability. -- **SHOULD**: strongly recommended. Both existing skills learned these the hard way. +- **SHOULD**: strongly recommended. - **MAY**: optional, useful in some SDKs. -Facts about the existing skills and SDKs are **as of 2026-10-07**: spec commit `12540dcf`; ably-cocoa at `b5074d9b` and ably-java at `0ff24017` (the checkouts reviewed). Remote `main`, fetched the same day (ably-cocoa `2072e0cd`, ably-java `8b7d3f5f`), has no changes to either skill or UTS harness since those commits. They illustrate patterns, and some may since have been fixed. +Statements about the existing skills and harnesses describe them when this guide was written, to illustrate patterns; some may since have been fixed. ## Contents @@ -104,7 +104,7 @@ The unit tier needs injection points in production code: Mocks and the fake clock are injected **at client construction**, so the harness must install them before the client is created. -**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when a retry ran on wall-clock time through `FakeClock.waitOn`, which does a real timed wait, commit `c4502c67`.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. Two workable shapes: route every SDK delay (`sleep`, `call_later`, timeouts) through the clock hook, so `ADVANCE_TIME` runs them; or run unit tests on a virtual-time event loop whose `time()` the fake clock controls, in which case the wall-clock timeout wrapper must not use the loop's clock (it measures virtual time). +**The clock hook must cover every time source the SDK uses**: timers, and also blocking waits (condition variables, timed `wait`, dispatch-after). If any wait runs on the real clock, retries fire on wall-clock time regardless of `ADVANCE_TIME`, and "nothing happened before the advance" becomes unassertable. (ably-java found this when a retry ran on wall-clock time through `FakeClock.waitOn`, which does a real timed wait.) Likewise, check that the fake clock drives your async runtime's timers: in asyncio, for example, faking the wall clock doesn't fire `loop.call_later` callbacks; in .NET, route `Task.Delay` and timers through a `TimeProvider`-style abstraction. Two workable shapes: route every SDK delay (`sleep`, `call_later`, timeouts) through the clock hook, so `ADVANCE_TIME` runs them; or run unit tests on a virtual-time event loop whose `time()` the fake clock controls, in which case the wall-clock timeout wrapper must not use the loop's clock (it measures virtual time). If a hook doesn't exist, add it to the SDK first. Without these hooks, the unit tier is impossible. @@ -203,7 +203,7 @@ The harness has two kinds of test of its own, both permanent and both required: - **Per-tier smoke tests:** end-to-end wiring of one tier through the real SDK hooks (unit with mocks and the fake clock; direct sandbox; proxy). - **Helper self-tests:** contract tests that each harness helper obeys its helper spec, or this guide's rule for it. -**Why both.** Spec-derived tests can't be the harness's acceptance gate. They are regenerated, they may legitimately be red (a UTS spec error), and a failing one has to be triaged as SDK, spec or harness. A smoke-test or self-test failure is unambiguously a harness or environment fault. Scenario-only smoke tests aren't enough: they follow a happy path, so a helper that breaks its contract (a mock `close()` that notifies synchronously, a declared mock event that is never emitted, a state wait that samples) still passes, and the defect surfaces later as a flaky spec-derived test. Both existing SDKs hit this (as of 2026-10-07). One had connect-only smoke tests, env-gated and never run in CI, then removed them; its sampling state wait, error-swallowing polls and teardown hangs were later found only through flaky spec-derived tests. The other's scenario smoke tests, run in CI, caught a fake clock that waited on the real clock, but not its synchronous mock `close()` or a mock event type it declares and never emits. +**Why both.** Spec-derived tests can't be the harness's acceptance gate. They are regenerated, they may legitimately be red (a UTS spec error), and a failing one has to be triaged as SDK, spec or harness. A smoke-test or self-test failure is unambiguously a harness or environment fault. Scenario-only smoke tests aren't enough: they follow a happy path, so a helper that breaks its contract (a mock `close()` that notifies synchronously, a declared mock event that is never emitted, a state wait that samples) still passes, and the defect surfaces later as a flaky spec-derived test. Both existing SDKs hit this. One had connect-only smoke tests, env-gated and never run in CI, then removed them; its sampling state wait, error-swallowing polls and teardown hangs were later found only through flaky spec-derived tests. The other's scenario smoke tests, run in CI, caught a fake clock that waited on the real clock, but not its synchronous mock `close()` or a mock event type it declares and never emits. **Rules (MUST):** @@ -427,7 +427,7 @@ SKILL.md A module notes file should cover: -1. **Source of truth and runtime status:** the spec's IDL that the notes apply, including any typed-SDK variant (for objects, the proposed `RTTS` points (RTTS1–RTTS11 at the branch tip, `c5cc6b8a`) that both existing skills' notes follow; as of `12540dcf` they are on the unmerged `feature/liveobjects-cross-sdk-types-spec` branch, not in `objects-features.md`), and whether the module is implemented (whether evaluate mode is possible). +1. **Source of truth and runtime status:** the spec's IDL that the notes apply, including any typed-SDK variant (for objects, the proposed `RTTS` points (RTTS1–RTTS11) that both existing skills' notes follow, which are on the unmerged `feature/liveobjects-cross-sdk-types-spec` branch, not in `objects-features.md`), and whether the module is implemented (whether evaluate mode is possible). 2. **Layers:** spec names that denote several things (e.g. `LiveMap` as a creation value type, a public view and an internal node). 3. **Entry point and setup:** plugin registration, channel modes. 4. **Async model:** how `AWAIT`, deferred futures and errors render. @@ -887,10 +887,10 @@ Regenerate the whole file when most tests changed; otherwise regenerate only the Treat a removed ID plus an added ID as a **possible rename**, keyed on the descriptive name and position, and **re-check the assertions**: a renamed point can invert its semantics (`RTL15b1/serial-cleared-suspended-1` became `RTL15b2/serial-retained-suspended-1`). -Examples of the pattern, as of 2026-10-07 (spec `12540dcf`): +Examples of the pattern: -- **Rename not followed.** Both SDKs carried `realtime/unit/RTN16g2/recovery-key-null-inactive-0` after the spec renamed it to `realtime/unit/RTN16g3/recovery-key-null-inactive-0` (spec commit `d0d1c02f`): ably-cocoa `Test/UTS/unit/realtime/ConnectionRecoveryTests.swift` and ably-java `lib/src/test/kotlin/io/ably/lib/uts/unit/realtime/ConnectionRecoveryTest.kt`. The audit reports one missing and one orphan, but nothing prompted a re-run. -- **Additions not followed.** In ably-java `RealtimeObjectTest.kt`, two `RTO27` tests remained where the spec has a single `RTO27/channel-state-data-lifecycle-0` (the only `RTO27` Test ID in the spec's history, added in `65e6dd5e`), and `RTO20d4/mixed-null-serials-applies-non-null-0` (added in `6425db00`) had no test. +- **Rename not followed.** Both SDKs carried `realtime/unit/RTN16g2/recovery-key-null-inactive-0` after the spec renamed it to `realtime/unit/RTN16g3/recovery-key-null-inactive-0`: ably-cocoa `Test/UTS/unit/realtime/ConnectionRecoveryTests.swift` and ably-java `lib/src/test/kotlin/io/ably/lib/uts/unit/realtime/ConnectionRecoveryTest.kt`. The audit reports one missing and one orphan, but nothing prompted a re-run. +- **Additions not followed.** In ably-java `RealtimeObjectTest.kt`, two `RTO27` tests remained where the spec has a single `RTO27/channel-state-data-lifecycle-0` (the only `RTO27` Test ID the spec has ever had), and `RTO20d4/mixed-null-serials-applies-non-null-0` had no test. ### 9.4 Local clone vs fetching `main` @@ -1061,27 +1061,26 @@ Both contain a `SKILL.md`, a mapping file, a resolver and an audit script, and a ### Reference implementations (last resort) -This guide and the UTS docs are meant to be complete. If they don't answer a question while you create or maintain a skill and its harness, you MAY consult the existing skills and harnesses, read-only, at the commits this guide reviewed: +This guide and the UTS docs are meant to be complete. If they don't answer a question while you create or maintain a skill and its harness, you MAY consult the existing skills and harnesses, read-only: -| Repo @ commit | Skill | Harness | Harness smoke tests | Generated tests | +| Repo | Skill | Harness | Harness smoke tests | Generated tests | |---|---|---|---|---| -| [ably-java](https://github.com/ably/ably-java) @ `0ff24017` | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-java/tree/0ff24017d2692accb4acf66217fd42000543cc37/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | -| [ably-cocoa](https://github.com/ably/ably-cocoa) @ `b5074d9b` | [`.claude/skills/uts-to-swift`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/.claude/skills/uts-to-swift) | The `UTS` test target, [`Test/UTS`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS) (helpers in `Test/UTS/infra`; README `Test/UTS/README.md`), plus the shared test-support targets it depends on: [`Test/AblyTesting`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/AblyTesting) and [`Test/AblyLiveObjectsTesting`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/AblyLiveObjectsTesting) | None: they were retired, a [pattern to avoid](#patterns-to-avoid) | [`Test/UTS/unit`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS/unit), [`Test/UTS/integration`](https://github.com/ably/ably-cocoa/tree/b5074d9b25f0b6d502835b027fc407e698cd5802/Test/UTS/integration) | +| [ably-java](https://github.com/ably/ably-java) | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-java/tree/main/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-java/tree/main/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | +| [ably-cocoa](https://github.com/ably/ably-cocoa) | [`.claude/skills/uts-to-swift`](https://github.com/ably/ably-cocoa/tree/main/.claude/skills/uts-to-swift) | The `UTS` test target, [`Test/UTS`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS) (helpers in `Test/UTS/infra`; README `Test/UTS/README.md`), plus the shared test-support targets it depends on: [`Test/AblyTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyTesting) and [`Test/AblyLiveObjectsTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyLiveObjectsTesting) | None: they were retired, a [pattern to avoid](#patterns-to-avoid) | [`Test/UTS/unit`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/unit), [`Test/UTS/integration`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/integration) | Rules: 1. **The authority doesn't change.** Consult the references only after this guide, the UTS docs and the helper specs fail to answer the question. On any conflict, this guide (and, for semantics, the UTS docs, per [How it relates to the other UTS docs](#how-it-relates-to-the-other-uts-docs)) wins; record the conflict. 2. **Learn the pattern; never copy.** Don't copy, port or translate code, scripts, prose, tables or examples verbatim. Take how a problem was structured or solved, and write your own version in your language's idioms, on your SDK's hooks and harness. -3. **Check against [Patterns to avoid](#patterns-to-avoid)** before adopting anything. Both skills and both harnesses have documented defects, often in exactly the files you would consult. -4. **Read the pinned commits** linked above, where Patterns to avoid applies exactly. `main` may since have improved; if you consult `main`, re-check what you take against Patterns to avoid, and record which revision you used. -5. **Record each consultation**: the repo, commit and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. -6. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. +3. **Check against [Patterns to avoid](#patterns-to-avoid)** before adopting anything. Both skills and both harnesses have documented defects, often in exactly the files you would consult. The list describes them when this guide was written; some may since have been fixed, so check what you take either way. +4. **Record each consultation**: the repo, revision and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. +5. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. -Reading GitHub is network access: the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#21-your-inputs) gates it, or offers local clones that contain the pinned commits instead. +Reading GitHub is network access: the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#21-your-inputs) gates it, or offers local clones instead. ### Patterns to avoid -Observed in the existing skills and their harnesses as of 2026-10-07 (the reviewed checkouts, ably-cocoa `b5074d9b` and ably-java `0ff24017`; unchanged on remote `main` as fetched that day); some may since be fixed. Each is a pattern a new skill shouldn't copy. Check anything taken from the [reference implementations](#reference-implementations-last-resort) against this list. When the "as of" commits in the introduction move, update the links in that table and this list together. +Observed in the existing skills and their harnesses when this guide was written; some may since have been fixed. Each is a pattern a new skill shouldn't copy. Check anything taken from the [reference implementations](#reference-implementations-last-resort) against this list. **In both skills** diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md index 10f6e3e09..05f1739ae 100644 --- a/uts/skills/uts-to-lang-skill-creator/SKILL.md +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -39,13 +39,13 @@ You work from four primary sources: and, only as a last resort, the reference implementations below. -**Reference implementations (last resort).** The guide distils the lessons of the two existing skills and harnesses, so you shouldn't need them. You MAY consult them, read-only, at the commits the guide pins. The guide's [Reference implementations (last resort)](../../docs/writing-uts-spec-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: +**Reference implementations (last resort).** The guide covers what the two existing skills and harnesses teach, so you shouldn't need them. You MAY consult them, read-only. The guide's [Reference implementations (last resort)](../../docs/writing-uts-spec-translator-skills.md#reference-implementations-last-resort) lists the repos, paths and links. Consult them only when all of the following hold: 1. You have searched this skill, the guide, the UTS docs and the helper specs, and they don't answer the question. -2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) with `git -C show :` and `git -C ls-tree -r --name-only -- `, never checking out, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. If a clone lacks the pinned commit, fetching it is network access too. +2. You can read them without unapproved network access. Either read them from local clones the user named (input in [1.3](#13-required-inputs)) at their current HEAD (read the files, or use `git -C show HEAD:`), never checking out, modifying, pulling or fetching on the user's behalf. Or read them from GitHub after **STOP-6**; the user may approve reference reads once for the whole run, and you record that approval. Fetching or pulling a clone is network access too. 3. You take a **pattern**, not text: how something was structured or solved. Never copy, port or translate code, scripts, prose, tables or examples verbatim. Write every hook, helper, script, rule and example yourself, in this language's idioms, from the guide, the UTS docs and the target repo. -4. You check what you took against the guide's [Patterns to avoid](../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). If you read `main` instead of the pinned commits, record the revision and re-check against Patterns to avoid. -5. You record each consultation in the Design record ("Reference-implementation reads": repo, commit, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. +4. You check what you took against the guide's [Patterns to avoid](../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). Patterns to avoid describes the existing skills when the guide was written; some may since have been fixed, so check what you take either way, and record the revision you read. +5. You record each consultation in the Design record ("Reference-implementation reads": repo, revision, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)). @@ -76,13 +76,13 @@ At a stop point, present what you found and the options, then wait for the user' | **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | | **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | | **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | -| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases, reading a reference implementation on GitHub or fetching its commit, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | +| **STOP-6** | 1 onwards | Before any network access (running network tests, reading or downloading proxy releases, reading a reference implementation on GitHub or fetching its clone, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | | **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | | **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | | **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | | **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](references/phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), [8.4](references/phase-6-validate-skill.md#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | | **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | -| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, commit, path) and what they showed; the reading you propose | +| **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, revision, path) and what they showed; the reading you propose | ### 2.4 Never guess @@ -154,7 +154,7 @@ Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to | Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | Located by `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and confirmed with the user; otherwise **stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | | User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | | Model tier | The session's model, Opus-class ([2.8](#28-model-tier-and-sub-agents)) | Take it from your environment if it states your model; otherwise ask. If the model isn't Opus-class: **STOP-1** | -| Local clones of ably-cocoa and ably-java (optional) | Clones that contain commits `b5074d9b` and `0ff24017` | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | +| Local clones of ably-cocoa and ably-java (optional) | Clones of either repo, read at their current HEAD (ideally a recent `main`) | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | ### 1.4 Outputs diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md index 82da43241..a296cb86b 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md @@ -46,7 +46,7 @@ |---|---|---|---|---|---|---|---| ## Reference-implementation reads (last resort, 2.1) -| # | Date | Question | Guide/UTS sections searched | Repo @ commit : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | +| # | Date | Question | Guide/UTS sections searched | Repo @ revision : path | Pattern learned (not copied) | Checked against Patterns to avoid | Access (local clone / GitHub, STOP-6 approval) | |---|---|---|---|---|---|---|---| ## Changelog (fixes made to the skill, notes or harness) diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md index e7d48702b..4ccc5699e 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md @@ -39,7 +39,7 @@ Tiers not ready; `not available` catalogue rows; harness capabilities missing; a - Spec-repo doc issues (contradictions between guide and UTS docs; guide gaps revealed by reference-implementation reads): … ## Reference-implementation reads -Each read from the Design record (question, repo @ commit : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. +Each read from the Design record (question, repo @ revision : path, pattern learned, how it was checked against Patterns to avoid), with the guide gap it reveals and a draft guide fix. "None" if the guide sufficed. ## Suggested CI changes (not applied unless approved at STOP-5) diff --git a/uts/skills/uts-to-lang-skill-creator/references/glossary.md b/uts/skills/uts-to-lang-skill-creator/references/glossary.md index c98355179..ca838a1a9 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/glossary.md +++ b/uts/skills/uts-to-lang-skill-creator/references/glossary.md @@ -18,7 +18,7 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs | **Self-test** | A harness test that checks one helper obeys its helper-spec contract (guide 2.7); permanent, in CI, no tag | | **Preflight** | The generated skill's step F: compile, run the tier's smoke tests and self-tests, check Known gaps; stop on red | | **Reference test** | A reviewed, spec-derived test per tier that the skill reads before generating | -| **Reference implementations** | The existing `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java) skills and their harnesses, at the commits the guide pins: a last-resort, read-only reference for patterns, never copied and never an authority ([2.1](../SKILL.md#21-your-inputs)). Not a reference test | +| **Reference implementations** | The existing `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java) skills and their harnesses, linked from the guide: a last-resort, read-only reference for patterns, never copied and never an authority ([2.1](../SKILL.md#21-your-inputs)). Not a reference test | | **Opus-class model** | The most capable model tier available (Claude Opus or an equivalent); required for this procedure and its sub-agents ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) | | **Notes** | `references/-mapping.md`: how ably-js-shaped pseudocode maps to this SDK for one module | | **Resolver** | `resolve_uts.py`: validates a module and derives targets and names; its output is the single source of truth | diff --git a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md index f2a2474e0..e4db49028 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md +++ b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md @@ -14,7 +14,7 @@ The harness and the skill must stay in step with five things that change indepen Rules for every maintenance run: -- Follow the same ground rules ([section 2](../SKILL.md#2-ground-rules)) and stop points, including the model tier ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](../SKILL.md#21-your-inputs)). If the guide's "as of" commits have moved since a recorded read, re-check that read's conclusion against the updated Patterns to avoid before relying on it again. +- Follow the same ground rules ([section 2](../SKILL.md#2-ground-rules)) and stop points, including the model tier ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](../SKILL.md#21-your-inputs)). If the guide's Patterns to avoid has changed since a recorded read, re-check that read's conclusion against it before relying on it again. - Start from the working records; update the Repo profile and the harness design if anything they describe has changed. - Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should)). - After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](phase-6-validate-skill.md#81-run-the-resolver-on-every-module) and [8.2](phase-6-validate-skill.md#82-test-the-audit-itself). diff --git a/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md index 55901d75a..c8c7183fd 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md +++ b/uts/skills/uts-to-lang-skill-creator/references/phase-6-validate-skill.md @@ -43,7 +43,7 @@ Use one known-good test file (the pilot output from [8.3](#83-pilot-translate-on | Delete the native assertion that renders a spec `FAILS WITH` (awaited or not) | A positive assertion shortfall for that test | | Delete the poll call that renders a spec `poll_until` (with no `AWAIT` on the spec line) | An await shortfall for that test (spec polls count as waits) | -**Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l`` (as of `12540dcf`: rest 571, realtime 554, objects 339). +**Whole-corpus sweep:** run the audit on every spec the resolver lists for each of the three modules, against an empty test file. Every run must print one parseable JSON object and exit with the "ID problems" code (or report "not verifiable" for ID-less specs such as `rest/unit/encoding/msgpack_interop.md`). No crashes. Cross-check the spec-side ID count per module against ``grep -rhoE '\*\*Test ID\*\*: `[^`]+`' /uts/ | sort -u | wc -l``. ## 8.3 Pilot-translate one spec per available tier From 361ec4a698b8ac92413e398bd9c817db4fee2baf Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Fri, 9 Oct 2026 15:55:58 +0530 Subject: [PATCH 09/33] UTS: drop the count date from uts/README.md The spec counts are a snapshot; point readers at the find command for current numbers instead of stamping a date. --- uts/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/uts/README.md b/uts/README.md index b347fedc9..d54487f01 100644 --- a/uts/README.md +++ b/uts/README.md @@ -66,7 +66,7 @@ uts/ | Helper specs | 4 | Mock and fixture infrastructure definitions | | **Total** | **150** | | -Counts as of 2026-10-08; `find uts/ -name '*.md'` is authoritative (it also lists `objects/PLAN.md`, which is not a spec). +For current counts, `find uts/ -name '*.md'` is authoritative (it also lists `objects/PLAN.md`, which is not a spec). ## Three Test Tiers From 06049e6532a515c0ae6250968e6162571c8ab598 Mon Sep 17 00:00:00 2001 From: sacOO7 Date: Fri, 9 Oct 2026 19:03:58 +0530 Subject: [PATCH 10/33] UTS: orient skill creator runs and scope them by SDK capability The skill creator now starts every run with a read-only Orient step: - check the repo is an Ably Pub/Sub SDK (owner ably; ably- or ably-pubsub-, with deny and allow lists in assets/eligibility.json) - detect REST and Realtime capabilities, door entry points and LiveObjects support, using class names read from the spec clone at run time, plus SDK-specific aliases in assets/capability-names.json - classify the repo state and always ask for the mode: create a new skill, or upgrade/fix an existing uts-to-* skill (diff-driven, full gap audit, or regenerate) Scope follows the detected capabilities: REST-only and realtime-only SDKs get a partial skill that a later upgrade can extend, and tests that need the absent client are marked capability-inapplicable. Also adds the existing-skill gap audit with per-item add/skip/defer, an always-asked LiveObjects decision with an SDK-blocked translate-only path, and the matching guide and template updates. --- uts/README.md | 6 +- .../writing-uts-spec-translator-skills.md | 39 +- uts/skills/uts-to-lang-skill-creator/SKILL.md | 107 ++- .../assets/capability-names.json | 214 +++++ .../assets/eligibility.json | 51 ++ .../assets/templates/acceptance-checklist.md | 7 +- .../assets/templates/design-record.md | 18 +- .../assets/templates/final-report.md | 17 +- .../assets/templates/repo-profile.md | 11 +- .../assets/templates/skill-gap-audit.md | 49 ++ .../assets/templates/uts-infra-design.md | 11 +- .../references/failure-modes.md | 11 + .../references/glossary.md | 21 +- .../references/liveobjects-support.md | 125 +++ .../references/maintenance.md | 21 +- .../references/orient.md | 318 ++++++++ .../references/phase-1-understand-repo.md | 27 +- .../references/phase-2-design-harness.md | 12 +- .../references/phase-3-build-harness.md | 10 +- .../references/phase-4-design-record.md | 15 +- .../references/phase-5-generate-skill.md | 24 +- .../references/phase-6-validate-skill.md | 9 +- .../references/upgrade-existing-skill.md | 123 +++ .../scripts/check_repo_eligibility.py | 279 +++++++ .../scripts/detect_capabilities.py | 593 ++++++++++++++ .../scripts/detect_liveobjects.py | 526 +++++++++++++ .../scripts/inspect_existing_skill.py | 742 ++++++++++++++++++ .../scripts/orient.py | 594 ++++++++++++++ .../scripts/spec_clone_info.py | 3 +- .../scripts/spec_names.py | 357 +++++++++ .../scripts/survey_repo.py | 18 +- 31 files changed, 4258 insertions(+), 100 deletions(-) create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/capability-names.json create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/eligibility.json create mode 100644 uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/liveobjects-support.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/orient.md create mode 100644 uts/skills/uts-to-lang-skill-creator/references/upgrade-existing-skill.md create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/check_repo_eligibility.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/detect_capabilities.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/detect_liveobjects.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/inspect_existing_skill.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/orient.py create mode 100755 uts/skills/uts-to-lang-skill-creator/scripts/spec_names.py diff --git a/uts/README.md b/uts/README.md index d54487f01..16ff1e3ae 100644 --- a/uts/README.md +++ b/uts/README.md @@ -46,7 +46,7 @@ uts/ │ ├── proxy.md # Proxy infrastructure spec (cross-module) │ └── completion-status.md # Spec coverage matrix ├── skills/ # Agent skills (Claude Code, Codex) -│ └── uts-to-lang-skill-creator/ # Builds a uts-to- skill and its harness (SKILL.md, references/, assets/, scripts/) +│ └── uts-to-lang-skill-creator/ # Builds or upgrades a uts-to- skill and its harness (SKILL.md, references/, assets/, scripts/) └── README.md # This file ``` @@ -155,8 +155,8 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo - **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes - **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations -- **[Writing UTS Spec Translator Skills](docs/writing-uts-spec-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync -- **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that, given an SDK repo, follows a step-by-step procedure to create its `uts-to-` skill: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations. See [Installing the skill creator](#installing-the-skill-creator) +- **[Writing UTS Spec Translator Skills](docs/writing-uts-spec-translator-skills.md)** — How to build a per-language `uts-to-` agent skill (Claude Code and Codex) and its harness: harness smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync, upgrading an existing skill +- **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that, given an SDK repo, follows a step-by-step procedure to create its `uts-to-` skill, or to audit and upgrade an existing one: understand the repo, design, build and test the UTS test infrastructure (harness), generate the skill, validate it with pilot translations. See [Installing the skill creator](#installing-the-skill-creator) - **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions - **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs diff --git a/uts/docs/writing-uts-spec-translator-skills.md b/uts/docs/writing-uts-spec-translator-skills.md index 75542cd5a..31d532aaa 100644 --- a/uts/docs/writing-uts-spec-translator-skills.md +++ b/uts/docs/writing-uts-spec-translator-skills.md @@ -2,7 +2,7 @@ This guide explains how to build a per-language agent skill (Claude Code and Codex) — `uts-to-python`, `uts-to-csharp`, `uts-to-go`, and so on — that translates UTS specs into native tests for one SDK. It is written for an SDK engineer building `uts-to-` in their own repo. Two existing skills, `uts-to-swift` (ably-cocoa) and `uts-to-kotlin` (ably-java), appear here only as short examples, as the origin of a lesson, or as patterns to avoid ([Appendix](#appendix-existing-skills-background-and-patterns-to-avoid)). You shouldn't need to read these skills: this guide and the UTS docs are the authority, and they are meant to be complete. If they don't answer a question while you create or maintain a skill or its harness, the existing skills and harnesses MAY be consulted as a last resort, under the rules in [Reference implementations (last resort)](#reference-implementations-last-resort). -If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, use the [`uts-to-lang-skill-creator` skill](../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building the skill and its harness; this guide remains the authority on what the skill must contain. +If you are an LLM agent asked to create a `uts-to-` skill for an SDK repo, or to audit and upgrade an existing one, use the [`uts-to-lang-skill-creator` skill](../skills/uts-to-lang-skill-creator/SKILL.md) (installation: [`uts/README.md`](../README.md#installing-the-skill-creator)), on an Opus-class model ([Model tier](#model-tier)). It is the step-by-step procedure for building or upgrading the skill and its harness; this guide remains the authority on what the skill must contain. **Intent.** A translator skill exists so that translation is near-mechanical: the same spec, translated twice, gives the same test. @@ -43,7 +43,7 @@ Statements about the existing skills and harnesses describe them when this guide ### What a translator skill is -A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that an agent (Claude Code or Codex) follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a harness in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). +A translator skill is a packaged procedure (`SKILL.md` plus a few scripts and notes) that an agent (Claude Code or Codex) follows to turn the pseudocode specs in one UTS module directory (`uts/rest`, `uts/realtime`, `uts/objects`) into runnable tests in your SDK's test suite. It optionally runs them and diagnoses failures. It is built on, and delivered with, a harness in the same repo: the native test library that implements the UTS helper specs, with its own smoke tests and self-tests (section 2). Translator skills, and the procedure that builds them, are for Ably Pub/Sub SDK repositories (`ably-`, or its renamed form `ably-pubsub-`). ### How it relates to the other UTS docs @@ -90,6 +90,8 @@ The harness is part of the deliverable, not an optional prerequisite: a `uts-to- **Design the harness from the repo's existing test setup.** For every capability in 2.1–2.7, first find what the SDK's native tests already have, then decide to **reuse** it, **wrap** it in a pseudocode-shaped facade, **extend** it, or **build** new (2.2, "Reuse before you build"). Build the harness, and its tests, **before** writing the skill. In both existing SDKs this took several iterations, and the harness co-evolved with the first real translation run. +**Scope follows the SDK's capabilities.** The harness and the skill cover the clients the SDK actually has: a REST client, and a Realtime client with its connection, channels and presence. An SDK without a Realtime client (a REST-only SDK) gets a REST-only harness and a partial skill: the `realtime` and `objects` modules (apart from the few REST-only tests filed under `uts/realtime`), and every harness capability or checklist item that needs a Realtime client, are not applicable (capability absent), not gaps. The REST-only forms of the sandbox and proxy smoke tests publish over REST and serve a REST request through an HTTP rule. A realtime-only SDK is the mirror case: the `rest` module is not applicable, but the harness still needs the HTTP hook and `mock_http`, because the Realtime client uses HTTP too. Where an SDK exposes its clients through separate doors (for example server and device factories), the skill constructs clients through one chosen side, or through each side as a harness parameter, and the scope follows the clients that side reaches. When the SDK gains a client, the skill is extended (9.5). + ### 2.1 SDK test hooks (MUST) The unit tier needs injection points in production code: @@ -209,6 +211,7 @@ The harness has two kinds of test of its own, both permanent and both required: - **Permanent.** Don't retire them when spec-derived tests cover a tier. - **In CI, ungated.** They run, ungated, in whichever CI job runs their tier ([2.5](#25-build-and-ci-wiring-must)); no environment variable may skip them (compile-time platform gating per 2.4 aside). +- **SDK-blocked objects smoke test.** Where the SDK exposes the LiveObjects public API but doesn't implement it yet, an objects smoke-test assertion that fails only on an SDK not-implemented path (the harness compiles, every self-test is green, and the error names the SDK symbol) may be listed in Known gaps as "SDK-blocked", kept running under a strict expected-failure marker, and leaves the smoke-test checklist item unmet; the objects tier is then translate-only until the assertion passes. - **Untagged and structural.** No `UTS:` tag; each is named after the capability it proves; each passes on its own under the single-class filter. - **Never copied.** The skill never copies their shape and never names them as reference tests. Its Harness reference ([3.4](#34-recommended-outlines-should)) may cite a smoke test as a wiring example only (client construction, suite fixtures, teardown). - **Separate.** They live with the harness (layout in 2.2), outside every directory the mapping gives the resolver as a target, so generated tests can't collide with or overwrite them. @@ -346,13 +349,15 @@ Keep two documents with separate roles: the test library's README describes **wh **The notes are a map, not an authority.** Where the notes, the spec's IDL and the SDK disagree about the SDK's API, the SDK source is the ground truth: fix the notes. +**Decide `objects` from evidence.** Whether the skill covers `uts/objects` is decided from the SDK's source, not assumed. If the SDK exposes the LiveObjects public API (the `RealtimeChannel#object` accessor, RTL27, and the public types of `objects-features.md`, or an earlier revision's names), the skill gets full `objects` notes, the mapping entry and the objects harness pieces (2.2, 2.7), even if the implementation is incomplete; objects runs then default to translate-only (section 4, step E), and once evaluated, a failing test is an SDK deviation, gated, or a skipped stub where the API is missing ([7.1](#71-three-acceptable-end-states)). Otherwise the module gets a placeholder notes file, or an entry with every tier marked not ready, and the reason is recorded. The [procedure](../skills/uts-to-lang-skill-creator/references/liveobjects-support.md#12-liveobjects-objects-support) gathers the evidence with a script and asks the owner every run. + ### 3.2 The mapping file Each tier value is **one path**, relative to the repo or to a declared root (Swift's `testRoot`), never machine-absolute. Derive everything else (namespace, package, build target) from it in the resolver, never by hand. ```json { - "_comment": "Maps each UTS module to its target test dir per tier. Used by scripts/resolve_uts.py.", + "_description": "Maps each UTS module to its target test dir per tier. Used by scripts/resolve_uts.py.", "harness": { "root": "", "readme": "/README.md", @@ -375,6 +380,7 @@ Each tier value is **one path**, relative to the repo or to a declared root (Swi - **Language-specific derivations live in the resolver.** Swift has no package, so the directory is the full story. Kotlin derives `package` from the path after `src/test/kotlin/` and the Gradle module from the first segment. A C# resolver might map `tests/IO.Ably.Tests.Uts/Unit/Realtime` to namespace `IO.Ably.Tests.Uts.Unit.Realtime` (PascalCase segments) and to the nearest ancestor `.csproj`; a Python resolver derives a dotted package path. - **Hand-maintained entries.** `--create` scaffolds only the repo's default layout. A module whose tests live elsewhere (ably-java's `objects`, in `:liveobjects`) is hand-maintained: the skill MUST say which entries are, and must not offer `--create` to "fix" them. - You MAY store per-module defaults as data, e.g. `"evaluate": true` or a list of white-box specs, instead of prose in the notes. +- **A module the SDK can't support** (section 2, "Scope follows the SDK's capabilities") keeps its entry with an `"unsupported": ""` marker (any tests in it the SDK can still run are listed in the entry, under `"extraTests"`), so the resolver refuses it with that reason instead of offering to create paths for it. Tests that need a client the SDK lacks are listed per module under `"capabilityInapplicable"`, as in [7.4](#74-language-inapplicable-inputs) case (d). ### 3.3 Frontmatter and arguments @@ -470,7 +476,7 @@ The workflow has two phases. Implement both. Runs SHOULD use an Opus-class model | **C. Choose the tier** | Offer only tiers that are present. Where the mapping records a tier as not ready (per-module data, 3.2), refuse it and name what blocks it. The tier fixes the target dir, the spec list **and** the translation flow (mocked / sandbox / proxy). Don't re-detect it per spec | **Ask** | | **D. Choose specs** | All specs in the tier, or a subset | **Ask** | | **E. Translate-only, or translate and evaluate?** | Translate-only: generate, compile, and run the step 7 audit and review, but don't run the tests (the SDK feature may not exist yet). Translate and evaluate: also run and diagnose. If unsure whether the implementation exists, ask rather than guess | **Ask** | -| **F. Harness preflight** | Before generating anything, for the chosen tier and specs, in both modes: compile (or collect) the harness and the test target; run the tier's smoke tests and self-tests with the resolver's `harness` command ([2.7](#27-harness-smoke-tests-and-self-tests-must)); check the harness README's Known gaps against the selected specs. Integration and proxy tiers need network (and the proxy binary or platform) for this. If anything is red, **stop** and report it as a harness or environment problem, never as an SDK deviation or a spec error; don't generate tests on a red harness. If a selected spec needs a known gap, stop and ask: extend the harness (a harness change, section 5), deselect the spec, or proceed recording a Mock Infrastructure Limitation for the affected lines. Never substitute. Record the result in the final report | Stop on red | +| **F. Harness preflight** | Before generating anything, for the chosen tier and specs, in both modes: compile (or collect) the harness and the test target; run the tier's smoke tests and self-tests with the resolver's `harness` command ([2.7](#27-harness-smoke-tests-and-self-tests-must)); check the harness README's Known gaps against the selected specs. Integration and proxy tiers need network (and the proxy binary or platform) for this. If anything is red, **stop** and report it as a harness or environment problem, never as an SDK deviation or a spec error; don't generate tests on a red harness. If a selected spec needs a known gap, stop and ask: extend the harness (a harness change, section 5), deselect the spec, or proceed recording a Mock Infrastructure Limitation for the affected lines. Never substitute. The only exception is an SDK-blocked objects smoke test (2.7): when the SDK-blocked items listed in Known gaps (under their expected-failure marker) are the only failures, translate-only runs proceed, and the preflight output lists it. Record the result in the final report | Stop on red | If the module's notes file exists but is only a placeholder, the skill MUST tell the user the module's mapping hasn't been authored and treat the module as not yet translatable, rather than guess a mapping. @@ -763,13 +769,14 @@ Using fake timers where the spec uses real ones, a queue-ordering workaround, pe ### 7.4 Language-inapplicable inputs -Per `uts/README.md`, an input that can't be constructed in your language makes that test or table row **not applicable**: note it **in the derived test file**, not in `deviations.md`, and don't count it as a coverage gap. Three cases: +Per `uts/README.md`, an input that can't be constructed in your language makes that test or table row **not applicable**: note it **in the derived test file**, not in `deviations.md`, and don't count it as a coverage gap. Three cases, plus a fourth that isn't about the language: | Case | Example | Rule | |---|---|---| | **(a) Inexpressible input** | A wrong-typed argument a static type system rejects (`increment("10")`); a non-string key in JavaScript | Not applicable. Omit the test or row with a comment at the omission | | **(b) Assertion guaranteed by the static type** *(extends the README)* | `ASSERT keys IS Array` on a method whose declared return type is a list | Keep the spec line as a comment noting the compile-time guarantee: an annotated omission, not a deviation | | **(c) Missing API** | The SDK has no such method | Not one of the README's cases: decide not-applicable vs deviation per `writing-derived-tests.md` "Check the SDK's API surface" ([7.1](#71-three-acceptable-end-states)) | +| **(d) Capability-inapplicable** | The test constructs a client the SDK doesn't have: a REST test that enters presence through a `Realtime` client, run against a REST-only SDK | Not applicable, and not a deviation. Note it in the test file, or list it in the mapping so it is never selected ([3.2](#32-the-mapping-file)). Not language-inapplicable either: it becomes applicable when the SDK gains the client | - Not every type mismatch is inapplicable. Calling `increment` on a map is still expressible by casting to the counter view and asserting the throw; translate it. - Dynamic languages rarely have case (a): translate wrong-type rows as runtime assertions. @@ -789,7 +796,7 @@ Per `uts/README.md`, an input that can't be constructed in your language makes t ## 8. Deterministic tooling -Path validation, mapping, spec discovery, naming and faithfulness checking are mechanical. **Scripts do them identically every run; a model eyeballing two files does them inconsistently.** In ably-java the audit's first use found three undocumented omissions in already-reviewed tests. +Path validation, mapping, spec discovery, naming and faithfulness checking are mechanical. **Scripts do them identically every run; a model eyeballing two files does them inconsistently.** In ably-java the audit's first use found three undocumented omissions in already-reviewed tests. Data that maintainers update (such as the mapping) belongs in a JSON file, not in script code; start each file with a `_description` key, and explain non-obvious entries with `_comment` keys, which the scripts ignore. ### 8.1 Resolver (`resolve_uts.py`): MUST @@ -904,6 +911,17 @@ Record the clone's HEAD SHA, whether it is dirty and which files are; a translat - *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. - *Divergence (existing skills):* both existing skills fetch `writing-derived-tests.md` from GitHub `main`, and tell the model to "fetch" the features spec with no pinned source, while reading specs from the local clone, so the two can skew. +### 9.5 Upgrading an existing skill (SHOULD) + +Spec changes aren't the only drift. A skill written by hand, by an earlier version of the procedure, or before a change to this guide, can drift from the guide itself. Before you change such a skill: + +1. **Audit** the skill, its harness and its UTS-derived tests against [section 13](#13-checklist-for-a-new-uts-to-lang-skill), and list each item that is missing or doesn't conform, with the evidence. +2. **Let the owner choose** for each item: add it, skip it, or defer it. A skipped or deferred MUST item leaves the skill not conforming, and the final report says so. The harness smoke tests and self-tests (2.7) of every tier the skill keeps offering are the exception: a tier without them is refused until they exist. +3. **Upgrade in place.** Keep what conforms: rules, names, notes, hand-maintained mapping entries and harness pieces. Close the chosen items one at a time, and compare against the results from before the change (resolver output, audit results, harness tests, and the existing UTS-derived tests and their compile status). +4. **Regenerate the skill from scratch** only if the owner chooses to. Change existing UTS-derived tests only by regenerating them through the skill (re-sync, 9.2), never by hand-editing them. + +The [procedure's upgrade mode](../skills/uts-to-lang-skill-creator/references/upgrade-existing-skill.md#11-upgrade-mode-audit-and-upgrade-an-existing-skill) does this; the procedure starts every run by detecting the repo's skills, their origin, harness and UTS-tagged tests, and recommends creating or upgrading. A repo's own skill and harness are its inputs, not [reference implementations](#reference-implementations-last-resort), even in ably-cocoa and ably-java. + --- ## 10. Verification and CI @@ -974,7 +992,7 @@ From the git histories of both existing skills. Most lessons are already rules i ## 13. Checklist for a new `uts-to-` skill -The skill and its harness are one deliverable: the skill isn't done until every MUST item below, harness items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. A MUST item that can't be met yet (for example, the repo owner declines the CI change that would run the harness tests) is also marked ✗ with the reason, and the skill doesn't conform to this guide until it is met. +The skill and its harness are one deliverable: the skill isn't done until every MUST item below, harness items included, is met for each tier it supports. An item whose source section is SHOULD (marked here or in the linked section) is met, or its omission is explained (✗ with a reason) in the final report. A MUST item that can't be met yet (for example, the repo owner declines the CI change that would run the harness tests) is also marked ✗ with the reason, and the skill doesn't conform to this guide until it is met. An item, or part of one, that needs a capability the SDK lacks (section 2, "Scope follows the SDK's capabilities") is marked n/a with that reason; n/a is not ✗. **Harness** @@ -998,7 +1016,7 @@ The skill and its harness are one deliverable: the skill isn't done until every - [ ] `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative to the skill dir, hand-maintained entries marked, `harness` entry - [ ] `resolve_uts.py`: validation, errors, output contract, path-based tiers, relative exclusions, runner-collectable naming, collision detection, validate-then-write `--create` preserving entries, `harness` output - [ ] `audit_translation.py`: the parsing contract, ID coverage, duplicates, separate assert/await counts, ignores commented assertions, never crashes, flags unverifiable specs; itself mutation-tested -- [ ] Per-module notes for every module whose API diverges from the pseudocode (at least `objects`, if the SDK implements it) +- [ ] Per-module notes for every module whose API diverges from the pseudocode; `objects` decided from evidence of the SDK's LiveObjects public API: full notes if it exposes the API, even if incomplete; a placeholder or an entry with every tier marked not ready otherwise ([3.1](#31-layout)) - [ ] Scripts run on every developer platform (path handling, UTF-8, LF) **Workflow** @@ -1045,12 +1063,13 @@ The skill and its harness are one deliverable: the skill isn't done until every - [ ] Skill and harness created on an Opus-class model (MUST); the skill states the model for its runs, or pins it where the tool supports that (SHOULD), and records it in its final report ([Model tier](#model-tier)) - [ ] Reads of the [reference implementations](#reference-implementations-last-resort), if any, made only as a last resort, recorded, and reported as guide gaps +- [ ] An existing skill, its harness and its UTS-derived tests audited against this checklist before being changed; gaps listed for the owner to choose; working parts preserved ([9.5](#95-upgrading-an-existing-skill-should)) (SHOULD) --- ## Appendix: Existing skills (background and patterns to avoid) -You shouldn't need to read these skills: this guide and the UTS docs are the authority. They are named so that the examples and lessons above can be traced, and as a last-resort reference for creating a skill and its harness ([Reference implementations (last resort)](#reference-implementations-last-resort)). +You shouldn't need to read these skills: this guide and the UTS docs are the authority. They are named so that the examples and lessons above can be traced, and as a last-resort reference for creating a skill and its harness ([Reference implementations (last resort)](#reference-implementations-last-resort)). Ably is renaming its SDK repositories from `ably-` to `ably-pubsub-` (`ably-java` is now [`ably/ably-pubsub-java`](https://github.com/ably/ably-pubsub-java)); this guide keeps the short names, and old clones' remotes still work through GitHub's redirect. | Skill | Repo and paths | Lessons this guide drew from it | |---|---|---| @@ -1065,7 +1084,7 @@ This guide and the UTS docs are meant to be complete. If they don't answer a que | Repo | Skill | Harness | Harness smoke tests | Generated tests | |---|---|---|---|---| -| [ably-java](https://github.com/ably/ably-java) | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-java/tree/main/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-java/tree/main/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-java/tree/main/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | +| [ably-java](https://github.com/ably/ably-pubsub-java) (now `ably/ably-pubsub-java`) | [`.claude/skills/uts-to-kotlin`](https://github.com/ably/ably-pubsub-java/tree/main/.claude/skills/uts-to-kotlin) | The `:uts` Gradle module (path `uts/`): [`uts/src/main/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/uts/src/main/kotlin/io/ably/lib/uts); README `uts/README.md` | [`uts/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/uts/src/test/kotlin/io/ably/lib/uts): one scenario smoke test per tier (unit, direct sandbox, proxy); no helper self-tests ([Patterns to avoid](#patterns-to-avoid)) | [`lib/src/test/kotlin/io/ably/lib/uts`](https://github.com/ably/ably-pubsub-java/tree/main/lib/src/test/kotlin/io/ably/lib/uts) (rest, realtime; built by `:java`); [`liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts`](https://github.com/ably/ably-pubsub-java/tree/main/liveobjects/src/test/kotlin/io/ably/lib/liveobjects/uts) (objects; `:liveobjects`) | | [ably-cocoa](https://github.com/ably/ably-cocoa) | [`.claude/skills/uts-to-swift`](https://github.com/ably/ably-cocoa/tree/main/.claude/skills/uts-to-swift) | The `UTS` test target, [`Test/UTS`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS) (helpers in `Test/UTS/infra`; README `Test/UTS/README.md`), plus the shared test-support targets it depends on: [`Test/AblyTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyTesting) and [`Test/AblyLiveObjectsTesting`](https://github.com/ably/ably-cocoa/tree/main/Test/AblyLiveObjectsTesting) | None: they were retired, a [pattern to avoid](#patterns-to-avoid) | [`Test/UTS/unit`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/unit), [`Test/UTS/integration`](https://github.com/ably/ably-cocoa/tree/main/Test/UTS/integration) | Rules: diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md index 05f1739ae..1bef11864 100644 --- a/uts/skills/uts-to-lang-skill-creator/SKILL.md +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -1,20 +1,20 @@ --- name: uts-to-lang-skill-creator -description: "Builds a uts-to-LANG translator skill, together with the UTS test harness it runs on, for one Ably SDK repository (C#, Python, Go, JavaScript, Ruby, Dart, Rust, ...). Use it whenever asked to create, bootstrap, scaffold or generate a uts-to-csharp, uts-to-python, uts-to-go or other uts-to-* skill; to set up UTS (Universal Test Suite) test infrastructure in an SDK repo (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, harness smoke tests and self-tests); to bring the UTS specs in uts/rest, uts/realtime and uts/objects to a new SDK; or to maintain, regenerate or re-validate an existing uts-to-* skill after the guide, UTS docs, corpus or SDK changed. Runs a procedure with user stop points: profile the repo, design and build the harness, generate the skill, pilot-validate it, report. Needs a local ably/specification clone. Not for translating specs once a uts-to-* skill exists (use that skill) or for writing UTS specs." +description: "Builds or upgrades a uts-to-LANG translator skill, with the UTS test harness it runs on, for one Ably Pub/Sub SDK repository (ably-LANG or ably-pubsub-LANG: C#, Python, Go, JavaScript, Ruby, ...). Use it whenever asked to create a uts-to-csharp, uts-to-python or other uts-to-* skill; to audit, fix, upgrade or refresh one after the guide, UTS docs, corpus or SDK changed; to add LiveObjects (uts/objects) support; or to set up UTS test infrastructure (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, harness smoke tests and self-tests). First checks the repo is an Ably Pub/Sub SDK, then detects what it has (uts-to-* skills and their origin, harness, UTS-tagged tests, LiveObjects API) and recommends Create or Upgrade/Fix, then runs a procedure with user stop points. Needs a local ably/specification clone. Not for Chat, Spaces, AI Transport, CLI or other non-Pub/Sub repos, for translating specs once a uts-to-* skill exists (use that skill), or for writing UTS specs." license: Apache-2.0 compatibility: "Claude Code or Codex, on an Opus-class model. Needs shell and file access, git, Python 3.8+, a local clone of ably/specification, and the target SDK's build toolchain." allowed-tools: Read Grep Glob metadata: team: "engineering" - version: "1.0.0" + version: "1.1.0" tags: "testing, uts, translator-skill, skill-creator, harness, ably-specification" marketplace: "false" - short-description: "Build a uts-to-LANG skill and its UTS harness" + short-description: "Detect, then build or upgrade a uts-to-LANG skill and its harness" --- # UTS-to-Lang Skill Creator -Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create a `uts-to-` translator skill, and the UTS test infrastructure (the **harness**) it runs on, for one Ably SDK repository. [Writing UTS Spec Translator Skills](../../docs/writing-uts-spec-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why; this skill says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this skill links to it, and you must read the linked section. +Run this skill on an Opus-class model ([2.8](#28-model-tier-and-sub-agents)). It is a step-by-step procedure for an LLM agent to create or upgrade a `uts-to-` translator skill, and the UTS test infrastructure (the **harness**) it runs on, for one Ably Pub/Sub SDK repository. It always starts with [Step 0: orient](#step-0-orient). [Writing UTS Spec Translator Skills](../../docs/writing-uts-spec-translator-skills.md) (below, **the guide**) says **what** a good translator skill is and why; this skill says **how** to build one, in what order, with what checks, and where to stop and ask the user. It doesn't repeat the guide: wherever the reason or the detailed requirement lives in the guide, this skill links to it, and you must read the linked section. The harness and the skill are **one deliverable** (guide [section 2](../../docs/writing-uts-spec-translator-skills.md#2-the-harness-uts-test-infrastructure)): the skill isn't complete until the harness's per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must)) are green, and wired into CI, at every tier the skill supports. The harness is designed from the repo's existing test setup: every capability is reused, wrapped, extended or built ([4.2](references/phase-2-design-harness.md#42-step-2b-assess-each-capability-reuse-wrap-extend-or-build)). The phases run in a fixed order, because each stands on the one before it; a harness designed without studying the repo's existing tests duplicates or fights them. @@ -26,6 +26,19 @@ The harness and the skill are **one deliverable** (guide [section 2](../../docs/ - **Files of the spec repo** (the guide, the UTS docs, the helper specs, the corpus) are linked relative to this skill's place in a spec clone, `uts/skills/uts-to-lang-skill-creator/`, so the links work on GitHub. A link that climbs out of the skill directory (`../../…` from `SKILL.md`, `../../../…` from `references/`) is a spec-repo path; every other link is a file of this skill. At run time, never resolve a spec-repo path against the directory you loaded this skill from. Run `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and read each file at `/uts/…`: a link to `…/docs/writing-uts-spec-translator-skills.md#31-layout` means `/uts/docs/writing-uts-spec-translator-skills.md`, section 3.1. Never fetch them from GitHub. - **Inputs** come from the user's words: the text after `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex), or their message. Don't rely on argument substitution. +## Step 0: orient + +Do this first, in every run, before any phase. It is read-only and quick: no build, no test, no network, no file written. [references/orient.md](references/orient.md) (section 13) has the detail. + +1. Run `python3 /scripts/orient.py [--spec-clone ]`. It runs `check_repo_eligibility.py`, `spec_clone_info.py`, `inspect_existing_skill.py`, `detect_liveobjects.py` and `detect_capabilities.py`, then classifies the repo. +2. **Eligibility (STOP-16).** This procedure supports only Ably Pub/Sub SDK repositories: `ably/ably-` or `ably/ably-pubsub-` (both names are the same repository), checked across all remotes ([13.2](references/orient.md#132-repository-eligibility-stop-16)). + - On a reject, show the script's message exactly and end the run. There is no override. + - On an ask (a fork, no remote, too few source files to judge), ask the question first. +3. **STOP-1:** settle the model tier (2.8) and anything the spec clone needs (2.6). +4. **STOP-17** if the capability profile needs confirming (REST-only, realtime-only, partial or unclear realtime, doors with sides, wraps native SDKs, an allow-list override, names from the fallback, or changed since the recorded run): the harness and skill scope follow the SDK's REST and Realtime clients, not its name, and for an SDK with doors, the side the UTS tests use ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)); record D-31. +5. **STOP-13, always.** Show the State summary: eligibility, capabilities and scope; the skills found, with origin and installs; inspector hints; harness; UTS-tagged tests; LiveObjects verdict; state class S0–S4, with re-entry if any. Recommend Create (S0, S1) or Upgrade/Fix (S2, S3), or ask which skill (S4), with a one-line reason ([13.3](references/orient.md#133-state-classes), [13.4](references/orient.md#134-the-state-summary-and-stop-13)). Record the answer as D-27. +6. Route per [13.5](references/orient.md#135-routing). After Orient (start of Phase 1), copy the design-record and repo-profile templates to the working-records directory, and copy the State summary into the Repo profile. + ## 2. Ground rules ### 2.1 Your inputs @@ -47,13 +60,13 @@ and, only as a last resort, the reference implementations below. 4. You check what you took against the guide's [Patterns to avoid](../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid), and against the guide and the UTS docs, which always win ([1.6](#16-which-document-wins)). Patterns to avoid describes the existing skills when the guide was written; some may since have been fixed, so check what you take either way, and record the revision you read. 5. You record each consultation in the Design record ("Reference-implementation reads": repo, revision, path, question, what you learned, how you checked it), and list it in the Final report as a guide gap: a candidate guide improvement. -A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)). +These rules govern other repos. The target repo's own `uts-to-*` skill, harness and UTS-derived tests are part of input 4, even when the target is ably-cocoa or ably-java: you read them in full and change them in place ([11.1](references/upgrade-existing-skill.md#111-when-this-mode-applies)). A read-only consultation needs no stop point of its own. Where the guide names another SDK's file, script, helper or test (as the origin of a lesson, or in the Swift and Kotlin cells of its construct tables), it is background, not a template. If the references don't resolve the question either, or the only answer they give departs from the guide, **STOP-12**. The generated skill never consults them (D-22), and the pilot never does either ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)). ### 2.2 What you may change | Action | Allowed? | |---|---| -| Create and edit files under the skill directory, its `.agents/skills` link or copy (D-01), and the working-records directory | Yes | +| Create and edit files under the skill directory, its `.agents/skills` link or copy (D-01), and the working-records directory | Yes (for an existing skill's files, see the row on existing skills below) | | Create new test-only harness code (shared test library, module helpers, smoke tests, UTS test-project wiring) | Only after the user approves the harness design at [STOP-3](#23-stop-points), and only within the scope they approved | | Change existing test-support code that the SDK's own tests use (shared fakes, fixtures, base classes) | Only with explicit approval for each change, and never in a way that changes behaviour for the existing tests. Prefer wrapping or extending to editing | | Change SDK production code, including adding test hooks or raising visibility | **Only with explicit approval for each change** ([STOP-4](#23-stop-points)). Present the proposal first ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | @@ -62,6 +75,7 @@ A read-only consultation needs no stop point of its own. Where the guide names a | Stage, commit, push, branch, or open a PR | **No, unless the user asks** | | Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | | Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | +| Edit, replace or delete files of an existing `uts-to-*` skill or UTS harness that this run didn't create | Only for the items the user chose at STOP-15 (D-29) or the harness rows approved at STOP-3; the objects files that the STOP-14 decision (D-28) changes; when regenerating, replacements the user approves ([11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)); the provenance stamp in its frontmatter (D-30). Never delete without asking | | Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | | Fetch from the network for anything other than what the user approved (sandbox, proxy binary, dependencies, reference-implementation reads under [2.1](#21-your-inputs)) | No | @@ -71,8 +85,8 @@ At a stop point, present what you found and the options, then wait for the user' | ID | Phase | When | What you present | |---|---|---|---| -| **STOP-1** | Start | A required input is missing, the session's model isn't Opus-class ([1.3](#13-required-inputs), [2.8](#28-model-tier-and-sub-agents)), or `spec_clone_info.py` fails or reports `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | -| **STOP-2** | 1 (end) | The Repo profile is written | The profile, for confirmation or correction | +| **STOP-1** | 0 (Orient) | A required input is missing, the session's model isn't Opus-class ([1.3](#13-required-inputs), [2.8](#28-model-tier-and-sub-agents)), or `spec_clone_info.py` fails or reports `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | +| **STOP-2** | 1 (step 1d) | The Repo profile is written | The profile, for confirmation or correction | | **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | | **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | | **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | @@ -83,6 +97,11 @@ At a stop point, present what you found and the options, then wait for the user' | **STOP-10** | 3, 6 | A Phase 3 build row or a pilot spec hits its attempt bound ([5.2](references/phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks), [8.4](references/phase-6-validate-skill.md#84-fix-the-cause-then-regenerate)), or the pilot reveals a suspected UTS spec error | A summary: the failure, what you tried, your diagnosis (per `writing-derived-tests.md` Phase 2 for a test), the options, and a draft spec fix where relevant | | **STOP-11** | Any | A conflict between the docs, or between the guide and the SDK, that changes what you would build or generate | Both statements, with paths and headings, and the reading you propose | | **STOP-12** | Any | The guide, the UTS docs and the reference implementations ([2.1](#21-your-inputs)) together don't answer a question, or the only answer the references give would depart from the guide | The question; the guide and UTS sections you searched; the references you read (repo, revision, path) and what they showed; the reading you propose | +| **STOP-13** | 0 (Orient), every run | Always ([13.4](references/orient.md#134-the-state-summary-and-stop-13)) | The State summary, the state class (S0–S4, re-entry) and the options for it, with the recommended one: Create; Upgrade/Fix (diff-driven or full gap audit); regenerate from scratch; resume or restart; pick a skill (S4); stop | +| **STOP-14** | 1 (step 1e), every run | Always, in every mode ([12.2](references/liveobjects-support.md#122-stop-14-recommend-then-ask)) | The `detect_liveobjects.py` evidence (public-API, legacy, internal and wire-only names, with files and counts), its recommendation and yours, and the options: full objects support (translate-only if incomplete), placeholder, none now | +| **STOP-15** | 1 (step 1f), Upgrade/Fix | The gap audit, or the list of detected changes (diff-driven), is written ([11.4](references/upgrade-existing-skill.md#114-step-u3-stop-15-choose-per-item), [section 10](references/maintenance.md#10-maintenance)) | Every non-conforming item or detected change, with the proposed change and the options add, skip or defer (a skipped or deferred MUST row leaves the skill not conforming); or regenerate from scratch instead | +| **STOP-16** | 0 (Orient), first check | `check_repo_eligibility.py` rejects the repo, or can't decide ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | Reject: the exact message that this procedure supports only Ably Pub/Sub SDK repositories and needs upgrading for this one; the run ends, with no override. Ask: the remotes found and the question (confirm a fork, name the GitHub repository, check the path) | +| **STOP-17** | 0 (Orient), with STOP-13 | The capability profile isn't full, is unclear, has door sides, wraps native SDKs, comes from an allow-list override, uses fallback names, or changed since the recorded run ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | The profile (REST, Realtime and its sub-areas, with evidence), the suggested scope (modules in scope; `unsupported` modules; capability-inapplicable tests), the delta if any; options: accept, correct with evidence and rerun, stop. With doors, also the side question: construct clients through the core constructors (sanctioned internal-access route), the server door, the device door, or both as a parameter; the scope follows the side | ### 2.4 Never guess @@ -99,7 +118,7 @@ Every choice you make that isn't forced by the guide or the UTS docs goes into t ### 2.6 Pin the spec clone -At the start of the run, run `python3 /scripts/spec_clone_info.py []`, passing the path the user gave, if any. It locates the clone (the path given; else `UTS_SPEC_CLONE`; else the clone this skill is symlinked from; a path inside a clone is walked up to its root), checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd' -- uts/docs/writing-uts-spec-translator-skills.md uts/skills/uts-to-lang-skill-creator`. It prints one JSON object. +In Orient (Step 0, which runs it for you), run `python3 /scripts/spec_clone_info.py []`, passing the path the user gave, if any. It locates the clone (the path given; else `UTS_SPEC_CLONE`; else the clone this skill is symlinked from; a path inside a clone is walked up to its root), checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd' -- uts/docs/writing-uts-spec-translator-skills.md uts/skills/uts-to-lang-skill-creator`. It prints one JSON object. - Any `ok: false` (for example `SPEC_CLONE_NOT_FOUND`, `NOT_A_SPEC_CLONE`, `GIT_ERROR`): **STOP-1**; ask for the clone's path. - `foundBy` other than `argument`: tell the user which clone was found and how, and confirm it before Phase 1. @@ -112,14 +131,14 @@ Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) a ### 2.7 Working style -- Do Phase 1 before anything else, and don't write harness code before the user approves the harness design (STOP-3), or skill files before the harness is verified (STOP-7). +- Do Step 0 (Orient) before anything else, then Phase 1; don't write harness code before the user approves the harness design (STOP-3), or skill files before the harness is verified (STOP-7). - Escalate verification in the repo's own order (compile, then build tests, then a filtered run, then wider runs), as recorded in the Repo profile. -- Keep the working records up to date as you go; they are also your memory if the session is interrupted. On resuming, read them first. +- Keep the working records up to date as you go, including the design record's Run status (13.7); they are also your memory if the session is interrupted. Orient finds them and offers to resume (13.6); then read them first. - Use the repo's own conventions (file headers, line endings, naming, test style) for every file you create. ### 2.8 Model tier and sub-agents -- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name): MUST.** The guide's [Model tier](../../docs/writing-uts-spec-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so before Phase 1 (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. +- **Run this procedure on the most capable model tier available (Claude Opus or an equivalent Opus-class model; in another tool, such as Codex, its most capable tier, recorded by name): MUST.** The guide's [Model tier](../../docs/writing-uts-spec-translator-skills.md#model-tier) gives the reasons: long context, multi-file reasoning, fidelity, and few silent omissions in rules that every later run inherits. If your environment states your model, record it; otherwise ask the user. If you aren't on an Opus-class model, say so in Orient (**STOP-1**). If the user still wants to proceed, record it in the decision log as a departure from the guide, and state it in the Final report. - **Sub-agents** you spawn for review, validation or the pilot ([8.3](references/phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)) MUST also run on an Opus-class model. Where your tooling lets you choose a sub-agent's model, choose it explicitly rather than relying on a default. Use a cheaper model only for a clearly mechanical step whose output a script verifies (for example, running the [8.2](references/phase-6-validate-skill.md#82-test-the-audit-itself) corpus sweep and collecting its JSON), if at all, and record which steps used one. - **Record** the model (name and ID) of this session and of every sub-agent in the Design record ("Inputs at generation time") and in the Final report. - **State** the generated skill's model tier in its opening lines and `compatibility`; pin it only as D-26 decides. @@ -140,6 +159,7 @@ The user starts a run in the target SDK repo with one sentence, for example: /uts-to-lang-skill-creator ~/src/specification realtime unit first (Claude Code) $uts-to-lang-skill-creator ~/src/specification realtime unit first (Codex) Create a uts-to-csharp skill for this repo; the spec clone is ~/src/specification. +Audit our uts-to-python skill against the guide and upgrade it; add LiveObjects support. Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to create a uts-to- skill for this repo. (any agent, skill not installed) ``` @@ -149,27 +169,28 @@ Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to | Input | Example | If missing | |---|---|---| -| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo; confirm it has SDK sources and tests | +| Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo: Orient checks that it is an eligible Ably Pub/Sub SDK repository with SDK sources and tests (STOP-16) | | Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | | Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | Located by `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and confirmed with the user; otherwise **stop and ask** ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | | User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | | Model tier | The session's model, Opus-class ([2.8](#28-model-tier-and-sub-agents)) | Take it from your environment if it states your model; otherwise ask. If the model isn't Opus-class: **STOP-1** | -| Local clones of ably-cocoa and ably-java (optional) | Clones of either repo, read at their current HEAD (ideally a recent `main`) | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | +| Local clones of ably-cocoa and ably-java (now `ably-pubsub-java`) (optional) | Clones of either repo, read at their current HEAD (ideally a recent `main`) | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after **STOP-6** | ### 1.4 Outputs 1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at [STOP-3](#23-stop-points). -2. **The skill**, at `.claude/skills/uts-to-/` (linked from `.agents/skills/uts-to-/` for Codex, guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)) unless the Design record says otherwise (D-01): `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). -3. **Pilot-translated tests**, one spec per available tier, produced by the new skill in Phase 6. -4. **Four working records** (*Procedure recommendation*), written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): +2. **The skill** (in upgrade mode, the existing skill, upgraded in place), at `.claude/skills/uts-to-/` (linked from `.agents/skills/uts-to-/` for Codex, guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)) unless the Design record says otherwise (D-01): `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if shipped (D-19), `scripts/scan_constructs.py` (layout: guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). +3. **Pilot-translated tests**, one spec per available tier (in upgrade mode, per tier whose rules changed, 11.6), produced by the new skill in Phase 6. +4. **Four working records** (five in Upgrade/Fix; *Procedure recommendation*), created after STOP-13, written as Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; confirmed in Phase 4, decision D-24): - `repo-profile.md` ([repo-profile template](assets/templates/repo-profile.md)); - `uts-infra-design.md`: the gap table, the harness design and the build log ([uts-infra-design template](assets/templates/uts-infra-design.md)); - `design-record.md` ([design-record template](assets/templates/design-record.md)), which also holds the decision log; - - `final-report.md` ([final-report template](assets/templates/final-report.md)), with the acceptance checklist ([acceptance-checklist template](assets/templates/acceptance-checklist.md)) pasted into its "Acceptance checklist" section. + - `final-report.md` ([final-report template](assets/templates/final-report.md)), with the acceptance checklist ([acceptance-checklist template](assets/templates/acceptance-checklist.md)) pasted into its "Acceptance checklist" section; + - in Upgrade/Fix, also `skill-gap-audit.md` ([skill-gap-audit template](assets/templates/skill-gap-audit.md)). -Create each record by copying its template from `assets/templates/` (the repo-profile, uts-infra-design, design-record and final-report templates) to the working-records directory under the same file name, then fill it in. +Create each record by copying its template from `assets/templates/` (the repo-profile, uts-infra-design, design-record and final-report templates, and in Upgrade/Fix the skill-gap-audit template) to the working-records directory under the same file name, then fill it in. -Write them at the default location from Phase 1 on, using the language inferred so far for ``; if STOP-2 changes the language or D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. +Write them from STOP-13 on, at the location Orient's routing gives (13.5), by default using the language inferred so far for ``; if STOP-2 changes the language or D-24 chooses another location, move the files then. They are working notes, not skill files, so writing them before STOP-7 is allowed. The working records are what a later agent reads to maintain or regenerate the skill ([section 10](references/maintenance.md#10-maintenance)), so keep them accurate as you go, not only at the end. @@ -194,33 +215,41 @@ If two documents disagree, follow the higher one in the table (and **STOP-11** i **Procedure recommendations.** Where this skill adds detail the guide doesn't have (for example: the working-records files and their location, tier-readiness data in the mapping, a mandatory `specRepo`, and requiring the audit's end-at-heading rule), it is marked *Procedure recommendation*: this skill's way of meeting the guide, not a guide requirement. Follow it unless the user decides otherwise, and record any departure in the decision log. Everything else either links to the guide or is procedure (order, checks, stop points). +## Run modes + +Orient (Step 0) recommends the mode, and the user decides at STOP-13 (D-27). There are two user-facing modes, and the same phases, ground rules and stop points serve both: + +| Mode | When | Read | +|---|---|---| +| **Create** | No `uts-to-*` skill yet (S0); or none, but UTS-tagged tests or harness code to adopt (S1, 2.2) | The phases below | +| **Upgrade/Fix** | A `uts-to-*` skill exists. **Diff-driven** for one this procedure built (S2): follow what changed since its recorded run ([section 10](references/maintenance.md#10-maintenance)). **Full gap audit** for one of unknown origin (S3), or by choice: audit it, its harness and its UTS-derived tests against the guide, choose per item, upgrade in place ([section 11](references/upgrade-existing-skill.md#11-upgrade-mode-audit-and-upgrade-an-existing-skill)). **Regenerate from scratch** only by the user's choice ([11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)) | Sections 10 and 11 | + +An interrupted run is resumed or restarted from its records (13.6). In every mode, Phase 1 also asks whether to add LiveObjects (`objects`) support (**STOP-14**), recommending an answer from `detect_liveobjects.py`'s evidence: [references/liveobjects-support.md](references/liveobjects-support.md) (section 12). + ## Phases at a glance -Before starting a phase, read its reference file in full: this table is a map, not enough to act on. On resuming an interrupted run, read the working records first, then the reference of the phase you are in. +Before starting a phase, read its reference file in full: this table is a map, not enough to act on. In Upgrade/Fix with a full gap audit, read [11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)'s row for each phase first (diff-driven: section 10): the phases are scoped to the chosen items, and acceptance items tied to a skipped or deferred gap-audit row are reported ✗ under D-29, not worked. In regenerate mode, read 11.5's last paragraph instead. On resuming an interrupted run, read the working records first, then the reference of the phase you are in. | Phase | Goal | Read first | Writes | Ends at | |---|---|---|---|---| -| 1. Understand the repo | A confirmed Repo profile: how the SDK is built, tested and mocked today | [references/phase-1-understand-repo.md](references/phase-1-understand-repo.md) | `repo-profile.md` (repo-profile template) | STOP-2 | +| 0. Orient | Eligibility, capabilities and scope, state class, mode | [Step 0](#step-0-orient); [references/orient.md](references/orient.md) | nothing | STOP-16 (if not eligible), STOP-1 (if needed), STOP-17 (if needed), STOP-13 | +| 1. Understand the repo | A confirmed Repo profile: how the SDK is built, tested and mocked today; the LiveObjects decision; in Upgrade/Fix, the gap audit or the change list | [references/phase-1-understand-repo.md](references/phase-1-understand-repo.md) (and sections 10, 11, 12) | `repo-profile.md`; in Upgrade/Fix `skill-gap-audit.md` | STOP-2, STOP-14 (STOP-15 in Upgrade/Fix) | | 2. Design the harness | Reuse, wrap, extend or build each capability; hooks; placement; tier feasibility | [references/phase-2-design-harness.md](references/phase-2-design-harness.md) | `uts-infra-design.md` (uts-infra-design template) | STOP-3 | | 3. Build and verify the harness | The approved harness, with green per-tier smoke tests and self-tests, wired into CI | [references/phase-3-build-harness.md](references/phase-3-build-harness.md) | harness code and README; build log | STOP-7 | -| 4. Skill design decisions | Decisions D-01 to D-26, confirmed | [references/phase-4-design-record.md](references/phase-4-design-record.md) | `design-record.md` (design-record template) | STOP-8 | +| 4. Skill design decisions | Decisions D-01 to D-31, confirmed | [references/phase-4-design-record.md](references/phase-4-design-record.md) | `design-record.md` (design-record template) | STOP-8 | | 5. Generate the skill files | Mapping, resolver, audit, construct catalogue, module notes, `SKILL.md` | [references/phase-5-generate-skill.md](references/phase-5-generate-skill.md) | the `uts-to-` skill | (STOP-9 if constructs are unmapped) | | 6. Validate the skill | Resolver, audit mutations, pilots, other paths, lint, examples | [references/phase-6-validate-skill.md](references/phase-6-validate-skill.md) | pilot tests; results | (STOP-10 at a bound) | | 7. Final report and handover | Report and acceptance checklist | [section 9](#9-phase-7-final-report-and-handover) below | `final-report.md` (final-report and acceptance-checklist templates) | — | ## 9. Phase 7: final report and handover -1. Write `final-report.md` from the [final-report template](assets/templates/final-report.md). -2. Fill the [acceptance checklist](assets/templates/acceptance-checklist.md): tick each item, or explain why it isn't met (scoped out, planned, not applicable). +1. Write `final-report.md` from the [final-report template](assets/templates/final-report.md). Set the design record's Run status to finished, and refresh the skill's provenance stamp (D-30, 13.7). +2. Fill the [acceptance checklist](assets/templates/acceptance-checklist.md): tick each item, mark it n/a with the reason ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)), or explain why it isn't met (scoped out, planned, skipped or deferred at STOP-15). 3. Show the user the report, the list of changed and new files (harness, harness tests, hooks, skill, pilot tests), and the suggested CI changes. The run is complete only if the harness tests are green, and in CI, at every tier the skill supports; otherwise the report says which tiers aren't and why. **Don't commit** unless asked. -4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. +4. If the user asks for a commit, follow the repo's commit conventions (P-08) and, if the user agrees, keep the SDK hooks, the harness, the skill and the pilot tests in separate commits. In Upgrade/Fix, offer one commit per closed gap-audit item (from the Design record's changelog), harness rows first. The acceptance checklist maps each item of the guide's [section 13 checklist](../../docs/writing-uts-spec-translator-skills.md#13-checklist-for-a-new-uts-to-lang-skill) to where this procedure produces and verifies it. Mark each ✓, ✗ (with the reason) or n/a (with the reason). -## Maintenance runs - -When the user asks to update, regenerate or re-validate an existing `uts-to-` skill or its harness (the guide, the UTS docs, the corpus, the harness or the SDK changed), read [references/maintenance.md](references/maintenance.md) first, then the working records. The ground rules and stop points above apply unchanged. - ## Reference index | File | Read it when | Sections | @@ -228,15 +257,19 @@ When the user asks to update, regenerate or re-validate an existing `uts-to-/scripts/.py`. +All read-only, Python 3, printing to stdout (no network); run them from anywhere as `python3 /scripts/.py`. What they know about Ably's repositories and SDK names that the spec can't give is data in `assets/eligibility.json` and `assets/capability-names.json`: update the data file, not the code ([section 10](references/maintenance.md#10-maintenance)). | Script | Use | Output | |---|---|---| +| `orient.py [--spec-clone P] [--skill-dir P] [--records P] [--full]` | Step 0: run the scripts below in order, classify the repo and recommend a mode ([13.1](references/orient.md#131-run-it)) | `state` (class S0–S4 or INELIGIBLE, re-entry, reasons, recommended, options), `stateSummaryText`, and the facts behind them | +| `check_repo_eligibility.py [--spec-clone P]` | Is this an Ably Pub/Sub SDK repository? All remotes, both name forms, canonical names, deny- and allow-lists, the definition gate ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | `decision` (accept, reject, ask), `code`, `owner`, `repo`, `nameForm`, `canonical`, `plannedRename`, `capabilityOverride`, `message`, `fingerprint` | +| `detect_capabilities.py [--spec-clone P] [--include-submodules] [--capability-override rest=full,realtime=absent]` | The REST and Realtime capability profile (connection, channels, presence, transport), the door sides and the suggested scope ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | `namesSource`, `specRevision`, `capabilities` (levels, clients, public reachability, evidence, `methodGapsAdvisory`), `source`, `sides`, `wrapsNativeSdk`, `scopeSuggestion` (`kind`, `modules`, `unsupported`, `capabilityInapplicable`, `extraTests`, `confirmAtStop17`), `reasons`, `summary` | +| `spec_names.py [--spec-clone P]` | The names the two detectors look for, read from the spec clone's IDL at run time, and the corpus's capability-inapplicable tests ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | `source` (`spec` or `fallback`), `revision`, `warnings`, the client, channel, connection and presence names and members, the LiveObjects name sets, `capabilityInapplicable` | | `spec_clone_info.py [SPEC_CLONE]` | Locate, validate and pin the spec clone ([2.6](#26-pin-the-spec-clone)). Discovery order: the path the user gave; `UTS_SPEC_CLONE`; the clone this skill is symlinked from | One JSON object: `specClone`, `foundBy`, `sha`, `dirty`, `dirtyFiles`, `guideLastChange`, `skillDir`, `skillRealpath`, `skillVersion`, `skillMatchesClone`, `warning` (if any), `paths` (guide, UTS docs, helper specs); or `ok: false` with a `code` | | `survey_repo.py [ …]` | A first pass over the target repo for the discovery checklist ([3.2](references/phase-1-understand-repo.md#32-step-1b-answer-the-discovery-checklist)) | HEAD and status, extension counts, test roots, submodules, candidate mocks, `UTS:` tags, existing skills | | `scan_constructs.py /uts/` | List the corpus's pseudocode constructs and harness calls ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); copy it into the generated skill's `scripts/` (D-19) | Keyword and snake_case call counts | +| `inspect_existing_skill.py []` | Inventory an existing `uts-to-*` skill, its harness entry and its UTS-derived tests, for STOP-13 and the gap audit ([11.1](references/upgrade-existing-skill.md#111-when-this-mode-applies)); also lists user-level installs | Skills and their installs (copies compared), frontmatter, files, mapping, harness (from the mapping, or candidate READMEs, smoke-test files and CI lines), regex checks keyed to checklist items (hints, not a review); tag, header-SHA and `deviations.md` counts, tags outside the mapped tiers; working records | +| `detect_liveobjects.py [--spec-clone P] [--include-submodules]` | Evidence for the LiveObjects decision at STOP-14 ([12.1](references/liveobjects-support.md#121-gather-the-evidence-step-1b-p-15)); names from `spec_names.py` | `namesSource`, `specRevision`; a one-paragraph `summary`; a recommendation with reasons; per category (public API, accessor, legacy, shared, internal, wire-only), symbols with counts, files and declaration sites; the public names not found; plugin key, packaging, nested packages; exclusions | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json b/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json new file mode 100644 index 000000000..aca028168 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/capability-names.json @@ -0,0 +1,214 @@ +{ + "_description": "Names the capability and LiveObjects detectors can't read from the spec: SDK client aliases, door factories (2.0 split SDKs), earlier-revision and spec-prose LiveObjects names, and the fallback used only when the spec can't be parsed. Update this file, not the scripts, when an SDK adds an alias or a door, or the spec renames something. Everything else (client, channel, connection, presence and LiveObjects names and members) is read at run time from the spec clone by scripts/spec_names.py (section 13.8). Keys starting with '_' are ignored.", + "aliases": { + "_comment": "SDK names for the REST and Realtime clients that aren't spec names or their per-language forms: the 2.0 split SDKs' client types (ably-dotnet integration/v2 PubSubHttpClient, PubSubRealtimeClient).", + "rest": [ + "PubSubHttpClient", + "DefaultPubSubHttpClient" + ], + "realtime": [ + "PubSubRealtimeClient", + "DefaultPubSubRealtimeClient", + "PubSubClient" + ] + }, + "doorFactories": { + "_comment": "Public entry points of split SDKs (2.0 'doors'). A door factory's capability comes from its return type, else its name (Http or Rest: REST; otherwise realtime). Its side comes from its enclosing type (sides.*.types, matched whole, e.g. static class PubSubDevice) or else a path segment (sides.*.paths, e.g. Ably.PubSub.Device, pubsub-device); a door with neither is core. Sides come only from doors: a client type under a 'device' path (push LocalDevice code) is not a side.", + "names": [ + "createClient", + "createHttpClient", + "createRestClient", + "createRealtimeClient", + "create_client", + "create_http_client", + "create_rest_client", + "create_realtime_client", + "CreateClient", + "CreateHttpClient", + "CreateRealtimeClient", + "NewClient", + "NewHTTPClient", + "NewRealtimeClient", + "clientBuilder", + "httpClientBuilder", + "realtimeClientBuilder", + "createClientWithOptions", + "createClientWithKey", + "createClientWithToken" + ], + "sides": { + "device": { + "types": [ + "PubSubDevice", + "AblyPubSubDevice" + ], + "paths": [ + "device", + "pubsub-device" + ] + }, + "server": { + "types": [ + "PubSubServer", + "AblyPubSubServer" + ], + "paths": [ + "server", + "pubsub-server" + ] + } + } + }, + "_comment": "legacyPublicApi: public LiveObjects names from earlier revisions of objects-features.md (before channel.object.get() and the LiveMap and LiveCounter value types). legacyNowInternal: public then, internal now; a public declaration counts as legacy. internalFromProse: internal names the spec gives only in prose (no IDL entry).", + "legacyPublicApi": [ + "RealtimeObjects", + "getRoot", + "createMap", + "createCounter", + "LiveObjectSubscription", + "LiveMapValueType", + "LiveCounterValueType" + ], + "legacyNowInternal": [ + "LiveObject", + "LiveObjectUpdate", + "LiveMapUpdate", + "LiveCounterUpdate" + ], + "internalFromProse": [ + "ObjectsPool" + ], + "fallback": { + "_comment": "Used only when the spec can't be parsed: reported as namesSource 'fallback' with a warning that Orient shows. Keep it minimal; never rely on it.", + "capability": { + "clients": { + "rest": [ + "Rest", + "RestClient" + ], + "realtime": [ + "Realtime", + "RealtimeClient" + ] + }, + "roles": { + "restClient": { + "names": [ + "RestClient" + ], + "members": { + "request": "RSC19", + "stats": "RSC6a", + "time": "RSC16" + }, + "from": [ + "fallback" + ] + }, + "realtimeClient": { + "names": [ + "RealtimeClient" + ], + "members": { + "connect": "RTC15", + "close": "RTC16" + }, + "from": [ + "fallback" + ] + }, + "restChannel": { + "names": [ + "RestChannel" + ], + "members": { + "publish": "RSL1", + "history": "RSL2" + }, + "from": [ + "fallback" + ] + }, + "channels": { + "names": [ + "RealtimeChannel", + "ChannelStateChange", + "ChannelState", + "ChannelEvent" + ], + "members": { + "attach": "RTL4", + "detach": "RTL5", + "whenState": "RTL25" + }, + "from": [ + "fallback" + ] + }, + "presence": { + "names": [ + "RealtimePresence" + ], + "members": { + "enterClient": "RTP4", + "history": "RTP12" + }, + "from": [ + "fallback" + ] + }, + "connection": { + "names": [ + "Connection", + "ConnectionStateChange", + "ConnectionState", + "ConnectionEvent" + ], + "members": { + "createRecoveryKey": "RTN16g", + "whenState": "RTN26" + }, + "from": [ + "fallback" + ] + } + } + }, + "liveobjects": { + "publicApi": [ + "RealtimeObject", + "PathObject", + "LiveMap", + "LiveCounter", + "compactJson" + ], + "internalApi": [ + "InternalLiveMap", + "InternalLiveCounter", + "LiveObject", + "ObjectsPool" + ], + "shared": [ + "ObjectMessage", + "ObjectOperation" + ], + "sharedConstants": [ + "MAP_SET", + "COUNTER_INC" + ], + "wireOnly": [ + "ObjectState", + "ObjectData" + ], + "wireConstants": [ + "OBJECT_SYNC" + ], + "accessorTypes": [ + "RealtimeObject" + ], + "from": [ + "fallback" + ] + } + } +} diff --git a/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json b/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json new file mode 100644 index 000000000..0bcfcfcbb --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/eligibility.json @@ -0,0 +1,51 @@ +{ + "_description": "Which repositories the skill creator accepts (STOP-16, section 13.2): read by scripts/check_repo_eligibility.py. Update this file, not the script, when Ably adds, renames or retires a repository. Keys starting with '_' are ignored.", + "owner": "ably", + "namePattern": "^ably-(pubsub-[a-z0-9]+|[a-z0-9]+)$", + "_comment": "namePattern is matched case-insensitively against the remote's repository name without .git. Both the legacy ably- and the renamed ably-pubsub- forms are the same repository (GitHub redirects the old URL, and local clones keep it), so both are accepted.", + "deny": { + "_comment": "Single-token names of known non-SDK ably- repositories that namePattern would accept. Legacy form only: ably-pubsub- isn't denied (the definition gate still applies). Multi-hyphen names (ably-chat-swift, ably-ai-transport-js) already fail namePattern. 'pubsub' because ably-pubsub alone names no language. The second group (from brand on) comes from a survey of the ably GitHub organisation: tools, docs or stubs that a name rule alone would accept.", + "_comment-server": "ably-server is an Ably-compatible server written in Go that *uses* the Go SDK. Its name looks like a language, and a usage-based gate would accept it.", + "_comment-os": "ably-os is an internal MCP server in TypeScript that uses AblyRealtime; not an SDK.", + "_comment-nativescript": "ably-nativescript is the legacy name of ably-js-nativescript, a wrapper that re-exports ably-js and defines no client. canonical maps it to that name (rejected by namePattern); the deny entry is a second guard.", + "names": [ + "common", "specification", "spec", "cli", "docs", "ui", "boomer", "chat", "spaces", "objects", "liveobjects", + "livesync", "sandbox", "proxy", "examples", "example", "website", "www", "laravel", "terraform", "pubsub", + "labs", "demo", "demos", "dashboard", "infra", "control", "mcp", "status", "benchmark", "benchmarks", "test", + "tests", "tools", "scripts", "ai", "asset", "assets", + "brand", "comply", "jmeter", "os", "research", "rss", "scan", "server", "titanium", "roku", "nativescript" + ] + }, + "allow": { + "_comment": "Exceptions to namePattern: Pub/Sub SDK repositories whose names it rejects. Eligibility isn't scope: a capabilityOverride replaces detection only where detection can't see the SDK, and is reported as the profile's source.", + "_comment-ably-ruby-rest": "ably-ruby-rest is the REST-only Ruby SDK: a gem wrapping an ably-ruby git submodule, whose entry point loads only ably/rest. Its own tree defines no client, so the definition gate and capability detection find nothing; the override supplies REST full and realtime absent.", + "ably-ruby-rest": { + "capabilityOverride": { "rest": "full", "realtime": "absent" } + }, + "ably-pubsub-ruby-rest": { + "capabilityOverride": { "rest": "full", "realtime": "absent" } + } + }, + "canonical": { + "_comment": "Legacy name -> current name (GitHub redirects the old one), reported as `canonical`. Both names are accepted, except where the current name fails namePattern: ably-nativescript is now ably-js-nativescript, a wrapper, so it is rejected.", + "ably-js": "ably-pubsub-js", + "ably-java": "ably-pubsub-java", + "ably-dotnet": "ably-pubsub-dotnet", + "ably-php": "ably-pubsub-php", + "ably-python": "ably-pubsub-python", + "ably-ruby": "ably-pubsub-ruby", + "ably-flutter": "ably-pubsub-flutter", + "ably-dart": "ably-pubsub-dart", + "ably-ios": "ably-cocoa", + "ably-nativescript": "ably-js-nativescript" + }, + "plannedRename": { + "_comment": "Renames in preparation, reported as `plannedRename`: either name is accepted.", + "ably-cocoa": "ably-pubsub-cocoa", + "ably-go": "ably-pubsub-go" + }, + "definitionGate": { + "_comment": "After the name rule, a REST or Realtime client entry point must be *defined* in non-test source (not just used, as load tools and servers do). With no definition: reject (NO_CLIENT_DEFINITION) when there are at least minSourceFilesToReject non-test source files, else ask (NO_SDK_FINGERPRINT: wrong path or sparse checkout?). An allow-list capabilityOverride satisfies the gate.", + "minSourceFilesToReject": 20 + } +} diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md index 0d20bed8d..4b1fc9951 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md @@ -1,4 +1,4 @@ - + | Guide §13 item | Produced in | Verified in | Status | |---|---|---|---| @@ -9,7 +9,7 @@ | Thread-safe capture, log sink, `assertContainsInOrder`, caller-attributed failures | 4.4; 5.2 row 2 | helper tests; G-12 | | | Harness designed from the repo's existing test setup (reuse, wrap, extend or build, 2.2); shared vs port-only placement; recommended harness layout; fixture-helper scope documented | 4.2; 4.7; 5.3; notes item 9 | G-13 | | | Harness README with a Known gaps section (MUST) | 5.3; 5.2 row 12 | G-14; 8.5 path 9 | | -| Harness smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.5 path 8 | | +| Harness smoke tests per tier and helper self-tests: permanent, in CI, ungated, untagged, outside generated-test directories (MUST) | 4.7; 5.2 rows 1–11, 13 | STOP-7; G-19, G-20; 8.5 path 8 (✗ "SDK-blocked" while an objects smoke test is SDK-blocked, 12.5) | | | `SandboxApp` (idempotent retries only); `ably-common` | 4.5; 5.2 row 9 | smoke integration test; G-15 | | | `ProxyManager` (pinned, every OS, verified download, health check, port handling); `ProxySession`; string `match.action`; proxy auth decided and commented | 4.5; 5.2 row 10; D-18 | smoke proxy test; G-16 | | | One CI home per suite (MUST); per-tier jobs (SHOULD) | 5.2 row 13 or suggested CI changes | G-17; Final report | | @@ -21,7 +21,7 @@ | `uts-package-mapping.json`: one path per tier, unique namespaces, `notes` relative, hand-maintained entries marked, `harness` entry | 7.1 | 7.1 acceptance; 8.1 | | | `resolve_uts.py`: validation, errors, output contract, path tiers, relative exclusions, collectable names, collisions, validate-then-write `--create` preserving other entries, `harness` output | 7.2 | 8.1 | | | `audit_translation.py`: parsing contract, ID coverage, duplicates, separate counts, ignores commented assertions, never crashes, flags unverifiable; mutation-tested | 7.3 | 8.2 | | -| Module notes for every diverging module (at least `objects`) | 7.5; D-23 | 7.5 acceptance | | +| Module notes for every diverging module; `objects` decided from evidence of the LiveObjects public API (full notes if exposed, even if incomplete; a placeholder or an entry with every tier marked not ready otherwise) | 7.5; D-23; 12.1–12.3; D-28 (decided at STOP-14) | 7.5 acceptance; 8.5 paths 3–4 | | | Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | | **Workflow** | | | | | Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | @@ -58,3 +58,4 @@ | **Creation process** | | | | | Skill and harness created on an Opus-class model (MUST); model stated for runs, or pinned where the tool supports that (SHOULD), and recorded in the skill's final report | 2.8; D-26; 7.6 (frontmatter, final report) | Design record "Inputs at generation time" (Model line); 8.3 (pilot model recorded); Final report header | | | Reference-implementation reads, if any, last resort only, recorded and reported as guide gaps | 2.1; design-record template | Final report "Reference-implementation reads" | | +| An existing skill, its harness and its UTS-derived tests audited against this checklist before being changed; gaps listed for the owner to choose; working parts preserved (SHOULD) | 13 (Orient); 10; 11.2–11.5; D-27; D-29; skill-gap-audit template | STOP-15; 11.6; Final report "Existing assets and upgrade summary" (n/a in create mode) | | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md index a296cb86b..1a0ccf787 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md @@ -6,11 +6,16 @@ - Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: - Harness README: - Model: (Opus-class: yes | no, departure in decision log #); sub-agents: , or none +- Eligibility: via (accept | fork-confirmed) | user-named: () +- Capabilities (D-31): rest ; realtime (); liveobjects ; side: (); scope: ([ + REST-only tests under realtime]); unsupported: (); capability-inapplicable: tests (listed in the notes); names: +- State class (Orient): S0 | S1 | S2 | S3 | S4 (+ re-entry) — ; run mode (D-27): create | upgrade: diff-driven | upgrade: full audit | upgrade: regenerate (resumed | restarted ); target skill: (origin …), or none +- Run status: in progress (Phase , ) | finished () +- LiveObjects evidence: detect_liveobjects.py → , ; decision at STOP-14 (D-28) ## Decisions | ID | Decision | Choice | Guide-default? | Decided by | Rationale | |---|---|---|---|---|---| -| D-01 | Skill name and location | | yes / no | user / agent | | +| D-01 | Skill name and location | | yes / no | user / agent / existing skill | | | D-02 | Script language and runtime | | | | | | D-03 | Mapping: module × tier → target dir (hand-maintained entries marked) | (table below) | | | | | D-04 | Naming and collection rules (with three worked examples) | | | | | @@ -32,10 +37,15 @@ | D-20 | Commit policy; project/solution-file edits (approved / left to the user) | | | | | | D-21 | Fix-attempt bound | | | | | | D-22 | allowed-tools; remote access | | | | | -| D-23 | Notes files per module (full / placeholder) | | | | | +| D-23 | Notes files per module (full / placeholder; objects per D-28) | | | | | | D-24 | Working-records location | | | | | | D-25 | Pilot specs and mode per tier | | | | | | D-26 | Model tier and pinning for skill runs | | | | | +| D-27 | Run mode and state class (STOP-13) | | | | | +| D-28 | LiveObjects support: full / placeholder / none; translate-only / evaluate; SDK-blocked tiers and evidence, if any (STOP-14, 12.5) | | | | | +| D-29 | Gap-audit choices: add / skip / defer per GA- or CH- item; preserve list (STOP-15; n/a in create mode) | | | | | +| D-30 | Provenance stamp (metadata.generated-by, metadata.records) and Run status | | | | | +| D-31 | Capability profile and skill scope (STOP-13 or STOP-17) | | | | | ### D-03 mapping | Module | unit | integration | proxy | notes | hand-maintained? | @@ -50,5 +60,5 @@ |---|---|---|---|---|---|---|---| ## Changelog (fixes made to the skill, notes or harness) -| # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Tests regenerated | -|---|---|---|---|---|---| +| # | Date | Found by | Cause (skill / notes / catalogue / script / harness) | Fix | Closes (gap-audit GA-nn / G-nn, if any) | Tests regenerated | +|---|---|---|---|---|---|---| diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md index 4ccc5699e..2a8ec5c6c 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md @@ -3,12 +3,20 @@ - Spec clone (clean | dirty: ); repo ; date - Model: (Opus-class: yes | no, and why); sub-agents: , or none - Scope: ; out of scope and why +- Conforms to the guide: yes | no (MUST rows ✗: …) +- Eligibility: via (accept | fork-confirmed) | user-named: () +- Capabilities (D-31): rest …; realtime …; side: ; scope: ; unsupported: … (capability absent); capability-inapplicable: tests; partial skill: yes, extend by Upgrade/Fix when appears | no +- State class: S0 | S1 | S2 | S3 | S4 (+ re-entry); run mode: create | upgrade: diff-driven | upgrade: full audit | upgrade: regenerate (resumed | restarted); LiveObjects: full | placeholder | none (translate-only | evaluate), from detector recommendation ## Files created or changed -| Path | New / changed | Kind (hook, harness, smoke test, skill, script, notes, pilot test, CI suggestion) | Approved at | -|---|---|---|---| +| Path | New / changed | Kind (hook, harness, smoke test, skill, script, notes, pilot test, CI suggestion) | Item (GA-nn / CH-nn / G-nn, upgrade mode) | Approved at | +|---|---|---|---|---| Hand-maintained project files the user must update: … +## Existing assets and upgrade summary +Create (S1): the UTS-derived tests and harness code found (P-20), whether they match D-03, D-06 and D-07, and the follow-ups proposed. +Upgrade/Fix: items added, skipped (with the reason) and deferred (with the plan), by GA-/CH-/G- ID, from skill-gap-audit.md; harness rows decided at STOP-3; preserve list kept; baseline comparison (resolver, audit, harness tests, existing UTS-derived tests): differences and the item that explains each; what the old skill had that the new one lacks (regenerate only). + ## Repo profile Link to repo-profile.md; corrections made at STOP-2. @@ -30,6 +38,9 @@ Link to design-record.md; decisions that depart from the guide's defaults, and w Fixes made to the skill, notes or harness during the pilot (from the changelog): … Reference tests named in SKILL.md: +## SDK-blocked objects tiers +: — ; translate-only until the smoke test passes; acceptance row "Harness smoke tests …" ✗ (SDK-blocked). "None" otherwise. + ## Known limitations Tiers not ready; `not available` catalogue rows; harness capabilities missing; audit SHOULD items not implemented; platforms the scripts weren't run on. @@ -45,7 +56,7 @@ Each read from the Design record (question, repo @ revision : path, pattern lear (not applied unless approved at STOP-5) ## Next steps -Which modules and tiers to translate next, in order, and why; harness work remaining. +Which modules and tiers to translate next, in order, and why; harness work remaining; deferred gap-audit items with their plan. ## Acceptance checklist (the acceptance-checklist template, filled) diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md index d769f70f7..11fce9759 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md @@ -1,5 +1,8 @@ # Repo profile: +## State summary (Orient, STOP-13) +(paste orient.py's stateSummaryText, completed with the model, the capability profile confirmed at STOP-13 or STOP-17 (D-31), and the option chosen at STOP-13) + - Repo: , HEAD (clean | dirty: ) - Spec clone: , HEAD (clean | dirty: ); guide/skill last changed in ; skill version - Date: ; confirmed by the user on (STOP-2) @@ -24,9 +27,9 @@ | P-10 | Threading, callback queues, internal-queue drain, compile-time concurrency checks | | | | | P-11 | Every time source (timers, timed waits, delayed callbacks, clock reads) | | | | | P-12 | Runner parallelism and how to serialise | | | | -| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global) | | | | +| P-13 | Hooks: WS factory / HTTP / clock / network monitor / randomness (symbol, how set, when, per-client or global, reachable from the SDK language: yes / no, a harness gap) | | | | | P-14 | Internal-access mechanism and current use | | | | -| P-15 | API surface vs pseudocode, per module (rest / realtime / objects) | | | | +| P-15 | API surface vs pseudocode, per module (rest / realtime / objects); objects: detect_liveobjects.py summary | | | | ## Existing native test support | # | Item | Answer | Evidence | Verified? | @@ -35,7 +38,7 @@ | P-17 | Existing test helpers (waits, polls, capture, log capture, assertions, factories) | | | | | P-18 | Existing sandbox/integration support; fault-injection tooling | | | | | P-19 | ably-common submodule (path, initialised, fixture files) | | | | -| P-20 | Existing UTS assets and their state | | | | +| P-20 | Existing UTS assets and their state (inspect_existing_skill.py: skills and installs, harness entry, tags, header SHAs, deviations.md) | | | | | P-21 | Sandbox access from developer machines and CI | | | | ## How the existing tests drive the SDK (step 1c) @@ -57,7 +60,7 @@ ## API divergences per module - rest: … - realtime: … -- objects: … (implemented? plugin? typed? IDL followed?) +- objects: … (public API exposed? implemented? plugin? typed? IDL followed?); detector recommendation: …; decision at STOP-14: … ## Unknowns and questions for the user - … diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md new file mode 100644 index 000000000..0e3d9be5b --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md @@ -0,0 +1,49 @@ +# Gap audit: uts-to- in + +- Existing skill: (installs: <.claude/skills/…, .agents/skills/… (symlink | copy)>); frontmatter version +- Repo (clean | dirty: ); spec clone (clean | dirty: ) +- State class (Orient): S2 | S3 | S4; mode at STOP-13: upgrade: diff-driven | upgrade: full audit | upgrade: regenerate — chosen by on +- inspect_existing_skill.py run: ; its checks are regex hints, every row below is confirmed by reading +- Capabilities: recorded D-31 vs Orient now ; delta: + +## A. Inventory +| Part | What exists (path, symbol, command → result) | Baseline result (resolver, audit, harness tests, compile) | +|---|---|---| +| SKILL.md (lines, headings, frontmatter fields) | | | +| uts-package-mapping.json (modules, tiers, notes, harness entry, unready data, `unsupported` / `capabilityInapplicable` / `extraTests` markers) | | | +| scripts/ (resolver, audit, scanner, others) | | | +| references/ (notes per module; placeholders) | | | +| Harness (root, README, Known gaps, smoke tests, self-tests, CI jobs) | | | +| UTS-derived tests (see D) | | | +| Working records | | | + +## B. Conformance (one row per acceptance-checklist row outside the Harness group; copy its first column) +A row (or part of one) needing a capability the SDK lacks is "n/a — capability absent: (D-31)"; one that was n/a when recorded and whose capability now exists is "missing (newly applicable)". Each row and each section-D follow-up gets an ID (GA-01, GA-02, …) used by the decision, the changelog and the Final report. Status: conforms / partial / missing / non-conforming / n/a (reason) / unverified (checked in ). Choice at STOP-15: add / skip / defer (a skipped or deferred MUST row leaves the skill not conforming). The objects half of the module-notes row comes from D-28; Creation-process rows for a skill of unknown origin: "origin unknown; this run meets it for the artifacts it changes". + +| # | Acceptance-checklist item | Level | Status | Evidence | Proposed change | Step | Size | Approval needed | Choice | Re-confirmed (STOP-8 or after Phase 6) | +|---|---|---|---|---|---|---|---|---|---|---| +| GA-01 | | MUST / SHOULD | | | | | | | | | + +## C. Harness (decided per row at STOP-3, not here; G-19/G-20 for offered tiers can't be skipped or deferred) +| Row | Capability | Quick status | Evidence | Note for Phase 2 | +|---|---|---|---|---| +| G-01 … G-20 | (from the uts-infra-design template) | | | | + +## D. UTS-derived tests +| # | Module × tier (or "outside mapped tiers") | Files | Tags | Tag style | Header SHAs (vs clone HEAD) | No SHA / links main | deviations.md and other deviation records | Compile | Audit | Follow-up proposed | Choice | +|---|---|---|---|---|---|---|---|---|---|---|---| + +## E. LiveObjects (decided at STOP-14) +detect_liveobjects.py → , ; decision D-28: full | placeholder | none, translate-only | evaluate; SDK-blocked tiers (12.5): … ; objects items implied: … + +## F. Preserve list +| What is kept as is | Where | Why it conforms | +|---|---|---| + +## G. Decision at STOP-15 +Chosen by on : add …; skip …; defer … (GA- and CH- items); or regenerate from scratch instead. Recorded as D-29. + +## H. Changes since the recorded run (diff-driven Upgrade/Fix, section 10) +| # | What changed (guide / UTS docs / helper specs / corpus / harness / SDK) | Evidence (diff paths) | Artifacts to update | Tests to regenerate | Choice | +|---|---|---|---|---|---| +| CH-01 | | | | | add / skip / defer | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md index 73b3f29d0..c8c1d3233 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/uts-infra-design.md @@ -1,5 +1,9 @@ # Harness design: @ +- Run mode (D-27): …; upgrade: rows seeded from skill-gap-audit.md section C on , each re-checked per 4.2 +- LiveObjects (D-28): full | placeholder | none +- Capabilities (D-31): rest …; realtime …; side: …; hooks reachable from the SDK language (P-13): yes | no; rows and tiers needing an absent capability are "n/a — capability absent: (D-31)", not gaps (13.8) + ## Gap table | # | Capability | Guide | Status | Evidence | Approach (reuse / wrap / extend / build / n/a) | Builds on | Blocks (module/tier) | Production change? | Size | Acceptance check | |---|---|---|---|---|---|---|---|---|---|---| @@ -12,7 +16,7 @@ | G-07 | MockWebSocket (+ alternative API, undocumented members, raw frames, ping) | 2.2, mock_websocket.md | | | | | | | | | | G-08 | MockVCDiff encoder/decoders | 2.2, mock_vcdiff.md | | | | | | | | | | G-09 | MockNetworkListener | 2.2 | | | | | | | | | -| G-10 | standard_test_pool.md implementation; plugin options builder | 2.2, standard_test_pool.md | | | | | | | | | +| G-10 | standard_test_pool.md implementation; plugin options builder (only if D-28 = full; else n/a (D-28)) | 2.2, standard_test_pool.md | | | | | objects/unit | | | | | G-11 | Wait helpers (AWAIT_STATE, poll_until, poll_until_success, process_pending_events, wall-clock wrapper, fake clock driver) | 2.2, 5.6, 5.7 | | | | | | | | | | G-12 | Thread-safe capture, log sink, assertContainsInOrder, caller-attributed failures | 2.2 | | | | | | | | | | G-13 | Placement and recommended harness layout (shared / port-only / module helpers); fixture-helper scope documented in the module notes | 2.2 | | | | | | | | | @@ -51,12 +55,13 @@ ## Tier feasibility | Module | unit | integration | proxy | |---|---|---|---| -| rest | ready / after G-.. / not possible: / n/a | | | +| rest | ready / after G-.. / not possible: / n/a / n/a — capability absent: (D-31) | | | | realtime | | | | -| objects | | | | +| objects | per D-28: full → as above; placeholder → not possible: placeholder notes; none → n/a (D-28) | | | ## Decision at STOP-3 (a) build now: | (b) plan only | (c) scope to ready tiers — chosen by on ; design changes requested: … +Upgrade mode: per row build / plan (defer) / skip: …; G-19/G-20 for every tier the skill offers can't be skipped or deferred ## Build log (Phase 3) / plan (option b) | Order | Row | Work | Acceptance check | Result | Date | diff --git a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md index 71cbf23ab..9bd11f35d 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md +++ b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md @@ -31,4 +31,15 @@ All from the guide and the UTS docs; follow the link for the detail. | Local build green, CI red | CI is stricter (warnings as errors, static analysis, type checking) | Compile as strictly as CI in the skill | guide [10](../../../docs/writing-uts-spec-translator-skills.md#10-verification-and-ci); `writing-derived-tests.md` "Build pipeline and CI checks" | | A test fixed by hand reverts on the next run | The fix was made only in the generated file | Fix the skill, notes or harness; regenerate | guide [5](../../../docs/writing-uts-spec-translator-skills.md#5-translation-rules) | | Stale tests after a spec update | Renamed or merged IDs not followed | Re-sync mode; pair missing and orphan IDs; re-check assertions | guide [9.3](../../../docs/writing-uts-spec-translator-skills.md#93-renamed-merged-and-added-ids) | +| An upgraded skill lost hand-tuned rules, or existing UTS tests changed shape | The skill was regenerated or rewritten instead of upgraded in place | Upgrade in place, closing one chosen item at a time; keep the preserve list; compare with the baseline | [11.5](upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped), [11.6](upgrade-existing-skill.md#116-step-u5-validate-without-regressions); guide [9.5](../../../docs/writing-uts-spec-translator-skills.md#95-upgrading-an-existing-skill-should) | +| The detector says `no` or `unclear`, yet the SDK has LiveObjects | The API is in another package or repo, a submodule, generated code, or uses names it doesn't know | Ask where the API lives; run it there (or with `--include-submodules`); read the files; the user decides at STOP-14 | [12.1](liveobjects-support.md#121-gather-the-evidence-step-1b-p-15) | +| A real Ably Pub/Sub SDK is rejected or asked about at STOP-16 | A fork-only clone (no `ably/…` remote), an SSH host alias not resolvable from the ssh config, the wrong path or a sparse checkout (`NO_SDK_FINGERPRINT`), a client the definition gate can't see (`NO_CLIENT_DEFINITION`: a wrapper gem, a new client name), or a name the rule doesn't know | Add an upstream remote for the `ably/…` repository, fix the ssh config, or point at the repository root. A name the rule rejects, or an SDK the gate can't see, needs `assets/eligibility.json` updated (an allow-list entry with a `capabilityOverride`) or the alias added to `assets/capability-names.json`: no override at run time | [13.2](orient.md#132-repository-eligibility-stop-16) | +| A skill this procedure built is classified S3 | Its records weren't kept, or are somewhere else, or it predates the provenance stamp | Rerun Orient with `--records `, or proceed with the full gap audit | [13.3](orient.md#133-state-classes), [13.7](orient.md#137-provenance-d-30) | +| A single skill is classified S4 | A dangling symlink, a `uts-to-*` directory without `SKILL.md`, or a skill whose language has no files in the repo | Name the skill with `--skill-dir` (Orient then classifies only that skill and lists the others under Problems), and route it as 13.5's S4 rows: diff-driven if its origin is `creator-records`, else the full gap audit. Or repair the install: approve each change (never delete without asking), then rerun Orient | [13.3](orient.md#133-state-classes), [13.5](orient.md#135-routing) | +| Orient offers to resume a run that finished | The design record's Run status wasn't set to finished at Phase 7 | Restart, or set the Run status after checking the Final report | [13.6](orient.md#136-re-entry-resume-or-restart) | +| A REST-only SDK reported with realtime, or a full SDK reported REST-only | Realtime names only in comments, tests or docs; renamed clients the detector doesn't know; a client only reachable through a factory in an excluded directory or submodule | Read the evidence; correct the profile at STOP-17 with evidence; add a new client name to `aliases` in `assets/capability-names.json` (not the code) | [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | +| Orient's Spec names line says FALLBACK, or a renamed spec type isn't found | The spec clone's IDL couldn't be parsed (a missing heading, a moved file, an old clone), so the detectors used the data file's minimal, possibly stale names | Fix or update the spec clone at STOP-1 and rerun; if the IDL's format changed, report the parser gap. Don't rely on the LiveObjects or capability verdicts until it says `spec` | [13.4](orient.md#134-the-state-summary-and-stop-13), [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | +| Sides reported for an SDK without doors, or a door's side wrong | A factory or door type whose name or path the data file doesn't know, or a door type under an unexpected path | Read the Side lines' evidence; correct the side at STOP-17; add the door type or path to `doorFactories.sides` in `assets/capability-names.json` | [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | +| A harness defect reported as SDK-blocked | The objects smoke test fails, and it is blamed on the SDK without checking | SDK-blocked needs every self-test green and evidence naming an SDK not-implemented symbol; otherwise it is a harness stop | [12.5](liveobjects-support.md#125-an-sdk-blocked-objects-tier) | +| `objects` translated with no notes in a later run | No mapping entry, so the skill's step B offered `--create` | Give `objects` an entry even when out of scope: every tier unready, with the reason | [12.2](liveobjects-support.md#122-stop-14-recommend-then-ask) | | Rules in the skill and the manual disagree | The skill inlined a copy of a semantic rule | Link to the source instead | guide ["What the skill owns"](../../../docs/writing-uts-spec-translator-skills.md#what-the-skill-owns-and-what-it-defers), [12](../../../docs/writing-uts-spec-translator-skills.md#12-lessons-learned) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/glossary.md b/uts/skills/uts-to-lang-skill-creator/references/glossary.md index ca838a1a9..ef497b5f0 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/glossary.md +++ b/uts/skills/uts-to-lang-skill-creator/references/glossary.md @@ -32,4 +32,23 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs | **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | | **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | | **Phase N / generated-skill Phase 1–2** | "Phase N" (1–7) is a phase of this procedure. "Generated-skill Phase 1" (selection, steps 0 and A–F) and "generated-skill Phase 2" (per spec, steps 1–7) are the generated `uts-to-` skill's workflow (guide [4](../../../docs/writing-uts-spec-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | -| **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report | +| **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report; in upgrade mode, also the gap audit | +| **Orient** | Step 0 of every run: eligibility, the state class and the recommended mode, before any phase ([section 13](orient.md#13-step-0-orient)) | +| **Run mode** | Create or Upgrade/Fix (diff-driven, full gap audit, or regenerate), chosen at STOP-13 (D-27); an interrupted run is resumed or restarted | +| **State class** | Orient's classification: INELIGIBLE, S0 (nothing), S1 (tests or harness, no skill), S2 (a skill this procedure built), S3 (a skill of unknown origin), S4 (ambiguous); re-entry is an overlay | +| **State summary** | The short report Orient shows at STOP-13 and copies into the Repo profile | +| **Origin** | Whether a skill was built by this procedure: `creator-records`, `creator-metadata` (the D-30 stamp) or `unknown` | +| **Run status** | The design record's line saying whether a run is in progress (and in which phase) or finished; drives re-entry | +| **Eligible repo** | An Ably Pub/Sub SDK repository: `ably/ably-` or `ably/ably-pubsub-` (one repository, renamed), not on the deny-list, or on the allow-list (`ably-ruby-rest`), that passes the definition gate ([13.2](orient.md#132-repository-eligibility-stop-16)) | +| **Definition gate** | Eligibility's check, after the name rule, that a REST or Realtime client entry point is *defined* in non-test source, not merely used: no definition in a sizeable tree rejects (`NO_CLIENT_DEFINITION`), in a small one asks (`NO_SDK_FINGERPRINT`); an allow-list `capabilityOverride` satisfies it ([13.2](orient.md#132-repository-eligibility-stop-16)) | +| **Data files** | `assets/eligibility.json` (which repositories are accepted) and `assets/capability-names.json` (SDK aliases, door factories, legacy names, the names fallback): what maintainers update as the org and the SDKs change, instead of the scripts; each starts with `_description`, and `_comment` keys explain entries ([section 10](maintenance.md#10-maintenance)) | +| **Names source** | Where the detectors' names came from: `spec` (parsed from the spec clone's IDL at run time, with `specRevision`) or `fallback` (the data file's minimal names, with a loud warning) ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)) | +| **Capability profile** | The SDK's REST and Realtime clients (with connection, channels and presence) as `detect_capabilities.py` finds them, confirmed in Orient and recorded as D-31; it decides the skill's scope ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)) | +| **Side, door** | In an SDK split into server and device packages, a *door* is a public factory that returns a client (`PubSubDevice.CreateClient`); a *side* is the set of clients reached through one door, or through the core constructors (`core`). D-31 records the side the UTS tests construct clients through, and the scope follows it ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)) | +| **Not applicable (capability absent)** | A module, spec, gap-table row or checklist row that needs a capability the SDK (or the chosen side) lacks: not a gap and never ✗; it becomes a gap when the capability appears | +| **Unsupported** | The mapping marker (`"unsupported": ""`) on a whole module the SDK can't support, such as `realtime` for a REST-only SDK; the resolver and the skill refuse it with the reason, except any `extraTests` it lists | +| **Capability-inapplicable** | A test that needs a client the SDK lacks (a REST test that uses a Realtime client, in a REST-only SDK): guide 7.4 case (d), listed in the mapping's `capabilityInapplicable` or noted in the test file; not a deviation, and not language-inapplicable | +| **Partial skill** | A skill scoped to the SDK's capabilities (REST-only or realtime-only), extended later by Upgrade/Fix | +| **Gap audit** | Upgrade/Fix's list of each checklist item an existing skill, its harness or its UTS-derived tests don't meet, with the user's choice per item (add, skip, defer) and a preserve list ([section 11](upgrade-existing-skill.md#11-upgrade-mode-audit-and-upgrade-an-existing-skill)) | +| **SDK-blocked** | An objects tier whose smoke test fails only on SDK not-implemented paths while every self-test is green; listed in Known gaps, kept running under a strict expected-failure marker, translate-only until it passes ([12.5](liveobjects-support.md#125-an-sdk-blocked-objects-tier)) | +| **LiveObjects evidence** | `detect_liveobjects.py`'s spec-derived regex findings (public-API, legacy, shared, internal, wire-only names), from which STOP-14 recommends whether to add `objects` support ([section 12](liveobjects-support.md#12-liveobjects-objects-support)) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/liveobjects-support.md b/uts/skills/uts-to-lang-skill-creator/references/liveobjects-support.md new file mode 100644 index 000000000..0e169eb97 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/liveobjects-support.md @@ -0,0 +1,125 @@ +# 12. LiveObjects (`objects`) support + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +**Goal:** an explicit decision, made by the user from evidence in the SDK's source, on whether the skill covers `uts/objects`. Guide [3.1](../../../docs/writing-uts-spec-translator-skills.md#31-layout) ("Decide `objects` from evidence") is the requirement. You ask this question (**STOP-14**) in every run and every mode (Create and Upgrade/Fix, including regenerate and diff-driven), even when the evidence is unambiguous, and even when an existing skill already covers `objects`. Orient's State summary has already shown a one-line verdict. A resumed run asks it only if D-28 isn't recorded yet (13.6). For an SDK without a Realtime client (D-31, [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)), still ask, and recommend (c) with the reason "capability absent: no realtime client". + +Contents: + +- [12.1 Gather the evidence (step 1b, P-15)](#121-gather-the-evidence-step-1b-p-15) +- [12.2 STOP-14: recommend, then ask](#122-stop-14-recommend-then-ask) +- [12.3 What each choice adds, phase by phase](#123-what-each-choice-adds-phase-by-phase) +- [12.4 What the objects notes typically cover](#124-what-the-objects-notes-typically-cover) +- [12.5 An SDK-blocked objects tier](#125-an-sdk-blocked-objects-tier) + +## 12.1 Gather the evidence (step 1b, P-15) + +During the discovery checklist (P-15), run `python3 /scripts/detect_liveobjects.py --spec-clone ` (Orient already has). It is a read-only regex heuristic, and it prints one JSON object: a one-paragraph `summary`, the `recommendation`, and the evidence. Every name it searches for comes from the spec in the clone, never from another SDK's code. `spec_names.py` parses them from the objects IDL at run time; the legacy and prose-only names are in `assets/capability-names.json`. `namesSource` says which: `fallback` means the IDL couldn't be parsed, and the verdict isn't reliable until that is fixed ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)): + +| Category | Names (from the spec) | What it shows | +|---|---|---| +| `publicApi` | The types and members of the [objects IDL](../../../../specifications/objects-features.md) that aren't marked `internal` and aren't also protocol types (`RealtimeObject`, `PathObject`, `LiveMap`, `LiveCounter`, `ObjectsEvent`, `ObjectsSyncState`, `StatusSubscription`, the subscription event and options types, `compactJson`; `Instance` is left out because the bare word is too common, and is covered by `InstanceSubscriptionEvent`) | The public API is exposed. Declarations carry a visibility heuristic for each language, and `publicDeclaredAt` gives up to three declaration sites per name; `missingPublicNames` lists the current-spec names with no public declaration | +| `accessor` | The `RealtimeChannel#object` accessor (RTL27), used on a channel-named receiver or declared with the accessor's type (`RealtimeObject` today; `accessorTypes`, read from the IDL's `RealtimeChannel` property of a LiveObjects type); the earlier spec's `channel.objects` form is counted here too (form `objects`) | The entry point exists. A public declaration alone is enough for a `yes` (medium) | +| `legacyPublicApi` | Public names from earlier revisions of that spec, before the entry point became `channel.object.get()` and the value types became `LiveMap` and `LiveCounter`: `RealtimeObjects`, `getRoot`, `createMap`, `createCounter`, `LiveObjectSubscription`, `LiveMapValueType`, `LiveCounterValueType`; and `LiveObject` and its `*Update` interfaces, which were public then and are internal now, when declared public | The SDK may follow an earlier spec revision, which notes item 1 must name. These names often survive in comments or internal code: public declarations count fully, and other uses in source only toward the medium rule. Members written in another case (`CreateMap`) count only as declarations, so an unrelated library call doesn't | +| `shared` | Public message types that are also protocol types (`PublicAPI::ObjectMessage`, `PublicAPI::ObjectOperation`, the operation types and the operation actions) | Neither for nor against a public API | +| `internalApi` | Names that are internal in the current spec: IDL types and members marked `internal` (`InternalLiveMap`, `InternalLiveCounter`, `LiveObject`, `publishAndApply`, …) and `ObjectsPool` from the spec prose | An implementation exists, but this alone doesn't show the public API | +| `wireOnly` | Protocol-only names from `features.md`: `OBJECT_SYNC`, `ObjectState`, `ObjectData`, `ObjectsMap`, `ObjectsCounter` and their parts, the object channel modes | Protocol-level support, not the public API | +| `pluginKey`, `packaging` | Identifiers naming the `LiveObjects` plugin (PC5, PT2b) in [`features.md`](../../../../specifications/features.md); `live[-_ ]objects` in build manifests and directory names; a bare `objects` directory as weak evidence; `nestedPackages`, directories with their own package manifest that hold public-API names | A plugin or module that may hold the API. Confirm each nested package is the SDK's own, not a vendored copy | + +Names are matched as whole identifiers: the spec name, the same name behind a short all-caps prefix (such as an Objective-C `ART` or a C# `I`), its snake_case form, and, for members, any capitalisation (`GetRoot`, `CompactJSON`). Hits on comment lines and in test files are counted apart from the rest, and don't drive the recommendation. + +It skips the following, and reports what it skipped: + +- vendored directories (`vendor`, `node_modules`, `Pods`, `Carthage`, `third_party`, `deps`, `external` and the like, in any case); +- agent skill directories (an existing skill's notes would be false positives); +- git submodules (pass `--include-submodules` if the SDK keeps code in one); +- any copy of the spec repo, and copies of this script; +- in a git repo, ignored files, such as build output and test results, which never reach the scan (without git, it skips them by directory name and counts them); +- untracked files, which it counts apart. + +It also reports not-implemented markers found in files that hold public-API names. + +Then **read** what it lists: the declaration sites, the accessor and the manifests. The heuristic can't see everything. For example, it misses re-exports, `__all__` and generated code, and it can't tell a stub from a real implementation. Its `suggestedMode` is advisory: the translate-only or evaluate choice is made below and at the skill's step E. If the user says the LiveObjects API lives in another repository or package, run the script on that path too (read-only, with the user's path). Record the summary (recommendation, confidence, reasons, the public symbols found, and a few files) in the Repo profile's P-15 objects line. + +Also establish whether the implementation is complete enough to evaluate. Use the not-implemented markers, the public types compared with the IDL's list, the SDK's own objects tests, and the user's answer. If you're unsure, ask. This decides translate-only or evaluate for `objects` (guide [4, step E](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af); notes item 1). + +## 12.2 STOP-14: recommend, then ask + +Present: + +- the script's `summary`, verbatim; +- a table with one row per category: the symbols, their counts (source and test), and where the public ones are declared (`publicDeclaredAt`), plus `missingPublicNames`; +- the script's recommendation (`yes`, `no` or `unclear`), its confidence and reasons, and your own reading after reading the files; +- the options below, with your recommendation. + +| Option | When to recommend it | Result | +|---|---|---| +| **(a) Full objects support** | The SDK exposes the LiveObjects public API, even if the implementation is incomplete | Objects notes, the `objects` mapping entry and the objects harness pieces ([12.3](#123-what-each-choice-adds-phase-by-phase)). If the implementation is incomplete, objects runs default to translate-only (the skill's step E); once evaluated, a failing test is an SDK deviation, gated, or a skipped stub where the API is missing (guide [7.1](../../../docs/writing-uts-spec-translator-skills.md#71-three-acceptable-end-states)) | +| **(b) Placeholder** | Only internal, wire-level or packaging evidence; or the public API is planned but not yet in the source | A placeholder notes file and mapping entry. The skill refuses the module until the notes are written (guide [4](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af)) | +| **(c) No objects support now** | No evidence, or the user doesn't want it | An `objects` mapping entry with the D-03 default path for each tier, every tier marked not ready ("out of scope, D-28"), and no notes. The skill's step C then refuses the module with that reason; without an entry, its step B would offer to create one and translate with no notes | + +If the result is `unclear`, ask the user where the public API lives before you recommend anything. When an existing skill already has full objects notes (Upgrade/Fix: full gap audit or diff-driven), the options become: + +- (a) keep the objects support and upgrade it with the gap audit's objects items ([11.3](upgrade-existing-skill.md#113-step-u2-write-the-gap-audit)): D-28 full; +- (b) keep it as it is: D-28 full, recorded as "kept as is", with the gap audit's objects rows defaulting to skip; +- (c) take it out of scope: D-28 none. This never deletes files without the user's approval for each one. + +An existing placeholder notes file gets the normal options above. + +Ask this after STOP-2 (step 1e, [3.5](phase-1-understand-repo.md#35-step-1e-decide-liveobjects-support-stop-14)). Record the answer as D-28 (with the detector's recommendation and the mode, translate-only or evaluate), add it to the Repo profile's objects line, and log it in the decision log. A later run that finds new evidence asks again (see [section 10](maintenance.md#10-maintenance)). + +## 12.3 What each choice adds, phase by phase + +| Phase | (a) Full | (b) Placeholder | (c) None | +|---|---|---|---| +| 1 | The P-15 objects divergence list (IDL followed, plugin or built-in, typed or dynamic). Read a plugin test in step 1c ([3.3](phase-1-understand-repo.md#33-step-1c-study-how-the-existing-tests-drive-the-sdk)) | P-15 says why it's a placeholder | P-15 says why there's no objects support | +| 2 | G-10 in scope: the [`standard_test_pool.md`](../../../objects/helpers/standard_test_pool.md) symbol map ([4.3](phase-2-design-harness.md#43-step-2c-map-each-helper-spec-to-native-code)) and a plugin client-options builder. Place the module helpers where white-box specs can see internals (P-14). Fill the objects rows of the tier matrix ([4.8](phase-2-design-harness.md#48-step-2h-derive-tier-feasibility)) | G-10 `n/a` (D-28). The objects rows read "not possible: placeholder notes" | G-10 and the objects rows `n/a` | +| 3 | [5.2](phase-3-build-harness.md#52-step-3b-build-in-order-with-acceptance-checks) row 11 (the objects unit smoke test and the serial-helper self-test). The objects REST provisioning helper is built with row 9. If the implementation is incomplete, the smoke test may end SDK-blocked ([12.5](#125-an-sdk-blocked-objects-tier)) | — | — | +| 4 | D-03 gets an `objects` entry (hand-maintained if the module lives in a separate plugin package or build module). D-12 lists the white-box specs, D-23 says full, and D-25 has an objects pilot | D-03 entry with placeholder notes. D-23 says placeholder | D-03 entry with every tier not ready. D-23: none | +| 5 | Notes per [7.5](phase-5-generate-skill.md#75-step-5e-write-the-module-notes) and 12.4. Catalogue rows for the constructs `scan_constructs.py` finds in `uts/objects` | A placeholder notes file saying why, and what would unblock it | — | +| 6 | An objects pilot ([8.3](phase-6-validate-skill.md#83-pilot-translate-one-spec-per-available-tier)), translate-only when D-28 says so, and always for an SDK-blocked tier (12.5) | 8.5 path 3 (placeholder refused) | 8.5 path 4 (unready tier refused, with the reason) | + +## 12.4 What the objects notes typically cover + +The requirement is guide [3.4](../../../docs/writing-uts-spec-translator-skills.md#34-recommended-outlines-should)'s thirteen items, in order, as [7.5](phase-5-generate-skill.md#75-step-5e-write-the-module-notes) says. For `objects`, they usually amount to the following. Use this list to know where to look, not as text to fill in. Write every mapping from the SDK source and the harness as built: the notes are a map, and the SDK source is the authority (guide 3.1). + +1. **Source of truth:** which IDL the SDK follows. It may be the current [`objects-features.md`](../../../../specifications/objects-features.md) (`channel.object.get()` returning a `PathObject`, RTO23), an earlier revision (the detector's legacy names, such as `channel.objects.getRoot()`), or the typed-SDK variant that guide 3.4 item 1 names. Also say whether the module is implemented: this is the translate-only or evaluate decision (D-28). +2. **Layers:** `LiveMap` and `LiveCounter` as creation value types (RTLMV, RTLCV), the public views (`PathObject`, `Instance`) and the internal nodes (`InternalLiveMap`, `InternalLiveCounter`). The specs use all three. +3. **Entry point and setup:** plugin registration in the client options (PC5, PT2b); the RTL27 accessor, and its error when the plugin is missing (RTL27b); the object channel modes. +4. **Async model:** `get()` waiting for sync (RTO23c); how operations marked `=> io` render. +5. **Type mapping:** the API map from the ably-js-shaped pseudocode to the SDK. This covers `PathObject` and `Instance` navigation and reads, which conversions throw and which return null, the write-value union and its wrapper type, enums, and numeric types. +6. **Mutations, subscriptions and events:** subscribe and unsubscribe, `StatusSubscription`, the subscription event types and their options. +7. **Messages and errors:** the public `ObjectMessage` fields, the error codes the objects spec defines (for example 92005, 92007 and 92008) with `statusCode` and `cause`, and timestamp units. +8. **Internal access:** the white-box specs. These are typically the `internal_*` specs and `objects_pool.md`, but confirm each one by reading it ([7.5](phase-5-generate-skill.md#75-step-5e-write-the-module-notes) item 8). Also the ladder (D-12) and the placement boundary. +9. **Helper coverage:** every `standard_test_pool.md` symbol mapped to its native symbol, and each fixture helper's scope (which client messages `setup_synced_channel` answers, and any option the fake-clock variant needs). +10. **Overrides:** the reading list (the `objects-features.md` sections), naming (never dropping the spec point, guide [5.1](../../../docs/writing-uts-spec-translator-skills.md#51-traceability)), the `deviations.md` location in the owning module, and expected audit shortfalls. +11. **Integration helpers:** REST provisioning (`provision_objects_via_rest`, closing its client) and sync waits. +12. **A worked example and a symbol index.** +13. **Shape deviations** (`S-1…S-n`), if the SDK is typed. + +The `objects` notes of the existing `uts-to-swift` and `uts-to-kotlin` skills are reference implementations: read them only as a last resort, under [2.1](../SKILL.md#21-your-inputs), for a pattern, never to copy. The guide's [Patterns to avoid](../../../docs/writing-uts-spec-translator-skills.md#patterns-to-avoid) lists defects found in exactly those files (a naming override that drops the spec point, and spec points cited where they don't exist). + +## 12.5 An SDK-blocked objects tier + +Guide [2.7](../../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must) ("SDK-blocked objects smoke test") and guide [4, step F](../../../docs/writing-uts-spec-translator-skills.md#phase-1-selection-steps-0-and-af) are the requirement. When the SDK exposes the LiveObjects public API but doesn't implement it yet, the objects unit smoke test (`setup_synced_channel` completes and a pool object reads its seeded value) can't pass. Without this rule, the objects tiers would stay unready, and objects specs could never be translated. + +**When it applies.** Only when all of these hold. Any other red stays a harness or environment stop, as everywhere else. + +1. D-28 is full objects support. +2. The objects harness compiles. +3. Every helper self-test of that tier is green. +4. Every failing smoke-test assertion traces to an SDK not-implemented path. Record the evidence for each: the error or stack trace naming the SDK symbol (for example a `NotImplementedError` or a `TODO` throw), with its file and line. + +**What you do:** + +- **Known gaps.** List each one in the harness README's Known gaps as `SDK-blocked: ()`. +- **Keep the test running.** Keep the failing assertion running in CI under the runner's *strict* expected-failure marker (or as an assertion that the not-implemented error occurs). It is never skipped, and once the SDK passes it, the marker fails, which forces the gap to be cleared (guide 2.7 "In CI, ungated"). +- **Readiness data.** Mark the tier in the mapping as translate-only, not unready. *Procedure recommendation:* a `translateOnly` object per module, for example `"translateOnly": {"unit": "SDK-blocked: RTO23 get()"}`, which the resolver reports as `tiers..translateOnly` with the reason ([7.1](phase-5-generate-skill.md#71-step-5a-create-the-layout-and-the-mapping-file)). +- **Generated skill, step E.** Don't offer evaluate for that tier. +- **Generated skill, step F.** Pass only when the SDK-blocked items listed in Known gaps (under their expected-failure marker) are the only failures, and print them clearly in the preflight output. Re-check them in every preflight. When the smoke test passes, clear the Known gap and the `translateOnly` entry; evaluate then becomes available. +- **Records.** + - D-28: "full; SDK-blocked: …". + - The Final report: its own "SDK-blocked" line. + - In upgrade mode, gap-audit section E. +- **Acceptance checklist.** The row "Harness smoke tests per tier and helper self-tests …" is ✗ with the reason "SDK-blocked: objects unit smoke test ()". This is the same pattern as the declined-CI case in [5.4](phase-3-build-harness.md#54-step-3d-stop-7-confirm-the-harness). Other rows aren't affected. +- **Pilot.** The objects pilot for that tier runs translate-only, and the report says so (8.3). diff --git a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md index e4db49028..a2ce4ab97 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/maintenance.md +++ b/uts/skills/uts-to-lang-skill-creator/references/maintenance.md @@ -2,6 +2,8 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. +This is **Upgrade/Fix, diff-driven**: the sub-case Orient recommends at STOP-13 for a skill this procedure built, whose last run finished (state S2, [13.3](orient.md#133-state-classes)). Orient has already computed what changed since the recorded run (the State summary's "Changes" line). + The harness and the skill must stay in step with five things that change independently. Record the current state of each in the Design record ("Inputs at generation time"), so a later agent can diff against it. | What changed | How to detect it | What to update | @@ -10,12 +12,27 @@ The harness and the skill must stay in step with five things that change indepen | **The UTS docs or helper specs** | `git -C diff -- uts/README.md uts/docs/ uts/objects/helpers uts/rest/unit/helpers uts/realtime/unit/helpers` | Re-run the Phase 2 assessment for the affected rows; update the harness, its smoke tests and self-tests, and its README (Known gaps), the catalogue rows and notes item 9; then re-sync the affected tests | | **The UTS corpus** | The skill's own re-sync mode (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should)), plus the construct scanner ([7.4](phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)) | New catalogue rows (stop and ask for any unmapped construct); new harness symbols the corpus uses; regenerated tests; renamed IDs handled per guide [9.3](../../../docs/writing-uts-spec-translator-skills.md#93-renamed-merged-and-added-ids) | | **The harness itself** (a helper, mock, fixture or harness test changes) | Its smoke tests and self-tests in CI; the skill's preflight on every run | Fix a red harness test in the harness before any further translation; update the README and Known gaps; regenerate the tests whose rendering depends on the changed helper (guide 9.2) | -| **The SDK** (API, hooks, test layout, CI) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | +| **The SDK** (API, hooks, test layout, CI, capabilities) | `git -C diff ` (includes staged and unstaged changes; list untracked files with `git status --porcelain -- `) over the paths the Repo profile names | Repo profile; harness and hooks; notes (the SDK source is ground truth); mapping; then a re-sync if the harness or module helpers changed shape | + +**Keeping this skill's own data current.** What Orient knows about Ably's repositories and SDK names is data: update the data file, not the code. + +- [`assets/eligibility.json`](../assets/eligibility.json): when Ably adds, renames or retires a repository (the deny-list, the allow-list and its capability overrides, the canonical and planned names, the definition gate's threshold). +- [`assets/capability-names.json`](../assets/capability-names.json): when an SDK adds a client alias or a door factory, or the spec renames a LiveObjects name its IDL no longer shows. + +Each file starts with `_description`; give a non-obvious entry a `_comment` key. The scripts ignore `_` keys, check the shape, and report a malformed file as `DATA_FILE_ERROR`, naming the file and the key. Client, channel, connection, presence and LiveObjects names come from the spec clone at run time (13.8) and need no edit. After an edit, rerun Orient on a repository the change affects. Rules for every maintenance run: - Follow the same ground rules ([section 2](../SKILL.md#2-ground-rules)) and stop points, including the model tier ([2.8](../SKILL.md#28-model-tier-and-sub-agents)) and the last-resort rules for reference-implementation reads ([2.1](../SKILL.md#21-your-inputs)). If the guide's Patterns to avoid has changed since a recorded read, re-check that read's conclusion against it before relying on it again. -- Start from the working records; update the Repo profile and the harness design if anything they describe has changed. +- Start from the working records; update the Repo profile and the harness design if anything they describe has changed. Orient has run and STOP-13 chose this mode. Of Phase 1, re-run the discovery rows the change touches, STOP-2 for the updated profile, and step 1e (STOP-14). +- Ask **STOP-14** again (Orient has re-run `detect_liveobjects.py`), showing the recorded D-28 and what changed in the evidence (for example, the SDK has gained the public API since a placeholder was chosen) ([section 12](liveobjects-support.md#12-liveobjects-objects-support)). +- After STOP-2 and STOP-14, before editing: copy the skill-gap-audit template into the working records if it's absent, list each detected change as an item of its section H (CH-01, CH-02, …), with the artifacts it maps to and the tests to regenerate, then ask **STOP-15**: add, skip or defer per item. The items chosen there are what permit editing the skill and harness (2.2). - Regenerate tests through the skill; the only per-test state that survives is the list in guide 9.2 step 4 (DEVIATION gates, adapted assertions, `deviations.md` entries and, per `writing-derived-tests.md`, UTS-spec-error fail-fast placeholders until the spec is fixed) (guide [9.2](../../../docs/writing-uts-spec-translator-skills.md#92-a-re-sync-mode-should)). - After any harness change, re-run every affected tier's smoke tests and self-tests, and regenerate the affected tests; after any script change, repeat [8.1](phase-6-validate-skill.md#81-run-the-resolver-on-every-module) and [8.2](phase-6-validate-skill.md#82-test-the-audit-itself). - Record the new spec SHA and repo SHA in the Design record and append a changelog entry. +- If the user asks for an audit or an upgrade, or the guide's section 13 checklist changed since the recorded SHA, copy the skill-gap-audit template into the working records, take the baseline of [11.2](upgrade-existing-skill.md#112-step-u1-understand-the-existing-skill) (steps 2 and 3), then run the gap audit of [11.3](upgrade-existing-skill.md#113-step-u2-write-the-gap-audit) and STOP-15 for the affected items, rather than diffing alone. +- A capability added since the recorded D-31 (Orient's "Capability changed" line) is a change item, "capability added: ". It supersedes D-31, removes the module's `unsupported` marker (and the `capabilityInapplicable` entries the new client makes runnable) and maps the new tiers, adds the module notes, turns the G-rows and checklist parts that were n/a for that capability into gaps (gap-audit sections B and C, marked "newly applicable"), adds the guide 2.7 sets for the new tiers, and asks STOP-14 again ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)). A removed capability is its own item; tests are never deleted without asking. +- A side change (the SDK gains doors, or the user wants another side or both) is a change item, "side changed: -> ". Ask the side question of STOP-17 again, supersede D-31, and change the scope, the mapping markers and the harness's construction path to follow the new side ([13.8](orient.md#138-capabilities-and-scope-stop-17-d-31)). +- In the generated skill, a scope change is a mapping edit (`unsupported`, `capabilityInapplicable`, `extraTests`), not a resolver change ([7.1](phase-5-generate-skill.md#71-step-5a-create-the-layout-and-the-mapping-file)). +- If STOP-14's answer differs from the recorded D-28, run [12.3](liveobjects-support.md#123-what-each-choice-adds-phase-by-phase)'s column for the new choice through Phases 2 to 6, for `objects` only, and supersede D-28. +- End with Phase 7: add a dated section to `final-report.md` (the changes detected, the artifacts updated, the checks re-run), and re-mark the acceptance-checklist rows they affect. diff --git a/uts/skills/uts-to-lang-skill-creator/references/orient.md b/uts/skills/uts-to-lang-skill-creator/references/orient.md new file mode 100644 index 000000000..4b10b6899 --- /dev/null +++ b/uts/skills/uts-to-lang-skill-creator/references/orient.md @@ -0,0 +1,318 @@ +# 13. Step 0: orient + +Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) first (ground rules, stop points, path convention). Links that climb out of the skill directory are spec-repo paths: resolve them against the spec clone, not against this file's installed location. + +**Goal:** before anything else, in every run, establish four things: + +- that the repository is one this procedure supports; +- what it already has; +- which mode to run in (**Create** or **Upgrade/Fix**); +- for Upgrade/Fix, which existing skill to work on. + +Orient is read-only and quick. It builds nothing, runs no tests, touches no network and writes no file. The working records are created only after STOP-13, because their location depends on the answer (D-24). + +Contents: + +- [13.1 Run it](#131-run-it) +- [13.2 Repository eligibility (STOP-16)](#132-repository-eligibility-stop-16) +- [13.3 State classes](#133-state-classes) +- [13.4 The State summary and STOP-13](#134-the-state-summary-and-stop-13) +- [13.5 Routing](#135-routing) +- [13.6 Re-entry: resume or restart](#136-re-entry-resume-or-restart) +- [13.7 Provenance (D-30)](#137-provenance-d-30) +- [13.8 Capabilities and scope (STOP-17, D-31)](#138-capabilities-and-scope-stop-17-d-31) + +## 13.1 Run it + +Run `python3 /scripts/orient.py [--spec-clone ]`. Pass the spec clone the user named, if any. Add `--skill-dir ` when the user names the skill, and `--records ` when they say where its working records are. The script runs this skill's read-only scripts in order and prints one JSON object: + +1. `check_repo_eligibility.py`: on a reject, nothing else runs. +2. `spec_clone_info.py`: settles STOP-1 ([2.6](../SKILL.md#26-pin-the-spec-clone)). +3. `inspect_existing_skill.py`: skills, their origin and records, harness, UTS-tagged tests. +4. `detect_liveobjects.py`: a one-line LiveObjects verdict. +5. `detect_capabilities.py`: the REST and Realtime capability profile, the sides, and the suggested scope (13.8), with eligibility's `capabilityOverride` if the repo is allow-listed. + +The spec clone (the argument, else the one `spec_clone_info.py` found) is passed to the eligibility check and to both detectors, which read their names from it (13.8). A malformed data file in `assets/` (`DATA_FILE_ERROR`, naming the file and the key) stops Orient with `ok: false`: fix the file before anything else. + +The JSON has `state` (class, re-entry, reasons, recommended option, options) and `stateSummaryText`. It gives a structured result that is the same under Claude Code and Codex. The facts behind it are in the other fields; `--full` adds the raw script outputs. + +Settle the stops in this order, in as few messages as possible: + +1. **STOP-16** (13.2), if the repo isn't plainly eligible. +2. **STOP-1** for the model (2.8) and for anything listed in `stop1`, such as the spec clone's path or a stale install of this skill. +3. **STOP-17** (13.8), when `stop17` is set: the capability profile needs confirming. +4. **STOP-13** (13.4). + +If the user's request conflicts with the state (for example "create a skill" when one exists), say so at STOP-13. + +## 13.2 Repository eligibility (STOP-16) + +This procedure applies only to Ably Pub/Sub SDK repositories. `check_repo_eligibility.py [--spec-clone P]` decides from the repo's remotes, all of them, not only `origin`. Its rules are data in [`assets/eligibility.json`](../assets/eligibility.json) (owner, name pattern, deny-list, allow-list, canonical and planned names, the definition gate's threshold): when Ably adds, renames or retires a repository, update that file, not the script. + +- **Name rule.** A `github.com/ably/` remote whose name matches `^ably-(pubsub-[a-z0-9]+|[a-z0-9]+)$` (case-insensitive, without `.git` or a trailing `/`) is eligible. + - Ably is renaming its SDK repositories from `ably-` to `ably-pubsub-` (`ably-js` is now `ably-pubsub-js`, and `ably-java` is now `ably-pubsub-java`). Both names are the same repository, and both are accepted. Old clones keep the old URL, which GitHub redirects. + - `nameForm` reports `legacy`, `pubsub` or `allow-list`. + - The script parses HTTPS, `ssh://` and scp-style (`git@github.com:ably/ably-js.git`) URLs, with the scheme and host in any case, resolves SSH host aliases from the local ssh config, and redacts credentials. +- **Canonical names**, reported only. `canonical` gives a renamed repository's current name: `ably-js`, `ably-java`, `ably-dotnet`, `ably-php`, `ably-python`, `ably-ruby`, `ably-flutter` and `ably-dart` are now `ably-pubsub-`, and `ably-ios` is now `ably-cocoa`. `plannedRename` gives a rename in preparation (`ably-cocoa` → `ably-pubsub-cocoa`, `ably-go` → `ably-pubsub-go`); either name is accepted. The one exception: a legacy name whose current name fails the name rule is rejected (`NOT_PUBSUB_SDK_NAME`). `ably-nativescript` is now `ably-js-nativescript`, a wrapper that re-exports ably-js. +- **Deny-list** (legacy single-token names only; `ably-pubsub-` is never denied, but still has to pass the definition gate): + - `common`, `specification`, `spec`, `cli`, `docs`, `ui`, `boomer`, `chat`, `spaces`, `objects`, `liveobjects`, `livesync`, `sandbox`, `proxy`; + - `examples`, `example`, `website`, `www`, `laravel`, `terraform`, `pubsub`, `labs`, `demo`, `demos`, `dashboard`, `infra`, `control`, `mcp`, `status`; + - `benchmark`, `benchmarks`, `test`, `tests`, `tools`, `scripts`, `ai`, `asset`, `assets`; + - `brand`, `comply`, `jmeter`, `os`, `research`, `rss`, `scan`, `server`, `titanium`, `roku`, `nativescript`. These are tools, docs repositories and stubs from a survey of the `ably` organisation that the name rule alone accepts. `ably-jmeter`, `ably-os` and `ably-server` *use* a client, so a usage-based check would accept them. `ably-titanium` is a README-only stub, and `ably-roku` implements no spec. `server` blocks only `ably-server`, never a server door inside an SDK repository (13.8). `nativescript` is a second guard for the wrapper above. + + These are known non-SDK `ably-` repositories; the data file's `_comment` keys give the reasons. Names with more hyphens, such as `ably-chat-swift`, `ably-chat-kotlin` and `ably-ai-transport-js`, already fail the name rule. +- **Allow-list:** `ably-ruby-rest` (the REST-only Ruby SDK) and its renamed form `ably-pubsub-ruby-rest`. These are explicit exceptions to the name rule. Each carries a `capabilityOverride` of REST `full` and realtime `absent`. The repository is a gem wrapping an `ably-ruby` git submodule whose entry point loads only `ably/rest`, so its own tree defines no client, and neither the gate nor the capability detector can see one. The override: + - is reported as `capabilityOverride`; + - satisfies the definition gate; + - is passed by Orient to `detect_capabilities.py --capability-override`, whose profile then shows `source: override (allow-list)`, and drops the detected reasons for each level it sets; + - is still confirmed by the user at STOP-17. +- **Definition gate**, checked after the name passes. A REST or Realtime client entry point must be *defined* in non-test source, not merely used. The gate counts only the scan's `gateTypes` (`detect_capabilities.py`, 13.8): client types declared in the repo (public or internal, including Go type aliases), plus door factories on a device or server side. It excludes functions (`def`, `func`, `fun`, `fn`, or any function that only returns or constructs another library's client), bindings (`var`, `let`, `const`, `val`, properties), and code under an `example(s)`, `sample(s)` or `demo(s)` path segment. Load tools and servers that import an SDK use a client but define none. `fingerprint` reports `sourceFiles`, `buildFiles`, `testRoot`, `clientDefinitions` (REST and Realtime) and `gate`. With no definition: + - with at least `definitionGate.minSourceFilesToReject` (20) non-test source files, the decision is reject, `NO_CLIENT_DEFINITION`; + - with fewer, it is ask, `NO_SDK_FINGERPRINT`: the wrong path, or a sparse checkout? +- **Eligibility isn't scope.** Only an allow-list override touches the capability profile. The scope comes from the SDK's capabilities ([13.8](#138-capabilities-and-scope-stop-17-d-31)). + +| `decision` (`code`) | What you do | +|---|---| +| `accept` (`ELIGIBLE`) | Continue; nothing to ask. Show `canonical`, `plannedRename` and `capabilityOverride` in the State summary | +| `reject` (`NOT_PUBSUB_SDK_NAME`, `DENIED_NON_SDK`, `NOT_ABLY_REPO`, `NO_CLIENT_DEFINITION`) | **STOP-16, hard stop.** Show `message` exactly, for example "uts-to-lang-skill-creator currently supports only Ably Pub/Sub SDK repositories (ably-pubsub-<lang> or ably-<lang>). No workflow exists yet for ably/ably-chat-swift; the skill creator needs to be upgraded to support it." Then end the run. There is no override | +| `ask` (`FORK_CONFIRM`) | **STOP-16.** Only a fork's remote matches (`/ably-js`). Ask the user to confirm it is a fork of `ably/`; record the answer in the decision log once records exist | +| `ask` (`NO_REMOTE`, `NON_GITHUB_REMOTE`, `NOT_GIT_REPO`) | **STOP-16.** Ask which `github.com/ably` repository this is, and check the answer by the same rule. Adding a remote is the user's action | +| `ask` (`NO_SDK_FINGERPRINT`) | **STOP-16.** The name passes, but the few source files define no client: is this the right path, or a sparse checkout? | + +## 13.3 State classes + +The first match wins: + +| Class | Rule (from `orient.py`) | Recommended | +|---|---|---| +| **INELIGIBLE** | `check_repo_eligibility.py` rejected the repo | Nothing: STOP-16 ends the run | +| **S4** ambiguous | More than one distinct `uts-to-*` skill; or a broken install (a dangling symlink, a `uts-to-*` directory without `SKILL.md`); or a skill whose language has no files in the repo; or finished records with no skill | No pre-selection: ask which skill is the target, and whether to repair the install first. With `--skill-dir`, Orient classifies only the named skill and lists the others under Problems | +| **S2** | One skill, with records from this procedure (`origin: creator-records`) | Upgrade/Fix, diff-driven; "Stop" if nothing changed since the recorded run | +| **S3** | One skill of unknown origin (hand-written, an older tool, or records not kept: `origin: unknown` or `creator-metadata`) | Upgrade/Fix, full gap audit | +| **S1** | No skill, but UTS-tagged tests or harness code (for example tagged tests in the SDK's native suite) | Create, keeping and matching the existing tests and harness | +| **S0** | Nothing | Create | + +**Re-entry** is an overlay on any class: records from a run that didn't finish (their `Run status` is "in progress", or, for older records, there is a design record but no final report). It is offered first (13.6). + +Some situations don't make a repo ambiguous: + +- A skill installed only for one tool (only `.claude/skills`) isn't S4. It is a gap-audit row ("Installed for both tools"). +- `uts-to-*` skills installed at user level are only reported. +- Other skills in the repo are listed by name only, so the user can say "that one is our translator" (then rerun with `--skill-dir`). + +`origin` comes from evidence this procedure writes: + +- `creator-records`: a design record (in `/generation/`, or the `--records` directory) whose header is "Design record: uts-to-…" and that names "uts-to-lang-skill-creator version". +- `creator-metadata`: the skill's `metadata.generated-by` stamp (13.7). + +A version number alone proves nothing. + +## 13.4 The State summary and STOP-13 + +Show `stateSummaryText`, completed with the session's model. Its layout: + +``` +WARNING SPEC NAMES FALLBACK: the spec clone's IDL wasn't parsed; the LiveObjects and capability verdicts below are unreliable; settle it at STOP-1 first (only on fallback) +Repo @ (clean|dirty); languages: +Eligibility accept: / via ( name)[; canonical ][; planned rename ][; capability override rest , realtime ] + | ask: [ /] (); STOP-16 +Spec clone @ (clean|dirty); found by ; creator (matches clone: ) +Model (Opus-class: yes|no) +Spec names from the spec clone's IDL @ | from the spec clone's IDL (revision unknown: not a git checkout) | FALLBACK, don't rely on the LiveObjects or capability verdicts: +Capabilities rest (); realtime (; connection …, channels …, presence …); [; sides: (rest yes|no, realtime yes|no), …][; source: override (allow-list)] +Method gaps advisory, realtime partial (): ., …; likely deviations or not-implemented, decided per test, never out of scope (only when realtime is partial) +Native SDK wraps native SDKs over a platform bridge: hooks probably unreachable from the SDK language: unit tier only if hooks exist (P-13); integration and proxy still offered (only if wrapsNativeSdk) +Side : rest ; realtime (one line per side; only for an SDK with doors) + STOP-17 must ask which side the UTS tests construct clients through: (a) core/internal constructor (the sanctioned internal-access route), (b) server door, (c) device door, (d) both doors as a harness parameter; the scope follows the side (D-31) +Scope : (objects per STOP-14)[; undecided: realtime, objects (ask at STOP-17)][; unsupported: ()][; capability-inapplicable test(s) or path(s)][; plus REST-only test(s) under realtime][; partial skill, extendable by Upgrade/Fix]; confirm +Capability changed since the recorded run: new>, …[; added: ][; lower than recorded: ] (only if it changed) +Skill [installs: …] name , version , origin (one per skill) + inspector hints: fail, warn,

pass (regex, not a review); mapping: ; objects notes: +Harness ; smoke/self-test files: [; offered tiers lacking them: | unknown (some files don't name a tier)] +Records

: run ; recorded spec , repo (only if the skill has records) +Skills none[; other repo skills: …] (instead of the Skill lines, when there is no skill) +Harness (with "Skills none") +Problems (only if any) +Changes since recorded spec : guide , UTS docs , helper specs , corpus files; checklist changed: [; repo files] +UTS tests tags in files (); ; header SHAs …; deviations.md: … +LiveObjects (): (decided at STOP-14) +User-level +State [ + re-entry]: +Recommended