diff --git a/uts/README.md b/uts/README.md index baeefcfac..c4a62986d 100644 --- a/uts/README.md +++ b/uts/README.md @@ -162,16 +162,22 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo ### Installing the skill creator -Install it once, at user level, by symlinking (if an earlier copy is there, remove it first: `ln -sfn` would create the link inside it) the skill directory from your local clone of this repo into both tools' skill directories. Both Claude Code and Codex follow directory symlinks, and a `git pull` of the clone updates both: +Keep one shared clone of this repo, `~/.ably/specification` (`%USERPROFILE%\.ably\specification` on Windows), or wherever `ABLY_SPEC_CLONE` points, and install the skill once, at user level, by symlinking its directory from that clone into both tools' skill directories. Both Claude Code and Codex follow directory symlinks. The clone's checkout must contain `uts/skills/uts-to-lang-skill-creator`; if its `main` doesn't yet, see the last bullet below. ```sh -SPEC=~/src/specification # your local clone of ably/specification +SPEC="${ABLY_SPEC_CLONE:-$HOME/.ably/specification}" # the shared clone +[ -d "$SPEC/.git" ] || git clone -c core.autocrlf=false --no-recurse-submodules https://github.com/ably/specification.git "$SPEC" mkdir -p ~/.claude/skills ~/.agents/skills ln -sfn "$SPEC/uts/skills/uts-to-lang-skill-creator" ~/.claude/skills/uts-to-lang-skill-creator # Claude Code ln -sfn "$SPEC/uts/skills/uts-to-lang-skill-creator" ~/.agents/skills/uts-to-lang-skill-creator # Codex ``` -Then, in the SDK repo, run `/uts-to-lang-skill-creator ` (Claude Code) or `$uts-to-lang-skill-creator ` (Codex), or ask for a `uts-to-` skill in your own words. Run it on the most capable model tier available (Opus-class). If symlinks aren't available (for example on Windows without Developer Mode), copy the directory instead and re-copy after each pull; the skill detects a stale copy and asks you to refresh it. Without installing, any agent can be told: "Read `/uts/skills/uts-to-lang-skill-creator/SKILL.md` and follow it." +- **Keeping it current.** Each run offers to fetch and fast-forward the shared clone's `main`, which also updates the skill; it does so only when `main` is checked out, clean and strictly behind `origin/main`, and otherwise reports the clone's state and asks. It never pulls, resets, stashes or switches branches. Set `ABLY_SPEC_SYNC=auto` to stop being asked, or `off` to stay offline (with no clone at all, the skill still offers to clone one, and asks first). The skill never sets it for you. +- **An earlier install.** If a copy of the skill directory is already at either install path, remove it first: `ln -sfn` would create the link inside it. +- **Windows.** Without symlinks (Developer Mode), copy the directory instead, and copy it again after each update; the skill detects a stale copy and asks you to refresh it. +- **Testing an unmerged spec or skill change.** Add a worktree of the shared clone (`git -C "$SPEC" worktree add `) and install the skill from it, or pass `--spec-clone `; keep the shared clone itself on `main`. The sync runs only when the clone in use is the shared clone, however it was found: a worktree, or any other clone you name, is never synced, and the skill warns you to keep it up to date yourself. Unless you name another clone, the skill reads the one it is installed from, so the guide it reads matches it. If the shared clone's `main` doesn't contain `uts/skills/uts-to-lang-skill-creator` yet, install from a clone or worktree of a branch that does; given only a clone that predates the skill, it stops with `SPEC_CLONE_INCOMPLETE` and says what to do. + +Then, in the SDK repo, run `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex), or ask for a `uts-to-` skill in your own words; no path is needed (`--spec-clone `, or a path in your words, still names another clone, which isn't synced). Run it on the most capable model tier available (Opus-class). Without installing, any agent can be told: "Read `/uts/skills/uts-to-lang-skill-creator/SKILL.md` and follow it." The `uts-to-` skills it generates use the same shared clone by default, with the same rule: a module path into any other clone is used as it is, with a warning. ## Go Test Proxy diff --git a/uts/docs/writing-uts-spec-translator-skills.md b/uts/docs/writing-uts-spec-translator-skills.md index b9b9ae31b..ea88e9c94 100644 --- a/uts/docs/writing-uts-spec-translator-skills.md +++ b/uts/docs/writing-uts-spec-translator-skills.md @@ -313,7 +313,8 @@ The minimum set per tier follows. A tier the skill doesn't support needs none of ├── scripts/ │ ├── resolve_uts.py # deterministic selection │ ├── audit_translation.py # deterministic faithfulness audit -│ └── scan_constructs.py # corpus construct scanner (SHOULD, section 6) +│ ├── scan_constructs.py # corpus construct scanner (SHOULD, section 6) +│ └── ensure_spec_clone.py # spec-clone sync, copied from the skill creator (SHOULD, 9.4) └── references/ └── -mapping.md # module notes (e.g. objects-mapping.md) ``` @@ -325,6 +326,7 @@ The minimum set per tier follows. A tier the skill doesn't support needs none of | `scripts/resolve_uts.py` | Validates the module dir, reads the mapping, lists specs and derives file/class/package names; prints one JSON object | MUST | | `scripts/audit_translation.py` | Compares one spec file with its generated test file; prints one JSON object | MUST | | `scripts/scan_constructs.py` | The corpus construct scanner ([section 6](#6-pseudocode-construct-catalogue)) | SHOULD | +| `scripts/ensure_spec_clone.py` | A copy of the skill creator's spec-clone sync ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)) | SHOULD | | `references/-mapping.md` | Maps ably-js-shaped pseudocode to your SDK's API for one module; may override parts of the generic flow | MUST wherever the SDK surface diverges from the pseudocode (in practice `objects` for every typed SDK) | Both existing skills use Python for scripts. Any language works, but the scripts MUST run on every developer platform (see [8.1](#81-resolver-resolve_utspy-must) and [section 10](#10-verification-and-ci)). @@ -385,7 +387,7 @@ Write the skill so the same directory loads in Claude Code and in OpenAI Codex. name: uts-to- description: "Translates Ably UTS (Universal Test Suite) pseudocode specs into runnable LANG tests in REPO. Use when asked to translate, port, derive, generate or re-sync tests from a UTS spec or module (uts/rest, uts/realtime, uts/objects), to evaluate UTS-derived tests against the SDK, or to check UTS coverage. Takes a UTS module directory, validates it, resolves the target test directories, asks for a tier (unit, integration, proxy) and specs, then derives one LANG test per spec Test ID. Not for creating the skill or its harness (use uts-to-lang-skill-creator), or for writing UTS specs." license: "" -compatibility: "Claude Code or Codex. Needs git, python3 and a local clone of ably/specification; run on an Opus-class model." +compatibility: "Claude Code or Codex. Needs git, python3 and a local clone of ably/specification (cloned or updated on approval); run on an Opus-class model." allowed-tools: Bash Read Edit Write metadata: version: "1.0.0" @@ -396,9 +398,9 @@ metadata: - **`name`** MUST be `uts-to-`, equal to the skill's directory name: lowercase letters, digits and hyphens, at most 64 characters. (`uts-to-kotlin` omits `name` and relies on the directory name; both tools accept that, but the Agent Skills format requires the field.) - **`description`** MUST be at most 1,024 characters and contain no `<` or `>`. Write `LANG` and `REPO`, not ``: skill packaging validators (for example skill-creator's `quick_validate.py`) reject angle brackets. It SHOULD be "pushy" and written in the third person: front-load the trigger phrases a user might type ("translate this UTS spec", "port uts/objects to LANG", "re-sync UTS tests") and end with a "Not for …" boundary. Put the usage line in the body, not in the description. Both existing skills' descriptions are procedural, which suits explicit invocation but triggers poorly otherwise. - **`metadata`** values MUST be strings (quote `"1.0.0"` and `"false"`). Codex shows `metadata.short-description` in its skill list. -- **`allowed-tools`** is space-separated. Claude Code pre-approves the listed tools while the skill runs; Codex ignores the field. Both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-local-clone-vs-fetching-main)). +- **`allowed-tools`** is space-separated. Claude Code pre-approves the listed tools while the skill runs; Codex ignores the field. Both existing skills declare `Bash, Read, Edit, Write, WebFetch` and search with `grep` through `Bash`. Add `Grep`/`Glob` only if your workflow uses those tools, and `WebFetch` only if the skill fetches anything remotely (see [9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). - **Claude Code-only fields** (`argument-hint`, `model`, `disable-model-invocation`, `when_to_use`, `context`, …) MAY be added only if the team accepts that they aren't portable: Codex ignores them, and they make claude.ai upload and standard packaging fail. Put Codex-only settings (display name, implicit-invocation policy) in an optional `agents/openai.yaml` in the skill directory. -- **Argument:** a single UTS module directory, directly under `uts/`. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. Don't depend on argument substitution (`$ARGUMENTS` works only in Claude Code): say in words where the argument comes from ("the module directory given after `/uts-to-LANG` in Claude Code or `$uts-to-LANG` in Codex, or named in the user's message"). If there is none, the skill MUST print the usage line and stop. +- **Argument:** a single UTS module directory, directly under `uts/`, or (MAY) a bare module name (`rest`, `realtime`, `objects`) resolved against the shared spec clone ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). Only the shared clone is synced; a module path into another clone is used as it is, with a warning. Both skills started with single-spec-file input and moved to modules; spec selection happens interactively. Don't depend on argument substitution (`$ARGUMENTS` works only in Claude Code): say in words where the argument comes from ("the module directory given after `/uts-to-LANG` in Claude Code or `$uts-to-LANG` in Codex, or named in the user's message"). If there is none, the skill MUST print the usage line and stop. - **Paths to the skill's own files** MUST be relative to the skill directory (the directory containing `SKILL.md`), for example "run `scripts/resolve_uts.py` from this skill's directory". Never write `.claude/skills/uts-to-/scripts/…`: Codex loads the skill from `.agents/skills/`, and `${CLAUDE_SKILL_DIR}` is Claude Code-only. The scripts follow the same rule: they locate the mapping file and the notes relative to their own file (for example `Path(__file__).parent.parent`), never relative to the working directory. - **Model (SHOULD):** no portable frontmatter field pins a model. State in the opening lines of `SKILL.md` (and in `compatibility`) that the skill should run on an Opus-class model ([Model tier](#model-tier)). Claude Code's `model` field is a non-portable option (see above). - **Placeholders, not personal paths,** in examples and usage text: `/uts/objects` in the body (angle brackets are fine in the body, but not in `description`). @@ -464,7 +466,7 @@ The workflow has two phases. Implement both. | Step | What happens | Interaction | |---|---|---| -| **0. Required reading** | Read `writing-derived-tests.md` and the "Pseudocode Conventions" section of `uts/README.md` once per run (plus `proxy.md` for the proxy tier; in evaluate mode, the features specs as needed), from the same spec clone as the module ([9.4](#94-local-clone-vs-fetching-main)). Read `writing-test-specs.md` whenever the construct table defers a meaning to it. Don't rely on memory or inlined summaries | — | +| **0. Required reading** | Sync the shared spec clone, or warn about another clone the argument names ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)), then read `writing-derived-tests.md` and the "Pseudocode Conventions" section of `uts/README.md` once per run (plus `proxy.md` for the proxy tier; in evaluate mode, the features specs as needed), from that same clone. Read `writing-test-specs.md` whenever the construct table defers a meaning to it. Don't rely on memory or inlined summaries | — | | **A. Resolve the module** | Run `resolve_uts.py ""`. If `ok` is false, relay `message` and stop. Treat the output as the single source of truth: never recompute a path or name by hand | Stop on error | | **B. Confirm or create the mapping** | If mapped, show each tier's target dir verbatim and ask the user to confirm. If unmapped, ask for the target base name (default: the source module name; suggest a rename only when the SDK uses different terminology) and run `resolve_uts.py --create `. If the user says an entry is wrong, re-run `--create ` to overwrite it (keeping its `notes`), unless it is hand-maintained (3.2), in which case edit it by hand | Ask | | **C. Choose the tier** | Offer only tiers that are present. Where the mapping records a tier as not ready (per-module data, 3.2), refuse it and name what blocks it. The tier fixes the target dir, the spec list and the translation flow (mocked / sandbox / proxy). Don't re-detect it per spec | Ask | @@ -802,9 +804,9 @@ Path validation, mapping, spec discovery, naming and faithfulness checking are m | Requirement | Detail | |---|---| -| Validate the module | Expand a leading `~`. The parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/` | +| Validate the module | Expand a leading `~`. The parent directory is named `uts` (compare path parts, so it works on Windows); the directory exists; it has `unit/` or `integration/`. A bare module name, if accepted (3.3), is resolved against the shared spec clone (9.4) first | | Errors | One JSON object with `ok: false`, a code and a `message`. Example codes (ably-cocoa): `NOT_A_UTS_MODULE_PATH`, `DIR_NOT_FOUND`, `NO_TIER_DIRS`, `MAPPING_NOT_FOUND`, `BAD_MAPPING`, `BAD_TARGET_NAME` | -| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, harness, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty, dirtyFiles}`, MAY) gives the spec clone's state (`dirtyFiles`: the modified or untracked files under `uts/` and `specifications/`, 9.1) for the file headers and the report (9.1, 9.4). `harness` (`{root, readme, tiers: {: {sources, tests}}}`) echoes the mapping's harness entry with paths validated (repo-relative), for the Harness reference and the preflight (3.2, 4 step F). Downstream steps read only these fields | +| Output contract | On success: `{ok: true, sourceModule, mapped, testRoot?, specRepo?, translationNotes, harness, tiers: {unit, integration, proxy: {present, sourceDir, targetDir, , , specs: [{file, className, testFile?}]}}}`. `testFile` (the target file name) is required wherever it isn't `.`, e.g. pytest `test_.py` with class `Test`. `present` says whether the source tier directory exists; `targetDir` is `null` and `mapped` false when the module has no mapping entry. `testRoot` appears when the mapping declares a root. `specRepo` (`{sha, dirty, dirtyFiles, branch, kind, onOriginMain}`, MAY) gives the spec clone's state (`dirtyFiles`: the modified or untracked files under `uts/` and `specifications/`, 9.1; `kind`: the shared clone or the one the argument names, which isn't synced unless it is the shared clone; `onOriginMain`: whether the SHA is on `origin/main`, 9.4) for the file headers and the report (9.1, 9.4). `harness` (`{root, readme, tiers: {: {sources, tests}}}`) echoes the mapping's harness entry with paths validated (repo-relative), for the Harness reference and the preflight (3.2, 4 step F). Downstream steps read only these fields | | Tier detection by path | `unit/**`; `integration/**` excluding `integration/proxy/**`; `integration/proxy/**`. Every tier can have sub-directories (e.g. `realtime/integration/channels/`). Use the path only | | Exclusions | `helpers/` (implement as harness, never translate), `README.md`, `PLAN.md`, `*_SUMMARY.md`. Match them relative to the tier base, so an ancestor directory in the checkout path can't trip them. Also skip any non-spec notes left inside a tier directory | | Deterministic naming | Strip a trailing `_test`, convert to the language's convention, add the suffix or prefix the runner needs (`objects_lifecycle_test.md` → Swift `ObjectsLifecycleTests`, Kotlin `ObjectsLifecycleTest`, pytest `test_objects_lifecycle.py` with class `TestObjectsLifecycle`) | @@ -871,7 +873,7 @@ Stamp each generated file's header with the spec-repo commit SHA it was derived // at ably/specification@ ``` -A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. +A per-module manifest (spec file → SHA → test file) MAY be kept as well. Link source specs at the pinned SHA, not `blob/main`. If the SHA isn't on `origin/main` (a branch run), list it in the final report: it may stop resolving once that branch is deleted. **A SHA identifies only committed content.** If a translated spec file differs from the recorded SHA (`git status --porcelain -- ` is non-empty: staged, unstaged or untracked), stop and ask. If the user proceeds, add `(locally modified; blob >)` to that file's header, list the file in the final report, and have re-sync compare the file with that blob rather than with the SHA. Local changes elsewhere in the clone (for example untracked notes outside the tier directories) only need recording, as 9.4 says. @@ -882,7 +884,7 @@ A per-module manifest (spec file → SHA → test file) MAY be kept as well. Lin Add a mode (for example `/uts-to- --resync` in Claude Code, `$uts-to- --resync` in Codex) that: 1. Runs the audit over every mapped spec in the module, not only the ones selected. -2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as new; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. Then re-scan the corpus for constructs the skill can't map (section 6). +2. Lists changed spec files against each file's recorded state, including uncommitted changes: `git diff --name-status -M -- uts/` (the working tree, staged and unstaged, against the SHA; group files by the SHA in their headers), or `git hash-object` for a file stamped with a blob (9.1). Untracked spec files have no committed state: compare one stamped with a blob (9.1) with that blob, and treat one with no test file as new; untracked files outside the tier directories are local notes and are ignored. Decide new and removed by comparing the resolver's spec list with the existing test files. If a recorded SHA isn't in the clone, fetch it (with approval, and only in the shared clone; the user fetches it into any other clone); otherwise review each spec stamped with it in full, as changed (base unknown). Then re-scan the corpus for constructs the skill can't map (section 6). 3. Classifies each spec as new, changed (any change to the spec file since its recorded state; report the kind: ID set, pseudocode, fixtures, or other content such as `## Protocol Variants`, `## Test Type` or prose), unchanged or removed. Review each changed file's diff to decide which tests to regenerate, and say why when a changed file is not regenerated. A spec whose helper specs or the UTS docs it relies on changed is reviewed too ([procedure section 10](../skills/uts-to-lang-skill-creator/references/upgrade-diff-driven.md#10-upgradefix-diff-driven)). 4. Regenerates the affected tests, preserving only DEVIATION gates, adapted assertions, UTS-spec-error fail-fast placeholders (they stay until the spec is fixed; `writing-derived-tests.md` "Resolution") and `deviations.md` entries, and updating those entries. 5. Updates the SHA in each regenerated file's header, and its blob note (9.1): re-stamped if the spec is still locally modified, removed if not (and in the manifest, if kept). @@ -899,16 +901,21 @@ Examples of the pattern: - **Rename not followed.** Both SDKs carried `realtime/unit/RTN16g2/recovery-key-null-inactive-0` after the spec renamed it to `realtime/unit/RTN16g3/recovery-key-null-inactive-0`: ably-cocoa `Test/UTS/unit/realtime/ConnectionRecoveryTests.swift` and ably-java `lib/src/test/kotlin/io/ably/lib/uts/unit/realtime/ConnectionRecoveryTest.kt`. The audit reports one missing and one orphan, but nothing prompted a re-run. - **Additions not followed.** In ably-java `RealtimeObjectTest.kt`, two `RTO27` tests remained where the spec has a single `RTO27/channel-state-data-lifecycle-0` (the only `RTO27` Test ID the spec has ever had), and `RTO20d4/mixed-null-serials-applies-non-null-0` had no test. -### 9.4 Local clone vs fetching `main` +### 9.4 One pinned local clone, brought up to date first -Pick one source for docs and specs, and pin it. This guide recommends **reading everything from the local spec clone** the module directory came from: +Pick one source for docs and specs, and pin it: the local spec clone. By default it is the shared clone that the skill creator also uses (`ABLY_SPEC_CLONE`, else `~/.ably/specification`); a module directory given as the argument names its own clone, which is used instead, so an explicit path into any clone keeps working. The sync below runs only when the clone in use is the shared clone, however it was found: a module path into any other clone is never synced or fetched, whatever `ABLY_SPEC_SYNC` says, and the skill shows a static warning instead ("Using the spec clone you named (``). It isn't synced; make sure its UTS specs are up to date."); keeping that clone current is the user's responsibility. -- the UTS docs (`/../docs/writing-derived-tests.md`, `proxy.md`); -- in evaluate mode, the features specs the decision tree needs (`/../../specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. +**Bring the shared clone up to date first (SHOULD).** Before reading anything, with the user's approval or their standing `ABLY_SPEC_SYNC=auto` (`off` means no network; the user sets the variable, never the skill), fetch `origin main` and fast-forward `main` with `git merge --ff-only --no-overwrite-ignore`, only if `main` is checked out, has no tracked changes, no merge, rebase or other git operation is in progress, and it is strictly behind `origin/main`. Never pull, reset, stash, rebase or switch branches: on another branch, a dirty tree, or an ahead or diverged `main`, report it and ask (use it as it is, fetch only, or stop). Offline, declined or failed: use the clone as it is, and say how far behind it was at its last fetch. A skill SHOULD do this with a copy of the skill creator's `scripts/ensure_spec_clone.py` ([procedure 13.9](../skills/uts-to-lang-skill-creator/references/orient.md#139-sync-the-spec-clone)). The fast-forward happens before the run, so the run is still pinned: check HEAD again before stamping headers and before the final report, and if it moved, stop and ask. -Record the clone's HEAD SHA, whether it is dirty and which files are; a translated spec that is locally modified is handled as in 9.1. You MAY warn when the clone is behind `origin/main`. +Then read everything from that clone, at the pinned SHA: + +- the UTS docs (`/uts/docs/writing-derived-tests.md`, `proxy.md`); +- in evaluate mode, the features specs the decision tree needs (`/specifications/features.md`, `objects-features.md`, `protocol.md`), not the GitHub URLs in `writing-derived-tests.md`. + +Record the clone's HEAD SHA, its branch, whether it is dirty and which files are, and the sync's outcome (or "not synced"); a translated spec that is locally modified is handled as in 9.1. - *Divergence from `writing-derived-tests.md`* (Phase 2, "2a. Is the UTS spec wrong?", which links the features specs on GitHub `main`): read them from the same clone as the UTS specs instead, because the UTS spec and the features spec it is judged against must come from one revision, and the clone's SHA is recorded in every header and report (9.1). Fetching `main` gives a revision that nothing records. +- Fetching docs or specs file by file (for example `gh api …/contents/…` or raw URLs) mixes revisions and records none: don't. ### 9.5 Upgrading an existing skill (SHOULD) @@ -1013,7 +1020,7 @@ The skill and its harness are one deliverable: the skill isn't done until every **Workflow** -- [ ] Required reading, including features specs, from the same clone as the specs +- [ ] Required reading, including features specs, from the same clone as the specs, the shared clone synced first per [9.4](#94-one-pinned-local-clone-brought-up-to-date-first) (SHOULD) - [ ] Steps 0 and A–F: the four user questions (mapping, tier, specs, translate vs evaluate) and the harness preflight, which stops on red - [ ] Steps 1–7 with batching; a reference test per tier; review in both modes - [ ] Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) @@ -1085,7 +1092,7 @@ Rules: 2. **Learn the pattern; never copy.** Don't copy, port or translate code, scripts, prose, tables or examples verbatim. Take how a problem was structured or solved, and write your own version in your language's idioms, on your SDK's hooks and harness. 3. **Check against [Patterns to avoid](#patterns-to-avoid)** before adopting anything. Both skills and both harnesses have documented defects, often in exactly the files you would consult. 4. **Record each consultation**: the repo, revision and path, the question, what you learned, and how you checked it. Report each one as a gap in this guide: a candidate guide improvement. -5. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-local-clone-vs-fetching-main)), and neither does the pilot run that validates it. +5. **Creation only.** This applies to creating or maintaining a skill and its harness. A generated skill, at run time, never consults another SDK's skill (it reads only the local spec clone, [9.4](#94-one-pinned-local-clone-brought-up-to-date-first)), and neither does the pilot run that validates it. Reading GitHub is network access: the [procedure](../skills/uts-to-lang-skill-creator/SKILL.md#reference-implementations-last-resort) gates it, or offers local clones instead. @@ -1095,12 +1102,16 @@ Observed in the existing skills and their harnesses when this guide was written, **In both skills** -- **Fetching the manual from `main`, and telling the model to "fetch" the features spec with no pinned source,** while reading specs from a local clone ([9.4](#94-local-clone-vs-fetching-main)). +- **Fetching the manual from `main`, and telling the model to "fetch" the features spec with no pinned source,** while reading specs from a local clone ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). - **Two sources of tier truth**: path-based in the resolver, content-based in the `SKILL.md` integration section. - **A `CONTAINS_IN_ORDER` mapping that isn't a subsequence check** ([6.7](#67-notes-on-the-catalogue)). - **No re-sync mode, no final report, no lint step, no collision detection**, and a silent `null` when a declared notes file is missing. - **Claude Code-only packaging** ([3.3](#33-frontmatter-and-arguments)): script paths hard-coded as `python3 .claude/skills//scripts/…` and reliance on `$ARGUMENTS`, both of which break under Codex; angle-bracket placeholders (`<…>`) in `description`, which skill packaging validators reject. +**In other skills** + +- **Fetching docs and specs file by file from the default branch** (`gh api …/contents/…`, raw URLs), unpinned: a run can mix revisions, the headers can record none, and it needs `gh` auth and a POSIX `base64` ([9.4](#94-one-pinned-local-clone-brought-up-to-date-first)). + **In the harnesses** - **Smoke tests env-gated and never run in CI, then retired** (ably-cocoa) once spec-derived tests covered their scenarios ([2.7](#27-harness-smoke-tests-and-self-tests-must)). diff --git a/uts/skills/uts-to-lang-skill-creator/SKILL.md b/uts/skills/uts-to-lang-skill-creator/SKILL.md index 146dda483..3257df547 100644 --- a/uts/skills/uts-to-lang-skill-creator/SKILL.md +++ b/uts/skills/uts-to-lang-skill-creator/SKILL.md @@ -1,8 +1,8 @@ --- name: uts-to-lang-skill-creator -description: "Builds or upgrades a uts-to-LANG translator skill, with the UTS test harness it runs on, for one Ably Pub/Sub SDK repository on this skill's whitelist (C#, Python, Go, JavaScript, Ruby, ...). Use it whenever asked to create a uts-to-* skill; to audit, fix, upgrade or refresh one after the guide, UTS docs, corpus or SDK changed; to add LiveObjects (uts/objects) support; or to set up UTS test infrastructure (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, harness smoke tests and self-tests). First checks the repo is on the whitelist, then detects what it has (uts-to-* skills and their origin, harness, UTS-tagged tests, REST and Realtime capabilities and scope, LiveObjects API) and recommends Create or Upgrade/Fix, then runs a procedure with user stop points. Needs a local ably/specification clone. Not for Chat, Spaces, AI Transport, CLI or other non-Pub/Sub repos, for translating specs once a uts-to-* skill exists (use that skill), or for writing UTS specs." +description: "Builds or upgrades a uts-to-LANG translator skill, with the UTS test harness it runs on, for one Ably Pub/Sub SDK repository on this skill's whitelist (C#, Python, Go, JavaScript, Ruby, ...). Use it whenever asked to create a uts-to-* skill; to audit, fix, upgrade or refresh one after the guide, UTS docs, corpus or SDK changed; to add LiveObjects (uts/objects) support; or to set up UTS test infrastructure (SDK test hooks, MockHttpClient, MockWebSocket, fake clock, sandbox and proxy helpers, harness smoke tests and self-tests). First checks the repo is on the whitelist, then detects what it has (uts-to-* skills and their origin, harness, UTS-tagged tests, REST and Realtime capabilities and scope, LiveObjects API) and recommends Create or Upgrade/Fix, then runs a procedure with user stop points. Needs a local (synced) ably/specification clone. Not for Chat, Spaces, AI Transport, CLI or other non-Pub/Sub repos, for translating specs once a uts-to-* skill exists (use that skill), or for writing UTS specs." license: Apache-2.0 -compatibility: "Claude Code or Codex, on an Opus-class model. Needs shell and file access, git, Python 3.8+, a local clone of ably/specification, and the target SDK's build toolchain." +compatibility: "Claude Code or Codex, on an Opus-class model. Needs shell and file access, git, Python 3.8+, a local clone of ably/specification (cloned or updated on approval), and the target SDK's build toolchain." allowed-tools: Read Grep Glob metadata: team: "engineering" @@ -30,12 +30,13 @@ The harness and the skill are one deliverable (guide [section 2](../../docs/writ ## Step 0: Orient -Do this first, in every run, before any phase. It is read-only: no build, no test, no network, no file written. Run `python3 /scripts/orient.py [--spec-clone ]` ([13.1](references/orient.md#131-run-it)), then settle its stops in this order; [references/orient.md](references/orient.md) (section 13) has the rules: +Do this first, in every run, before any phase. `orient.py` is read-only: no build, no test, no network, no file written. Run `python3 /scripts/orient.py [--spec-clone ]` ([13.1](references/orient.md#131-run-it)), then settle its stops in this order; [references/orient.md](references/orient.md) (section 13) has the rules: 1. **STOP-16**, if the repo isn't plainly eligible ([13.2](references/orient.md#132-repository-eligibility-stop-16)). On a reject, show the script's message exactly and end the run; there is no override. -2. **STOP-1**, if needed: the model tier (2.8) and anything in `stop1`, such as the spec clone (2.6). -3. **STOP-17**, when `stop17` is set: the capability profile and scope ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)). Record D-31. -4. **STOP-13**, always: the State summary and the recommended mode ([13.4](references/orient.md#134-the-state-summary-and-stop-13)). Record D-27, then route per [13.5](references/orient.md#135-routing). +2. **The spec-clone sync** ([13.9](references/orient.md#139-sync-the-spec-clone)): Step 0's only network use, for the shared clone only, after STOP-6 (not asked if `ABLY_SPEC_SYNC` is `auto` or `off`); rerun `orient.py` if HEAD moved. +3. **STOP-1**, if needed: the model tier (2.8) and anything in `stop1`, such as the spec clone (2.6). +4. **STOP-17**, when `stop17` is set: the capability profile and scope ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)). Record D-31. +5. **STOP-13**, always: the State summary and the recommended mode ([13.4](references/orient.md#134-the-state-summary-and-stop-13)). Record D-27, then route per [13.5](references/orient.md#135-routing). ## 2. Ground rules @@ -63,7 +64,7 @@ and, only as a last resort, the reference implementations. | Change CI workflows, build scripts that CI runs, or dependency manifests (including adding a test dependency) | **Only with explicit approval** ([STOP-5](#23-stop-points)). Otherwise write the change as a suggestion in the Final report | | Edit project or solution files needed to register new test files or projects (`.sln`, `.csproj`, `.projitems`, an Xcode project) | Only with explicit approval ([STOP-5](#23-stop-points)), recorded in the Design record (D-20). If the user declines, give them the exact entries to add, wait until they confirm, and list them in the Final report | | Stage, commit, push, branch, or open a PR | **No, unless the user asks** | -| Edit anything in the spec clone | No. Report spec errors and doc gaps in the Final report with a draft fix | +| Edit anything in the spec clone | No, except the shared clone's sync (13.9). Report spec errors and doc gaps in the Final report with a draft fix | | Delete or rewrite existing tests or existing UTS-derived tests | No, unless the user asks. Report their state in the Repo profile | | Edit, replace or delete files of an existing `uts-to-*` skill or UTS harness that this run didn't create | Only for the items the user chose at STOP-15 (D-29) or the harness rows approved at STOP-3; the objects files that the STOP-14 decision (D-28) changes; when regenerating, replacements the user approves ([11.5](references/upgrade-existing-skill.md#115-step-u4-run-the-phases-scoped)); the provenance stamp in its frontmatter (D-30). Never delete without asking | | Run the full test suite, or any integration or proxy test (network, sandbox, proxy binary download) | Ask first ([STOP-6](#23-stop-points)) | @@ -75,12 +76,12 @@ At a stop point, present what you found and the options, then wait for the user' | ID | Phase | When | What you present | |---|---|---|---| -| **STOP-1** | 0 (Orient) | A required input is missing, the session's model isn't Opus-class ([2.8](#28-model-tier-and-sub-agents)), or `spec_clone_info.py` fails or reports `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | +| **STOP-1** | 0 (Orient) | A required input is missing, the session's model isn't Opus-class ([2.8](#28-model-tier-and-sub-agents)), or no usable spec clone is found (offer to clone it, 13.9), or `skillMatchesClone: false` ([2.6](#26-pin-the-spec-clone)) | The missing input (or the model found) and why it's needed | | **STOP-2** | 1 (step 1d) | The Repo profile is written | The profile, for confirmation or correction | | **STOP-3** | 2 (end) | The harness design is written | The gap table, the design and three options: (a) build the missing harness now, (b) plan it only, (c) scope the skill to the tiers possible today | | **STOP-4** | 3 onwards | Before any SDK production-code change | The hook or visibility proposal ([4.6](references/phase-2-design-harness.md#46-step-2f-propose-the-sdk-test-hooks)) | | **STOP-5** | 3 onwards | Before any CI, build-script, dependency-manifest or project/solution-file change (including creating a test project or adding one to a solution) | The diff you propose and why | -| **STOP-6** | 1 onwards | Before any network access (network tests, proxy releases, a reference implementation on GitHub or fetching its clone, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads, whether the approval covers the whole run | +| **STOP-6** | 0 onwards | Before any network access (the shared clone's sync, 13.9, unless `ABLY_SPEC_SYNC` is `auto` or `off`; network tests, proxy releases, a reference implementation on GitHub or fetching its clone, [2.1](#21-your-inputs)), or running a full suite | The command, the expected duration, what it touches; for reference reads and the sync, whether the approval covers the whole run | | **STOP-7** | 3 (end) | The harness is built | The build results (smoke tests and self-tests per tier, CI wiring, updated tier feasibility), before any skill file is written | | **STOP-8** | 4 (end) | The Design record is written | The Design record, for confirmation | | **STOP-9** | 5 | Pseudocode constructs, helper symbols or spec shapes you can't map ([7.4](references/phase-5-generate-skill.md#74-step-5d-fill-the-construct-catalogue)); batch everything one scan finds into one stop | Each construct, where it occurs, a proposed rendering | @@ -90,7 +91,7 @@ At a stop point, present what you found and the options, then wait for the user' | **STOP-13** | 0 (Orient), every run | Always ([13.4](references/orient.md#134-the-state-summary-and-stop-13)) | The State summary, the state class (S0–S4, re-entry) and the options for it, with the recommended one: Create; Upgrade/Fix (diff-driven or full gap audit); regenerate from scratch; resume or restart; for S4, pick a skill, create a new skill or repair the installs first; stop | | **STOP-14** | 1 (step 1e), every run | Always, in every mode ([12.2](references/liveobjects-support.md#122-stop-14-recommend-then-ask)) | The `detect_liveobjects.py` evidence (names per category, with files and counts), its recommendation and yours; options: full objects support (translate-only if incomplete), placeholder, none now | | **STOP-15** | 1 (step 1f), Upgrade/Fix | The gap audit, or the list of detected changes (diff-driven), is written ([11.4](references/upgrade-existing-skill.md#114-step-u3-stop-15-choose-per-item), [section 10](references/upgrade-diff-driven.md#10-upgradefix-diff-driven)) | Every non-conforming item or detected change, with the proposed change and the options add, skip or defer (a skipped or deferred MUST row leaves the skill not conforming); or regenerate from scratch instead | -| **STOP-16** | 0 (Orient), first stop after the spec clone resolves | `check_repo_eligibility.py` rejects the repo, or can't decide ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | Reject: the script's exact message (not on the whitelist; or, for `NO_CLIENT_DEFINITION`, no client defined in this checkout); the run ends, with no override. Ask: the remotes found and the question (confirm a fork, name the GitHub repository, check the path) | +| **STOP-16** | 0 (Orient), first stop, before the spec-clone sync | `check_repo_eligibility.py` rejects the repo, or can't decide ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | Reject: the script's exact message (not on the whitelist; or, for `NO_CLIENT_DEFINITION`, no client defined in this checkout); the run ends, with no override. Ask: the remotes found and the question (confirm a fork, name the GitHub repository, check the path) | | **STOP-17** | 0 (Orient), with STOP-13 | Orient sets `stop17`: the capability profile needs confirming ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31) lists the triggers) | The profile (REST, Realtime and its sub-areas, with evidence), the suggested scope (modules; `unsupported` modules; capability-inapplicable tests) and any delta; options: accept, correct with evidence and rerun, stop. With doors, also the side question: the core constructors (sanctioned internal-access route), the server door, the device door, or both as a parameter; the scope follows the side | ### 2.4 Never guess @@ -108,11 +109,7 @@ Every choice you make that isn't forced by the guide or the UTS docs goes into t ### 2.6 Pin the spec clone -Orient runs `scripts/spec_clone_info.py` with the path the user gave, if any. It locates the clone, checks that the guide, the UTS docs and the helper specs exist, and reads the clone's git state; [13.1](references/orient.md#131-run-it) gives the discovery order, the outcomes and the stop or confirmation each needs. - -Record the SHA, whether the clone is clean (and, if not, which files under `uts/` or `specifications/` are modified or untracked), the guide's and this skill's last-change commit (`guideLastChange`) and `skillVersion`. Read every spec-repo file at the absolute paths the script reports (`paths`), or under `specClone`, and from this clone for the whole run. A spec you will translate that is itself modified or untracked is handled as in guide [9.1](../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must) (stop and ask; if the user proceeds, stamp its blob hash). Compare with `origin/main` only if the user agrees to a `git fetch`; never pull or check out on their behalf. If the clone changes during the run, stop and ask whether to restart from the new SHA. - -Untracked local notes in the clone (for example files matching `uts/UTS_*.md`) are not part of the corpus. Don't read them as specs. +Step 0 syncs the shared clone (no other clone is synced), then Orient pins it: [13.9](references/orient.md#139-sync-the-spec-clone) gives the precedence, the update rule and the outcomes. Record what the sync and `spec_clone_info.py` report: the SHA, the branch, the sync result, the dirty files under `uts/` and `specifications/`, `guideLastChange` and `skillVersion`. Read every spec-repo file at the paths it reports (`paths`), from this clone, for the whole run. A spec you will translate that is modified or untracked is handled as in guide [9.1](../../docs/writing-uts-spec-translator-skills.md#91-record-what-you-translated-from-must). Never fetch, pull or check out outside 13.9. If HEAD moves during the run, stop and ask whether to restart from the new SHA. Untracked notes in the clone (such as `uts/UTS_*.md`) aren't specs. ### 2.7 Working style @@ -148,7 +145,7 @@ Before starting a phase, read its reference file in full: this table is a map, n | 1. Understand the repo | A confirmed Repo profile: how the SDK is built, tested and mocked today; the LiveObjects decision; in Upgrade/Fix, the gap audit or the change list | [phase-1](references/phase-1-understand-repo.md) (and sections 10, 11, 12) | `repo-profile.md`; in Upgrade/Fix `skill-gap-audit.md` | STOP-2, STOP-14 (STOP-15 in Upgrade/Fix) | | 2. Design the harness | Reuse, wrap, extend or build each capability; hooks; placement; tier feasibility | [phase-2](references/phase-2-design-harness.md) | `uts-infra-design.md` | STOP-3 | | 3. Build and verify the harness | The approved harness, with green per-tier smoke tests and self-tests, wired into CI | [phase-3](references/phase-3-build-harness.md) | harness code and README; build log | STOP-7 | -| 4. Skill design decisions | D-01 to D-26 decided; D-27 to D-31 confirmed | [phase-4](references/phase-4-design-record.md) | `design-record.md` | STOP-8 | +| 4. Skill design decisions | D-01 to D-26 and D-32 decided; D-27 to D-31 confirmed | [phase-4](references/phase-4-design-record.md) | `design-record.md` | STOP-8 | | 5. Generate the skill files | Mapping, resolver, audit, construct catalogue, module notes, `SKILL.md` | [phase-5](references/phase-5-generate-skill.md) | the `uts-to-` skill | (STOP-9 if constructs are unmapped) | | 6. Validate the skill | Resolver, audit mutations, pilots, other paths, lint, examples | [phase-6](references/phase-6-validate-skill.md) | pilot tests; results | (STOP-10 at a bound) | | 7. Final report and handover | Report and acceptance checklist | [section 9](#9-phase-7-final-report-and-handover) | `final-report.md` with the acceptance checklist | — | @@ -166,12 +163,12 @@ Before starting a phase, read its reference file in full: this table is a map, n The user starts a run in the target SDK repo with one sentence, for example: ``` -/uts-to-lang-skill-creator ~/src/specification realtime unit first (Claude Code; $uts-to-lang-skill-creator in Codex) +/uts-to-lang-skill-creator realtime unit first (Claude Code; $uts-to-lang-skill-creator in Codex) Audit our uts-to-python skill against the guide and upgrade it; add LiveObjects support. Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to create a uts-to- skill for this repo. (skill not installed) ``` -`` is the user's local clone of `ably/specification`. Take it, and any goals ("objects unit tier first", "rest and realtime only"), from the user's words, and record the goals as inputs. Installation is described in [`uts/README.md`](../../README.md#installing-the-skill-creator). +`` is a local clone of `ably/specification`: by default the shared clone ([13.9](references/orient.md#139-sync-the-spec-clone)), so naming one is optional; a clone the user names isn't synced unless it is the shared clone, and otherwise Orient warns them to keep it up to date. Take any path the user names (`--spec-clone `, or a path in their words), and any goals ("objects unit tier first", "rest and realtime only"), from the user's words, and record the goals as inputs. Installation is described in [`uts/README.md`](../../README.md#installing-the-skill-creator). ### 1.3 Required inputs @@ -179,7 +176,7 @@ Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to |---|---|---| | Target SDK repo path | the current working directory | Ask. Don't assume the working directory is the SDK repo: Orient checks that it is an eligible Ably Pub/Sub SDK repository with SDK sources and tests (STOP-16) | | Language (and the skill name it implies) | C# → `uts-to-csharp`; Python → `uts-to-python` | Infer from the repo in Phase 1, then confirm | -| Local spec clone path | `~/src/specification` (the directory that contains `uts/` and `specifications/`) | Located by `scripts/spec_clone_info.py` ([2.6](#26-pin-the-spec-clone)) and confirmed with the user; otherwise stop and ask ([STOP-1](#23-stop-points)). Never fetch the spec or the UTS docs from GitHub instead | +| Local spec clone path | `~/.ably/specification` (the directory that contains `uts/` and `specifications/`) | Optional. Located by `scripts/spec_clone_info.py` (precedence: argument, this skill's clone, `ABLY_SPEC_CLONE`, `~/.ably/specification`), or cloned there on approval ([13.9](references/orient.md#139-sync-the-spec-clone)). Only the shared clone is synced. Never fetch the spec or the UTS docs file by file from GitHub | | User goals: modules and tiers wanted first | "realtime unit, then rest unit; objects later" | Ask. Default to every module (`rest`, `realtime`, `objects`) and every tier the SDK can support, prioritised by the user | | Model tier | The session's model | See [2.8](#28-model-tier-and-sub-agents) (STOP-1 if it isn't Opus-class) | | Local clones of ably-cocoa and ably-java (now `ably-pubsub-java`) (optional) | Clones of either repo, read at their current HEAD (ideally a recent `main`) | Not needed to start. Ask only if you reach the last-resort reference reads ([2.1](#21-your-inputs)); without clones, those reads go to GitHub after STOP-6 | @@ -187,7 +184,7 @@ Read /uts/skills/uts-to-lang-skill-creator/SKILL.md and follow it to ### 1.4 Outputs 1. **The harness** for this language, in the target repo: the shared test library implementing the helper specs, wait and assertion helpers, sandbox and proxy helpers, module helpers, the per-tier smoke tests and helper self-tests (guide [2.7](../../docs/writing-uts-spec-translator-skills.md#27-harness-smoke-tests-and-self-tests-must); permanent and run in CI), a harness README with its Known gaps and, with explicit approval, SDK test hooks. Built in Phase 3, unless the user scopes it down at STOP-3. -2. **The skill** (in Upgrade/Fix, the existing skill, upgraded in place), at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-/` for Codex, unless D-01 says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md` and, if D-19 ships it, `scripts/scan_constructs.py` (guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). +2. **The skill** (in Upgrade/Fix, the existing skill, upgraded in place), at `.claude/skills/uts-to-/`, linked from `.agents/skills/uts-to-/` for Codex, unless D-01 says otherwise: `SKILL.md`, `uts-package-mapping.json`, `scripts/resolve_uts.py`, `scripts/audit_translation.py`, `references/-mapping.md`, `scripts/scan_constructs.py` if D-19 ships it, and `scripts/ensure_spec_clone.py` per D-32 (guide [3.1](../../docs/writing-uts-spec-translator-skills.md#31-layout)). 3. **Pilot-translated tests**, one spec per available tier (in Upgrade/Fix, per tier whose rules changed, 11.6), produced by the new skill in Phase 6. 4. **Working records** (*procedure recommendation*): Markdown files in the working-records directory (default `.claude/skills/uts-to-/generation/`; D-24, confirmed in Phase 4). Copy each template from `assets/templates/` there under the same file name, then fill it in: - [repo-profile](assets/templates/repo-profile.md) and [design-record](assets/templates/design-record.md) (which holds the decision log), at the start of Phase 1; @@ -252,12 +249,13 @@ Each phase's reference (sections 3 to 8) is linked from [Phases at a glance](#ph ## Scripts -All are read-only Python 3, with no network access; they print to stdout and run from anywhere as `python3 /scripts/.py`. `--help` prints a script's docstring (purpose, usage, output and exit codes) and exits 0; a usage error prints the usage to stderr and exits 2. Each row's linked section has the rules. +All but `ensure_spec_clone.py` are read-only Python 3, with no network access; they print to stdout and run from anywhere as `python3 /scripts/.py`. `--help` prints a script's docstring (purpose, usage, output and exit codes) and exits 0; a usage error prints the usage to stderr and exits 2. Each row's linked section has the rules. | Script | Use | |---|---| | `orient.py [--spec-clone P] [--skill-dir P] [--records P] [--include-submodules] [--full]` | Step 0: runs `spec_clone_info.py`, the eligibility check, the inspector and both detectors, classifies the repo, recommends a mode ([13.1](references/orient.md#131-run-it)) | | `spec_clone_info.py [SPEC_CLONE \| --spec-clone P]` | Locate, validate and pin the spec clone; `warnings` is a list, possibly empty ([13.1](references/orient.md#131-run-it)) | +| `ensure_spec_clone.py [--spec-clone P] [--status \| --fetch \| --update \| --bootstrap]` | Step 0's sync of the shared clone (no other clone): the only script that uses the network or changes the spec clone, and only on approval ([13.9](references/orient.md#139-sync-the-spec-clone)); copied into the generated skill (D-32) | | `check_repo_eligibility.py [--spec-clone P]` | Is the repository on the whitelist? ([13.2](references/orient.md#132-repository-eligibility-stop-16)) | | `detect_capabilities.py [--spec-clone P] [--include-submodules] [--capability-override rest=full,realtime=absent]` | Capability profile, door sides, suggested scope ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | | `spec_names.py [--spec-clone P]` | The detectors' names, from the spec clone's IDL, and the capability-inapplicable tests ([13.8](references/orient.md#138-capabilities-and-scope-stop-17-d-31)) | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md index 6e65d8957..2da4d83f9 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/acceptance-checklist.md @@ -24,7 +24,7 @@ | Module notes for every diverging module; `objects` decided from evidence of the LiveObjects public API (full notes if exposed, even if incomplete; a placeholder or an entry with every tier marked not ready otherwise) | 7.5; D-23; 12.1–12.3; D-28 (decided at STOP-14) | 7.5 acceptance; 8.5 paths 3–4 | | | Scripts run on every developer platform (path handling, UTF-8, LF) | 7.1–7.3; D-02 | 8.1 (path parts, `~`), 8.6 (platforms) | | | **Workflow** | | | | -| Required reading incl. features specs, from the same clone | 7.6 (step 0) | 8.3 pilot | | +| Required reading incl. features specs, from the same clone, synced first (SHOULD; D-32) | 7.6 (step 0) | 8.3 pilot; 8.5 path 11 | | | Steps 0 and A–F: the four questions and the harness preflight (stop on red) | 7.6 | 8.3 pilot (green preflight first); 8.5 paths 8–9 | | | Steps 1–7 with batching; a reference test per tier; review in both modes | 7.6; 8.8 | 8.3 pilot; 8.8 | | | Unmapped constructs stop and ask; corpus scanner in `scripts/` (SHOULD) | 7.4; 7.6; D-19 | 8.3 pilot; 8.5 (re-sync) | | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md index 812cbddd4..845472147 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/design-record.md @@ -1,7 +1,7 @@ # Design record: uts-to- ## Inputs at generation time -- Spec clone: (clean | dirty: ); guide/skill: ; uts-to-lang-skill-creator version +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; guide/skill: ; uts-to-lang-skill-creator version - Repo: - Repo profile confirmed: ; harness option at STOP-3: (a | b | c); harness confirmed: - Harness README: @@ -47,6 +47,7 @@ | D-29 | Gap-audit choices: add / skip / defer per GA- or CH- item; preserve list (STOP-15; n/a in create mode) | | | | | | D-30 | Provenance stamp (metadata.generated-by, metadata.records) and Run status | | | | | | D-31 | Capability profile and skill scope (STOP-13 or STOP-17) | | | | | +| D-32 | Spec-clone sync in the generated skill (ensure_spec_clone.py copied; ABLY_SPEC_SYNC) | | | | | ### D-03 mapping | Module | unit | integration | proxy | notes | hand-maintained? | diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md index c2ac1dd1d..cd304ab24 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/final-report.md @@ -1,6 +1,7 @@ # Final report: uts-to- for -- Spec clone (clean | dirty: ); repo ; date +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; repo ; date +- Spec SHA not on origin/main (a branch run): - Model: (Opus-class: yes | no, and why); sub-agents: , or none - Scope: ; out of scope and why - Conforms to the guide: yes | no (MUST rows ✗: …) diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md index 11fce9759..2ecbfb3da 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/repo-profile.md @@ -4,7 +4,7 @@ (paste orient.py's stateSummaryText, completed with the model, the capability profile confirmed at STOP-13 or STOP-17 (D-31), and the option chosen at STOP-13) - Repo: , HEAD (clean | dirty: ) -- Spec clone: , HEAD (clean | dirty: ); guide/skill last changed in ; skill version +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)>; guide/skill last changed in ; skill version - Date: ; confirmed by the user on (STOP-2) - User goals: diff --git a/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md index ea52bab38..5c56c54d0 100644 --- a/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md +++ b/uts/skills/uts-to-lang-skill-creator/assets/templates/skill-gap-audit.md @@ -1,7 +1,8 @@ # Gap audit: uts-to- in - Existing skill: (installs: <.claude/skills/…, .agents/skills/… (symlink | copy)>); frontmatter version -- Repo (clean | dirty: ); spec clone (clean | dirty: ) +- Repo (clean | dirty: ) +- Spec clone: ; on (clean | dirty: ); on origin/main: ; sync: .. | up to date | not updated: , behind as of last fetch | not synced (not the shared clone)> - State class (Orient): S2 | S3 | S4; mode at STOP-13: upgrade: diff-driven | upgrade: full audit | upgrade: regenerate — chosen by on - inspect_existing_skill.py run: ; its checks are regex hints, every row below is confirmed by reading - Capabilities: recorded D-31 vs Orient now ; delta: diff --git a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md index 4ae3e8084..e948e8411 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md +++ b/uts/skills/uts-to-lang-skill-creator/references/failure-modes.md @@ -39,6 +39,13 @@ All from the guide and the UTS docs; follow the link for the detail. | Orient offers to resume a run that finished | The Design record's Run status wasn't set to finished at Phase 7 | Restart, or set the Run status after checking the Final report | [13.6](orient.md#136-re-entry-resume-or-restart) | | A REST-only SDK reported with realtime, or a full SDK reported REST-only | Realtime names only in comments, tests or docs; renamed clients the detector doesn't know; a client only reachable through a factory in an excluded directory or submodule | Read the evidence; correct the profile at STOP-17 with evidence; add a new client name to `aliases` in `assets/capability-names.json` (not the code) | [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | | Orient's Spec names line says FALLBACK, or a renamed spec type isn't found | The spec clone's IDL couldn't be parsed (a missing heading, a moved file, an old clone), so the detectors used the data file's minimal, possibly stale names | Fix or update the spec clone at STOP-1 and rerun; if the IDL's format changed, report the parser gap. Don't rely on the LiveObjects or capability verdicts until it says `spec` | [13.4](orient.md#134-the-state-summary-and-stop-13), [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | +| The spec-clone sync says `FETCH_FAILED` although the machine is online | A sandbox denies writes under the clone's `.git` or network access; or a credential or SSH prompt was suppressed | Use the clone as it is for this run; the user allows the directory and network (Claude Code `/add-dir` or `permissions.additionalDirectories`; Codex a writable root and network), or fixes the remote's auth, then reruns | [13.9](orient.md#139-sync-the-spec-clone) | +| The sync says `NOT_SYNCED_NAMED_CLONE`, or Orient shows SPEC CLONE NOT SYNCED | The clone in use isn't the shared clone (a path the user named, or a skill installed from another clone): only the shared clone is synced | Expected. The user keeps that clone up to date; to have it synced, use the shared clone instead (no path, a path into it, or `ABLY_SPEC_CLONE` pointing at it) | [13.9](orient.md#139-sync-the-spec-clone) | +| The sync says `LOCKED` | Another run is syncing the same clone, or a crashed run left its lock | Use the clone as it is; if no other run is syncing, the user removes the reported lock file | [13.9](orient.md#139-sync-the-spec-clone) | +| The sync says `NOT_ON_MAIN`, and nobody meant to leave the shared clone on a branch | The clone was switched to test a spec change and left there | Use it as it is, or stop while the user switches it back; the sync never switches. To test branches, use a worktree of the shared clone instead | [13.9](orient.md#139-sync-the-spec-clone) | +| The spec clone's HEAD moved mid-run | This run's sync, or another run's fast-forward of the shared clone | Stop and ask whether to restart from the new SHA; never stamp the old SHA on files read after the move | [13.9](orient.md#139-sync-the-spec-clone), [2.6](../SKILL.md#26-pin-the-spec-clone) | +| Orient says the recorded spec SHA isn't in this clone | The records came from another clone, or from a branch whose commits were since deleted | In the shared clone, fetch the SHA after STOP-6 and rerun Orient; if it is still missing, run the full gap audit. In any other clone, never fetch: the user fetches it, or run the full gap audit | [13.9](orient.md#139-sync-the-spec-clone) | +| `SPEC_CLONE_INCOMPLETE` | The clone's checkout predates the guide or this skill (for example a `main` from before they were merged) | Follow the message: for the shared clone, sync it if `origin/main` has them; otherwise name a clone or worktree of a branch that has them | [13.9](orient.md#139-sync-the-spec-clone) | | Sides reported for an SDK without doors, or a door's side wrong | A factory or door type whose name or path the data file doesn't know, or a door type under an unexpected path | Read the Side lines' evidence; correct the side at STOP-17; add the door type or path to `doorFactories.sides` in `assets/capability-names.json` | [13.8](orient.md#138-capabilities-and-scope-stop-17-d-31) | | A harness defect reported as SDK-blocked | The objects smoke test fails, and it is blamed on the SDK without checking | SDK-blocked needs every self-test green and evidence naming an SDK not-implemented symbol; otherwise it is a harness stop | [12.5](liveobjects-support.md#125-an-sdk-blocked-objects-tier) | | `objects` translated with no notes in a later run | No mapping entry, so the skill's step B offered `--create` | Give `objects` an entry even when out of scope: every tier unready, with the reason | [12.2](liveobjects-support.md#122-stop-14-recommend-then-ask) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/glossary.md b/uts/skills/uts-to-lang-skill-creator/references/glossary.md index 1cffff2d3..0c6e9e6c9 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/glossary.md +++ b/uts/skills/uts-to-lang-skill-creator/references/glossary.md @@ -31,6 +31,8 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs | **Delegating spec/test** | One that says to run another module's tests against this client | | **Re-sync** | Re-auditing a module against a newer spec SHA and regenerating what changed | | **Spec SHA** | The commit of the local spec clone that a test, or the skill, was derived from | +| **Shared spec clone** | The one local clone of `ably/specification` that every run and every SDK repo reads by default: `ABLY_SPEC_CLONE`, else `~/.ably/specification`; a path the user names, or the clone this skill is installed from, comes first, and is the shared clone only if it is that same directory ([13.9](orient.md#139-sync-the-spec-clone)) | +| **Spec-clone sync** | Step 0's only network use (including fetching a recorded spec SHA missing from the clone), for the shared clone only (any other clone gets a static warning and is never fetched): with approval (or the user's `ABLY_SPEC_SYNC=auto`), fetch `origin main` and fast-forward a clean, checked-out `main` that is strictly behind; anything else is reported and left alone. `ensure_spec_clone.py` does it ([13.9](orient.md#139-sync-the-spec-clone)) | | **Phase N / generated-skill Phase 1–2** | "Phase N" (0–7) is a phase of this procedure; Phase 0 is Orient, also called Step 0 (the generated skill's own step 0 is named with its context: "guide 4 step 0" or "generated-skill step 0"). "Generated-skill Phase 1" (selection, steps 0 and A–F) and "generated-skill Phase 2" (per spec, steps 1–7) are the generated `uts-to-` skill's workflow (guide [4](../../../docs/writing-uts-spec-translator-skills.md#4-the-workflow-the-skill-must-implement)). `writing-derived-tests.md` Phase 1/2 is always named with that document | | **Working records** | Repo profile, Harness design (with the gap table), Design record, Final report; in Upgrade/Fix, also the gap audit | | **Stop point** | A STOP-n point where you present what you found and the options, wait for the user's answer and record it ([2.3](../SKILL.md#23-stop-points)) | diff --git a/uts/skills/uts-to-lang-skill-creator/references/orient.md b/uts/skills/uts-to-lang-skill-creator/references/orient.md index 4854a085b..fef212f8c 100644 --- a/uts/skills/uts-to-lang-skill-creator/references/orient.md +++ b/uts/skills/uts-to-lang-skill-creator/references/orient.md @@ -10,7 +10,7 @@ Part of the `uts-to-lang-skill-creator` skill: read [SKILL.md](../SKILL.md) firs - which mode to run in (**Create** or **Upgrade/Fix**); - for Upgrade/Fix, which existing skill to work on. -Orient is read-only and quick. It builds nothing, runs no tests, touches no network and writes no file. The working records are created only after STOP-13, because their location depends on the answer (D-24). +Orient is read-only and quick. It builds nothing, runs no tests, touches no network and writes no file. The spec-clone sync after STOP-16 (13.9) is separate. The working records are created only after STOP-13, because their location depends on the answer (D-24). Contents: @@ -22,34 +22,37 @@ Contents: - [13.6 Re-entry: resume or restart](#136-re-entry-resume-or-restart) - [13.7 Provenance (D-30)](#137-provenance-d-30) - [13.8 Capabilities and scope (STOP-17, D-31)](#138-capabilities-and-scope-stop-17-d-31) +- [13.9 Sync the spec clone](#139-sync-the-spec-clone) ## 13.1 Run it Run `python3 /scripts/orient.py [--spec-clone ]`. Pass the spec clone the user named, if any. Add `--skill-dir ` when the user names the skill, `--records ` when they say where its working records are (with `--skill-dir`, it applies to the named skill only), `--include-submodules` to scan submodule sources too, and `--full` for the raw script outputs. The script runs this skill's read-only scripts in order and prints one JSON object: -1. `spec_clone_info.py`: resolves the spec clone every other script reads, and settles STOP-1 ([2.6](../SKILL.md#26-pin-the-spec-clone)). -2. `check_repo_eligibility.py`: on a reject, nothing else runs. +1. `spec_clone_info.py`: resolves the spec clone every other script reads, offline ([2.6](../SKILL.md#26-pin-the-spec-clone)). +2. `check_repo_eligibility.py`, with or without a clone: on a reject, nothing else runs. 3. `inspect_existing_skill.py`: skills, their origin and records, harness, UTS-tagged tests. 4. `detect_liveobjects.py`: a one-line LiveObjects verdict. 5. `detect_capabilities.py`: the REST and Realtime capability profile, the sides, and the suggested scope (13.8), with eligibility's `capabilityOverride` if the whitelist gives one. -The clone `spec_clone_info.py` resolved (the argument, walked up to the clone root, else `UTS_SPEC_CLONE`, else the clone this skill is symlinked from) is passed to the eligibility check and to both detectors, which read their names from it (13.8); the State summary's Spec clone line shows that path. An explicit path (the argument or `UTS_SPEC_CLONE`) that isn't in a clone stops Orient with `NOT_A_SPEC_CLONE`: it is never replaced by another clone. A malformed data file in `assets/` (`DATA_FILE_ERROR`, naming the file and the key) stops Orient with `ok: false`: fix the file before anything else ([Maintaining this skill](../SKILL.md#maintaining-this-skill)). +The clone `spec_clone_info.py` resolved (the argument, walked up to the clone root, else the clone this skill is installed from, else `ABLY_SPEC_CLONE`, else the shared default `~/.ably/specification`; 13.9) is passed to the eligibility check and to both detectors, which read their names from it (13.8); the State summary's Spec clone line shows that path. An explicit path (the argument or `ABLY_SPEC_CLONE`) that isn't in a clone stops Orient with `NOT_A_SPEC_CLONE`: it is never replaced by another clone. When no clone is found, Orient still runs the eligibility check, so STOP-16 is settled before any offer to clone one. A malformed data file in `assets/` (`DATA_FILE_ERROR`, naming the file and the key) stops Orient with `ok: false`: fix the file before anything else ([Maintaining this skill](../SKILL.md#maintaining-this-skill)). -`spec_clone_info.py [ | --spec-clone ]` checks that the guide, the UTS docs and the helper specs exist, and runs `git rev-parse HEAD`, `git status --porcelain -- uts specifications` and `git log -1 --format='%H %cd'` on the guide and this skill ([2.6](../SKILL.md#26-pin-the-spec-clone) says what to record). Its outcomes: +`spec_clone_info.py [ | --spec-clone ]` checks that the guide, this skill, the UTS docs and the helper specs exist, reads the clone's git state offline (`ensure_spec_clone.py`'s status: branch, tracked changes, the last fetch of `origin/main`), and runs `git log -1 --format='%H %cd'` on the guide and this skill ([2.6](../SKILL.md#26-pin-the-spec-clone) says what to record). Its outcomes: -- Any `ok: false` (for example `SPEC_CLONE_NOT_FOUND`, `NOT_A_SPEC_CLONE`, `GIT_ERROR`): STOP-1; ask for the clone's path. -- `foundBy` other than `argument`: tell the user which clone was found and how, and confirm it before Phase 1. +- `SPEC_CLONE_NOT_FOUND` (nothing at `ABLY_SPEC_CLONE`, or at the default): after STOP-16, the STOP-1 bootstrap offer (13.9). +- `SPEC_CLONE_INCOMPLETE` (a revision that predates the guide or this skill): STOP-1, with the script's message (13.9). +- Any other `ok: false` (`NOT_A_SPEC_CLONE`, `SPEC_CLONE_ENV_CONFLICT`, `GIT_ERROR`): STOP-1; ask for the clone's path. +- `foundBy` other than `argument`: shown in the State summary (with the clone's kind) and confirmed with the STOP-13 answer. - `skillMatchesClone: false` (an installed copy that differs from this clone's copy of the skill, also reported in `warnings`): STOP-1; ask the user to refresh the install, or to confirm which version to follow. -- `skillMatchesClone: null` (the clone has no copy of this skill, for example an older checkout): tell the user and confirm before Phase 1. The JSON has `state` (class, re-entry, reasons, recommended option, options) and `stateSummaryText`, the same under Claude Code and Codex. The facts behind them are in the other fields. If Orient stopped with `NOT_A_SPEC_CLONE`, settle the clone's path (STOP-1) and rerun Orient first. Then settle the stops in this order, in as few messages as possible: 1. **STOP-16** (13.2), if the repo isn't plainly eligible. -2. **STOP-1** for the model (2.8) and for anything listed in `stop1`, such as the spec clone's path or a stale install of this skill. -3. **STOP-17** (13.8), when `stop17` is set: the capability profile needs confirming. -4. **STOP-13** (13.4). +2. **The spec-clone sync** (13.9), as `specSync` describes; rerun Orient if it moved HEAD or created the clone. +3. **STOP-1** for the model (2.8) and for anything listed in `stop1`, such as the spec clone's path or a stale install of this skill. +4. **STOP-17** (13.8), when `stop17` is set: the capability profile needs confirming. +5. **STOP-13** (13.4). If the user's request conflicts with the state (for example "create a skill" when one exists), say so at STOP-13. @@ -115,11 +118,15 @@ A version number alone proves nothing. Show `stateSummaryText`, completed with the session's model. Its layout: ``` +WARNING SPEC CLONE NOT SYNCED: Using (). It isn't synced; make sure its UTS specs are up to date. (only for a clone that isn't the shared clone, 13.9) +WARNING STALE SPEC CLONE: ; offer the sync at STOP-6 (13.9) | the sync is off (ABLY_SPEC_SYNC=off) (only for the shared clone, when the last fetch is over 30 days old, or unknown) WARNING SPEC NAMES FALLBACK: the spec clone's IDL wasn't parsed; the LiveObjects and capability verdicts below are unreliable; settle it at STOP-1 first (only on fallback) Repo @ (clean|dirty); languages: Eligibility accept: / via ( name)[; canonical ][; planned rename ][; capability override rest , realtime ] | ask: [ /] (); STOP-16 -Spec clone @ (clean|dirty); found by ; creator (matches clone: ) +Spec clone @ (clean|dirty); ; branch | detached; found by ; creator (matches clone: ) + HEAD committed days ago; origin/main as last fetched days ago was commit(s) ahead; the real gap may be larger; sync: | not synced (not the shared clone) + | NOT FOUND (SPEC_CLONE_NOT_FOUND): no clone at ; after STOP-16, offer to clone it there … | NOT USABLE (): ; settle it at STOP-1 | (not usable: : ) Model (Opus-class: yes|no) Spec names from the spec clone's IDL @ | from the spec clone's IDL (revision unknown: not a git checkout) | FALLBACK, don't rely on the LiveObjects or capability verdicts: Capabilities rest (); realtime (; connection …, channels …, presence …); [; sides: (rest yes|no, realtime yes|no), …][; source: override (whitelist)] @@ -131,12 +138,13 @@ Scope : (objects per Capability changed since the recorded run: new>, …[; added: ][; lower than recorded: ] (only if it changed) Skill [installs: …] name , version , origin (one per skill) inspector hints: fail, warn,

pass (regex, not a review); mapping: ; objects notes: -Harness ; smoke/self-test files: [; offered tiers lacking them: | unknown (some files don't name a tier)] +Harness ; smoke/self-test files: [ by name; candidate(s) by location (read to confirm)][; offered tiers lacking them: | unknown (some files don't name a tier)] Records

: run ; recorded spec , repo (only if the skill has records) Skills none[; other repo skills: …] (instead of the Skill lines, when there is no skill) Harness (with "Skills none") Problems (only if any) Changes since recorded spec : guide , UTS docs , helper specs , corpus files; checklist changed: [; repo files] + | recorded spec isn't in this clone: fetch it (STOP-6, 13.9) | the user fetches it (this clone isn't synced), or run a full gap audit | the records name no spec SHA: run a full gap audit UTS tests tags in files (); ; header SHAs …; deviations.md: … LiveObjects (): (decided at STOP-14) User-level @@ -146,6 +154,8 @@ Recommended