Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
6cfeb3f
UTS: add guide to writing uts-to-<lang> translator skills
sacOO7 Oct 8, 2026
a693524
UTS: add agent procedure for generating a translator skill
sacOO7 Oct 8, 2026
8a1bb95
UTS: rename translator skill procedure to uts-to-lang-skill-creator
sacOO7 Oct 8, 2026
5016c00
UTS: make the harness part of translator skill setup
sacOO7 Oct 8, 2026
0df3556
UTS: address review feedback on translator skill docs
sacOO7 Oct 8, 2026
a7fc036
UTS: package the translator skill procedure as an agent skill
sacOO7 Oct 8, 2026
9857c3f
UTS: move the translator skill guide to uts/docs
sacOO7 Oct 9, 2026
b56c819
UTS: drop provenance notes from the translator skill guide and skill
sacOO7 Oct 9, 2026
361ec4a
UTS: drop the count date from uts/README.md
sacOO7 Oct 9, 2026
06049e6
UTS: orient skill creator runs and scope them by SDK capability
sacOO7 Oct 9, 2026
8f4d04a
UTS: trim data-file comments and process wording in the skill creator
sacOO7 Oct 10, 2026
c3318e1
UTS: limit the skill creator to a whitelist of Pub/Sub SDK repositories
sacOO7 Oct 10, 2026
c179725
UTS: fix spec-clone resolution and PHP detection in skill creator scr…
sacOO7 Oct 10, 2026
fc13cfd
UTS: rename the skill creator's section 10 file to upgrade-diff-drive…
sacOO7 Oct 10, 2026
8931905
UTS: retitle the skill creator's section 11 as Upgrade/Fix, full gap …
sacOO7 Oct 10, 2026
355da6b
UTS: tidy the skill creator scripts' CLI, output and docstrings
sacOO7 Oct 10, 2026
80014c3
UTS: clean up the skill creator's references and templates
sacOO7 Oct 10, 2026
8ace95c
UTS: tighten the skill creator's SKILL.md
sacOO7 Oct 10, 2026
542f2a5
UTS: tighten the translator skill guide
sacOO7 Oct 10, 2026
0028278
UTS: describe the skill creator's Orient step and modes in the README
sacOO7 Oct 10, 2026
c9422de
UTS: address validation findings in the skill creator
sacOO7 Oct 10, 2026
9d67e21
UTS: keep the skill creator's stop points and ground rules within the…
sacOO7 Oct 10, 2026
0b0e59e
UTS: add a shared spec clone and its sync script to the skill creator
sacOO7 Oct 14, 2026
b7f1c91
UTS: fix the skill creator's inspector and Orient misreports
sacOO7 Oct 14, 2026
addbdba
UTS: document the skill creator's spec-clone sync
sacOO7 Oct 14, 2026
1ecceb2
UTS: have generated translator skills use the shared spec clone
sacOO7 Oct 14, 2026
f09a855
UTS: install the skill creator from a shared spec clone
sacOO7 Oct 14, 2026
68afd51
Merge pull request #565 from ably/uts/shared-spec-clone
sacOO7 Oct 15, 2026
7303a64
UTS: shorten the skill creator's install section
sacOO7 Oct 15, 2026
e4a232f
UTS: show Orient's summary as Markdown, and STOP-17 as the infrastruc…
sacOO7 Oct 15, 2026
6648bd5
UTS: ask Orient's questions one at a time, run mode first, in plain w…
sacOO7 Oct 15, 2026
1b2b5ba
UTS: apply the skill creator review batch
sacOO7 Oct 16, 2026
dfac3b1
UTS: rename the UTS harness to the UTS infra
sacOO7 Oct 16, 2026
0e52e65
UTS: fix the review findings on the skill creator batch
sacOO7 Oct 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 38 additions & 4 deletions uts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ uts/
│ │ ├── push/ # RSH — push admin
│ │ └── types/ # T* — type definitions
│ └── integration/ # REST integration tests (Ably sandbox)
│ └── proxy/ # Proxy-based fault injection tests
├── realtime/
│ ├── unit/ # Realtime unit tests (mocked WebSocket)
│ │ ├── helpers/
Expand All @@ -29,27 +30,43 @@ uts/
│ │ └── presence/ # RTP — realtime presence
│ └── integration/ # Realtime integration tests
│ ├── proxy/ # Proxy-based fault injection tests
│ ├── auth/, channels/, connection/, presence/ # Direct sandbox tests, by area
│ └── *.md # Direct sandbox tests
├── objects/
│ ├── helpers/
│ │ └── standard_test_pool.md # Standard LiveObjects test pool and fixtures
│ ├── unit/ # LiveObjects unit tests (mocked WebSocket)
│ └── integration/ # LiveObjects integration tests
│ └── proxy/ # Proxy-based fault injection tests
├── docs/ # Guides and reference
│ ├── writing-test-specs.md # How to write UTS specs
│ ├── writing-derived-tests.md # How to translate specs into SDK tests
│ ├── writing-uts-spec-translator-skills.md # What a uts-to-<lang> translator skill and its UTS infra must contain
│ ├── integration-testing.md # Integration testing policy
│ ├── proxy.md # Proxy infrastructure spec (cross-module)
│ └── completion-status.md # Spec coverage matrix
├── skills/ # Agent skills (Claude Code, Codex)
│ └── uts-to-lang-skill-creator/ # Builds or upgrades a uts-to-<lang> skill and its UTS infra (SKILL.md, references/, assets/, scripts/)
└── README.md # This file
```

## Spec File Counts

| Category | Count | Description |
|----------|-------|-------------|
| REST unit | 40 | Mocked HTTP client tests |
| REST integration | 11 | Ably sandbox tests |
| REST unit | 41 | Mocked HTTP client tests |
| REST integration (direct) | 11 | Ably sandbox tests |
| REST integration (proxy) | 1 | Fault injection via Go proxy |
| Realtime unit | 54 | Mocked WebSocket tests |
| Realtime integration (direct) | 13 | Direct sandbox tests |
| Realtime integration (proxy) | 7 | Fault injection via Go proxy |
| Helper specs | 4 | Mock infrastructure definitions |
| **Total** | **129** | |
| Objects unit | 15 | LiveObjects, mocked WebSocket |
| Objects integration (direct) | 3 | LiveObjects sandbox tests |
| Objects integration (proxy) | 1 | LiveObjects fault injection |
| Helper specs | 4 | Mock and fixture infrastructure definitions |
| **Total** | **150** | |

For current counts, `find uts/<module> -name '*.md'` is authoritative (it also lists `objects/PLAN.md`, which is not a spec).

## Three Test Tiers

Expand Down Expand Up @@ -138,9 +155,26 @@ See [docs/writing-test-specs.md](docs/writing-test-specs.md) for the full pseudo

- **[Writing Test Specs](docs/writing-test-specs.md)** — How to author UTS specs: mock patterns, pseudocode conventions, proxy test structure, common mistakes
- **[Writing Derived Tests](docs/writing-derived-tests.md)** — How to translate UTS specs into SDK-specific tests, diagnose failures, and record deviations
- **[Writing UTS Spec Translator Skills](docs/writing-uts-spec-translator-skills.md)** — How to build a per-language `uts-to-<lang>` agent skill (Claude Code and Codex) and its UTS infrastructure: infra smoke tests and self-tests, workflow, construct mapping, audit tooling, re-sync, upgrading an existing skill
- **[UTS-to-Lang Skill Creator](skills/uts-to-lang-skill-creator/SKILL.md)** — An agent skill (Claude Code and Codex) that builds or upgrades an SDK repo's `uts-to-<lang>` skill and the UTS test infrastructure (UTS infra) it runs on. Each run starts with Orient: it checks the repo against the whitelist in [`assets/eligibility.json`](skills/uts-to-lang-skill-creator/assets/eligibility.json) (Ably Pub/Sub SDK repositories only), detects any existing `uts-to-*` skill, UTS infra and UTS-tagged tests, classifies the repo's state (S0–S4), and detects its REST and Realtime clients, which set the scope: a REST-only or realtime-only SDK gets a partial skill. It then recommends Create (design, build and test the UTS infra, generate the skill, validate it without generating tests) or Upgrade/Fix (diff-driven for a skill it built, or a full gap audit against the guide), and always asks whether to add LiveObjects (`uts/objects`) support. See [Installing the skill creator](#installing-the-skill-creator)
- **[Integration Testing Policy](docs/integration-testing.md)** — When to write integration vs unit tests, proxy test design principles, test structure conventions
- **[Completion Status](docs/completion-status.md)** — Coverage matrix tracking which spec items have UTS test specs

### Installing the skill creator

Clone this repo to `~/.ably/specification` and link the skill into Claude Code and Codex:

```sh
git clone https://github.com/ably/specification.git ~/.ably/specification
mkdir -p ~/.claude/skills ~/.agents/skills
ln -sfn ~/.ably/specification/uts/skills/uts-to-lang-skill-creator ~/.claude/skills/uts-to-lang-skill-creator # Claude Code
ln -sfn ~/.ably/specification/uts/skills/uts-to-lang-skill-creator ~/.agents/skills/uts-to-lang-skill-creator # Codex
```

The skill offers to keep that clone up to date each time it runs. If either path already holds a copied folder rather than a link, delete it first. On Windows, copy the folder to both paths instead of linking it.

Then, in the SDK repo, run `/uts-to-lang-skill-creator` (Claude Code) or `$uts-to-lang-skill-creator` (Codex) on an Opus-class model.

## Go Test Proxy

The programmable proxy for integration testing lives in a separate repository: [ably/uts-proxy](https://github.com/ably/uts-proxy). It sits between the SDK and the Ably sandbox, transparently forwarding traffic while allowing rule-based fault injection.
Expand Down
2 changes: 2 additions & 0 deletions uts/docs/writing-derived-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ The process has two phases:

Not every situation has an existing implementation. Tests may be written ahead of the implementation (test-first development), or for a new SDK that doesn't yet exist. In those cases, only the translation phase applies.

To automate translation for one SDK with an agent skill (`uts-to-<lang>`), see [Writing UTS Spec Translator Skills](writing-uts-spec-translator-skills.md). That guide packages this document's rules into a repeatable procedure. This document remains the authority on translation and evaluation semantics.

---

## Phase 1: Translation
Expand Down
Loading
Loading