From 5392a9c517192ed1b62970707c256694dce6c450 Mon Sep 17 00:00:00 2001 From: Sergey S Date: Mon, 14 Sep 2026 03:54:48 +0200 Subject: [PATCH 1/2] =?UTF-8?q?0.24.3=20=E2=80=94=20the=20budget=20stops?= =?UTF-8?q?=20being=20estimated,=20and=20two=20skills=20stop=20naming=20a?= =?UTF-8?q?=20file=20they=20do=20not=20ship?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hygiene from the 2026-09-13 family audit (HK-11). - the body budget is MEASURED. This gate divided by a calibrated 3.9 chars/token and the calibration drifted: agent-evals estimated ~4961 and measures 4374, agent-orchestrator ~4762 against 4309 — two files reported past the working limit with 300+ tokens of real headroom. With no tokenizer the check discloses rather than issuing a verdict from the wrong instrument. - agent-harness and agent-interop named `test/validate.py`, a repository file that does not ship inside a skill; both say "this pack's repository validator" now - agent-harness declares compatibility — it ships scripts/audit_agent.py - plugin.json's homepage matches package.json and the forge - the README names what loads on demand (27 files, counted per skill from the tree — the first draft of that table was wrong on three of four rows) and both invocation forms; its installer line said "both skills" over four Gate: npm test EXIT=0; OK: agent-stack structurally valid (15 checks, v0.24.3). Co-Authored-By: Claude Opus 5 (1M context) --- .claude-plugin/marketplace.json | 2 +- CHANGELOG.md | 23 ++++++++++++ README.md | 22 ++++++++++- SKILL-CARD.md | 2 +- docs/evidence/verification.md | 12 ++++++ package.json | 2 +- .../agent-stack/.claude-plugin/plugin.json | 4 +- .../agent-stack/skills/agent-harness/SKILL.md | 5 ++- .../agent-stack/skills/agent-interop/SKILL.md | 2 +- test/validate.py | 37 ++++++++++++++++--- 10 files changed, 98 insertions(+), 13 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 89c6eb2..b6206fe 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -12,7 +12,7 @@ "displayName": "Agent Stack", "source": "./plugins/agent-stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.24.2", + "version": "0.24.3", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" diff --git a/CHANGELOG.md b/CHANGELOG.md index 3eca3b5..8ce73de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,26 @@ +## 0.24.3 — the budget stops being estimated, and two skills stop naming a file they do not ship + +Hygiene from the 2026-09-13 family audit (HK-11). + +- **The body budget is MEASURED.** This gate divided by a calibrated 3.9 chars/token; the + calibration drifted. Measured 2026-09-14 against the authority's tokenizer: + `agent-evals` estimated **~4961** and measured **4374**; `agent-orchestrator` estimated + **~4762** and measured **4309** — two files reported past the working limit with 300+ + tokens of real headroom. make-skill v0.28.0 closed the same defect in the family + auditor by measuring. With no tokenizer installed the check now **discloses** rather + than issuing a verdict from the wrong instrument. +- **`agent-harness` and `agent-interop` named `test/validate.py`** — a repository file + that does not ship inside a skill, so it resolves at the repo root and nowhere an + installed reader stands. Both say "this pack's repository validator" now. +- **`agent-harness` declares `compatibility`**: it ships `scripts/audit_agent.py`, and a + skill that needs an interpreter says so in front matter. +- **`plugin.json`'s homepage** matches `package.json` and the forge + (`skills.sshlg.me/skills/agent-stack/`) instead of the GitHub URL. +- **The README names what loads on demand** — 27 reference files, counted per skill from + the tree, not stated — and both invocation forms (`/agent-stack:` and + `/`), because which one resolves depends on how the pack was installed. The + installer line said "both skills" over a pack of four. + ## v0.24.2 — the ledger section names the version it ships The previous patch wrote the ledger section BEFORE bumping the version, so the diff --git a/README.md b/README.md index d97c2cf..8907782 100644 --- a/README.md +++ b/README.md @@ -160,7 +160,7 @@ waterfall, and model-routing precedence. /plugin install agent-stack@agent-stack ``` -**npm installer** — copies both skills into `~/.claude/skills/`: +**npm installer** — copies all four skills into `~/.claude/skills/`: ```bash npx @ssheleg/agent-stack @@ -180,6 +180,26 @@ npx --yes sshlg-skills@latest update Restart your agent afterwards — skills load at session start. +**Reaching a skill by name.** Claude Code routes on the description, and both +invocation forms work: `/agent-stack:agent-orchestrator` names the plugin's copy +explicitly, `/agent-orchestrator` the plain one. Which resolves depends on how this +pack was installed, so both are written here rather than one promised. + +### What loads on demand + +Twenty-seven reference files ship with the four skills, each with a load trigger in +its skill body. By skill: + +| Skill | References | +|---|---| +| `agent-orchestrator` | 12 — the tool-calling loop, memory and retrieval, context budgets, provider routing, the work graph, the resale wallet and the saga across database and provider API | +| `agent-harness` | 7, plus `scripts/audit_agent.py` | +| `agent-interop` | 6 — one per protocol surface, each pinning the revision it was read at | +| `agent-evals` | 2 — `otel-genai.md` (the semantic conventions an eval reads) and `statistics.md` (when a difference is a difference) | + +A file with no load trigger in a body is a defect, not a bonus: the skill audit +refuses one. + --- ## When it triggers diff --git a/SKILL-CARD.md b/SKILL-CARD.md index d2fdcb8..38c3c09 100644 --- a/SKILL-CARD.md +++ b/SKILL-CARD.md @@ -5,7 +5,7 @@ | Field | Value | |---|---| | Pack | `agent-stack` | -| Version | `0.24.2` | +| Version | `0.24.3` | | Skills | `agent-orchestrator`, `agent-evals`, `agent-interop`, `agent-harness` | | License | MIT | | Source | https://github.com/ssheleg/agent-stack | diff --git a/docs/evidence/verification.md b/docs/evidence/verification.md index c978767..f8f6a6d 100644 --- a/docs/evidence/verification.md +++ b/docs/evidence/verification.md @@ -10,6 +10,18 @@ This file exists because its absence read as zero exposure. `sshlg-skills` board --- +## Shipped state — v0.24.3 (2026-09-14) + +Hygiene from the 2026-09-13 family audit (HK-11). + +| REQ | What ships | How it was confirmed | Confirmed | +|---|---|---|---| +| HY-measure | The body budget is measured, never estimated | `_measure_body_tokens` uses tiktoken where installed and discloses where it is not. The drift it replaces: agent-evals ~4961 estimated / 4374 measured, agent-orchestrator ~4762 / 4309 | **observed** — two files were being reported past the working limit with 300+ tokens of headroom | +| HY-resolve | No skill body names a file that does not ship inside it | `agent-harness/SKILL.md:122` and `agent-interop/SKILL.md:44` said `test/validate.py`, a repository file; both now say "this pack's repository validator" | **observed** | +| HY-compat | A skill that needs an interpreter declares it | `agent-harness` ships `scripts/audit_agent.py` and now carries `compatibility` in front matter; `audit_skill.py --house` 0 GAP, 19 PASS | **observed** | +| HY-shelf | The README names what loads on demand, counted from the tree | 27 reference files, per skill: orchestrator 12, harness 7, interop 6, evals 2 — the first draft of this table stated 12/4/7/4 from memory and was corrected against `ls` | **observed** — the counted number and the remembered one disagreed on three of four rows | +| Gate | The whole suite on this tree | `npm test` EXIT=0; `OK: agent-stack structurally valid (15 checks …, v0.24.3)` | **observed** | + ## Shipped state — v0.24.2 (2026-09-10) Sherlock external-v3 (24 findings), each carrying its own executable regression. diff --git a/package.json b/package.json index bfaf894..ddcdefa 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@ssheleg/agent-stack", - "version": "0.24.2", + "version": "0.24.3", "scripts": { "test": "python3 test/validate.py && python3 test/plant_guard_test.py && node test/installer_test.js && npm run test:audit", "test:audit": "for t in test/audit_regressions/*.py; do python3 \"$t\" || exit 1; done" diff --git a/plugins/agent-stack/.claude-plugin/plugin.json b/plugins/agent-stack/.claude-plugin/plugin.json index 040e00d..1268618 100644 --- a/plugins/agent-stack/.claude-plugin/plugin.json +++ b/plugins/agent-stack/.claude-plugin/plugin.json @@ -3,12 +3,12 @@ "name": "agent-stack", "displayName": "Agent Stack", "description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.", - "version": "0.24.2", + "version": "0.24.3", "author": { "name": "ssheleg", "url": "https://x.com/sshlg93" }, - "homepage": "https://github.com/ssheleg/agent-stack", + "homepage": "https://skills.sshlg.me/skills/agent-stack/", "repository": "https://github.com/ssheleg/agent-stack", "license": "MIT", "keywords": [ diff --git a/plugins/agent-stack/skills/agent-harness/SKILL.md b/plugins/agent-stack/skills/agent-harness/SKILL.md index 60757ea..f02e472 100644 --- a/plugins/agent-stack/skills/agent-harness/SKILL.md +++ b/plugins/agent-stack/skills/agent-harness/SKILL.md @@ -14,6 +14,9 @@ description: >- "аудит агента", "встроить агента". Not for the loop's plumbing, its evals, or its protocols — those are siblings. license: MIT +compatibility: >- + scripts/audit_agent.py runs with python3 (standard library only); no network, + no package install. Everything else in this skill is prose and needs nothing. --- # Agent harness — what the agent is told, and how to audit what someone else told theirs @@ -119,7 +122,7 @@ because a decision table with two homes is one that will disagree with itself. ## References Each opens with its own **Load this when** line and a revision stamp — this material moves, -and `test/validate.py` fails the build on a reference that does not say when it was read. +and this pack's repository validator fails the build on a reference that does not say when it was read. | File | Read it when | |---|---| diff --git a/plugins/agent-stack/skills/agent-interop/SKILL.md b/plugins/agent-stack/skills/agent-interop/SKILL.md index b8bbe69..212dfa9 100644 --- a/plugins/agent-stack/skills/agent-interop/SKILL.md +++ b/plugins/agent-stack/skills/agent-interop/SKILL.md @@ -41,7 +41,7 @@ against the live specification rather than recalled: still introduced `binds` as a core concept as of 2026-08-13. So: **every reference in this skill opens with a `**Spec pinned:**` line**, and -`test/validate.py` fails the build without one. That is a mechanical check, not an +This pack's repository validator fails the build without one. That is a mechanical check, not an aspiration — the class of error it prevents is the one where prose reads as current because nothing on the page says otherwise. diff --git a/test/validate.py b/test/validate.py index 5ac74fe..c3fe51a 100644 --- a/test/validate.py +++ b/test/validate.py @@ -93,6 +93,23 @@ def front_matter(path): return m.group(1), text +def _measure_body_tokens(body): + """(count, how) — a real tokenizer where one is installed, the estimate otherwise. + + The authority is the family auditor's `cl100k_base`. The divisor below was calibrated + against it once and drifted: on 2026-09-14 it read agent-evals at ~4961 and + agent-orchestrator at ~4762 where the tokenizer read 4374 and 4309. + """ + try: + import tiktoken + except Exception: # noqa: BLE001 - optional + return int(len(body) / 3.9), "estimate" + try: + return len(tiktoken.get_encoding("cl100k_base").encode(body)), "tiktoken:cl100k_base" + except Exception: # noqa: BLE001 - a broken install is not a verdict + return int(len(body) / 3.9), "estimate" + + def scalar(block, key): """Read one front-matter scalar without a YAML dependency. @@ -217,12 +234,22 @@ def scalar(block, key): # -> 3.9), so this tracks the authority closely and slightly high. Re-derive it if the # auditor's tokenizer changes; do not widen it to make a failing file pass. body = text.split("---", 2)[2] if text.count("---") >= 2 else text - body_tokens = int(len(body) / 3.9) - if body_tokens > BODY_BUDGET_TOKENS: - fail(f"{name}/SKILL.md: body ~{body_tokens} tokens, past the {BODY_BUDGET_TOKENS} " - "budget — the answer at this point is a split, not a trim") + body_tokens, how = _measure_body_tokens(body) + if how == "estimate": + # A calibrated divisor is still an estimate, and the calibration expires: measured + # 2026-09-14, this one read agent-evals at ~4961 and agent-orchestrator at ~4762 + # while the authority's tokenizer read 4374 and 4309 — two files reported past the + # working limit with 300+ tokens of real headroom. make-skill v0.28.0 closed the + # same defect in the family auditor by MEASURING; a verdict from the wrong + # instrument gets quoted as if it were one, so with no tokenizer this discloses + # instead of judging. + notes.append(f"{name}/SKILL.md: body budget NOT MEASURED (~{body_tokens} tokens by " + "chars/3.9) — `pip install tiktoken` to gate it") + elif body_tokens > BODY_BUDGET_TOKENS: + fail(f"{name}/SKILL.md: body {body_tokens} tokens ({how}), past the " + f"{BODY_BUDGET_TOKENS} budget — the answer at this point is a split, not a trim") elif body_tokens > BODY_WORKING_TOKENS: - notes.append(f"{name}/SKILL.md: body ~{body_tokens} tokens, past the " + notes.append(f"{name}/SKILL.md: body {body_tokens} tokens ({how}), past the " f"{BODY_WORKING_TOKENS} working limit ({BODY_BUDGET_TOKENS} budget) — " "displace before the next addition") From 53d309ccf452867ebc59c95f4e8825e8ef6756b0 Mon Sep 17 00:00:00 2001 From: Sergey S Date: Mon, 14 Sep 2026 03:55:59 +0200 Subject: [PATCH 2/2] ci: install the tokenizer before the validator here too The body budget check now measures or discloses; a runner without tiktoken would disclose on every skill and gate nothing. Same step the skill-audit job has carried since make-skill v0.28.0, in the job that runs the repo validator. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/validate.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index bad3d84..dcc406c 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -31,6 +31,13 @@ jobs: steps: - uses: actions/checkout@v4 + # The body budget is MEASURED, never estimated: `test/validate.py` refuses to issue + # a verdict from chars/3.9 and discloses instead, so without this step the budget + # check stops gating quietly. The skill-audit job installs the same package for the + # same reason (make-skill v0.28.0). + - name: Install a real tokenizer (the budget is MEASURED, never estimated) + run: python3 -m pip install --quiet tiktoken + - name: Structural validator run: python3 test/validate.py