Skip to content

test(client): integrity verification and the accessor telemetry sweep - #32

Merged
XieX merged 2 commits into
xie/skills-03-core-accessorsfrom
xie/skills-04-verification-tests
Sep 18, 2026
Merged

XieX merged 2 commits into
xie/skills-03-core-accessorsfrom
xie/skills-04-verification-tests

Conversation

@XieX

@XieX XieX commented Aug 25, 2026 •

Copy link
Copy Markdown

Stacked PR 4 of 7 — merge bottom-up into xie/agent-skills-feature-ac9ac7.

  1. feat(client): Agent Skills value types #29 — value types
  2. feat(client): symlink-refusing filesystem primitives #30 — symlink-refusing filesystem primitives
  3. feat(client): Agent Skills store seam, verification, and accessors #31 — store seam, verification, accessors
  4. test(client): integrity verification and the accessor telemetry sweep #32 — integrity + telemetry tests ← you are here
  5. feat(client): writeSkills materialization and the abuse matrix #33 — writeSkills + abuse matrix
  6. test(client): the rest of the writeSkills suite #34 — remaining writeSkills tests
  7. docs(client): document Agent Skills #35 — docs

Each PR targets the one below it, so GitHub already shows only this PR's own diff.


Fourth of seven. Test-only — this is the adversarial half of the previous PR, split out rather than asking anyone to review 1,500 lines in one sitting.

Integrity verification

Covers what a hostile or broken store can serve: a content hash that doesn't match, a hash of the wrong shape, content over the size cap (measured on the encoded bytes), and a key or version that fails revalidation. Verified content comes back as the exact bytes that were hashed — the multi-byte UTF-8 case asserts byte equality against the TextEncoder encoding, not string equality.

The case worth reading carefully is the UTF-8 round-trip one. TextEncoder substitutes U+FFFD for an unpaired surrogate rather than throwing, so the test pins contentHash to the sha256 of the substituted bytes. That means the hash comparison cannot be what rejects it — only the round-trip guard can be, which is what makes the test fail against an implementation missing that guard.

Telemetry sweep

Asserts the seam's shape rather than any one signal:

  • The allowlist arrives here: a sweep over every recorded name proves the set is exactly three, and that the two names this SDK must never emit — AgentControl Skill SDK Reference Returned and AgentControl Skill Content Retrieved — appear nowhere.
  • No skill content and no filesystem path reaches a signal.
  • An emitter that throws cannot fail the operation it was observing.

getSkillResult (security review LA-2)

One case per reason token, each store shaped to reach a different construction site in resolveFromStore, so the mapping is under test and not just the union. Plus:

  • store_unavailable is distinct from absent — a throwing store and an empty store must not read the same, or a caller cannot tell "retry" from "never configured".
  • A version pin against an inline fake store holding two versions resolves to the pinned version rather than the newest. This is the test that fails if the version is not threaded into getObject: the store would answer with the newer version, the equality check would refuse it, and the outcome would report wrong_version for a pin the store could have satisfied.
  • A store that ignores or lies about the pin still gets its answer refused, which is what keeps the equality check a defense rather than dead code.
  • getSkill still resolves to null, and still never rejects, for all four failure outcomes — the no-behaviour-change guarantee.
  • Both accessors throw the same error for a missing store, asserted by comparing the messages rather than by matching a pattern twice.
  • Exactly one ld.skills.integrity_failure record and one signal per failed retrieval, so the new accessor provably does not double-log.
  • detail never contains the skill content, including when a hostile store routes the body through contentHash.
  • createSkillOutcome returns a frozen object, like every sibling factory.

Verification

Client package 395 → 476 tests. typecheck and biome clean.

🤖 Generated with Claude Code


Note

Overview
Test-only expansion of skills.test.ts (~900 lines) that adversarially pins skill accessor security and observability contracts from the prior implementation PR.

Adds integrity verification cases for hostile store payloads: hash mismatch, size cap on encoded bytes, key/version revalidation, UTF-8 round-trip (unpaired surrogates, BOM), and stores that throw (mapped to empty/null results).

Adds telemetry seam checks: integrity signal shape and redaction (no skill body in signals or smuggled via key/hash), allowlist of three signal names and explicit absence of removed signals, no client.track from accessors, and fault-tolerant emitters.

Adds ld.skills.integrity_failure log record contract tests: eight reason_code values, stable JSON field set and key ordering, redaction, and logging even without telemetry.

Adds getSkillResult coverage distinguishing absent vs integrity_failure vs store_unavailable vs wrong_version, version pinning through getObject, unchanged getSkill null semantics, no double-logging, safe detail, plus createSkillOutcome immutability and package export checks for the five SkillOutcomeReason tokens.

Reviewed by Cursor Bugbot for commit 8644e92. Bugbot is set up for automated code reviews on this repo. Configure here.


class ThrowingEmitter {
record(): void {
throw new Error('emitter exploded');

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😂

XieX and others added 2 commits September 16, 2026 14:19
Splitting these out of the accessor commit rather than reviewing 1,500 lines at
once. Both suites test code that already landed; they are the adversarial half
of it.

Integrity verification covers what a hostile or broken store can serve: a
content hash that does not match, a hash of the wrong shape, content over the
64 KB cap, a key or version that fails revalidation, and the UTF-8 round-trip
case where `TextEncoder` substitutes U+FFFD for an unpaired surrogate. That last
one pins `contentHash` to the sha256 of the *substituted* bytes, so the hash
comparison cannot be what rejects it — only the round-trip guard can be, which
is what makes the test fail against an implementation missing it.

The telemetry suite asserts the seam's shape rather than any one signal. The
allowlist arrives with it: a sweep over every recorded name proves the set is
exactly three and that the two names this SDK must never emit —
`AgentControl Skill SDK Reference Returned` and
`AgentControl Skill Content Retrieved` — appear nowhere. Also covered: no skill
content and no filesystem path reaches a signal, and an emitter that throws
cannot fail the operation it was observing.

Client package: 395 -> 433 tests. typecheck and biome clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds the regression test for the BOM fix in the previous commit, placed
next to the lone-surrogate test it is the counterpart of: authentic
content beginning with U+FEFF verifies and returns its exact bytes.
Fails without `ignoreBOM: true`.

Comment sweep, per review feedback: the vocabulary and key-order tests
now describe their cross-SDK contract without naming the Python SDK, and
the surrogate test explains the U+FFFD substitution as a Node behavior
rather than by contrast with str.encode. No assertions changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@XieX
XieX force-pushed the xie/skills-04-verification-tests branch from a718d95 to 8644e92 Compare September 16, 2026 18:26
@XieX
XieX merged commit 2d5712d into xie/agent-skills-feature-ac9ac7 Sep 18, 2026
8 checks passed
@XieX
XieX deleted the xie/skills-04-verification-tests branch September 18, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants