Skip to content

test(sdk): shared conformance suite for pre-sign policy engines (#226) - #246

Merged
Eras256 merged 2 commits into
nirium-protocol:mainfrom
fernando-mendoza:feat/226-policy-conformance
Nov 4, 2026
Merged

Eras256 merged 2 commits into
nirium-protocol:mainfrom
fernando-mendoza:feat/226-policy-conformance

Conversation

@fernando-mendoza

Copy link
Copy Markdown
Collaborator

What

A shared conformance test any pre-sign policy engine can run (#226). runPolicyConformance(target) registers seven node:test cases. Each one runs the engine through the SDK's real gate (createPolicyGatedSigner) and stellar-sdk's real authorizeEntry signing path, with no RPC and no network:

Case Expected
ALLOW echoing the exact contextHash signs exactly once, onDecision = ALLOWED,SIGNED
DENY no signature, DENY
WAIT no signature, WAIT
ALLOW with a wrong contextHash no signature, UNBOUND
ALLOW past its expiresAt no signature, EXPIRED
malformed answer (null / unknown verdict / non-numeric expiresAt) no signature, MALFORMED
answer after the timeout no signature, TIMEOUT, engine signal aborted

The engine provides one payment it must ALLOW and one it must DENY, plus one it answers WAIT to if it ever does. The last four cases start from the engine's own ALLOW and change only the fault each case names. Each case first asserts that the engine really did ALLOW, so none of them passes trivially.

Files

  • packages/sdk/src/x402-policy-conformance.ts: the suite. It also exposes policyConformanceCases() as { name, run } for other runners.
  • packages/sdk/test/x402-policy-conformance.test.ts: meta-tests (see below).
  • examples/x402-policy-daily-budget/test/conformance.test.ts: the daily-budget engine runs the suite.
  • .github/workflows/policy-conformance.yml: builds the SDK and runs the daily-budget example's tests, on PRs touching the SDK or that example. Until now those tests weren't run in CI.
  • SDK README: a short "Conformance test for your engine" section. The example README now says the tests need the SDK built.

The hook itself is unchanged (out of scope per the issue).

Design choices (happy to change any)

  • Location / API: like the gate, it isn't exported from the package index. It ships in the tarball as dist/x402-policy-conformance.js, so third-party engines can import it from nirium/dist/..., the same way the examples import the gate. It's marked experimental.
  • WAIT: daily-budget never answers WAIT. When target.wait isn't given, the suite turns the engine's ALLOW into a WAIT. When it is given, the engine must answer WAIT itself.
  • Fresh hook per case (createHook), so budget state doesn't leak between cases.

How it was tested

  • Meta-tests (teeth):
    • A correct reference engine passes all cases, both with a native and with an injected WAIT.
    • Each of these broken engines fails exactly the expected cases:
      • ignores contextHash → ALLOW
      • omits contextHash → ALLOW
      • ALLOWs everything → DENY
      • DENYs everything → ALLOW plus all derived cases
      • declares WAIT but ALLOWs → WAIT
      • hangs → all cases
      • throws → all cases
      • returns an already-expired ALLOW → ALLOW
      • leaves out expiresAt while requireExpiry is set → ALLOW
      • sends a stale policyVersion → ALLOW
      • returns an unknown verdict → ALLOW
    • A gate that signs despite a refusal fails every refusal case, and a gate that never signs fails ALLOW.
  • Daily-budget sabotage, run locally and not committed: making the engine ignore contextHash fails the ALLOW case. Making it ALLOW over the cap fails the DENY case.
  • packages/sdk: npm run build && npm test: jest 56/56, node:test 160/160. scripts/clean-install-check.mjs ok, and the tarball contains the new dist files and no *.test.*.
  • examples/x402-policy-daily-budget: npm ci && npm test 16/16, npm run typecheck ok. Both pass with the file: link and with the packed SDK tarball.
  • .github/scripts/typecheck-example.sh against the packed tarball: all 8 examples ok.

Closes #226

@M0nsxx M0nsxx left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work on this, Fer. The scope matches #226, CI is green and I can see all 9 cases running against daily-budget in the log. The meta-tests are the best part.

I'd change two things before merging. I found both by reading the code and haven't run them, so please check me on it.

  1. The timeout case can fail a correct engine. It sets timeoutMs to 100 and then waits for the engine's real answer under that same deadline. An HTTP engine that takes 200 ms gets TIMEOUT from the gate before engineDecisions has anything in it, so expectEngineAllowed reports "the engine must ALLOW" even though nothing is wrong with the engine. #226 is meant for any engine, and remote ones are the common case. Could the case get the engine's answer outside the gate's deadline and await it before asserting? If you'd rather keep it as is, at least document the timeoutMs option in the README. A meta-test with a 200 ms engine would cover it either way.

  2. A class-based hook can pass when it shouldn't. { ...engine } only copies own properties, so a currentVersion on the prototype is dropped and the version check never runs. A stale policyVersion would then pass. onDecision is also called unbound, and the gate swallows the resulting error. Binding those fields explicitly would fix both, or createHook could reject anything that isn't a plain object.

One small README ask: please say that passing means the engine answers correctly and the gate fails closed around it. It doesn't validate the policy logic itself, and four of the seven cases exercise mostly the gate.

Drafted with Claude (Opus 5.5) and reviewed by @Eras256.

fernando-mendoza and others added 2 commits November 3, 2026 17:03
…um-protocol#226)

Adds runPolicyConformance / policyConformanceCases (dist/x402-policy-conformance.js):
seven cases through the real gate and stellar-sdk signing path. Only an ALLOW
with the exact contextHash signs; DENY, WAIT, wrong contextHash, expired,
malformed and timeout sign nothing. Meta-tests prove broken engines and a
leaky gate fail the right cases. The daily-budget example runs it in CI
(Policy Conformance workflow); SDK README documents how to run it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ium-protocol#226)

Addresses review on nirium-protocol#246:
- Timeout case: ask the engine outside the gate's deadline (with its own
  timeoutMs budget, default 5000 ms) and await its answer before asserting,
  so a correct engine slower than 100 ms no longer fails the case.
- Build the wrapped hook field by field and call methods on the engine, so a
  class-based engine keeps prototype currentVersion (stale-version check runs)
  and onDecision keeps its `this`.
- README: what passing does and does not prove; timeoutMs and class engines.
Meta-tests: 200 ms and 600 ms correct engines pass every case; a class engine
passes, a class engine with a stale policyVersion fails ALLOW, and a class
onDecision sees ALLOWED,SIGNED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@fernando-mendoza
fernando-mendoza force-pushed the feat/226-policy-conformance branch from 7810d79 to dc80200 Compare November 3, 2026 23:07

@Eras256 Eras256 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green, scoped to its own issue (#226), test-only additions, no touch on auth/secrets/CORS. Merging.

@Eras256
Eras256 merged commit f1551ad into nirium-protocol:main Nov 4, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pre-sign policy hook: a conformance test any policy engine can run

3 participants