Repository navigation
test(sdk): shared conformance suite for pre-sign policy engines (#226) - #246
Conversation
M0nsxx
left a comment
There was a problem hiding this comment.
Nice work on this, Fer. The scope matches #226, CI is green and I can see all 9 cases running against daily-budget in the log. The meta-tests are the best part.
I'd change two things before merging. I found both by reading the code and haven't run them, so please check me on it.
-
The timeout case can fail a correct engine. It sets timeoutMs to 100 and then waits for the engine's real answer under that same deadline. An HTTP engine that takes 200 ms gets TIMEOUT from the gate before engineDecisions has anything in it, so expectEngineAllowed reports "the engine must ALLOW" even though nothing is wrong with the engine. #226 is meant for any engine, and remote ones are the common case. Could the case get the engine's answer outside the gate's deadline and await it before asserting? If you'd rather keep it as is, at least document the timeoutMs option in the README. A meta-test with a 200 ms engine would cover it either way.
-
A class-based hook can pass when it shouldn't. { ...engine } only copies own properties, so a currentVersion on the prototype is dropped and the version check never runs. A stale policyVersion would then pass. onDecision is also called unbound, and the gate swallows the resulting error. Binding those fields explicitly would fix both, or createHook could reject anything that isn't a plain object.
One small README ask: please say that passing means the engine answers correctly and the gate fails closed around it. It doesn't validate the policy logic itself, and four of the seven cases exercise mostly the gate.
Drafted with Claude (Opus 5.5) and reviewed by @Eras256.
…um-protocol#226) Adds runPolicyConformance / policyConformanceCases (dist/x402-policy-conformance.js): seven cases through the real gate and stellar-sdk signing path. Only an ALLOW with the exact contextHash signs; DENY, WAIT, wrong contextHash, expired, malformed and timeout sign nothing. Meta-tests prove broken engines and a leaky gate fail the right cases. The daily-budget example runs it in CI (Policy Conformance workflow); SDK README documents how to run it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ium-protocol#226) Addresses review on nirium-protocol#246: - Timeout case: ask the engine outside the gate's deadline (with its own timeoutMs budget, default 5000 ms) and await its answer before asserting, so a correct engine slower than 100 ms no longer fails the case. - Build the wrapped hook field by field and call methods on the engine, so a class-based engine keeps prototype currentVersion (stale-version check runs) and onDecision keeps its `this`. - README: what passing does and does not prove; timeoutMs and class engines. Meta-tests: 200 ms and 600 ms correct engines pass every case; a class engine passes, a class engine with a stale policyVersion fails ALLOW, and a class onDecision sees ALLOWED,SIGNED. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7810d79 to
dc80200
Compare
What
A shared conformance test any pre-sign policy engine can run (#226).
runPolicyConformance(target)registers sevennode:testcases. Each one runs the engine through the SDK's real gate (createPolicyGatedSigner) and stellar-sdk's realauthorizeEntrysigning path, with no RPC and no network:contextHashonDecision=ALLOWED,SIGNEDDENYWAITcontextHashUNBOUNDexpiresAtEXPIREDexpiresAt)MALFORMEDTIMEOUT, engine signal abortedThe engine provides one payment it must ALLOW and one it must DENY, plus one it answers WAIT to if it ever does. The last four cases start from the engine's own ALLOW and change only the fault each case names. Each case first asserts that the engine really did ALLOW, so none of them passes trivially.
Files
packages/sdk/src/x402-policy-conformance.ts: the suite. It also exposespolicyConformanceCases()as{ name, run }for other runners.packages/sdk/test/x402-policy-conformance.test.ts: meta-tests (see below).examples/x402-policy-daily-budget/test/conformance.test.ts: the daily-budget engine runs the suite..github/workflows/policy-conformance.yml: builds the SDK and runs the daily-budget example's tests, on PRs touching the SDK or that example. Until now those tests weren't run in CI.The hook itself is unchanged (out of scope per the issue).
Design choices (happy to change any)
dist/x402-policy-conformance.js, so third-party engines can import it fromnirium/dist/..., the same way the examples import the gate. It's marked experimental.target.waitisn't given, the suite turns the engine's ALLOW into a WAIT. When it is given, the engine must answer WAIT itself.createHook), so budget state doesn't leak between cases.How it was tested
contextHash→ ALLOWcontextHash→ ALLOWexpiresAtwhilerequireExpiryis set → ALLOWpolicyVersion→ ALLOWcontextHashfails the ALLOW case. Making it ALLOW over the cap fails the DENY case.packages/sdk:npm run build && npm test: jest 56/56, node:test 160/160.scripts/clean-install-check.mjsok, and the tarball contains the new dist files and no*.test.*.examples/x402-policy-daily-budget:npm ci && npm test16/16,npm run typecheckok. Both pass with thefile:link and with the packed SDK tarball..github/scripts/typecheck-example.shagainst the packed tarball: all 8 examples ok.Closes #226