Skip to content

Add quantus exercise chain smoke/fuzz suite#115

Open
illuzen wants to merge 4 commits into
mainfrom
illuzen/runtime-test
Open

Add quantus exercise chain smoke/fuzz suite#115
illuzen wants to merge 4 commits into
mainfrom
illuzen/runtime-test

Conversation

@illuzen

@illuzen illuzen commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

@ -0,0 +1,53 @@

feat: add quantus exercise chain smoke/fuzz suite

Summary

Adds a single command — quantus exercise — that runs a scenario-based smoke and fuzz suite against a live node, covering every call path the CLI wraps. It is designed as a pre-runtime-upgrade regression gate: run it before and after an upgrade and any extrinsic that stops working (or any validation that silently loosens) fails the suite with a nonzero exit code, making it CI-ready.

quantus exercise                              # default phases against ws://127.0.0.1:9944
quantus exercise --json --fuzz-iterations 25  # CI mode
quantus exercise --upgrade-wasm runtime.wasm  # full governance upgrade + post-upgrade re-run

What it covers

Phase Scenarios
setup connect, compatibility check, dev wallets, derive + fund ephemeral accounts
reads runtime version, metadata pallets, chain properties, blocks/events decode, treasury, high-security status, scheduler storage, balances
balances single transfer, Utility::batch, transfer with tip, manual nonce, remark_with_event
reversible schedule + owner-cancel (one-time delay), schedule with delay, set_high_security + HS default-delay transfer + guardian cancel
multisig 2-of-3 lifecycle: create, fund, propose, approve, execute, verify recipient balance
recovery config reads; initiate_recovery pinned as clean NotRecoverable (CLI lacks create_recovery — documented limitation)
preimage note_preimage + status verify; request_preimage pinned as BadOrigin
governance membership reads, full TechReferenda flow (submit, decision deposit, 3 aye votes), add_member pinned as BadOrigin
negative over-balance, below-ED, overflow amount, malformed SS58 (client-side), stale nonce, delay-too-short, self-guardian, default-delay-without-HS, scheduler-calls-disabled canary, removed-pallets canary
fuzz seeded loop of random transfers / remarks / batches / reversible delays with boundary amounts (0, 1, ED±1, u128::MAX); every iteration must succeed cleanly or reject cleanly
wormhole multiround flow (5 rounds, 5 proofs each); included by default — proving is fast on release builds. Use --skip wormhole on debug builds
upgrade (opt-in) note WASM preimage, System::set_code via TechReferenda on a fast-governance node, poll for spec bump, then automatically re-run all phases against the upgraded runtime

Negative scenarios assert the specific expected error, so a runtime upgrade that accidentally loosens validation is caught, not just crashes.

Flags

--phases / --skip, --fuzz-iterations N (default 25), --seed <u64> (reproducible fuzzing; seed is always printed), --upgrade-wasm <path>, --upgrade-timeout-secs, --ephemeral-accounts N, --fail-fast, --json.

Also in this PR

  • Metadata regenerated for spec 134 (was 131): community Referenda/ConvictionVoting pallets removed, Scheduler dispatchables disabled. The stale referenda CLI modules and scheduler dispatch code are removed accordingly; compatibility table updated.
  • Quiet logging mode (src/log.rs): the exercise suite suppresses per-transaction chatter (spinners, expected-failure error banners) so the per-step report is the single source of truth. --verbose restores full output. Normal CLI commands are unaffected.
  • Nightly CI workflow (.github/workflows/exercise.yml): builds quantus-node from the chain repo with fast-governance, starts a dev node, runs the suite with --json, uploads report + node log as artifacts. Nightly + workflow_dispatch (with chain ref and fuzz-iteration inputs) rather than per-PR, since node build + QPoW block times dominate.

Test plan

  • cargo +nightly fmt --check, clippy --all-targets (no warnings), cargo test --locked (210 passed)
  • Full suite green against a local spec-134 fast-governance dev node (all phases except wormhole/upgrade)
  • One manual end-to-end run of the upgrade phase with a candidate WASM (--upgrade-wasm) before relying on it for the real upgrade
  • First nightly exercise.yml run on CI to validate timing

Known limitations

  • Recovery coverage is reads + expected-failure only (CLI has no create_recovery wrapper).
  • The upgrade phase has not yet been exercised end-to-end; it needs a WASM with a higher spec version and a fast-governance node.
  • Full run takes tens of minutes wall-clock, dominated by QPoW block times, not suite overhead.

illuzen and others added 2 commits July 3, 2026 19:37
Adds a scenario-based exercise command that drives every call path the
CLI wraps against a live node, for pre-runtime-upgrade regression
testing and CI:

- Phases: reads, balances, reversible, multisig, recovery, preimage,
  governance, negative, seeded fuzz; opt-in wormhole and a
  governance-driven runtime-upgrade phase (--upgrade-wasm, requires a
  fast-governance node) with automatic post-upgrade re-run.
- Negative scenarios pin expected rejections (incl. scheduler-disabled
  and removed-pallet canaries) so upgrades that loosen validation fail.
- CI-friendly: --json report, nonzero exit on failure, --seed for
  reproducible fuzzing, --fail-fast; nightly exercise.yml workflow that
  builds a fast-governance node and runs the suite.
- Regenerate subxt metadata for spec 134 and drop the removed
  Referenda/ConvictionVoting modules and disabled Scheduler calls.
- Add quiet logging mode so expected-failure transaction chatter does
  not drown the per-step report output.

Co-authored-by: Cursor <cursoragent@cursor.com>
Proof generation is fast enough on release builds; use --skip wormhole
on debug builds. The post-upgrade re-run now also covers wormhole.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread src/cli/exercise/scenarios/negative.rs Dismissed
Comment thread src/log.rs Dismissed
Comment thread src/log.rs Dismissed
illuzen and others added 2 commits July 3, 2026 19:57
Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the truncated secret prefix from check-nullifiers output and the
full secret / generated mnemonic from verbose multiround logs, so no
secret material reaches logs (flagged by CodeQL cleartext-logging).

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants