Add quantus exercise chain smoke/fuzz suite#115
Open
illuzen wants to merge 4 commits into
Open
Conversation
Adds a scenario-based exercise command that drives every call path the CLI wraps against a live node, for pre-runtime-upgrade regression testing and CI: - Phases: reads, balances, reversible, multisig, recovery, preimage, governance, negative, seeded fuzz; opt-in wormhole and a governance-driven runtime-upgrade phase (--upgrade-wasm, requires a fast-governance node) with automatic post-upgrade re-run. - Negative scenarios pin expected rejections (incl. scheduler-disabled and removed-pallet canaries) so upgrades that loosen validation fail. - CI-friendly: --json report, nonzero exit on failure, --seed for reproducible fuzzing, --fail-fast; nightly exercise.yml workflow that builds a fast-governance node and runs the suite. - Regenerate subxt metadata for spec 134 and drop the removed Referenda/ConvictionVoting modules and disabled Scheduler calls. - Add quiet logging mode so expected-failure transaction chatter does not drown the per-step report output. Co-authored-by: Cursor <cursoragent@cursor.com>
Proof generation is fast enough on release builds; use --skip wormhole on debug builds. The post-upgrade re-run now also covers wormhole. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Remove the truncated secret prefix from check-nullifiers output and the full secret / generated mnemonic from verbose multiround logs, so no secret material reaches logs (flagged by CodeQL cleartext-logging). Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@ -0,0 +1,53 @@
feat: add
quantus exercisechain smoke/fuzz suiteSummary
Adds a single command —
quantus exercise— that runs a scenario-based smoke and fuzz suite against a live node, covering every call path the CLI wraps. It is designed as a pre-runtime-upgrade regression gate: run it before and after an upgrade and any extrinsic that stops working (or any validation that silently loosens) fails the suite with a nonzero exit code, making it CI-ready.What it covers
Utility::batch, transfer with tip, manual nonce,remark_with_eventset_high_security+ HS default-delay transfer + guardian cancelinitiate_recoverypinned as cleanNotRecoverable(CLI lackscreate_recovery— documented limitation)note_preimage+ status verify;request_preimagepinned asBadOriginadd_memberpinned asBadOrigin--skip wormholeon debug buildsSystem::set_codevia TechReferenda on a fast-governance node, poll for spec bump, then automatically re-run all phases against the upgraded runtimeNegative scenarios assert the specific expected error, so a runtime upgrade that accidentally loosens validation is caught, not just crashes.
Flags
--phases/--skip,--fuzz-iterations N(default 25),--seed <u64>(reproducible fuzzing; seed is always printed),--upgrade-wasm <path>,--upgrade-timeout-secs,--ephemeral-accounts N,--fail-fast,--json.Also in this PR
Referenda/ConvictionVotingpallets removed,Schedulerdispatchables disabled. The stalereferendaCLI modules and scheduler dispatch code are removed accordingly; compatibility table updated.src/log.rs): the exercise suite suppresses per-transaction chatter (spinners, expected-failure error banners) so the per-step report is the single source of truth.--verboserestores full output. Normal CLI commands are unaffected..github/workflows/exercise.yml): buildsquantus-nodefrom the chain repo withfast-governance, starts a dev node, runs the suite with--json, uploads report + node log as artifacts. Nightly +workflow_dispatch(with chain ref and fuzz-iteration inputs) rather than per-PR, since node build + QPoW block times dominate.Test plan
cargo +nightly fmt --check,clippy --all-targets(no warnings),cargo test --locked(210 passed)--upgrade-wasm) before relying on it for the real upgradeexercise.ymlrun on CI to validate timingKnown limitations
create_recoverywrapper).