Skip to content

docs(rebalance): add a runbook and fix the alert's next-step text - #80

Merged
robertleifke merged 1 commit into
mainfrom
ops/rebalance-runbook
Sep 23, 2026
Merged

robertleifke merged 1 commit into
mainfrom
ops/rebalance-runbook

Conversation

@robertleifke

Copy link
Copy Markdown
Contributor

check --alert fires and the procedure to act on it lived only in a session
transcript. This venue has twice lost days to an alert that fired correctly while
nobody acted on it, so an alert whose response isn't written down is the same
failure shape.

The bug in the alert text

The message told the operator to:

Withdraw USDC from the subaccount to the rebalance signer

The chain cannot do that. action.data is abi.encode(address asset, uint256 amount) —
64 bytes, no recipient field — so a withdrawal always pays
Matching.subAccountToOwner(15), the MM wallet 0x3448ac0A…. Never the rebalance
signer 0x1661AA54…. An operator following it literally goes looking for an argument
that doesn't exist, at exactly the moment they're under pressure.

It is two moves: withdraw (pays the owner), then forward to the signer. Fixed, and
the message now points at the runbook.

Pinned, and the pin is verified

Two tests assert the corrected wording. I reverted the message to the old text and
confirmed both fail, then restored it and confirmed both pass — a test that can't
fail would have been worse than none here. 31/31 green.

The runbook

services/rebalance/RUNBOOK.md — the four steps with addresses, and the trap list.
Notable contents:

  • SubAccounts.ownerOf(15) returns Matching, not the owner. Read
    Matching.subAccountToOwner(15).
  • Withdrawal amounts are in the token's 6 decimals; subaccount balances read back in
    18. Don't copy a ledger figure into the amount.
  • Step 2 (forward MM wallet → signer) is the one most likely to be skipped: the alert is
    about sub 15 and that step touches neither the subaccount nor the CLI.
  • A successful op can report as failed via RPC replica lag — check the chain before
    retrying, or pay twice.
  • Don't route funds to the legacy CashAsset 0x6B232A21… (~2 USDC held vs ~69 claimed).
    The escrow in this flow, 0x364058aF…, is a different contract and is solvent
    (326 held vs 321.0004 claimed, summed across all 19 subaccounts).

Both withdrawal routes verified on chain

Not asserted from the code — checked against history on this escrow (333 USDC in,
6.999575 out across 2 withdrawals, 326 held; the arithmetic closes):

Route Selector Applies to
Matching.verifyAndMatch 0x74d906c3 deposited subaccounts — sub 15
escrow direct 0x0ad58d2f subaccounts not deposited in Matching

The precedent for sub 15's route is sub 19's owner withdrawing 1.999575 USDC in
0xffff17a9…5e0175 (block 51301502) — same escrow, same structure, owner-signed.

No behaviour change beyond the alert string.

The alert said to withdraw "to the rebalance signer", which the chain cannot
do -- withdrawal data is (asset, amount) with no recipient, so it always pays
the subaccount owner. It is two moves. Withdrawal route confirmed against the
sub 19 precedent, tx 0xffff17a9814e32bedd2bcdeffb214dc1293bac59d634560af73200293a5e0175.
@robertleifke
robertleifke merged commit df943c0 into main Sep 23, 2026
1 check passed
@robertleifke
robertleifke deleted the ops/rebalance-runbook branch September 23, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant