feat(escrow_manager): reclaim idle escrow via thaw and withdraw

crates/bin/escrow_manager/README.md

@@ -39,10 +39,75 @@ the effective margin is wider.
 
 Lowering the margin frees capital but leaves less headroom to absorb query volume between cycles.
 Note that total deposits are capped at 10,000 GRT per cycle (`MAX_ADJUSTMENT`), which bounds how
-fast a thin margin can be refilled. **The manager never withdraws**, so raising
-`balance_fill_factor` only affects receivers still being topped up — balances already above their
-target drain only as receivers collect. Validate changes with `dry_run: true` first, and watch
-`escrow_target_grt` against `escrow_balance_grt`.
+fast a thin margin can be refilled. Unless reclamation is enabled (see below) the manager only ever
+deposits, so raising `balance_fill_factor` only affects receivers still being topped up — balances
+already above their target drain only as receivers collect. Validate changes with `dry_run: true`
+first, and watch `escrow_target_grt` against `escrow_balance_grt`.
+
+## Reclaiming Escrow
+
+Deposits are one-directional, so a receiver's balance is a high-water mark of its past debt. When
+query volume drops the target falls but the balance does not follow, leaving escrow idle. Setting
+`withdraw_enabled: true` lets the manager reclaim that idle escrow back to the payer wallet.
+
+| Field | Default | Meaning |
+|---|---|---|
+| `withdraw_enabled` | `false` | Reclaim escrow above the target balance |
+| `withdraw_margin` | `0.25` | Headroom kept above target before reclaiming |
+| `min_withdraw_grt` | `500` | Minimum excess required to start reclaiming |
+
+`dry_run: true` covers reclamation along with everything else: the manager logs the full plan each
+cycle — grep for `dry run: skipping` — without sending any transaction, deposits included.
+
+Escrow is only thawed above `target * (1 + withdraw_margin)`, so at the defaults the manager holds
+roughly 1.7x debt and reclaims the rest. There is no separate handling for abandoned accounts: a
+receiver with no debt has a target of 2 GRT, so it falls out of the same rule as the large-excess
+end of the range.
+
+Reclaiming is not instant. It goes through the escrow contract's `thaw` and `withdraw`, so funds
+only return after the contract's thawing period — **28 days on mainnet**.
+
+Each cycle a receiver gets exactly one action, decided in this order:
+
+1. **Withdraw** — a thaw has matured, so the escrow returns to the payer.
+2. **Deposit** — the balance net of anything thawing is short of the target, so top it up.
+3. **Thaw** — nothing is thawing and the balance clears `target * (1 + withdraw_margin)` by at
+   least `min_withdraw_grt`, so start reclaiming the excess.
+4. Otherwise nothing.
+
+Because the branches are exclusive, the three transaction batches share no receivers and none
+depends on another having landed first.
+
+A thaw is never resized once running: the contract cannot grow one without resetting its 28-day
+timer, so escrow that builds up meanwhile waits for the next round. Debt that grows during a thaw
+is answered by the deposit branch instead, which is why the thawing amount does not need to track
+it. The trade is that money occasionally round-trips through the payer wallet — the full thawed
+amount leaves at maturity and a deposit covers the difference — rather than being kept in place.
+That costs one deposit against the alternative of rewriting the thaw every cycle for 28 days.
+
+Two consequences worth knowing:
+
+- Because the thawing period matches the 28-day receipt window the debt estimate is built from, the
+  debt picture rolls over completely before a thaw matures. The margin is what absorbs debt growth
+  over that period, so size it against how much a single receiver's debt can grow in a month — not
+  just against how much idle capital you want to reclaim.
+- A receiver collecting against its escrow shrinks a pending thaw, and a collection that drains the
+  balance cancels it outright. This is normal and needs no intervention, but it means a reclamation
+  in flight is not guaranteed to complete.
+
+Funding always uses the balance net of anything thawing, matching the escrow contract's own
+`getBalance`, so coverage holds throughout a reclamation and across the withdrawal that ends it.
+
+Setting `withdraw_enabled` back to `false` is inert with respect to the thaw itself: anything
+already thawing is left exactly as it is and logged as a warning each cycle, rather than being
+cancelled or withdrawn. It is not free, though — that escrow stays permanently excluded from the
+effective balance and is never reclaimed, so the manager tops the account up around it. To stop
+reclaiming without stranding work in flight, leave it enabled and raise `withdraw_margin` instead,
+which prevents new thaws while letting pending ones complete.
+
+Validate with `dry_run: true` first, and watch `escrow_total_thawing_grt` and
+`escrow_thawing_count`. Given the 28-day round trip, that observation period is the only practical
+way to tune these values.
 
 ## Sender and Signers
 
@@ -132,12 +197,21 @@ curl http://localhost:9090/metrics
 | `escrow_total_balance_grt` | Gauge | Total escrow balance across all receivers |
 | `escrow_total_target_grt` | Gauge | Total target escrow balance across all receivers |
 | `escrow_total_adjustment_grt` | Gauge | Total GRT deposited in the last cycle |
+| `escrow_total_thawing_grt` | Gauge | Total escrow pending withdrawal across all receivers |
 | `escrow_receiver_count` | Gauge | Number of receivers being tracked |
+| `escrow_thawing_count` | Gauge | Number of receivers with escrow pending withdrawal |
 | `escrow_loop_duration_seconds` | Histogram | Duration of each polling cycle |
 | `escrow_debt_grt{receiver}` | Gauge | Outstanding debt per receiver |
 | `escrow_balance_grt{receiver}` | Gauge | Escrow balance per receiver |
 | `escrow_target_grt{receiver}` | Gauge | Target escrow balance per receiver |
 | `escrow_adjustment_grt{receiver}` | Gauge | Last adjustment per receiver (0 if at or above target) |
+| `escrow_thawing_grt{receiver}` | Gauge | Escrow pending withdrawal per receiver |
 | `escrow_deposit_ok` | Counter | Successful deposit transactions |
 | `escrow_deposit_err` | Counter | Failed deposit transactions |
 | `escrow_deposit_duration` | Histogram | Deposit transaction duration |
+| `escrow_thaw_ok` | Counter | Successful thaw transactions |
+| `escrow_thaw_err` | Counter | Failed thaw transactions |
+| `escrow_thaw_duration` | Histogram | Thaw transaction duration |
+| `escrow_withdraw_ok` | Counter | Successful withdrawal transactions |
+| `escrow_withdraw_err` | Counter | Failed withdrawal transactions |
+| `escrow_withdraw_duration` | Histogram | Withdrawal transaction duration |

crates/bin/escrow_manager/src/config.rs

@@ -49,6 +49,24 @@ pub struct Config {
     /// Must be in the range (0, 1].
     #[serde(default = "default_balance_fill_factor")]
     pub balance_fill_factor: f64,
+    /// Reclaim escrow that sits above a receiver's target balance. When disabled the manager only
+    /// ever deposits, and a receiver's balance is a high-water mark of its past debt. Reclaiming
+    /// takes effect via `thaw` and `withdraw`, so funds only return after the escrow contract's
+    /// thawing period (28 days on mainnet) has elapsed.
+    #[serde(default)]
+    pub withdraw_enabled: bool,
+    /// Headroom kept above the target balance before any escrow is reclaimed: escrow is only
+    /// thawed above `target * (1 + withdraw_margin)`. This absorbs debt growth over the thawing
+    /// period and provides hysteresis against the deposit step ladder, so it should comfortably
+    /// exceed the debt growth expected for a single receiver over that period. Must be in the
+    /// range [0, 1].
+    #[serde(default = "default_withdraw_margin")]
+    pub withdraw_margin: f64,
+    /// Minimum excess, in whole GRT, required to start thawing a receiver's escrow. Excess below
+    /// this is left alone, since reclaiming it is not worth the thawing period. It gates starting a
+    /// thaw only: a thaw already running is never resized, so the threshold is not reapplied to it.
+    #[serde(default = "default_min_withdraw_grt")]
+    pub min_withdraw_grt: u64,
 }
 
 fn default_port_metrics() -> u16 {
@@ -59,6 +77,14 @@ fn default_balance_fill_factor() -> f64 {
     0.8
 }
 
+fn default_withdraw_margin() -> f64 {
+    0.25
+}
+
+fn default_min_withdraw_grt() -> u64 {
+    500
+}
+
 #[derive(Debug, Deserialize)]
 pub struct Kafka {
     pub config: BTreeMap<String, String>,

crates/bin/escrow_manager/src/contracts.rs

@@ -1,6 +1,7 @@
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
 
 use alloy::{
+    eips::BlockId,
     network::EthereumWallet,
     primitives::{keccak256, Address, BlockNumber, Bytes, U256},
     providers::{DynProvider, Provider as _, ProviderBuilder, WalletProvider},
@@ -38,6 +39,7 @@ sol!(
 use GraphTallyCollector::{GraphTallyCollectorErrors, GraphTallyCollectorInstance};
 
 pub struct Contracts {
+    provider: DynProvider,
     payments_escrow: PaymentsEscrowInstance<DynProvider>,
     graph_tally_collector: GraphTallyCollectorInstance<DynProvider>,
     token: ERC20Instance<DynProvider>,
@@ -63,6 +65,7 @@ impl Contracts {
             GraphTallyCollectorInstance::new(graph_tally_collector, provider.clone());
         let token = ERC20Instance::new(token, provider.clone());
         Self {
+            provider,
             payments_escrow,
             graph_tally_collector,
             token,
@@ -74,6 +77,13 @@ impl Contracts {
         self.payer
     }
 
+    /// Collector every escrow call in this module targets. Escrow accounts are scoped by
+    /// `(payer, collector, receiver)`, so anything reading account state has to filter on this
+    /// exact address or it will plan against a different collector's balances.
+    pub fn collector(&self) -> Address {
+        *self.graph_tally_collector.address()
+    }
+
     pub async fn allowance(&self) -> anyhow::Result<u128> {
         self.token
             .allowance(self.payer(), *self.payments_escrow.address())
@@ -133,13 +143,97 @@ impl Contracts {
         Ok(block_number)
     }
 
-    pub async fn authorize_signer(&self, signer: &PrivateKeySigner) -> anyhow::Result<()> {
-        let chain_id = self
-            .graph_tally_collector
-            .provider()
-            .get_chain_id()
+    /// Timestamp of the latest block, for planning against contract-side time checks.
+    ///
+    /// Block timestamps are non-decreasing, so this is a lower bound on the timestamp of whatever
+    /// block a transaction sent now lands in. Planning against it can only be conservative.
+    pub async fn latest_block_timestamp(&self) -> anyhow::Result<u64> {
+        let block = self
+            .provider
+            .get_block(BlockId::latest())
+            .await
+            .context("get latest block")?
+            .ok_or_else(|| anyhow!("no latest block"))?;
+        Ok(block.header.timestamp)
+    }
+
+    /// Start thawing `tokens` for each receiver, batched into one transaction.
+    ///
+    /// Only ever called for receivers with nothing currently thawing. `thaw` restarts the thawing
+    /// period on every call, and the contract will not grow a running thaw without resetting its
+    /// timer either, so escrow that builds up while one is in flight waits for the next.
+    pub async fn thaw_many(
+        &self,
+        thaws: impl IntoIterator<Item = (Address, u128)>,
+    ) -> anyhow::Result<BlockNumber> {
+        let calls: Vec<Bytes> = thaws
+            .into_iter()
+            .map(|(receiver, tokens)| {
+                self.payments_escrow
+                    .thaw(
+                        *self.graph_tally_collector.address(),
+                        receiver,
+                        U256::from(tokens),
+                    )
+                    .calldata()
+                    .clone()
+            })
+            .collect();
+
+        let receipt = self
+            .payments_escrow
+            .multicall(calls)
+            .send()
             .await
-            .context("get chain ID")?;
+            .map_err(decoded_err::<PaymentsEscrowErrors>)?
+            .with_timeout(Some(Duration::from_secs(30)))
+            .with_required_confirmations(1)
+            .get_receipt()
+            .await?;
+
+        receipt
+            .block_number
+            .ok_or_else(|| anyhow!("invalid thaw receipt"))
+    }
+
+    /// Withdraw each receiver's matured thawing amount back to the payer, batched into one
+    /// transaction.
+    ///
+    /// The contract requires the maturity timestamp to be strictly in the past and reverts with
+    /// `PaymentsEscrowStillThawing` otherwise, which in a multicall fails the whole batch. Callers
+    /// must only include receivers whose thaw has definitely matured.
+    pub async fn withdraw_many(
+        &self,
+        receivers: impl IntoIterator<Item = Address>,
+    ) -> anyhow::Result<BlockNumber> {
+        let calls: Vec<Bytes> = receivers
+            .into_iter()
+            .map(|receiver| {
+                self.payments_escrow
+                    .withdraw(*self.graph_tally_collector.address(), receiver)
+                    .calldata()
+                    .clone()
+            })
+            .collect();
+
+        let receipt = self
+            .payments_escrow
+            .multicall(calls)
+            .send()
+            .await
+            .map_err(decoded_err::<PaymentsEscrowErrors>)?
+            .with_timeout(Some(Duration::from_secs(30)))
+            .with_required_confirmations(1)
+            .get_receipt()
+            .await?;
+
+        receipt
+            .block_number
+            .ok_or_else(|| anyhow!("invalid withdraw receipt"))
+    }
+
+    pub async fn authorize_signer(&self, signer: &PrivateKeySigner) -> anyhow::Result<()> {
+        let chain_id = self.provider.get_chain_id().await.context("get chain ID")?;
         let deadline_offset_s = 60;
         let deadline = U256::from(
             SystemTime::now()