inbox: widening poll backoff on an unreachable server, without deactivating the mailbox

[Ahmustufa]

PR 2 of P1.8 (PR 1 was #253). The last sub-item, and the one whose brief was wrong.

The problem is not what the parity plan says it is

The plan says "widening backoff … without deactivating the mailbox", implying something deactivates it. Nothing does — re-verified here, not taken on trust: mailboxes.status is written by exactly one query, reached from one function, reachable only from Pause/Resume, only ever with paused/active.

So an hour-long outage does not cost the connection. It costs ~60 dial+auth attempts and ~20 dead-letter rows per mailbox per hour — and that reconnect hammering is itself what provokes the per-IP sign-in throttles the fleet architecture exists to avoid, while burying real failures in dead-letter noise.

The trap, avoided: status = 'error' is a documented valid value, and writing it would have stopped sending too, because MailboxExists and ListActiveMailboxes both gate on active. The backoff therefore lives in two new columns read by one query. Verified: RecordInboxPollFailure writes only inbox_poll_failures and inbox_poll_retry_after, workspace-pinned — it is structurally incapable of touching status.

Four signals, three answers

ClassifyConnectFailure never inspects error text — every branch is a sentinel, a type, or a status code.

Signal	Class	Schedule
Refused / reset / EOF / DNS failure	transport	widening ladder
Accepts-then-stalls (deadline, net.Error.Timeout)	transport	widening ladder
TLS handshake failure	transport	widening ladder
Provider 429 / 5xx	transport	widening ladder
Wrong password, revoked token, no usable mechanism, 401/403	auth	straight to the cap + SkipRetry
SSRF guard refusal	policy	straight to the cap
Our own context.Canceled	aborted	nothing recorded
Non-dial failure (cursor write, job fetch)	—	nothing recorded

Auth goes to the cap immediately, and the argument is the same one already made one level down in this codebase: imapauth.go caps itself at two attempts per connection because "walking every mechanism would turn one wrong password into four rejected sign-ins per poll, every three minutes, which is how a provider decides to lock an account." A retry cannot fix a wrong password, and each one is another rejected sign-in.

Transport is read before the auth sentinel, deliberately: a connection dropping mid-LOGIN is wrapped as auth but is transport in substance, and reading the sentinel first would park a mailbox on a flaky network at the hour cap as though its password were wrong. A test pins that ordering.

The ladder

transport/unknown:  3m → 6 → 12 → 24 → 48 → 60m (clamped)
auth/policy:        [60m]

The first rung equals inboxSweepInterval, so one failed poll changes nothing — a blip costs a healthy mailbox no latency. An hour's outage drops from ~20 fan-outs to 5.

The cap is the requirement, not the rungs. An uncapped exponential reaches a day, then a week, and a mailbox nobody checks stops detecting replies for good — a deactivation with extra steps. Capped, it is still polled hourly and recovers on its own.

What a backed-off mailbox can still do

Everything except be fanned out this tick. It still sends — sequence steps, warm-up, manual replies. status stays active. It rejoins automatically when retry_after passes, and pause → resume clears the counter, so an operator who fixes a password doesn't wait out the hour. (That reset was added beyond the sketch; without it there was no prompt manual recovery.)

Two details worth the review time

The delay is computed on the database clock — now() + make_interval(...) — because ListActiveMailboxes compares against the database clock. Computing it host-side would have let the exact host↔DB skew that bit this project twice decide whether a mailbox is due. No time.Now() appears in any new assertion; every time test measures the delta the database computed.

The ladder crosses the wire as seconds and is validated on both sides. An empty ladder would leave retry_after NULL and silently disable the whole feature.

coreapi.Client was not widened — PollBackoffCore is consumer-defined and resolved by type assertion once at wiring time, matching the neighbouring WarmupEvidenceClient check. Per-message assertion is what let a missing capability degrade in silence before.

Scope extended once, deliberately

graphinbox.go reported every non-2xx as fmt.Errorf("unexpected status %d"), so a revoked M365 token would have classified as unknown and taken the gentle ladder instead of the cap — half the feature, invisibly. Now carried as *APIError, same as the send path. Gmail needed nothing; its client already returns *googleapi.Error and the wrapper %ws it, which was verified rather than assumed.

Verification

Gate	Result
sqlc generate	no drift
go build / go vet / go vet -tags=integration	clean
gofmt -l internal cmd	empty
golangci-lint 2.12.2 (pinned, cache cleaned)	0 issues
TZ=UTC go test -race -count=1 -p 4 -tags=integration ./...	102 packages, 0 FAIL, 0 races

Rebased onto b5d2571 (#242) and re-verified; the migration remains the newest, so no version collision. Guarantee-carrying tests run explicitly: TestPollBackoffSuppressesSchedulingButNeverEligibility, TestAnUnreachableServerBacksOffWithoutDeactivatingTheMailbox, TestARejectedCredentialCapsImmediatelyAndStillSends, TestPollBackoffMigrationRollsBack — all PASS.

Every guard was removed and watched to fail, then restored. The classifier is driven through the real IMAP client against PR 1's in-process server — no mocked reader; a new stopListening closes the listener so the next dial gets a real ECONNREFUSED from the kernel. Migration up→down→up verified, including that a pre-existing mailbox lands eligible after the up, so an upgrade does not silently stop polling every mailbox in the install.

Flagged, not fixed

1. Pre-existing tenancy gap, found in passing and left alone: MailboxExists is WHERE id = $1 AND status = 'active' with no workspace_id filter. It is an existing recorded exception and was leaned on as the "still sends" gate, not changed. Independently verified. Worth its own ticket.
2. GetInboxPollJob failures are not backed off. For Gmail/M365 that call performs the OAuth refresh in the control plane, so a revoked refresh token still means a rejected token request on every sweep — real hammering this does not stop. Fixing it properly means classifying in the control plane and returning a typed error.
3. ConnectFailureAborted has a bounded blind spot. Cancellation is only observable on the legs taking a context; once a session is up, a cancel surfaces as an i/o timeout and classifies as transport — one rung, cleared by the next success. Found because the first version of that test failed; the test was narrowed and the limitation written into the doc rather than the test deleted.
4. Nothing surfaces the failure count or retry time in the API or UI. Counters are persisted and the classification is logged (ids and class only — never error text or anything derived from a credential), so a genuine failure is visible to an operator with database or log access, not to a user. That is the obvious PR 3, and it pairs with the permanently-0 mailbox-error tile.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VECELVAKe8Xcp7GH9wGR5t