Fix batch 3 of the v0.4.0 review: account pass retries, lifecycle, and database outages

[AsyncAssassin]

Summary

Batch 3 of the fixes from the merged review of v0.4.0: operations. It covers the sync run lifecycle, the account pass, and behavior while the database is down. B18 got a short design first: which address failures are that address's and which are the provider's. There is one commit per fix.

Fix	What changes	Was
B18	An address whose provider fetch fails retryably, such as a timeout or a 5xx, waits on a retry list with the attempts and backoff of a run (5, from 30 s doubling), on its own schedule stored in the pass, while the others sync. It is recorded as failed once its attempts are spent. Due retries go first in each claim. Pages committed before a failure count as progress. Three addresses failing in a row are a provider outage, which costs them nothing; throttling fails the claim like Retry-After. While addresses wait, lastError says so (ADDRESS_RETRY, changeset 018). Failed and interrupted claims save the pass.	the claim failed at that address, and the next claim began there again. The run failed after five attempts, the addresses after that one were never synced, and every failed claim lost the progress of the pass
Disabled chain (K20)	Syncs skip an address whose chain is disabled, as they skip a disabled address. POST /api/v1/addresses/{id}/sync answers 404 Unsupported chain, and a run queued before fails without calling the provider.	every account sync that included such an address failed with it, and disabling the chain was not enough
B17	An interrupted or pool-rejected run is due again at once.	it waited a backoff that grew with every claim, up to 15 minutes after a deploy
B37	HTTP Basic reads users through Spring Security's UserCache. A user is fresh for a minute, a password mismatch reads the store again, and writes through the service apply at once. While the database is down, a user read in the last ten minutes stays accepted, re-read by one request at a time. A wrong password then gets 503, like an unknown name. The outbox gauges read counts that a background job refreshes.	every request with credentials and every scrape read the database, so the metrics vanished exactly when the database was down
B38	A database failure inside an Alchemy fetch is a database error of the run and leaves provider health alone.	it turned Alchemy DOWN and put raw exception text, which can quote SQL, into the health error
B23	Only the domain model's checks of event values (DomainInvariantException) count as bad provider data.	any IllegalArgumentException in ingestion failed the run terminally and blamed the provider

B37 goes beyond the agreed one-minute cache. A plain TTL would not have kept the metrics reachable during an outage, which is what the finding is about. So a cached user outlives a failed read by up to ten minutes, and a scrape no longer counts the outbox in the database. One request per user and minute still waits for the pool timeout while it reads the database again (B43, recorded).

Visible changes:

• a new last_requeue_reason value, ADDRESS_RETRY;
• max-attempts and the retry backoff now also apply to each address of an account run;
• an old password, or a user removed directly in the users table, keeps working for up to a minute; a new password works at once;
• ASSET_SYNC_OUTBOX_GAUGES_ENABLED and ASSET_SYNC_OUTBOX_GAUGES_FIXED_DELAY control the gauge refresh.

Verification

• ./gradlew clean check: 372 tests, 0 failures, 2 skipped (the env-gated Alchemy live smoke), no compiler warnings.
• Every fix has a test that fails when the fix is reverted. The review fixes alone add 18 such checks: 11 for the account pass, 6 for the user cache, 1 for the gauge job.
• A unit test fills every list of the pass with the longest errors and checks that the checkpoint fits the 16 KiB of run_checkpoint. That is why the retry list holds 50 addresses and the pass lists 10 failures.
• Live on the jar of this branch, under prod against real PostgreSQL and a stub bridge, with retry-backoff-base-delay=2s:
  • An address always answering 503, between two good ones: the run was FAILED after 29 seconds with 1 of 3 addresses failed terminally: <id>: failed 5 times: Provider returned HTTP 503. and failure_attempts 0. The bridge saw five requests for that address, and the other two addresses were stored.
  • An address that answered 503 twice and then a page: the run SUCCEEDED with no failure attempt spent.
  • An account whose first three requests failed: the claim failed as Provider unavailable: 3 addresses in a row failed, the last with: Provider returned HTTP 503. with failure_attempts 1, and the pass kept the three addresses with no attempt counted. The retry took them first, then the fourth, and SUCCEEDED.
  • A 429 with Retry-After, and one without, each failed the claim as throttling, five times until FAILED, rather than being put on one address. The address before each was fetched once.
  • docker pause on PostgreSQL: a scrape of /actuator/prometheus with credentials loaded a moment earlier answered 200 in 0.07 s. A minute later three concurrent scrapes all answered 200: one waited 30 seconds while it read the user again, the other two answered in 0.07 s. After docker unpause, reads worked again and db health was UP.

Review

One /code-review round ran on this PR, at medium, with a verifier for every candidate. It found 15 problems; 14 were confirmed, and the four most serious reproduced against PostgreSQL. All are fixed in three commits, without another round:

• 3555bfd, the account pass retries:
  • A single address left to retry took a full outage for its own failure and failed in 90 s. Now each address gets the attempts and backoff of a run.
  • Two stubborn addresses looked like an outage in every claim. The "two failures without a page" rule is gone.
  • Committed pages counted for nothing. They now count as progress.
  • A 429 or rate limit was put on one address. It now fails the claim.
  • Retries came seconds apart. Their due time is now stored per address, and due retries go first.
  • A full retry list stalled the scan. It now waits for the retries to drain.
  • Rolling back to 0.4.1 would break on ADDRESS_RETRY. An unknown reason now reads as none, with SQL for the rollback in docs/database.md.
  • A waiting run showed no error. lastError now says what it waits for.
  • A queued address run on a disabled chain still called the provider. It no longer does.
• d8b6101, the user cache:
  • It is now Spring Security's UserCache, so a password mismatch reads the store again.
  • A load that raced a write no longer stores the old user.
  • The outage fallback ends ten minutes after the last read.
  • A wrong password during an outage gets 503 like an unknown name, so the answer no longer tells which names were used.
• a8daa83: the gauge job has a switch and a delay, and both test profiles turn it off.

Recorded rather than done:

• B43: during an outage, one request per user and minute waits for the pool timeout while it reads the user again.
• B44: the gauge job shares the scheduler thread.
• Low items the review cut, listed in the plan: the changeset 018 table scan, a slow database blamed on Alchemy, and cleanups.

Left out

• B19, the wording recheck, is optional; the wording itself was done in batch 1. B20–B22 are optional and not done by default.
• Batch 4 follows: B24–B27, B30–B32, B35, B36, B39–B42, the docs items, then release 0.4.2.