Answer 503 while the database is down, also behind HTTP Basic (B02)

[AsyncAssassin]

Summary

B02 from the merged review of v0.4.0, with D05 and D10 from the same review; 0.4.1 is released once all fixes are merged. docs/api.md promises 503 database-unavailable while PostgreSQL is unavailable, but the service answered otherwise:

• local and test: 7 of the 9 endpoints answered 500 internal-error. A transaction that cannot get a connection fails with CannotCreateTransactionException, and a rollback on a connection the outage broke fails with TransactionSystemException; neither is a DataAccessException. An SQL error Spring cannot classify, such as a write to a read-only database, stayed a jOOQ exception and got 500 as well.
• demo, prod, and every other protected profile: a request with valid credentials got 401 with a Basic challenge, because HTTP Basic could not read its user store (InternalAuthenticationServiceException).
• A server that stops answering (paused container, network partition): a query in flight waited forever, because pgjdbc has no socket timeout by default and statement_timeout needs a live server.

What changes:

• One rule, isDatabaseFailure(), decides what a database failure is for the API, the credential check, and the sync worker: a Spring DataAccessException, a transaction that could not begin or roll back, or an exception whose direct cause is an SQLException. Other transaction exceptions, such as UnexpectedRollbackException, stay 500. A sync run that meets such a failure records Database error (<class>). instead of Unexpected error (<class>)..
• The entry point answers a user store failure caused by the database with the same 503, and any other with the generic 500, both without a challenge. A request without credentials keeps its 401, and so does a username the store cannot hold, such as one with a NUL character. Both paths write the same database_operation_failed line.
• The pool sets socketTimeout=40, above the 30-second statement_timeout, and connectTimeout=5. The 30-second Hikari connection-timeout stays: a shorter one would give false 503s under load.
• Docs (D05, D10):
  • docs/failure-modes.md section 6 is the one full description of the outage. It covers the timeouts, and a COMMIT that the socket timeout cuts off but that may still commit, with what that means for a retry. It also notes that every request with credentials waits for the pool during the outage. The other files link to it.
  • The immutable-conflict examples show the fields the service writes, in its order.
  • The error list names the 404 for an unsupported asset and the 409 for a duplicate account.
  • docs/failure-modes.md names the meters and log lines that exist instead of promising them "when metrics are implemented".

CHANGELOG.md lists the changes under [Unreleased] → Fixed.

Verification

• ./gradlew clean check: 280 tests, 0 failures, 2 skipped (the env-gated Alchemy live smoke).

• DatabaseOutageIntegrationTests starts its own PostgreSQL and boots a test and an e2e context against it:

  • with the database up, the operator's credentials work, and a username with a NUL character gets 401 with the challenge;
  • the pool carries the connect timeout and a socket timeout above the statement_timeout its connections run with;
  • after the container stops, all 9 endpoints answer 503 database-unavailable without a challenge in both contexts, the anonymous request keeps its 401, and health turns 503.

  On the first version's handlers the same test reports 7 × 500 in test and 9 × 401 in e2e. It now runs in 9 s instead of 29 s.

• Unit tests use the real exception classes:

  • CannotCreateTransactionException, TransactionSystemException, and a jOOQ exception around an SQLException give 503; UnexpectedRollbackException and jOOQ's TooManyRowsException give 500;
  • the entry point gives 503 for a lookup that failed on CannotGetJdbcConnectionException, 401 with the challenge for one that failed on a DataIntegrityViolationException, and 500 otherwise.

• SyncApiIntegrationTests: a TransactionSystemException during a run leaves it QUEUED with Database error (TransactionSystemException)..

• Mutation check: dropping the NUL guard, the SQLException rule, or the worker rule, a socket timeout below statement_timeout, or a misspelt connectTimeout each turns a test red.

• Live demo against its own PostgreSQL container, the main jar against this branch:

  Scenario	main	This branch
  database up, username with a NUL character	401 with challenge	401 with challenge
  docker stop, demo-reader GET	401 with challenge	503 database-unavailable, no challenge
  docker stop, demo-operator POST	401 with challenge	503 database-unavailable, no challenge
  docker stop, no credentials	401 with challenge	401 with challenge
  docker start again	404 (recovered)	404 (recovered)
  docker pause with a query in flight	no answer after 100 s	503 after 40 s

  The docker pause row bypasses Hikari's connection check (-Dcom.zaxxer.hikari.aliveBypassWindowMs=3600000) to reproduce a connection taken right before the pause. A request that waits for a new connection gets its answer after the 30-second Hikari connection-timeout in both versions.

A regression review of the first version (/code-review) found that a NUL username on a healthy database got the outage answer. It also found untranslated jOOQ errors, the worker's run error, the COMMIT caveat, and weaker tests and docs; all are fixed in 0665722. Left as a documented limit: every request with credentials reads the user store, so during an outage even authenticated metrics requests wait for the pool. A credential cache or a separate management port would change that, and it is a decision of its own.

This PR, #15, #16, #18, and #19 each add a section under [Unreleased] in CHANGELOG.md, so each later merge needs a one-file rebase. The other files merge cleanly with all of them.

[AsyncAssassin]

Combined into #23 together with the other batch-1 fixes, so they merge without a rebase per PR. This description keeps the detailed evidence for its fix.