[grempe]

Fixes #1251.
Part of #1111: Tessera now closes what it opens, but there's no way yet to wait for that cleanup (see "Not included").

This replaces #1252, which let callers pass in their own *sql.DB. As @mhutchinson pointed out there, the root bug is better fixed by Tessera closing what it opens, which also covers everyone who passes a DSN.

Why

We found this through our own log's stress testing. Its burn-in restarts the log in-process, and its tests and fuzzers open many logs in one process. Tessera never closed the MySQL pools it opened, so they added up to more connections than MySQL allows by default, and we had to cap the burn-in's restarts and raise max_connections. The same pattern turned out to be in all three drivers:

	Opens	Left open on each reopen, before this change
AWS driver	a MySQL pool per appender	about 2 goroutines and 1 MySQL connection (64 idle allowed)
AWS antispam	a MySQL pool	about 5 goroutines and 4 MySQL connections
GCP driver	a Spanner client, unless one is passed in	25–37 goroutines, plus its gRPC connections and sessions
GCP antispam	a Spanner client, unless one is passed in	the same
POSIX antispam (Badger)	a Badger database	17 goroutines, its memory, and its directory lock, so the same directory can't be opened again in that process
POSIX driver	nothing it keeps	—

Construction that failed part way leaked what it had opened too, as in my comment on #1251.

What changes

• Each appender closes what it opened once its context is done and its background jobs have returned. This follows the documented shutdown: call the shutdown function, then cancel the context. The background jobs now run under a sync.WaitGroup, and a goroutine closes the pool or client once they've returned.
  • AWS: the sequencer's MySQL pool.
  • GCP: a Spanner client the appender created. A client passed in Config.SpannerClient is left open. The appender no longer stores the client it creates in its Config, so the driver's other lifecycles don't share a client it closes.
  • The GCS client is left alone: its Close makes any later use panic, and it holds only idle HTTP connections.
• Failed construction closes what it opened.
  • newMySQLSequencer and the AWS NewAntispam close their pool when Ping, initDB or checkDataCompatibility fails.
  • The AWS and GCP newAppender close the sequencer when they fail.
  • The GCP Appender closes a client it created when initDB or newSpannerCoordinator fails.
• Each antispam gets Close() error, since the caller creates it.
  • The GCP antispam closes only a client it created, as its docs already say of one passed in.
  • The Badger antispam stops its compaction goroutine, waits for it, then closes the database, which releases the lock.
  • Calling Close twice is safe.

Behaviour changes

• After the appender's context ends, the LogReader's IntegratedSize and NextIndex return errors on AWS and GCP, since they read the closed pool or client. Reads of checkpoints, tiles and bundles are unaffected.
• A caller using an antispam should call Close after shutting the appender down and cancelling its context. One that doesn't is no worse off than before.

Not included

• A way to wait for the cleanup, the larger part of Storage lifecycle instances need a shutdown hook #1111. Cleanup happens shortly after the context ends, but the shutdown function doesn't wait for it. Making it waitable would mean the shutdown function stopping the background jobs itself, and drivers signalling when they've finished, which is an API decision for you.
• The mirror and migration writers. They also open pools and clients they don't close, but they're one-off runs, and their contexts' lifetimes are less clear.

Testing

• New unit tests, using CI's setups (MySQL for AWS, spannertest for GCP, temporary directories for POSIX). Each fails without its fix:
  • TestMySQLSequencerClosesPoolOnFailure and TestAppenderClosesSequencer (AWS);
  • TestNewAntispamClosesPoolOnFailure and TestAntispamClose (AWS antispam);
  • TestAppenderClosesSpannerClient (GCP: its own client, when the context ends and when newAppender fails; a caller's client stays open);
  • TestAntispamClose (GCP antispam: its own client closes; a caller's stays open);
  • TestAntispamClose (Badger: open, close and reopen the same directory three times, with all goroutines gone).
• Suites:
  • go test -race ./... passes on macOS, apart from TestConstructAndUseAppender (/tmp is a symlink) and TestPosixMirrorIntegration (TestPosixMirrorIntegration fails on macOS: its burst of connections overflows the listen queue #1253, fixed by integration/mirror: limit the test's connections #1254), which fail the same way on main here.
  • go test -p=1 -race ./storage/aws/... -is_mysql_test_optional=false passes against MySQL 8.4.
  • golangci-lint v2.13.2 reports 0 issues.
• Stress testing, with a harness that isn't part of this PR:
  • Reopening: a log reopened 30 times in one process on each driver, under -race, with 8 concurrent adders, re-adds of earlier entries, and every third round cancelled mid-flight without calling shutdown.
    • Setups: AWS against MySQL 8.4 and an S3-compatible store (RustFS), as a MySQL user capped at 30 connections; GCP against the Cloud Spanner emulator and fake-gcs-server, and against spannertest; POSIX with the Badger antispam.
    • With this change: nothing leaked. Every acknowledged entry (about 6,600 per run) ended up in the log at its index, and every re-add came back as a duplicate at its original index.
    • On main: the same runs fail at the second reopen. AWS leaks 9 to 13 MySQL connections a reopen and hits the connection limit by the fourth, GCP leaks about 50 goroutines a reopen on the emulator, and Badger can't reopen its directory.
  • Cancelled construction: 40 openings on each driver, under contexts that end at a random point within 300 ms. With this change nothing was left behind. On main, AWS leaked 28 MySQL connections, GCP 2,000 goroutines, and Badger couldn't reopen at all.
  • Edge cases: Close twice, an add after Close, Close while the follower runs, and the LogReader after the context ends all return errors, with no panics or races.

This touches storage/aws/aws.go, as #1257 does, and the two merge cleanly in either order. #1252 conflicts with it, and I'll close #1252.

---

Reviewed and submitted by me, a human. The investigation, the fix and its testing were developed with the assistance of an AI coding agent.