Skip to content

storage/aws: keep idle MySQL connections by default - #1244

Merged
roger2hk merged 1 commit into
transparency-dev:mainfrom
grempe:mysql-idle-conns
Nov 5, 2026
Merged

roger2hk merged 1 commit into
transparency-dev:mainfrom
grempe:mysql-idle-conns

Conversation

@grempe

@grempe grempe commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1243.

newMySQLSequencer and NewAntispam applied MaxIdleConns whenever it was >= 0. A config that left it unset therefore called SetMaxIdleConns(0), and the pool kept no idle connections: every query opened a new MySQL connection and closed it afterwards.

This change applies it whenever it's != 0. An unset value then keeps the database/sql default, which is currently 2, and a negative value still keeps no idle connections, as database/sql does for any n <= 0. MaxOpenConns and MaxIdleConns are now documented on both aws.Config and AntispamOpts.

Testing

  • New TestMySQLSequencerIdleConnections and TestAntispamIdleConnections. Each makes 10 sequential queries on a new pool and checks DBStats.MaxIdleClosed:
    • with the setting unset, none of the 10 queries closes its connection; on main, all 10 do;
    • with -1, all 10 do; on main, none do, because main ignores a negative value.
  • Mutants: setting either half back to >= 0, or to > 0, fails only that half's test.
  • Race tests: go test -race ./storage/aws/... -is_mysql_test_optional=false against MySQL 8.4 passes on Go 1.26.10, and the tests also pass on Go 1.27.3.
  • Lint: golangci-lint v2.13.2 reports 0 issues.
  • Full suite: it passes on macOS apart from TestConstructAndUseAppender (/tmp is not a directory), which fails the same way on main here.
  • Under load: 100,000 entries through the AWS driver to RustFS and MySQL 8.4. The table counts the MySQL server's new connections:
MaxIdleConns main this change
unset 1,697–1,774 (3 runs) 13–177, median 20 (5 runs)
-1 23 1,699–1,804 (2 runs)
64 5 5

The spread when unset comes from the default's 2 idle connections: when more queries run at once, the extra connections are closed afterwards.

A behaviour change to note: an explicit MaxIdleConns: 0 used to keep no idle connections; it now keeps the default, and -1 keeps none. The same goes for --db_max_idle_conns=0 in the AWS conformance and migrate binaries. Their help text ("defaults to 2") still holds, so this PR leaves them unchanged.


Reviewed and submitted by me, a human. The investigation, the fix and its testing were developed with the assistance of an AI coding agent.

@grempe

grempe commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main and moved the new tests to the middle of their files, so this merges cleanly with #1252 and the other open PRs in any order. The change itself is the same.

Comment thread storage/aws/aws.go Resolved
Comment thread storage/aws/aws.go Resolved
Comment thread storage/aws/antispam/aws.go Resolved
@grempe

grempe commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Switched to != 0 with your comments, added your doc comments to both aws.Config and AntispamOpts, and rebased onto main. The tests now also check that -1 keeps no idle connections. I applied the first suggestion by hand: its range ends just above MaxIdleConns int, so committing it as-is would have declared the field twice. As for the conformance and migrate binaries, --db_max_idle_conns=0 now means the default too; their help text still holds, so I've left them alone, but I'm happy to spell out zero and negative values there.

newMySQLSequencer and NewAntispam applied MaxIdleConns whenever it was
>= 0, so a config that left it unset called SetMaxIdleConns(0) and the
pool kept no idle connections: every query opened a new connection and
closed it afterwards. Under load that meant about 1,700 new MySQL
connections per 100,000 entries.

Apply it whenever it is != 0, so that an unset value keeps the
database/sql default, and a negative value still keeps no idle
connections. Document MaxOpenConns and MaxIdleConns on aws.Config and
AntispamOpts.
@roger2hk
roger2hk merged commit dd633a0 into transparency-dev:main Nov 5, 2026
20 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS: an unset MaxIdleConns keeps no idle MySQL connections, so every query opens a new one

2 participants