Skip to content

fix: steady the release tests and add cheap database upkeep - #42

Merged
bashgeek merged 4 commits into
mainfrom
fix/release-prep-flaky-tests-db
Nov 1, 2026
Merged

bashgeek merged 4 commits into
mainfrom
fix/release-prep-flaky-tests-db

Conversation

@bashgeek

@bashgeek bashgeek commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Summary

Last prep before tagging a release.

  • It fixes the flaky tests that failed the v1.1.3 and v1.2.0 release runs.
  • It adds the cheap database upkeep items from the large-install review.
  • It gives Postgres enough shared memory in the shipped compose files.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Docs / config only

Changes

Flaky tests

  • Fixed sleeps replaced: 13 fixed 300 to 500 ms sleeps in the ingest, transaction and flamegraph tests, and 2 in internal/ingest/txbuffer_test.go, raced the 200 ms ingest flush. They now use waitPersisted, which waits until the buffer reports the rows committed, with a 10 s deadline.
  • "Nothing stored" tests no longer wait: the handler drops session-only envelopes, malformed profiles and profiles from projects with profiling off before anything reaches a buffer. Those tests now assert on the buffer's counters instead of sleeping.
  • Makefile: COMPOSE ran docker compose version on every make call. That slowed the Makefile tests on CI runners until they hit bun's 5 s default timeout. The variable is now deferred and kept out of the exported environment, so only the db* targets call Docker.
  • Bun Makefile tests: they get a 30 s timeout, and a new test puts a stub docker on PATH to check that migrate-create, build, lint and a mistyped target never call it.

Migration 0038 (no table rewrites; every statement can safely run again)

  • Drops logs_attributes. No query uses a JSONB operator on log attributes, and its GIN entries were a large share of log insert time and WAL.
  • Adds issues_regressed_at for the alert check that runs every 15 s.
  • Vacuums and analyzes events, event_tags, transactions, spans, logs and issues at 2% change instead of 20%.

Compose

  • shm_size: 1g for Postgres in docker-compose.yml and in the compose file install.sh writes. Docker's default /dev/shm of 64 MB is too small for parallel queries.

Left out on purpose: I also tried replacing the usage_update statement triggers with a row trigger limited to project_id and received_at, so the grouper's per-event update would never fire them. A single UPDATE that moves many rows then fires the row trigger once per row. Each firing updates the same counter row again in one transaction, so the cost grows quadratically; a retention test that moves 205k rows hung. The right fix for that overhead is batching the grouper.

Test plan

  • Flaky tests: the affected API and ingest tests passed three times, each in a fresh process.
  • Migration 0038: TestMigration0038IsIdempotent re-runs the whole migration. It checks that logs_attributes is gone and issues_regressed_at exists, that the planner uses the new index for regressed_at > $1, and that the autovacuum settings are on all six tables.
  • Makefile: make -n db db-stop db-logs still prints the compose commands.
  • Compose: the new installer and compose-file tests check shm_size, and docker compose config accepts the file.
  • Full suite: go vet, golangci-lint (0 issues), go test -race ./... (all 17 packages) and the four bun script test files (11 tests) pass.

Checklist

  • Tests added or updated
  • No new dependencies introduced (or justified in the PR description)
  • I assign copyright in this contribution to Blendbyte GmbH, as required by the
    Contributor License Agreement

Fixed 300 to 500 ms sleeps raced the 200 ms ingest flush on slow CI runners
and failed release runs. Tests now wait until the buffer reports the rows
committed. Envelopes the handler drops before buffering (session-only, a
malformed profile, profiling off) are asserted on the buffer's counters, with
nothing to wait for.
COMPOSE ran docker compose version on every make call, which made the
Makefile tests slow on CI runners and timed them out at bun's 5 s default.
It is now deferred and kept out of the exported environment, so only the
db targets call Docker. The Makefile tests get a 30 s timeout and check
that migrate-create, build and lint never call Docker.
- Drop logs_attributes: no query uses a JSONB operator on attributes, and
  its GIN entries were a large share of log insert time and WAL
- Index issues.regressed_at for the alert evaluator, which asks for recent
  regressions every 15 seconds
- Vacuum and analyze events, event_tags, transactions, spans, logs and
  issues at 2% change instead of 20%, so large tables keep their visibility
  maps fresh
Docker's 64 MB /dev/shm default is too small for parallel queries on a
large database. Both the shipped compose file and the one install.sh writes
now set shm_size.
@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@bashgeek
bashgeek merged commit 38da58f into main Nov 1, 2026
8 checks passed
@bashgeek
bashgeek deleted the fix/release-prep-flaky-tests-db branch November 1, 2026 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant