ci: bencher v1, benches/perf/{lb,sharding,pooler}, detect perf regressions by jkaczman · Pull Request #1617 · pgdogdev/pgdog

jkaczman · 2026-09-22T20:58:50Z

Automatically detect performance regressions in CI using Bencher

Runs our pre-existing load-balancer, sharding and pooler tests (loc. in benches/perf) in a workflow; measures mean TPS over 3 minutes, and looks at the history (from main) to see if the mean TPS dropped by more than 3%. If it does, it comments on the PR with the numbers, and emits an error in CI.

I tested Bencher many times, and the results were always <1% variance! Contrary to this, Blacksmith, GH Actions had variance of over 9%, and did not seem suitable for what we want. It's also free for open source :)

Implementation wise: this re-uses our pgdog-base-runtime image, re-builds with a built (via Blacksmith) pgdog binary off the tested PR's branch, copies over perf scripts, asks Bencher to run it, and Bencher handles everything from there.

Many future plans to add more tests on top of this

Something notable to mention:

This will not work when a fork is being merged in because of GitHub secrets. This is fixable, it will just require implementing Bencher's two-step workflow for that case. I'm going to follow up with another PR fixing that.

re #1585

github-actions · 2026-09-22T21:08:36Z

Bencher Report

Project	PgDog
Branch	jk-bencher-v1
Testbed	Intel v1

Click to view all benchmark results

Benchmark	Throughput	operations / second (ops/s) x 1e3
lb	📈 view plot 🚷 view threshold	29.35 ops/s x 1e3
pooler	📈 view plot 🚷 view threshold	30.77 ops/s x 1e3
sharding	📈 view plot 🚷 view threshold	19.91 ops/s x 1e3

🐰 View full continuous benchmarking report in Bencher

levkk · 2026-09-22T21:44:20Z

+        #
+        # Why --github-actions? This allows it to comment on the PR with the result.
+        # Why --adapter json? This allows us to use a 'custom harness'; basically nothing like rust's cargo test / criterion, we're running custom scripts
+        run: |


I think you want this to be fire and forget (don't wait for job to start/end). Otherwise, we're sitting in the action wasting minutes.

Yep. You're right! I missed that it edited the message.

I forgot about this: GITHUB_TOKEN expires when the job finishes or after its effective maximum lifetime.

github-actions · 2026-09-22T23:52:04Z

Bencher Report

Project	PgDog
Branch	jk-bencher-v1
Testbed	Intel v1

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%)	Lower Boundary operations / second (ops/s) x 1e3 (Limit %)
pooler	📈 view plot 🚷 view threshold	30.88 ops/s x 1e3 (+0.47%) Baseline: 30.74 ops/s x 1e3	29.82 ops/s x 1e3 (96.55%)

🐰 View full continuous benchmarking report in Bencher

github-actions · 2026-09-22T23:55:29Z

Bencher Report

Project	PgDog
Branch	jk-bencher-v1
Testbed	Intel v1

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%)	Lower Boundary operations / second (ops/s) x 1e3 (Limit %)
lb	📈 view plot 🚷 view threshold	29.61 ops/s x 1e3 (+0.13%) Baseline: 29.57 ops/s x 1e3	28.68 ops/s x 1e3 (96.87%)

🐰 View full continuous benchmarking report in Bencher

github-actions · 2026-09-22T23:58:53Z

Bencher Report

Project	PgDog
Branch	jk-bencher-v1
Testbed	Intel v1

Click to view all benchmark results

Benchmark	Throughput	Benchmark Result operations / second (ops/s) x 1e3 (Result Δ%)	Lower Boundary operations / second (ops/s) x 1e3 (Limit %)
sharding	📈 view plot 🚷 view threshold	19.97 ops/s x 1e3 (+0.15%) Baseline: 19.94 ops/s x 1e3	19.34 ops/s x 1e3 (96.85%)

🐰 View full continuous benchmarking report in Bencher

jkaczman · 2026-09-23T12:04:52Z

in addition to the fire-and-forget, also leaving a note here that I've contacted the author of Bencher via email to try to figure out why our runs aren't being parallelized

jkaczman · 2026-09-23T22:55:04Z

Ultimately:

we're now able to have 2 benchmark runs in parallel until they get more hardware on hand
- it might be better for now to only run benchmarks when a tag is added to a PR to prevent long waiting times from non-sensitive PRs
fire-and-forget feature is being worked on by them, but I'm not sure of an ETA

levkk · 2026-09-23T23:35:20Z

TBH doesn't look like this product is quite ready. Let's setup our own GH runner instead. If we use a C7 with like 4-8 cores, we should get pretty good results imo.

codecov · 2026-09-28T13:41:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

jkaczman · 2026-09-28T19:46:12Z

I believe this will have to be merged into main to be tested with the new fire-and-forget functionality. Bencher now internally fires repository_dispatch as a callback to trigger a secondary workflow, however, that's only ran off the default (in our case, main) branch.

…essions

In addition to pgdogdev#1617, also run perf regression tests on fork PRs. This uses [Bencher's recommended 2-step PR workflow](https://bencher.dev/docs/how-to/github-actions/#pull-requests-from-forks) (with some minor tweaks) to safely make sure that forks can't abuse `GITHUB_TOKEN` / `BENCHER_API_KEY` / etc. Steps: - Workflow 1: runs on the PR branch, builds the Docker image with the new pgdog binary, uploads image as an artifact (+ what PR it's from) - Workflow 2: runs off main branch after Workflow 1 concludes, uses secrets to upload the image, and run the Bencher CLI.

jkaczman marked this pull request as ready for review September 22, 2026 20:58

levkk reviewed Sep 22, 2026

View reviewed changes

Comment thread benches/perf/lb/run.sh Outdated

levkk reviewed Sep 22, 2026

View reviewed changes

jkaczman mentioned this pull request Sep 23, 2026

Run CI "fire-and-forget" with --github-actions on Bencher Cloud bencherdev/bencher#1046

Open

jkaczman mentioned this pull request Sep 23, 2026

ci: also run Bencher perf tests on fork PRs #1619

Merged

jkaczman added this pull request to stack #1620 September 23, 2026 12:30

jkaczman force-pushed the jk-bencher-v1 branch 2 times, most recently from 4b5d726 to 4285b8a Compare September 23, 2026 22:46

jkaczman closed this Sep 23, 2026

jkaczman reopened this Sep 28, 2026

jkaczman force-pushed the jk-bencher-v1 branch from 4285b8a to 28d085c Compare September 28, 2026 13:38

jkaczman force-pushed the jk-bencher-v1 branch 3 times, most recently from ebda5bb to 245188e Compare September 28, 2026 17:32

jkaczman added 7 commits September 28, 2026 17:44

ci: bencher v1, benches/perf/{lb,sharding,pooler}, error on perf regr…

bce31c4

…essions

-T 30 -> 150

3cc5590

-T 150 -> 90 (free tier limit)

4f4ea12

Run all 3, in parallel, for 3 mins each.

2995ea7

Only comment on regression

4ed2b03

Update Bencher to latest; fire-and-forget repository dispatch workflow

39c99e0

Only comment on alert (followup). --start-point-reset

8cfe197

jkaczman force-pushed the jk-bencher-v1 branch from c069afe to 8cfe197 Compare September 28, 2026 21:44

jkaczman added this pull request to the merge queue Sep 28, 2026

Merged via the queue into main with commit f353ba0 Sep 28, 2026
31 of 34 checks passed

jkaczman deleted the jk-bencher-v1 branch September 28, 2026 22:33

Navigation Menu

Repository navigation

ci: bencher v1, benches/perf/{lb,sharding,pooler}, detect perf regressions - #1617

ci: bencher v1, benches/perf/{lb,sharding,pooler}, detect perf regressions#1617
jkaczman merged 7 commits into
mainfrom
jk-bencher-v1

jkaczman commented Sep 22, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Sep 22, 2026 •

edited

Loading

Uh oh!

levkk Sep 22, 2026

Uh oh!

jkaczman Sep 22, 2026

Uh oh!

jkaczman Sep 23, 2026

Uh oh!

github-actions Bot commented Sep 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 22, 2026 •

edited

Loading

Uh oh!

jkaczman commented Sep 23, 2026

Uh oh!

jkaczman commented Sep 23, 2026 •

edited

Loading

Uh oh!

levkk commented Sep 23, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 28, 2026

Uh oh!

jkaczman commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jkaczman commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

levkk Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

jkaczman Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

jkaczman Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

github-actions Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

github-actions Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bencher Report

Uh oh!

jkaczman commented Sep 23, 2026

Uh oh!

jkaczman commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

levkk commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 28, 2026

Codecov Report

Uh oh!

jkaczman commented Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jkaczman commented Sep 22, 2026 •

edited

Loading

github-actions Bot commented Sep 22, 2026 •

edited

Loading

github-actions Bot commented Sep 22, 2026 •

edited

Loading

github-actions Bot commented Sep 22, 2026 •

edited

Loading

github-actions Bot commented Sep 22, 2026 •

edited

Loading

jkaczman commented Sep 23, 2026 •

edited

Loading

levkk commented Sep 23, 2026 •

edited

Loading