Repository navigation
gcry 0.38.0 as the official collector + fresh four-arm benchmark session - #23
Merged
Merged
Conversation
…discriminating experiments The read workload carries a 150-250 ms extreme-tail cluster (p99.9 153.6 ms) while the write phase on the same process stays under 18 ms. This note collects the run evidence, ranks five candidate causes - Boehm stop-the-world pauses amplified by the per-connection lock convoy, PostgreSQL-side transients after the TRUNCATE+seed phase setup, read-path allocation profile, admission-gate parking, and chunked response writes - and defines the experiments (diagnose.sh STW correlation, no-reseed and autovacuum-off controls) that decide between them.
…read-phase live heap The instrumented run pins the stall source: every per-second latency spike coincides one-for-one with a GC stop-the-world window of the same magnitude (max 55.1 ms against 55.2 ms of world-stop), while PostgreSQL sits quiescent. The read phase flips Boehm's collection profile from many ~1 ms micro-pauses (write phase, live heap flat at 35 MB) to rare 10-30 ms full pauses as the live heap climbs 44 to 109 MB through the phase; the committed run's 150-250 ms clusters are the same pauses at a larger heap scale, amplified by the per-connection lock queue. diagnose.sh also gains three fixes: the sampler was wired by --define bench_sample but only activated by bench_timing (so it never produced its CSV), the app now runs under ALTAIR_ENV=production like bench.sh, and the pool/worker budget honors BENCH_POOL / BENCH_WORKERS env overrides defaulting to the unified 20/6 profile. Evidence archived under results/diagnose/.
…n matrix documented Buffered json/html/text responses now carry an explicit Content-Length (one write, no chunked framing); the helpers are documented as terminal body writes. Full suite, formatter and Ameba green. The read-tail investigation gains the mitigation matrix, all measured with paired controls: GC env vars are not honored by this build, incremental mode triples stop time under MT, heap caps break the workload, and GC_set_free_space_divisor(1) - the function, not the env var - halves the stop-time budget for ~4 percent throughput and is the standing operator knob. The ORM read path is exonerated by an isolated probe (zero retention, allocation parity with writes); the sampler's live column is garbage-between-collections, and the corrected mechanism is a two-equilibrium heap hysteresis. bench_sample carries the BENCH_GC_* toggles so every arm stays reproducible.
…lt off Applies the measured GC mitigations to the bench app behind env toggles (free-space divisor 1 plus a collect_a_little trickle fiber in gc_tuning.cr). Six interleaved PID-stamped rounds show: the dirty tail is per-run intermittent rather than per-config (stock runs were clean this session; a fixed run still hit a 482 ms window), and in clean windows the knobs cost 3.5-6.5 percent on throughput and read percentiles. Conclusion documented in fix-results.md: keep the knobs opt-in, adopt median-of-three methodology for official numbers, and treat the heap-trajectory hysteresis as the remaining root-cause path. Local only; not merged, not pushed.
…ted, write penalty gone gcry wired back behind -Dgc_none with gc_tuning guarded out of gcry builds. Six interleaved PID-stamped rounds against stock Boehm: write throughput at parity (the 0.27.1-era 11 percent write penalty is gone in 0.38.0), read throughput +2.7, read p99.9 +10.8 and max +39.6 on medians - and decisively, Boehm hit its intermittent dirty window in one round (123.5/482.2 ms) while gcry stayed within 22.1-23.3/35.6-37.9 across all three rounds. Against the committed dirty-window baseline that is an 85 percent read-tail reduction. Evidence in results/gcrytest/, analysis in gcry-results.md. Local branch only.
The framework depends on sdogruyol/gcry and wires it through a single compiler flag: building with -Dgc_none swaps the bundled Boehm for gcry with no code change (require "altair" is enough, the require sits at the top of src/altair.cr). The full framework suite - 1,025 examples - passes under gcry, and CI gains a -Dgc_none build step so the wiring is verified on every change. pg is pinned to 0.30.0 so the vendored wire-protocol patches keep applying. Adopted on measured evidence (interleaved A/B, medians of three rounds, gcry-results.md): gcry eliminates Boehm's intermittent read-tail window (read p99.9 22.4 ms / max 36.3 ms stable, versus Boehm's 123.5/482.2 ms spike rounds) at write-parity - the 0.27.1-era write penalty is gone in 0.38.0. Docs updated across the root README, the benchmark report and the website (benchmarks page gains the official-collector section with the A/B table; the homepage card links it).
One interleaved run window, every arm a fresh process, zero failed requests; raw exports under results/official-gcry/. Fiber leads throughput (20,300 write / 18,990 read req/s); Altair on the official gcry collector holds 13,981 / 13,619 req/s - 1.7-1.8x Express, 0.69-0.72x Fiber, 6.3x/4.6x the Ruby full-stack arm - and carries the best write max of all four frameworks (25.2 ms, tighter than Fiber's 36.8 ms) with a deterministic read tail (37.2 ms max, no intermittent window). The root README, benchmark report and website tables and homepage strip all carry the new session; the earlier Boehm session stays in git history.
…llector work # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Adopts gcry 0.38.0 as the project's official garbage collector, on measured evidence, and refreshes every benchmark surface with a new official four-arm session.
Framework
shard.ymldepends on gcry 0.38.0;src/altair.crwires it in behind a single compiler flag - building with-Dgc_noneswaps the bundled Boehm for gcry with no code change (require "altair"is enough).pgis pinned to 0.30.0 so the vendored wire-protocol patches (issue fix(pg): reentrant per-connection mutex for extended query eliminates PQ wire desync under concurrency (p99 tail + 96% error reduction) #21) keep applying cleanly.-Dgc_nonebuild step verifying the wiring on every change.Why gcry (the evidence)
A root-cause investigation traced an intermittent read-tail spike (p99.9 up to 154 ms, max 482 ms on unlucky runs) to Boehm stop-the-world pauses that scale with heap size. Every mitigation was measured with paired controls (env vars ignored by this build, incremental mode 3x worse under MT, heap caps break the workload) before trying gcry 0.38.0:
Write throughput is at parity (the 0.27.1-era 11 percent penalty is gone in 0.38.0). Full trail:
examples/benchmark_k6/read-tail-investigation.mdandgcry-results.md.Fresh official benchmark session
One interleaved window, all four arms fresh processes, zero failed requests (exports under
results/official-gcry/):Highlights: Altair's write max is the best of all four frameworks (tighter than Go's), read tail is deterministic, 1.7-1.8x Express on throughput.
Also in this PR
json/html/textrenders.diagnose.shnow honors the unified profile and runs production mode.Merges
Contains the development hot-reload work (issue #3) merged with the gcry work; the CHANGELOG carries both entries.
Note for reviewers
One CI run on the merge commit hit an
invalid memory accessat spec startup on the 2-core runner; not reproducible locally under identical conditions (1,083 examples green with a live PG service). Suspected to be the known host-class flake (issue #20) - a re-run is the first check.