Skip to content

Traffic retention job + cheaper actions observer - #419

Merged
espadonne merged 10 commits into
trunkfrom
perf/traffic-retention
Sep 2, 2026
Merged

espadonne merged 10 commits into
trunkfrom
perf/traffic-retention

Conversation

@espadonne

@espadonne espadonne commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Phase 3 leftovers from docs/internal/retro/2026-09-02-availability-sitrep.md
(root causes #7 and #10).

Traffic retention (#7)

repo_traffic_* had no purge since 2026-05-18: 881 MB of a 988 MB prod DB,
~30k rows/day/table, mostly crawler paths. The Traffic UI reads 14 days.

  • ccb1172f + 6acd7fa0 — 0129: repo_traffic_paths_day_idx, repo_traffic_referrers_day_idx,
    CONCURRENTLY + NO TRANSACTION (both tables are written synchronously in
    the request path). repo_traffic_uniques reuses its existing created_at
    index instead of getting a second one; repo_traffic_daily already had
    (day DESC). No bulk DELETE in the migration on purpose.
  • c178a8c4 — four :execrows batched deletes in internal/repos/queries/repo_traffic.sql,
    WHERE ctid IN (SELECT ctid ... LIMIT $2); make sqlc-generate output included.
  • fe6cf73a — traffic.Purge: 30-day window for _uniques/_paths/_referrers,
    400 days for _daily. 5k rows per statement, each its own transaction,
    capped at 2000 batches per table per run.
  • 1eccf91d — traffic:purge handler; payload overrides the windows and bounds.
    Idempotent: the cutoff is recomputed per run and rows inside the window are
    never touched. Re-enqueues itself when it stops on the batch cap so the
    backlog drains in one night rather than over 480 cron beats.
  • ed912f8a — registered in cmd/shithubd/worker.go, ExecStart added to
    shithubd-cron.service.

Decisions: 30 days rather than the sitrep's 14 so a purge can never truncate a
bar the chart draws; _daily kept at 400 days (one row per repo per day — it is
the only long-term history and is small); the 2.5M-row backfill is the job's
first run, not a migration, so no deploy holds a long transaction.

actionsobserver (#10)

  • 27d5ba92 — sum(octet_length(chunk)) over the hot log-chunk table (a full
    scan + detoast, independent of traffic) moved off the 15 s tick onto its own
    5 min cadence. Count and queue-depth gauges are unchanged at 15 s. Metric
    names unchanged.
  • 42be95b8 — the 5 min lag noted in the two runbooks that list the gauge.

Tests

  • make ci locally (golangci-lint 2.13.2, gofumpt, policy/markdown/
    secret-log/spdx/migration/systemd lints, unit tests, build)
  • internal/repos/traffic unit — batch loop bounds: short batch, exact
    multiple, MaxBatches cap, non-positive bounds, partial progress on
    error, canceled context, option normalization, retention > UI window
  • internal/infra/metrics unit — tick ratio, slow refresh every N ticks,
    cancel, clamp (interval injected, no DB)
  • CI "Integration tests" step (now blocking), run 33600849877 on the
    rebase onto trunk 3c9e1dcd — every step green, zero FAIL lines.
    TestTrafficPurgeKeepsTheRetentionWindow (30-day boundary kept, older
    row dropped, _daily on its own 400-day window, second run a no-op) and
    TestTrafficPurgeBoundsDeletesPerRun (BatchSize×MaxBatches cap,
    second pass resumes) both pass; internal/worker/jobs,
    internal/repos/traffic, internal/infra/metrics and
    internal/migrationsfs all ok.
  • Rebased after Fix the bugs the new Postgres CI service surfaced #418 merged 0127/0128; my migration is renumbered to 0129
    (6acd7fa0), migration-version lint green.
  • 6f36eda1 fixed the one real miss an earlier run caught: assertTrafficDays
    expected days in the wrong order against ORDER BY day ASC. The purge had
    kept exactly the right rows.

Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
Signed-off-by: mfwolffe <wolffemf@dukes.jmu.edu>
@mfwolffe
mfwolffe force-pushed the perf/traffic-retention branch from 2546f7f to 6acd7fa Compare September 2, 2026 06:52
@espadonne
espadonne merged commit fc54e7e into trunk Sep 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants