Skip to content

fix(data-warehouse): stop a repartition retry abandoning the table - #95956

Merged
trunk-io[bot] merged 1 commit into
masterfrom
posthog/dwh-repartition-retry-keeps-its-attempt
Sep 7, 2026
Merged

fix(data-warehouse): stop a repartition retry abandoning the table#95956
trunk-io[bot] merged 1 commit into
masterfrom
posthog/dwh-repartition-retry-keeps-its-attempt

Conversation

@Gilbert09

Copy link
Copy Markdown
Member

Problem

  • A large warehouse table stops repartitioning for good the first time its final sync run is retried, so it keeps syncing on the layout that made it fail.
  • The retry cap counts sync runs, not the Temporal retries inside one run, and an attempt is charged before the rewrite starts.
  • On the run that charges the third attempt, the stored count already reads the cap, so that run's own retry reads the cap as spent.
  • The retry then abandons the rewrite instead of running it. It clears the pending target, engages the daily cooldown, and reports RepartitionAttemptsExhausted with final=true.
  • A heartbeat timeout is enough to trigger this, and that is the ordinary reason Temporal retries this activity.

Changes

  • A table now keeps its rewrite when the run holding its last attempt is retried, so a heartbeat blip on the final chance no longer abandons it.
  • The cap check discounts a charge the current run made itself, by matching the stored charged_job_id against this run's job id.
  • The give-up at the cap is unchanged for a retry that fails cleanly, because the failure path counts the charge itself, and for a later sync run, which sees a charge from another job.
  • Nothing in the UI changes.

How did you test this code?

  • New test test_a_retry_inside_the_run_that_spent_the_cap_still_rewrites: it fails when the cap check ignores the current run's charge, which is the defect fixed here. Its sibling test_retries_inside_one_sync_run_burn_a_single_attempt starts from zero attempts, so no existing case reaches the cap inside one run.
  • Ran the cap check over its inputs directly in this sandbox: no marker, a count under the cap, a count at the cap, a charge from another job, and a charge from this job. Only the last one is newly exempt.
  • Not run: test_repartition_controller.py itself. Its session setup needs ClickHouse, which this sandbox has no server for. The file collects, and CI runs it.
  • No manual or production verification was performed.

Automatic notifications

  • Publish to changelog?

Docs update

None. The README already describes the cap as consecutive failed sync runs, which is what this restores.

🤖 Agent context

Autonomy: Fully autonomous

  • Written by Claude Opus 5 in a PostHog cloud task, triaging a day of warehouse_repartition_failed events. The terminal one was an RepartitionAttemptsExhausted give-up, and reading the charge and give-up bookkeeping found the retry that never rewrites.
  • Skills invoked: /writing-tests, /writing-code-comments, /writing-pr-descriptions.
  • No duplicate: no open PR touches the attempt cap. #95861 fixes the other failure mode from the same day, budget-exceeded rewrites that cannot converge, by holding imports while a checkpoint is live. It leaves the give-up path alone, and the two diffs do not overlap.
  • Public artifact: the diff carries no material from the agent session. The test uses the fixture helpers already in the file and invented ids.

The attempt cap is charged before the rewrite runs, so on the sync run that
charges the last attempt the stored count already reads the cap. That run's own
Temporal retry read it as spent and gave up, which cleared the pending target,
engaged the daily cooldown and reported RepartitionAttemptsExhausted, without
the retry ever running the rewrite.

The cap check now discounts a charge the current run made itself, so the last
attempt gets its retries. A retry that fails cleanly still gives up at the cap,
because the failure path counts the charge itself.

Generated-By: PostHog Desktop
Task-Id: 525d61af-596c-448f-8879-7bc0b96922f8
Copilot AI lite review requested due to automatic review settings September 7, 2026 10:07
@trunk-io

trunk-io Bot commented Sep 7, 2026

Copy link
Copy Markdown

😎 Merged successfully - details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane (py:product:warehouse_sources)

This PR is assigned to the backend Python lane (py:product:warehouse_sources). It runs backend Python tests and may merge in parallel with PRs in other lanes.

⚠️ Backend coverage — 95.0% of changed backend lines covered — 1 uncovered

🧪 Backend test coverage

Patch coverage — changed backend lines (products + core): ███████████████████░ 95.0% (19 / 20)

File Patch Uncovered changed lines
products/warehouse_sources/backend/temporal/data_imports/workflow_activities/repartition_table.py 87.5% 713

🤖 Agents: add a test covering the lines above, or note why under "How did you test this code?". Machine-readable gap list: the patch-coverage artifact on this run (gh run download 34109754203 -n patch-coverage), or the coverage-data block at the end of this comment.

Per-product line coverage (touched products)
Product Coverage Lines
warehouse_sources ███████████████████░ 97.2% 409,547 / 421,449

Report-only. Patch coverage = changed backend lines covered vs origin/master. Sorted lowest first.
Known gaps: lines covered only by Temporal tests show as uncovered; core line numbers may drift if master changed the same file.

@Gilbert09 Gilbert09 added the stamphog Request AI approval (no full review) label Sep 7, 2026 — with PostHog

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved.

Small, focused bugfix by an author on the owning team, with a new test covering the exact scenario described; no schema/API/security surface touched, and the only review comment is a bot that couldn't actually review (quota limit), not a substantive concern.

  • Author wrote 0% of the modified lines and has 108 merged PRs in these paths (familiarity MODERATE).
  • copilot-pull-request-reviewer[bot] reviewed the current head.
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list no deny categories matched
size 19L, 1F substantive, 49L/2F incl. docs/generated/snapshots — within ceiling
tier T1-agent / T1b-small (49L, 2F, single-area, fix)
stamphog 2.0.0b4 .stamphog/policy.yml @ 1820958 · reviewed head 1820958

@trunk-io

trunk-io Bot commented Sep 7, 2026

Copy link
Copy Markdown

Static BadgeStatic BadgeStatic Badge

View Full Report ↗︎Docs

@talyn-app

talyn-app Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

/trunk merge

@trunk-io
trunk-io Bot merged commit 07778f7 into master Sep 7, 2026
242 checks passed
@trunk-io
trunk-io Bot deleted the posthog/dwh-repartition-retry-keeps-its-attempt branch September 7, 2026 12:12
@deployment-status-posthog

deployment-status-posthog Bot commented Sep 7, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-09-07 12:32 UTC Run
prod-us ✅ Deployed 2026-09-07 12:42 UTC Run
prod-eu ✅ Deployed 2026-09-07 12:45 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stamphog Request AI approval (no full review)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants