Skip to content

fix(demo): deduplicate materialization and parallelize result uploads - #407

Merged
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test
Sep 11, 2026
Merged

fix(demo): deduplicate materialization and parallelize result uploads#407
suguanYang merged 1 commit into
mainfrom
fix/wangbinqi/remove-duplicate-migration-test

Conversation

@suguanYang

Copy link
Copy Markdown
Contributor

Summary

  • Add short-lived demo materialization claims with stable 409 responses for completed/in-progress duplicates.
  • Narrow namespace snapshot locking so different demo sources can publish concurrently.
  • Batch publication writes and reuse a process-level S3 client/upload executor.
  • Upload raw result files with bounded process-level concurrency (default 20) and increase the shared S3 pool to 20.

Verification

  • Demo contract tests: 10 passed
  • Result storage focused tests: 4 passed
  • Storage config contract: 7 passed
  • Ruff: passed
  • Pyright: 0 errors
  • git diff --check: passed
  • Local TSLA bundle timing: median 10.25s for 73 files / ~92 MiB on LocalStack

Notes

  • Full worker contract suite requires local pytest_postgresql/fakeredis dependencies and was not rerun in this environment.
  • Production S3 throughput still needs post-deploy internal-network measurement.

import sqlalchemy as sa


revision: str = "0b1c2d3e4f5a"


revision: str = "0b1c2d3e4f5a"
down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"

revision: str = "0b1c2d3e4f5a"
down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
branch_labels: Sequence[str] | None = None
revision: str = "0b1c2d3e4f5a"
down_revision: str | Sequence[str] | None = "e4f5a6b7c8d9"
branch_labels: Sequence[str] | None = None
depends_on: Sequence[str] | None = None
@suguanYang
suguanYang merged commit 85d5b2b into main Sep 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants