Skip to content

fix: deduplicate concurrent upload parse dispatch - #351

Open
Frankie-Xu wants to merge 1 commit into
Ontos-AI:mainfrom
Frankie-Xu:fix/Frankie-Xu/dedupe-parse-dispatch
Open

fix: deduplicate concurrent upload parse dispatch#351
Frankie-Xu wants to merge 1 commit into
Ontos-AI:mainfrom
Frankie-Xu:fix/Frankie-Xu/dedupe-parse-dispatch

Conversation

@Frankie-Xu

@Frankie-Xu Frankie-Xu commented Aug 28, 2026

Copy link
Copy Markdown

Summary

  • Dispatch parsing only when this handoff owns the successful waiting-file -> pending transition.
  • Treat a CAS retry that succeeds as pending -> pending as an idempotent no-op.
  • Recreated on current main (85d5b2ba): main already no-ops a rejected CAS from pending, but a successful same-state retry still dispatched.
  • Add direct and HTTP-interleaving regression coverage for the S3 notification / confirm-upload race.
  • confirm-upload remains the fallback when the storage event is absent. RedisJobLock and the worker terminal-state gate are unchanged.

Closes #286

Test plan

  • make lint
  • uv run pytest apps/api/tests/contract/test_s3_event_contract.py -k "pending_to_pending or cas_winner or localstack" -q — 3 passed (no FastAPI import)
  • HTTP interleaving cases are in the same file; they import apps/api/main.py and currently fail on Windows because fcntl is missing (project P3.2). Upstream CI is Ubuntu.

@Frankie-Xu

Copy link
Copy Markdown
Author

This is a first-time fork workflow and GitHub currently reports no checks. The focused regression, concurrent HTTP cases, affected contract file, lint, and typecheck pass locally; the full API contract result and its unrelated environment-specific failure are documented in the PR body. Please approve the workflow run when convenient.

A successful pending-to-pending CAS retry was still enqueueing a second parse_task when the S3 notification raced confirm-upload.
@Frankie-Xu
Frankie-Xu force-pushed the fix/Frankie-Xu/dedupe-parse-dispatch branch from ad0e247 to 6912ba3 Compare September 11, 2026 16:29
@Frankie-Xu

Copy link
Copy Markdown
Author

Rebased/recreated on current main (85d5b2ba). The remaining hole was a successful pending -> pending CAS retry still calling start_uploaded_file_parse. Only waiting-file -> pending now dispatches.

@Frankie-Xu

Copy link
Copy Markdown
Author

Follow-up: #412 lands a Windows-safe telemetry file lock, so test_s3_event_contract.py can be collected on Windows instead of dying on import fcntl. Ubuntu CI for this PR is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Avoid duplicate parse dispatch when S3 notification races confirm-upload

1 participant