Skip to content

feat(platform): reconcile uncertain Spark commits - #27

Open
zhouning wants to merge 1 commit into
feat/ar1-metadata-fabric-spark-commit-failure-recoveryfrom
feat/ar1-metadata-fabric-spark-uncertain-commit-reconciliation
Open

feat(platform): reconcile uncertain Spark commits#27
zhouning wants to merge 1 commit into
feat/ar1-metadata-fabric-spark-commit-failure-recoveryfrom
feat/ar1-metadata-fabric-spark-uncertain-commit-reconciliation

Conversation

@zhouning

Copy link
Copy Markdown
Owner

Summary

  • add a fail-closed M3-13 contract and live Docker Desktop rehearsal for provider-committed Spark/Iceberg response loss
  • return HTTP 504 after Gravitino commits, suppress the transport retry, and classify the outcome through snapshot, row, and referenced-file readback without resubmitting the logical write
  • commit fingerprinted local evidence, ADR-059, runtime inventory registration, roadmap boundaries, tests, and CI validation

Evidence

  • contract fingerprint: 7a8d75a1d6b4558b982c6c3242d8d356c5046955f8aae7a45e5c297b6f4d4132
  • evidence fingerprint: d6462fff78d07047311b1f715d5f2c7f08c0ce8fbdd5c8b26a3d95ddc3474786
  • dependency M3-12 evidence: 39571cdac1e4043bcfc2d03a73b2b12ff925210daf8ae36bc640b8cb14d89401
  • final state: 2 parent-linked append snapshots, 3 rows, 2 referenced Parquet files, and 9 MinIO objects
  • decision: committed_do_not_resubmit; write_resubmitted=false
  • Spark Job completed 1/1; rehearsal namespace, both PVs, port-forwards, and temporary test database were cleaned up

Verification

  • M3-13 validator and tests: 23 passed
  • M3-12 regression: 23 passed
  • platform truth: 11 passed
  • required platform suite: 787 passed
  • route evaluation: 41 passed
  • PostgreSQL contract tests: 5 passed
  • runtime dependency constraints: 4 passed
  • migration catalog, Python compile, shell syntax, JSON/YAML parse, both Kustomize renders, and staged diff checks passed

Scope Boundary

This is local Docker Desktop evidence for one append and immediate deterministic readback. It does not prove a durable reconciliation controller, crash or concurrent-writer recovery, network exactly-once, protected identity/TLS, production object storage, full Spark/Flink conformance, production ingestion, or production readiness. Broad and production claims remain false.

Stacked on #26.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant