Skip to content

feat(platform): verify Spark object-store interoperability - #24

Open
zhouning wants to merge 1 commit into
feat/ar1-metadata-fabric-spark-iceberg-rest-interoperabilityfrom
feat/ar1-metadata-fabric-spark-object-store-interoperability
Open

feat(platform): verify Spark object-store interoperability#24
zhouning wants to merge 1 commit into
feat/ar1-metadata-fabric-spark-iceberg-rest-interoperabilityfrom
feat/ar1-metadata-fabric-spark-object-store-interoperability

Conversation

@zhouning

Copy link
Copy Markdown
Owner

Summary

  • replace the same-node shared Iceberg warehouse PVC with a cross-node MinIO S3-compatible boundary
  • verify Spark create/read/write, schema evolution, snapshots and time travel, then match the result through Gravitino and direct object-store metadata inspection
  • add fail-closed contract/evidence validation, Kubernetes rehearsal manifests, runtime inventory, CI coverage and ADR-056

Verified

  • live Docker Desktop Kubernetes rehearsal: MinIO on desktop-control-plane; PostgreSQL, Gravitino and Spark on desktop-worker; no Spark/Gravitino warehouse PVC
  • required platform suite: 709 passed, 9 warnings
  • PostgreSQL integration: 5 passed
  • PR route-evaluation contract: 41 passed, 2 warnings
  • focused M3-10/platform-truth tests: 28 passed, 4 warnings
  • runtime dependency constraints: 4 passed
  • all 28 CI contract validators, evidence verify/validate, kubectl kustomize, git diff --check and frontend production build passed

Boundaries

This proves local cross-node S3-compatible interoperability inside one Docker Desktop host and cluster. Production object storage, protected workload identity, OIDC/TLS, commit failure injection, cancel/reconcile/lineage, Flink and complete Spark conformance remain unverified and fail closed.

Stacked on #23. This PR must not be merged before its base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant