test(delta-harness): add merge-on-read matrix - #705
Draft
mkuchenbecker wants to merge 24 commits into
Draft
Conversation
Adds a self-contained Scala behavioral test harness that characterizes OpenHouse + Apache Iceberg table behavior end-to-end. The harness crosses a large matrix of table layouts (partitioning, MoR/CoW, ordered writes, nested types) with DDL, DML, maintenance, branching/WAP, streaming, and negative-path operations, asserting deltas against observed pre-state so each case holds under any layout. It runs locally against a real embedded OpenHouse catalog (harness/openhouse/Env.scala boots OpenHouseLocalServer + the OpenHouse Spark catalog; see run-openhouse.sh and HARNESS-GUIDE.md). The scenario and framework sources are also structured as a publishable Gradle library module (openhouse-spark-delta-harness_2.12) that excludes the embedded-only Env so downstream environments can supply their own adapter. Genuine product or upstream bugs are tagged in Plan.knownBugs with a prose explanation and skipped rather than silently passed, so the suite stays green while documenting the defect. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds TESTING-MATRIX.md, a living reference that explains the harness as a cross product of independent axes (operation family, data file format, partitioning, write mode, schema, preparation lineage, and reference routing). Documents how a case id reads, the CoreTable/NestedTypesTable/TypesTable schemas, the table layouts, the preparation lineages, and each operation family including the DDL sub-families. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move each test's preparation, action, and assertions into its scenario file so the complete behavior is readable in one place. Keep reusable preparation recipes while creating a fresh table for every case. Preserve the exact 2,574-case catalog, ordering, and known-bug behavior with regression tests for the catalog fingerprint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explain the scenario-owned test structure, immutable preparations, and fresh-table isolation used by the localized test cases. Describe the matrix as living documentation for the current harness architecture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the harness guide and testing matrix from the implementation PR so the documentation can be reviewed in a separate stacked change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Separate table preparations from DML operations so each case shows its starting state, mutation, and relative assertions in one place. Keep feature-owned scenarios in removable RTAS, merge-on-read, and branch layers while preserving the exact ordered 2,572-case catalog. Run the same published sources through the local Gradle task and the acceptance adapter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the standard branch focused on copy-on-write behavior, shared table preparations, bespoke DDL coverage, and the local execution framework. Remove RTAS, merge-on-read, branch, and WAP scenario ownership from this layer. Pin the resulting ordered standard catalog at 1,181 cases so each child branch can add one reviewable feature delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep runtime case metadata limited to stable identifiers and execution state. Put preparation and test explanations beside their Scala behavior so reviewers can read each case without tracing string registries. Generate a fresh UUID for every table and begin cleanup only after the preparation creates it, which preserves any pre-existing table on a name conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract the owned-table cleanup state machine behind a package-private boundary so its failure paths can be tested without starting Spark. Pin conflict preservation, successful cleanup, and suppression of cleanup failure behind the primary test failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin the remaining ownership outcome: when the test body succeeds and cleanup fails, the cleanup failure must surface to the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 1, 2026
Reflow harness documentation to the repository's 120-column target and explain the DML operation and preparation matrix at its source. Name the reusable date column independently from partitioning so layouts, not column identifiers, express partition choices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace provenance and consequence buckets with capability-owned scenario files whose public contribution surfaces explain the catalog at a glance. Separate local runner code from the publishable harness, make preparations show creation and standard seeding explicitly, and reindex generic case IDs. Use generated table names and failure-preserving ownership boundaries for every case-owned table, view, registration, rename, and lock lifecycle. Move column-default coverage out of the standard layer for a dedicated follow-up PR while pinning the remaining 1,177-case catalog. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Name every scenario source and trait ScenarioFoo so scenario files group together and the framework files remain visually distinct. Preserve the catalog contributions, IDs, ordering, count, and fingerprint unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep PR 682 focused on reusable DDL and DML coverage while moving orthogonal capabilities to extension branches. - retain 642 Parquet and ORC foundation cases - extract reusable changelog and concurrency support - preserve Plan and Scenarios consumer compatibility - add extension-stable catalog and support contract tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 264 replacement-specific cases on the 642-case foundation. - run every reusable DML operation across four RTAS preparations - cover schema, partition, policy, history, rename, order, and identity - require precise lineage rejection and typed concurrency outcomes - expose silent narrowing corruption as a known product bug Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rebuild merge-on-read DML, changelog, delete-file, maintenance, and snapshot-history contracts on the focused RTAS catalog. The scenarios prove exact current file state, procedure effects, and write-mode behavior across Parquet and ORC. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mkuchenbecker
force-pushed
the
mkuchenbecker/delta-harness-mor
branch
from
September 2, 2026 05:55
69c7207 to
0c224c0
Compare
Keep the replace-versus-append assertion limited to serializable outcomes while recording the rare lost-replace result as a known bug. Both format cases remain available for re-enablement when concurrent RTAS commits report conflicts or preserve the replace. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep only generated ID uniqueness and the ownership and cleanup failure behaviors that can fail independently. Scenario definitions are validated by running their behavior cases instead of restating source declarations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate replace-table behavior by running its scenarios instead of restating their generated catalog entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Validate merge-on-read behavior by running its scenarios instead of restating its generated catalog entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds merge-on-read preparations, DML cases, maintenance cases, and reader behavior on top of the RTAS layer. The source branch adds the merge-on-read layer after the standard and RTAS commits. Because this PR targets
main, its current diff also includes PRs 682 and 704 until those dependencies merge.The layer adds 466 cases, increasing the catalog from 1,542 to 2,008.
Stack
All OpenHouse PRs target
main. Review them in dependency order; each upper diff shrinks as its dependencies merge. The documentation PR depends on the standard framework, and the Airflow PR consumes the branch and WAP artifact.mainScope
Validation
9e12720fcb83450a7197f5fdc6911390589baaa359df2ad42d50a2b81324e861.interact.mor.alterToMorpassed locally.surface.proc.rewritePositionDeletespassed locally.